KR20160115068A - Method and apparatus for hierarchical stereo matching - Google Patents

Abstract

The hierarchical stereo matching apparatus determines the image pyramid level based on the resolution of the stereo image, downsamples the stereo image, and generates an image pyramid corresponding to the image pyramid level. Determining a different cost aggregation scheme for stereo matching at each image pyramid level and using a cost aggregation method determined for each of the stereo images of the image pyramid level to sequentially generate the lowest resolution stereo image to the highest resolution stereo image And performs the stereo matching sequentially.

Description

[0001] METHOD AND APPARATUS FOR HIERARCHICAL STEREO MATCHING [0002]

The present invention relates to a hierarchical stereo matching method and apparatus.

Stereo matching is a technique for detecting correspondence points in two or more images. Using the stereo matching technique, the disparity of corresponding points between images at two or more viewpoints can be obtained, and the depth of an object in the image can be calculated based on the disparity value. The disparity map can be used in various fields such as image-based rendering, robot vision, and next-gen realistic broadcasting.

Stereo matching technology has been studied in the field of image processing and various methods have been studied. Stereo matching techniques are divided into a global algorithm and a local algorithm. The global algorithm method uses a global optimization method such as belief propagation and graph-cut, and the local algorithm method divides the image into block-based methods to calculate and optimize the local cost. Both methods basically have a process of accumulating costs when calculating the cost of stereo matching. This is called cost aggregation and various methods have been proposed. The cost integration method affects the accuracy of the stereo matching and the computation time, and usually has a complementary relationship between the accuracy of the stereo matching and the computation time.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a hierarchical stereo matching method and apparatus capable of providing efficient stereo matching in terms of matching accuracy and calculation time.

According to one embodiment of the invention, a hierarchical stereo matching method of a hierarchical stereo matching device is provided. A hierarchical stereo matching method includes the steps of determining an image pyramid level based on a resolution of a stereo image, generating an image pyramid corresponding to the image pyramid level by downsampling a stereo image, Determining a different cost integration method, and sequentially performing stereo matching from a lowest resolution stereo image to a highest resolution stereo image sequentially using a cost integration method determined for the stereo image of the image pyramid level .

According to the embodiments of the present invention, it is possible to compensate for the disadvantages of the specific cost integration method by applying the cost integration method that is different for each layer or each frame according to the characteristics of the image in the stereo matching of the hierarchical structure. Accuracy can be increased.

1 is a diagram illustrating a hierarchical stereo matching apparatus according to an embodiment of the present invention.
2 is a view for explaining a method of hierarchical stereo matching with an image pyramid according to an embodiment of the present invention.
Fig. 3 is a view showing the stereo matching unit shown in Fig. 1. Fig.
4 is a flowchart illustrating a hierarchical stereo matching method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification and claims, when a section is referred to as "including " an element, it is understood that it does not exclude other elements, but may include other elements, unless specifically stated otherwise.

A method and an apparatus for hierarchical stereo matching according to an embodiment of the present invention will now be described in detail with reference to the drawings.

1 is a diagram illustrating a hierarchical stereo matching apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the hierarchical stereo matching apparatus 100 includes an image pyramid generation unit 110 and a stereo matching unit 120.

The image pyramid generation unit 110 determines an image pyramid level based on the resolution of the stereo image and downsamples the stereo image to generate an image pyramid corresponding to the image pyramid level. If the level of the stereo image is the lowest level (Level 1) and the level of the image pyramid is n, the image pyramid generation unit 110 reduces the horizontal and vertical directions by 1/2, respectively, based on the stereo image corresponding to the lowest level sampling and level 2 to create a stereo image of the resolution for the (level 2), and the horizontal and vertical, based on the stereo image that corresponds to the lowest level, each of ^{2 2} by the down sampling to level 3 (level 3) Thereby generating a stereo image having a resolution of. In this way, the image pyramid generator 110 downsamples the horizontal and vertical images by 1/2 ^(n-1) on the basis of the stereo image corresponding to the lowest level, and generates the image pyramid corresponding to the highest level n (Level n) Resolution stereo image to complete the image pyramid.

That is, the stereo image having the resolution corresponding to the level 2 (Level 2) is downsampled by 1/4 based on the stereo image of the lowest level (Level 1), and the stereo image having the resolution corresponding to the level 3 And is downsampled by 1/4 based on the stereo image of level 2 (Level 2). And the stereo image having the resolution corresponding to the level n (Level n) is downsampled by 1/4 based on the stereo image of the previous level [Level (n-1)].

The stereo matching unit 120 performs stereo matching using the image pyramid to calculate the variation, and generates the final variation image using the calculated variation. In stereo matching, one of the left and right images of the stereo image is set as a reference image, the other is set as a comparison image, and then matching points corresponding to each other in the reference image and the comparison image are found. Such stereo matching may be block-based matching, pixel-based matching, feature-based matching, or the like. Pixel-based matching is the task of finding corresponding points in the comparison image for each pixel in the reference image. Block-based matching is a task of dividing an image into fixed-size blocks and finding corresponding points in the comparison image for each block.

2 is a view for explaining a method of hierarchical stereo matching with an image pyramid according to an embodiment of the present invention.

Referring to FIG. 2, an image pyramid is generated by downsampling the reference image. As described above, when the reference image is at the lowest level (Level 1) and the image pyramid level is n, the reference image of the resolution corresponding to the level (Level k) is the previous level [(Level (k-1) ], Where k is an integer value from 2 to n.

The stereo matching unit 120 performs stereo matching on a stereo image of the highest level (Level n), that is, the lowest resolution among the image pyramids, and estimates a variation value. The stereo matching unit 120 performs a stereo matching on a stereo image having a resolution corresponding to a next higher level (Level n-1) by using a variation value estimated from an image having the lowest resolution corresponding to the highest level (Level n) Used as initial input value. That is, the stereo matching unit 120 obtains the reference image of the level (Level k-1) as the center of the position when the variation estimated from the stereo image of the resolution corresponding to the immediately preceding level (Level k) is multiplied by m And a stereo matching with the comparison image is performed on the search area to estimate a variation value. Here, m is determined based on the resolution between the lower level and the upper level (Level k-1, Level k). As described above, the horizontal resolution difference between the highest level (Level n) and the next level (Level k- / 2, so that m can be 2.

In this manner, the stereo matching unit 120 performs stereo matching on the stereo image having the highest resolution corresponding to the lowest level (Level 1) to estimate the variation value, and uses the variation value estimated from the highest- And generates a mutation image.

By using such a hierarchical image pyramid, the execution time of the stereo matching can be shortened and the accuracy can be improved as compared with the non-hierarchical image structure. In other words, when a variation is searched at the next lower level using a variation value estimated from a stereo image at a higher level, all regions need not be searched when a candidate search region is determined, so that search time at a lower level can be reduced . In addition, since the variation value can be corrected for each level through this process, the accuracy of stereo matching also increases.

Generally, block - based Sum of Absolute Differences (SAD) method is used as a cost integration method when performing stereo matching in non - hierarchical stereo matching. The SAD determines the SAD value between each matching block in the search area and the corresponding comparison block in the reference image and determines the pixel in the center of the matching block having the smallest SAD value as the corresponding point to the search area, Variation is the way in which it is estimated. In addition, there is a census method as a cost integration method. The census is a method of using the relative ordering among the pixels in the block rather than the pixel values between the blocks as in the SAD. The census is a method of finding a most similar value by generating a specific pattern by displaying it as binarized information representing a large value and a small value compared with a reference pixel of the block (usually the pixel at the center position) and comparing these patterns.

In the SAD method, the matching rate is low for textureless regions with little pattern or texture information, and the matching rate is low for the region where repeated patterns appear.

As a result, the result of the cost integration method may vary depending on the characteristics of the image, and accordingly, it is very important to adopt a cost integration method suitable for the characteristics of the image in order to improve the accuracy of the matching. There is also an AD-census method which improves the accuracy of the registration by mixing the cost integration methods with each strength, such as the SAD method and the census method. All of these methods are methods used in non-hierarchical stereo matching.

In the hierarchical stereo matching according to the embodiment of the present invention, the accuracy of matching is increased by using different cost integration methods for each level.

A cost accumulation method in hierarchical stereo matching according to an embodiment of the present invention will now be described in detail with reference to FIG.

Fig. 3 is a view showing the stereo matching unit shown in Fig. 1. Fig.

3, the stereo matching unit 120 includes a cost integration method determination unit 122, a search unit 124, and a misjudgment rate calculation unit 126. [

The cost integration method determination unit 122 determines the cost integration method in each level of the stereo image before performing the stereo matching in each level of the stereo image. The cost integration method determination unit 122 can determine different cost integration methods for each level. The cost integration method decision unit 122 can determine the cost integration methods having different strengths in a cost integration manner at different levels.

For example, the cost integration method determination unit 122 may determine the SAD method in a cost-integrated manner at the highest level, and determine the census method in a cost integration manner at a next-level level. Since the variation value is estimated through the census method at the next level based on the variation value estimated by the SAD method at the highest level, the portion not detected by the SAD method can be compensated based on the census method at the next level. SADs and censuses are just one example of a cost-intensive approach, and other cost-intensive approaches can be applied at each level to reduce false matching.

When selecting a particular integration method from among various cost integration methods, not only the accuracy of matching but also the amount of computation are also considered. Although SAD and Census methods have strengths in specific patterns, the census method is generally more accurate than the SAD method. Nevertheless, the reason why SAD is used more than Census method is that computation time complexity and implementation complexity are lower than Census method. Therefore, when the computation time is important in determining the cost integration method of each level, the cost integration method determination unit 122 selects a cost integration method that requires a large amount of computation only in a stereo image of a specific level but gives an accurate result, Level stereo image, a cost integration method requiring a small amount of computation can be selected, so that the efficiency with respect to time can be increased.

The search unit 124 performs stereo matching using the cost integration method determined in each level of the stereo image. The search unit 124 performs stereo matching using an image pyramid as described in FIG. 2 by applying the cost integration method determined for each level, and generates a final mutation image at the lowest level.

The erroneous matching ratio calculator 126 calculates the erroneous matching ratio to analyze the accuracy of the stereo matching performed for each level. The mismatch rate can be calculated through a consistency check on the reference image and the comparison image. In other words, the misalignment rate calculation unit 126 obtains a corresponding point existing in the comparison image using the variation value of the reference image, and then returns the reference point to the reference image position using the variation value of the corresponding point in the comparison image. Check how much the variation values match. The erroneous matching ratio calculator 126 may calculate the ratio of the mismatching pixels to the total number of pixels of the entire image as an erroneous matching ratio. The cost integration method determination unit 122 can select a more appropriate cost integration method at each level based on the calculated misregistration rate. The cost integration method determination unit 122 may select a cost integration method in the next layer based on the misalignment rate calculated in the corresponding layer or may select a cost integration method in the corresponding layer based on the calculated misalignment rate in the corresponding layer It is possible. For example, in the case of moving images, since the image characteristics are similar in the images in which the scene transition does not occur largely, it is necessary to experimentally determine the optimum integration method by calculating the misalignment rate by varying the cost integration method, The cost integration method can be selected.

On the other hand, the cost integration method in the cost integration method determination unit 122 may determine a cost integration method suitable for each frame of an image, not a level. The cost integration method determination unit 122 may adaptively calculate the cost of each frame based on the misalignment rate calculated by the different cost integration method for each frame since the characteristics and the scene structure of the image are similar in the near frame in which the scene change is not performed You can also choose a method.

4 is a flowchart illustrating a hierarchical stereo matching method according to an embodiment of the present invention.

Referring to FIG. 4, the hierarchical stereo matching apparatus 100 determines the level of the image pyramid based on the resolution of the stereo image (S410).

The hierarchical stereo matching apparatus 100 generates an image pyramid corresponding to the level of the image pyramid (S420). The stereo image is set to the highest resolution of the first level, the second level resolution image is generated by downsampling the stereo image, and the resolution image of the second level is downsampled to generate the third level resolution image. In this way, an image pyramid corresponding to the image pyramid level is generated.

The hierarchical stereo matching device 100 determines different cost aggregation schemes for stereo matching at each level of the image pyramid (S430). The cost integration method can be determined as described above with reference to FIG.

The hierarchical stereo matching apparatus 100 sequentially performs stereo matching from the lowest resolution stereo image to the highest resolution stereo image sequentially using the cost integration method determined for the stereo image of the image pyramid level (S440) . At this time, a search area is set based on a variation value estimated from a stereo image having a resolution corresponding to a previous level, and a stereo matching with a comparison image is performed on the search area with respect to the reference image of the current level at which the stereo matching is to be performed, Value is estimated.

In this way, the hierarchical stereo matching apparatus 100 performs stereo matching on the stereo image having the highest resolution and generates the final mutation image using the estimated mutation value (S450).

At least some functions of the hierarchical stereo matching method and apparatus according to the embodiment of the present invention described above may be implemented in hardware or software combined with hardware. For example, a processor implemented as a central processing unit (CPU), other chipset, microprocessor, or the like may perform the functions of the image pyramid generator 110 and the stereo matching unit 120.

The embodiments of the present invention are not limited to the above-described apparatuses and / or methods, but may be implemented through a program for realizing functions corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded, Such an embodiment can be readily implemented by those skilled in the art from the description of the embodiments described above.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

Claims (1)

In a hierarchical stereo matching method of a hierarchical stereo matching device,
Determining an image pyramid level based on the resolution of the stereo image,
Generating an image pyramid corresponding to the image pyramid level by downsampling the stereo image,
Determining a different cost aggregation scheme for stereo matching at each image pyramid level, and
Sequentially performing stereo matching from the lowest resolution stereo image to the highest resolution stereo image sequentially using the cost integration method determined for the stereo image of the image pyramid level,
/ RTI >

Images