KR101766431B1 - Method and apparatus for detecting disparity by using hierarchical stereo matching - Google Patents

Method and apparatus for detecting disparity by using hierarchical stereo matching Download PDF

Info

Publication number
KR101766431B1
KR101766431B1 KR1020150086143A KR20150086143A KR101766431B1 KR 101766431 B1 KR101766431 B1 KR 101766431B1 KR 1020150086143 A KR1020150086143 A KR 1020150086143A KR 20150086143 A KR20150086143 A KR 20150086143A KR 101766431 B1 KR101766431 B1 KR 101766431B1
Authority
KR
South Korea
Prior art keywords
image
mutation
weighted
upper layer
layer
Prior art date
Application number
KR1020150086143A
Other languages
Korean (ko)
Other versions
KR20160149088A (en
Inventor
신홍창
Original Assignee
한국전자통신연구원
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 한국전자통신연구원 filed Critical 한국전자통신연구원
Priority to KR1020150086143A priority Critical patent/KR101766431B1/en
Publication of KR20160149088A publication Critical patent/KR20160149088A/en
Application granted granted Critical
Publication of KR101766431B1 publication Critical patent/KR101766431B1/en

Links

Images

Classifications

    • H04N13/0014—
    • H04N13/0011—
    • H04N13/0018—
    • H04N13/0022—

Abstract

A method for extracting a variation using hierarchical stereo matching and an apparatus therefor are provided. Up-scale stereo matching of the upper layer based on the transition image of the previous lower layer to acquire the transition image of the upper layer, and up-sampling is performed based on the transition image of the upper layer Obtains a mutated image with improved resolution. The range term is calculated using the inductive image of the same layer to obtain a weight, and the obtained weight is applied to the mutation image of the layer to obtain a weighted mutation image.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for extracting a mutation using hierarchical stereo matching,

The present invention relates to stereo matching, and more particularly, to a method and apparatus for detecting a variation using hierarchical stereo matching.

Stereo matching is a technology that detects corresponding points on two or more images. Using the stereo matching technique, we can obtain the disparity of the correspondence points between the images at more than two viewpoints. Based on this difference, the depth of the object in the image can be calculated through triangulation. Such a disparity map can be utilized in various fields such as image-based rendering, robot vision, and next-generation realistic broadcasting.

Stereo matching technology has been studied in the field of image processing and various methods have been studied. Stereo matching techniques are divided into a global algorithm and a local algorithm.

The global algorithm uses global optimization methods such as belief propagation and graph-cut. In the local method, the image is divided into blocks and the local cost calculation such as block matching is performed. This is how to optimize it. The global algorithm can extract relatively more precise than the local method, but it takes a long time to optimize the algorithm, making it difficult to use it in real time.

In recent years, there have been widely used methods of performing initial matching through a local method and improving variation through edge-preserving filtering. In stereo matching, object boundary between color image and mutation image does not match. In order to solve this problem, boundary preservation filtering is applied to mutation image. One of the most representative methods of boundary preservation filtering is bilateral filtering. This method is a method using a non-linear edge preserving smoothing filter. The filtered image can be expressed as follows.

Figure 112015058750858-pat00001

Here, I represents an input image and O represents an output image. n s is a neighboring pixel of the center pixel p, w (x) is a domain kernel for calculating a distance difference between pixels, and c (x) is a range kernel.

The characteristics of the bidirectional filter, which smoothens the variation values while preserving the boundaries, are utilized to improve the occlusion or blurring caused by the occlusion at the boundaries found during stereo matching. do. Basically, the original image is used as a guidance image, and a weight is calculated based on an inductive image, and filtering is performed by applying the weight to a variation value. A variety of related techniques have been proposed.

However, the nonlinear method has a lot of computational complexity due to the nature of the algorithm, and there have been many attempts to speed up the method to approximate the same result faster. Recently, a hardware optimization bidirectional filtering method using hierarchical structure has been proposed.

The hardware-optimized bidirectional filtering method approximates the existing bidirectional filter to a structure capable of parallel processing in a hardware or GPU (graphics processing unit) device, preserving the boundary and improving the variation. However, this method is disadvantageous in that the left / right consistency is shifted in the stereo matching.

Specifically, in stereo matching, the left / right side image is obtained by calculating the variation value from the left image to the right image and the variation value from the right side to the left image, respectively. The filtering that improves the variation using the inductive image does not consider the correlation of the left / right image, but calculates the weight using only the information of only one image. Therefore, when filtering is separately applied to each of the left and right side images, inconsistency may be inconsistent between the filtered left and right side images. Here, the consistency of the left and right images means that the mutual values of the left side image and the right side image coincide with each other in both directions.

A problem to be solved by the present invention is to provide a mutation extraction method and apparatus that can maintain stereo coherence more accurately and quickly while maintaining left and right side consistency.

According to an aspect of the present invention, there is provided a mutation extraction method comprising: performing stereo matching on two or more images of an image pyramid type in a plurality of layers to obtain a mutation image; Performing an up-scale stereo matching of the upper layer based on the transition image of the previous lower layer to obtain a transition image of the upper layer; Performing up-sampling on the disparity image of the upper layer to obtain a disparity image having improved resolution; And calculating a range term by using the derived image of the same layer to obtain a weight, and applying the obtained weight to the mutation image of the layer to obtain a weighted mutation image.

In addition, the method may further include normalizing the weighted mutation image.

A mutation image may be finally obtained by obtaining the mutation image of the upper layer, acquiring the enhanced mutation image, acquiring the weighted mutation image, and performing the normalizing process repeatedly to the highest layer .

Wherein normalizing the weighted shifted image comprises weighting the shifted image as a weighted image, weighting the weighted image as a weighted value based on the range term, and applying the weighted image to the derived image of the corresponding layer, A variation image can be obtained.

The step of performing the stereo matching to obtain a disparity image may include performing stereo matching on the left and right images of the lowest layer of the image pyramid to obtain a left and right disparity image, And performing coherence correction on the left and right images based on weights considering coherence between the left and right images.

The step of acquiring the disparity image of the upper layer includes: setting a search range including a predetermined number of pixels around the position when the disparity value of the disparity image of the lower layer is multiplied by a predetermined number; And performing a stereo matching on the search range to obtain a mutation image of the upper layer.

The step of acquiring the image having the improved resolution may convert the image based on the weighting of the upper layer to perform adaptive upsampling.

The step of acquiring the image having the improved resolution includes: a step of down sampling the image of the upper layer to obtain a lower layer image; Obtaining a weighted disparity image of a lower layer based on the disparity image of the lower layer; Obtaining a weighted mutation image of the upper layer according to a weight of an upper layer obtained based on the weight of the lower layer; And obtaining a weighted disparity image interpolated using the weighted disparity image of the lower layer and the weighted disparity image of the upper layer.

According to another aspect of the present invention, there is provided a deviation extracting apparatus including: an image obtaining unit that obtains a plurality of images; And a processor for obtaining a disparity image based on the images, the processor comprising: a stereo matching unit for performing stereo matching on two or more images to obtain a disparity image; An upscale matching unit for performing an up-scale stereo matching of the upper layer based on the transition image of the previous lower layer to acquire a transition image of the upper layer; An up-sampling unit for performing up-sampling based on the disparity image of the upper layer to acquire a disparity image having improved resolution; And a range term calculation unit for calculating a weighted range term by calculating a range term using an inductive image of the same layer and applying the weighted weight to the mutation image of the corresponding layer to obtain a weighted mutation image.

The processor may further include a variation normalization unit that normalizes the weighted variation image.

Wherein the stereo matching unit performs stereo matching on the left and right images of the lowest layer of the image pyramid at the highest position of the image pyramid to obtain a left side image and a right side image and considers the consistency of the left image and the right image It is possible to perform the consistency correction on the left side image and the right side image based on the weights.

Wherein the upscale matching unit sets a search range including a predetermined number of surrounding pixels around a position at which a variation value of a lower layer shift image is multiplied by a predetermined multiple and performs stereo matching on the search range, A variation image of a layer can be obtained.

The upsampling unit may perform adaptive upsampling by converting the upsampling unit based on a weight variation of an upper layer.

Wherein the upsampling unit obtains a weighted mutation image of the upper layer according to a weight of an upper layer obtained based on a weight of a lower layer's mutation image obtained by downsampling the mutation image of the upper layer, The adaptive upsampling can be performed by obtaining the weighted disparity image interpolated using the disparity image and the weighted disparity image of the upper layer.

According to the embodiment of the present invention, stereo matching is performed in a process of performing hardware-optimized bidirectional filtering in which bidirectional filtering is performed using a hierarchical structure by preserving a boundary portion and performing a smoothing process, By reducing the time and using the improved variation results of each layer for stereo matching, the variation can be extracted while improving the boundary part.

In addition, the left / right consistency correction that maintains consistency of the left and right sides is also included, so that it is possible to extract faster and more accurate variations than before.

Figure 1 is a flow diagram of a hardware optimized bidirectional filtering method.
2 is an exemplary view showing an image pyramid.
Fig. 3 is a conceptual diagram showing a calculation of a weight based on an inductive image.
Figure 4 is a diagram illustrating performing adaptive upsampling.
5 is a flowchart of a variation extraction method according to an embodiment of the present invention.
FIG. 6 is an exemplary view showing consistency correction according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating that stereo matching is performed in an upper layer using a variation value obtained in a lower layer according to an embodiment of the present invention. FIG.
8 is a structural diagram of a mutation extraction apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise.

Hereinafter, a method of extracting a variation using hierarchical stereo matching and an apparatus thereof according to an embodiment of the present invention will be described.

Figure 1 is a flow diagram of a hardware optimized bidirectional filtering method.

A mutation image is acquired through stereo matching that detects corresponding points on two or more images. For example, a correspondence point is searched for a pixel position of the same object existing in two images acquired at different points in time, and a disparity of a corresponding point between two images is obtained. That is, one of a left image and a right image constituting a stereo image is set as a reference image and the other is set as a comparison image, and corresponding points corresponding to each other in the reference image and the comparison image are searched to find the corresponding point variation. Based on the obtained displacement values, the depth of the object is calculated based on the geometric relationship between the two images, and a depth map is obtained. Such stereo matching may be block-based matching, pixel-based matching, feature-based matching, or the like. Pixel-based matching is the task of finding corresponding points in the comparison image for each pixel in the reference image. Block-based matching is a task of dividing an image into fixed-size blocks and finding corresponding points in the comparison image for each block.

The original image and the mutated image are respectively down-sampled to generate an image pyramid (S100, S110). For example, the image pyramid is generated by downsampling each of the original image and the mutation image in the horizontal direction by ½ and down-sampling in the vertical direction by ½. The number of layers constituting the image pyramid depends on the size of the original image. When the number of layers constituting the image pyramid is N, the image corresponding to the level N is an image down-sampled by 1/4 based on the image of the immediately preceding level (n-1).

2 is an exemplary view showing an image pyramid.

Fig. 2 shows a case where the number of layers of the image pyramid is three layers. Level 1 is the original image, level 2 is the downsampled image of the original image to 1/4 size, and level 3 is the downsampled image of the level 2 image to 1/4 size. When downsampling the original image, downsampling is performed in such a manner that only one of adjacent pixels is selected, as shown in the right upper end of Fig. 2, in order to preserve the pixel value at the boundary portion.

On the other hand, when downsampling the disparity image, downsampling is performed in such a manner that the mean of adjacent pixels is obtained.

After generating the image pyramid for the original image and the image pyramid for the mutation image, the level 3 original image (referred to as an inductive image G) (S120, S130), which is the lowest layer of the image pyramid of the original image, A weight for a difference between pixel values between adjacent pixels is calculated (S140).

Fig. 3 is a conceptual diagram showing a calculation of a weight based on an inductive image.

The weight based on the difference of pixel values between adjacent pixels q based on the center pixel p can be calculated as follows.

Figure 112015058750858-pat00002

When the original image (inductive image G) is three channels of RGB, the absolute value of the difference between the largest value among the RGB value differences is selected and the weight is calculated.

Using Equation (2), the weight of the center pixel of the original image can be calculated as follows.

Figure 112015058750858-pat00003

Then, the weights calculated according to Equation (2) are substituted into the adjacent pixels

Figure 112015058750858-pat00004
(Q) of the pixel value between adjacent pixels (q) based on the center pixel (p) of the mutation image,
Figure 112015058750858-pat00005
Can be calculated.

Figure 112015058750858-pat00006

Also, the weight value

Figure 112015058750858-pat00007
Is applied to the mutation image, and a new mutation image to which a weight is applied to each mutation is obtained as follows.

Figure 112015058750858-pat00008

As described above, the weight value calculated from the original image of level 3 (derived image G) is used to calculate the weight value from the upper layer (original image of level 2).

And up-sampling is performed on the original image of the next higher layer level (S150 to S170).

Figure 4 is a diagram illustrating performing adaptive upsampling.

4,

Figure 112015058750858-pat00009
Represents the original image of the lower layer,
Figure 112015058750858-pat00010
Indicates a transition image of a lower layer. And
Figure 112015058750858-pat00011
Represents an original image of an upper layer,
Figure 112015058750858-pat00012
Represents a transition image of the upper layer.

First, the weight of the lower layer (e.g., level 3) original image is calculated as shown in Equation (6).

Figure 112015058750858-pat00013

As shown in Equation 6, the weight of the lower layer (here, level 3)

Figure 112015058750858-pat00014
And then calculates the weight of the upper layer
Figure 112015058750858-pat00015
Is calculated as follows.

Figure 112015058750858-pat00016

The weight of the upper layer

Figure 112015058750858-pat00017
The weight of the original image at the level 2 is calculated.

Figure 112015058750858-pat00018

Next, the weight variation image in the lower hierarchy

Figure 112015058750858-pat00019
And the weight variation image
Figure 112015058750858-pat00020
The weighted mutation image
Figure 112015058750858-pat00021
Is calculated as follows.

Figure 112015058750858-pat00022

Finally, the final variation image at level 2

Figure 112015058750858-pat00023
Is calculated as follows.

Figure 112015058750858-pat00024

This process is repeated for each layer according to the flowchart of FIG. 1, and the final image finally output at the highest hierarchical level (for example, level 1) is preserved to obtain a smoothed image.

This hardware-optimized bidirectional filtering method preserves the boundaries and improves the variation by approximating the existing bidirectional filter to a structure capable of parallel processing in hardware or GPU devices.

However, there is a phenomenon in which the left / right consistency deviates in the stereo matching. Stereo matching is usually obtained by calculating the variation value from the left image to the right image and the variation value from the right image to the left image, respectively. However, as described above, since the filtering for improving the variation using the inductive image is performed by calculating the weight using only the information of only one image without considering the correlation between the left and right images, If filtering is separately applied to each of the images, inconsistency may be inconsistent between the filtered left and right images. That is, there occurs a phenomenon that the mutual relationship between the mutation value of the left side image and the mutation value of the right side image does not coincide in both directions.

In the embodiment of the present invention, a mutual image is obtained by performing stereo matching based on a hierarchical structure, and bi-directional filtering is performed for each layer to maintain consistency of left and right sides. Here, the consistency of the left and right sides indicates that the correspondence relation between the mutation value of the left side image and the side image of the right side image coincide with each other in both directions.

5 is a flowchart of a variation extraction method according to an embodiment of the present invention.

The image pyramid is constructed by down-sampling two or more images acquired at different points of view (S300). The number of layers constituting the image pyramid depends on the size of the original image. When downsampling the original image, downsampling can be performed by selecting a near pixel value (2i, i is the coordinate value of the pixel) so that the pixel value of the boundary portion is stored, as described above. However, the present invention is not limited thereto.

When the image pyramid having a hierarchical structure with respect to the original image is configured, stereo matching is performed from the lowest hierarchical image having the smallest resolution (S310, S320).

Stereo matching is performed on the lowest hierarchical image to obtain an initial variation image (S330). A reference image is set as a left image and a right image constituting a stereo image and the other is set as a comparison image. Then, corresponding points corresponding to each other in the reference image and the comparison image are found, the variation of the corresponding point is found, The initial variation image is obtained based on the value of For example, an initial variation image can be obtained using local stereo matching using block matching. The cost aggregation method used in this case may be a sum of squared differences (SSD), a sum of absolute differences (SAD), or a relative ordering between pixels in a block. , It is possible to obtain the variation using the Winner-Takes-It-All method which takes one of the values showing the highest similarity cost by using one of various cost direct methods such as a census method using the same method. Since it is a well-known technique to obtain an initial mutation image based on a local algorithm for finding corresponding points on a pixel-by-pixel basis, a detailed description thereof will be omitted here, and the local variation algorithm is not limited to using the local variation algorithm.

At this time, the consistency correction is performed simultaneously while matching the weights considering the left / right consistency.

FIG. 6 is an exemplary view showing consistency correction according to an embodiment of the present invention.

Speaking as in Fig accompanying 6, the initial disparity image that is, the initial L / based on the right disparity image, the left side is the image (D L) point (x) the disparity D L (x) as in the initial right side corresponding points in the image (D R) initial right-hand side is the image (D R) for a point (x) D L (x) point pixel by distance, a point (x) of the initial left side image (D L) from this do. If you and the initial right-hand side is called image (D R) D L (x) pixels by off point [x + D L (x)] Variation value of D R (x) from a point (x) in, D R (x ') = D L (x), it can be seen that the left-right consistency is guaranteed.

6,

Figure 112015058750858-pat00025
Based on the difference value of the absolute value of the deviation, the left and right consistency value
Figure 112015058750858-pat00026
Can be calculated.

When the variation of the pixel p is corrected due to the inconsistency of the left and right coincidence, basically, the variation value selected as the most suitable pixel in adjacent pixels is used. As shown in FIG. 6, weights are calculated for adjacent pixels q by multiplying the similarity weight and the left and right consistency reliability C LR (x). The similarity weight is the cost obtained using the cost integration method selected for stereo matching.

The left-right coherence reliability is the left-

Figure 112015058750858-pat00027
Is greater than a certain threshold value (T), the left-right consistency reliability value C LR (x) becomes 0, and the left-right consistency value
Figure 112015058750858-pat00028
Is 0, the left-right coherence reliability value C LR (x) is 1 since it means that there is left-right coherence. And left and right coherence values
Figure 112015058750858-pat00029
Is less than or equal to a certain threshold value (T), the left-right consistency reliability value C LR (x) is inversely proportional to the left-right consistency value
Figure 112015058750858-pat00030
.

For the neighboring pixel q, the most appropriate weight among the weights obtained by multiplying the similarity weight and the left and right consistency reliability C LR (x) is selected, and the variation of the pixel q is replaced with the variation value of the center pixel p based on the weight.

When an initial variation image is given through the initial stereo matching, a range term is calculated using an inductive image of the same layer to obtain a weight value map and a weighted-disparity map. That is, when an initial mutation image for the lowest layer is given, a weight value for the lowest layer image is obtained using the original image of the lowest layer, i.e., an inductive image (see Equation 2), and the weighted image (See Equation 3) and a weighted disparity image (see Equations 4 and 5) in which the obtained weight is applied to the disparity image (S340).

Then, the weighted mutation image is divided into weighted images to obtain a normalized image (S350). The normalized image thus obtained is a weighted image through a range term and a smoothed image.

After this process is performed, an up-scale stereo matching is performed on the image of the upper layer of the lowest layer (S360, S370) based on the variation value of the normalized image of the lowest layer obtained previously (S380 ).

FIG. 7 is a diagram illustrating that stereo matching is performed in an upper layer using a variation value obtained in a lower layer according to an embodiment of the present invention. FIG.

A searching area including a predetermined number of neighboring pixels corresponding to a centered on the position when the variation value of the normalized image of the lowest layer (for example, level 3) obtained above is multiplied by a predetermined number is set And performs a stereo matching with the comparison image with respect to the search range to estimate the variation value.

The stereo matching is performed at an upper layer (for example, level 2) based on the lower layer variation value. As shown in FIG. 7, the level 2 original image (one of the left image and the right image, I) is divided by the scale variable S to obtain the coordinates of the previous transition value. The scale variable S

Figure 112015058750858-pat00031
Where W s represents the width of the source image (here, the original image of level 2), and W d represents the width of the variation image (here, the variation image of level 3). Where the coordinates of the previous displacement value represent the corresponding coordinates in the level 3 normalized image.

The coordinates of the previous variation value are obtained, and then the variation value of the position corresponding to the obtained coordinates

Figure 112015058750858-pat00032
And the mutation candidates including the mutation values of neighboring points.
Figure 112015058750858-pat00033
. At this time, the candidate of transition includes the variation of pixels located within the range of ± n pixels based on the position corresponding to the obtained coordinates.

Then, the search range is obtained. The search range can be calculated as follows.

Figure 112015058750858-pat00034

Here,? Represents a variable for search range correction considering an error range caused by an integer multiple (S) of the variation. For example, assuming that d3 is 3 and is an accurate value, d2 may be one of {5, 6, 7} instead of 6 (= 3 x 2). d3 = 3 may not be accurate, so the range of ± n pixels may be widened as necessary.

Stereo matching is performed within the search range to select the mutation value of the position with the highest degree of similarity to obtain the mutation image of the upper layer, that is, the level 2.

When the variation image at the level 2 is obtained, the variation image of the level 2 is downsampled to obtain the variation image of the level 3 again (S390). At this time, the level 3 variation image obtained by downsampling is different from that obtained by stereo matching.

Based on the variation image of level 3 obtained by downsampling, the weight variation image is obtained again (S400). That is, a range term for obtaining a weight is performed, a weight value for the corresponding image is obtained using the level 3 mutation image, and the weighted mutation image is obtained by applying the obtained weight to the mutation image.

Then, adaptive up-sampling is performed to improve the resolution (S410). As described above, the weights between the adjacent layer images are calculated. Based on the weight of the lower layer, level 3, the weight of the upper layer, that is, level 2, is obtained. And obtains a weighted variation image of level 2. Then, the weighted disparity image is calculated using the weighted disparity image in the lower layer and the weighted disparity image in the upper layer.

Here, the weight variation image in the lower layer

Figure 112015058750858-pat00035
Is a weighted disparity of the lower layer, so it is converted to a weight variation based on the upper layer to perform adaptive upsampling.

Thereafter, a range term for obtaining a weight value in the same layer is performed, a weight value for the corresponding image is obtained using the level 2 derived image, a weight image obtained by applying the obtained weight value to the guided image, and a weight value obtained are applied to the interpolated weight value variation image To obtain a weighted variation image (S340). Subsequently, the weighted mutation image is divided into weighted images to obtain a normalized image (S350).

This process is repeatedly performed up to the highest layer to finally obtain an improved mutation image.

The time complexity of the stereo matching can be reduced by using the hierarchical structure of the stereo matching method which finds the corresponding points in the stereo image and the disparity enhancement method which improves the initial matching result.

8 is a structural diagram of a mutation extraction apparatus according to an embodiment of the present invention.

7, the edge extraction apparatus 100 according to the embodiment of the present invention includes an image acquisition unit 110, a processor 120, a memory 130, and an output unit 140. [

The image obtaining unit 110 receives images including binocular or multi-view images acquired at the same time from at least two binocular or multi-view cameras. The input images may be stored in the memory 120.

The processor 120 may be configured to implement the methods described above with reference to Figures 5 and 6 above. The processor 110 includes a downsampling unit 121, a stereo matching unit 122, a range term computing unit 123, a normalization unit 124, an upscale matching unit 125, and an upsampling unit 126).

The downsampling unit 121 downsamples two or more images acquired at different points of time to construct an image pyramid.

The stereo matching unit 122 performs stereo matching on the original images of the respective layers constituting the image pyramid of the hierarchical structure to acquire the mutated images. At this time, the consistency correction is performed simultaneously while matching the weights considering the consistency of the left image and the right image.

The range term calculator 123 calculates a range term by using an inductive image (original image) of the same layer when a variation image is given to an arbitrary layer, obtains a weight value, and applies the obtained weight to the inductive image And obtains weighted images by applying the weighted values to the mutated images.

The variation normalization unit 124 divides the weighted variation image obtained by the range term calculation unit 123 into weighted images to obtain a normalized image. The normalized image is a roughly smoothed image.

On the other hand, the upscale matching unit 125 performs an up-scale stereo matching on the image of the upper layer based on the normalized image variation value obtained for the lower layer to obtain the variation image. The upscale matching unit 125 sets a search range with respect to the comparison image of the upper layer on the basis of the position when the variation value of the lower layer mutation image at the position corresponding to the arbitrary pixel of the upper layer reference image is multiplied by m , Performs stereo matching between the reference image and the comparison image in the search range, and selects the variation value of the position with the highest similarity to obtain the variation image of the corresponding layer (upper layer).

The upsampling unit 126 performs adaptive upsampling based on the disparity image transmitted from the upscale matching unit 125 and the original image of the corresponding layer. The weights between adjacent layers, that is, the lower layer and the upper layer image are obtained, and the weighted mutation images in the lower layer and the weighted mutation images in the lower layer based on the weights are obtained. The weighted mutation images are obtained by interpolating the weighted mutation images, , A weighted disparity image at an upper layer is obtained through Equation (10).

The memory 130 is coupled to the processor 110 and stores various information related to the operation of the processor 120. [

The output unit 140 is connected to the processor 120 and displays an input image.

The embodiments of the present invention described above are not implemented only by the apparatus and method, but may be implemented through a program for realizing the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

Claims (14)

Performing stereo matching on two or more images of a plurality of image pyramid types to obtain a mutation image;
Performing an up-scale stereo matching of the upper layer with respect to a search range set based on a transition value of a previous lower layer, and acquiring a transition image of the upper layer;
Performing up-sampling on the disparity image of the upper layer to obtain a disparity image having improved resolution; And
Calculating a range term using an inductive image of the same layer to obtain a weight, and applying the obtained weight to the mutation image of the layer to obtain a weighted mutation image
. The method of claim 1, wherein
Normalizing the weighted mutation image
Further comprising the steps of: 3. The method of claim 2,
Obtaining a mutated image of the upper layer, acquiring an improved mutated image, acquiring the weighted mutation image, and obtaining a mutation image by repeatedly performing the normalizing step up to the highest layer Extraction method. The method according to claim 2, wherein
Wherein normalizing the weighted shifted image comprises weighting the shifted image as a weighted image, weighting the weighted image as a weighted value based on the range term, and applying the weighted image to the derived image of the corresponding layer, A mutation extraction method for obtaining a mutation image. The method according to claim 1,
The step of acquiring the mutation image of the upper layer
Performing stereo matching on the left and right images of the lowest layer of the image pyramid to obtain a left side image and a right side image; And
Performing coherence correction on the left side image and the right side image based on a weight that considers the consistency of the left image and the right image,
. The method according to claim 1,
The step of acquiring the mutation image of the upper layer
Setting a search range including a predetermined number of surrounding pixels around a position at which a variation value of a lower layer hierarchical transition image is multiplied by a predetermined number; And
Performing stereo matching on the search range to obtain a mutation image of the upper layer
. The method according to claim 1,
Wherein the step of acquiring the disparity image having the enhanced resolution further includes the step of performing adaptive upsampling by converting the disparity image based on the weight variation of the upper layer. 8. The method of claim 7,
Wherein the step of acquiring the image having the improved resolution comprises:
Sampling the disparity image of the upper layer to obtain a disparity image of the lower layer;
Obtaining a weighted disparity image of a lower layer based on the disparity image of the lower layer;
Obtaining a weighted mutation image of the upper layer according to a weight of an upper layer obtained based on the weight of the lower layer; And
Obtaining a weighted disparity image interpolated using the weighted disparity image of the lower layer and the weighted disparity image of the upper layer
. An image acquiring unit acquiring a plurality of images;
And a processor for acquiring a variation image based on the images,
The processor comprising:
A stereo matching unit for performing stereo matching on two or more images to obtain a variation image;
An upscale matching unit for performing an up-scale stereo matching of the upper layer with respect to a search range set based on a transition value of a transition image of a previous lower layer to acquire a transition image of the upper layer;
An up-sampling unit for performing up-sampling based on the disparity image of the upper layer to acquire a disparity image having improved resolution; And
Calculating a range term using an inductive image of the same layer to obtain a weight, and applying the obtained weight to the mutation image of the layer to obtain a weighted mutation image,
And a subtractor. The method of claim 9, wherein
The processor
A normalization unit for normalizing the weighted mutation image,
Further comprising: The method of claim 9, wherein
Wherein the stereo matching unit performs stereo matching on the left and right images of the lowest layer of the image pyramid in two or more images of a plurality of image pyramid types to obtain a left side image and a right side image, And a coherence correction unit for performing coherence correction on the left and right side images based on weights considering coherence of the left and right images. The method of claim 9, wherein
Wherein the upscale matching unit sets a search range including a predetermined number of surrounding pixels around a position at which a variation value of a lower layer shift image is multiplied by a predetermined multiple and performs stereo matching on the search range, A mutation extraction apparatus for obtaining a mutation image of a hierarchy. 10. The method of claim 9,
Wherein the upsampling unit performs adaptive upsampling by converting the upsampling unit based on a weight variation of an upper layer. 14. The method of claim 13,
The up-
A weighted mutation image of the upper layer is obtained according to a weight of an upper layer obtained on the basis of a weight of a lower layer image obtained by downsampling the image of the upper layer, Wherein the adaptive upsampling is performed by obtaining an interpolated weighted disparity image using the weighted disparity image of the layer.














KR1020150086143A 2015-06-17 2015-06-17 Method and apparatus for detecting disparity by using hierarchical stereo matching KR101766431B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020150086143A KR101766431B1 (en) 2015-06-17 2015-06-17 Method and apparatus for detecting disparity by using hierarchical stereo matching

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020150086143A KR101766431B1 (en) 2015-06-17 2015-06-17 Method and apparatus for detecting disparity by using hierarchical stereo matching

Publications (2)

Publication Number Publication Date
KR20160149088A KR20160149088A (en) 2016-12-27
KR101766431B1 true KR101766431B1 (en) 2017-08-08

Family

ID=57736708

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020150086143A KR101766431B1 (en) 2015-06-17 2015-06-17 Method and apparatus for detecting disparity by using hierarchical stereo matching

Country Status (1)

Country Link
KR (1) KR101766431B1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108460795A (en) * 2018-03-05 2018-08-28 成都通甲优博科技有限责任公司 Inverted pyramid formula binocular solid matching process and system

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109492649B (en) * 2018-10-31 2021-09-21 华南理工大学 Image pyramid distance measurement-based neighbor propagation stereo matching method
CN111899326A (en) * 2020-06-18 2020-11-06 苏州小优智能科技有限公司 Three-dimensional reconstruction method based on GPU parallel acceleration
CN116740161B (en) * 2023-08-14 2023-11-28 东莞市爱培科技术有限公司 Binocular stereo matching aggregation method

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100267259B1 (en) 1998-02-20 2000-10-16 박호군 Dense depth extraction method using multiple camera images and intermediate images synthesis method using said method

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100267259B1 (en) 1998-02-20 2000-10-16 박호군 Dense depth extraction method using multiple camera images and intermediate images synthesis method using said method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Min, Dongbo, and Kwanghoon Sohn. "Cost aggregation and occlusion handling with WLS in stereo matching." IEEE Transactions on Image Processing 17.8 (2008.08): 1431-1442.

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108460795A (en) * 2018-03-05 2018-08-28 成都通甲优博科技有限责任公司 Inverted pyramid formula binocular solid matching process and system

Also Published As

Publication number Publication date
KR20160149088A (en) 2016-12-27

Similar Documents

Publication Publication Date Title
US10818026B2 (en) Systems and methods for hybrid depth regularization
CN106651938B (en) A kind of depth map Enhancement Method merging high-resolution colour picture
Yang et al. Color-guided depth recovery from RGB-D data using an adaptive autoregressive model
CN108596975B (en) Stereo matching algorithm for weak texture region
CN107025660B (en) Method and device for determining image parallax of binocular dynamic vision sensor
CN107622480B (en) Kinect depth image enhancement method
JP2016508652A (en) Determining object occlusion in image sequences
Lo et al. Joint trilateral filtering for depth map super-resolution
US20140035909A1 (en) Systems and methods for generating a three-dimensional shape from stereo color images
KR101766431B1 (en) Method and apparatus for detecting disparity by using hierarchical stereo matching
CN107689050B (en) Depth image up-sampling method based on color image edge guide
KR20130120730A (en) Method for processing disparity space image
Tan et al. Stereo disparity through cost aggregation with guided filter
JP2018181047A (en) Three-dimensional shape model generating device, three-dimensional shape model generating method and program
CN108805841B (en) Depth map recovery and viewpoint synthesis optimization method based on color map guide
Keaomanee et al. Implementation of four kriging models for depth inpainting
Damjanović et al. Local stereo matching using adaptive local segmentation
Lee et al. Automatic 2d-to-3d conversion using multi-scale deep neural network
Yang et al. Hierarchical joint bilateral filtering for depth post-processing
Zhang et al. A new closed loop method of super-resolution for multi-view images
Zhang et al. Local stereo disparity estimation with novel cost aggregation for sub-pixel accuracy improvement in automotive applications
KR20160138847A (en) Method and apparatus for generating disparity image
KR101532642B1 (en) Depth information upsampling apparatus and method
Wang et al. Depth map restoration and upsampling for kinect v2 based on ir-depth consistency and joint adaptive kernel regression
KR101607504B1 (en) Stereo Matching Method using Summed Area Table Scheme

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
E701 Decision to grant or registration of patent right
GRNT Written decision to grant