CN108830895A - Differentially expanding moving method based on segmentation in a kind of Stereo matching - Google Patents

Abstract

The invention discloses a segmentation-based local expansion and movement method in stereo matching, which adopts an image segmentation method, can fully consider the edge information of the image during processing, and avoids the influence of parallax mutation on the result. The invention can effectively reduce the matching error of the image in the region lacking in texture, and has better parallax calculation result at the edge of the object than the local expansion moving method, and the overall accuracy is also higher. The present invention applies 3D labels when calculating parallax, and compared with discrete parallax labels, the obtained parallax is more accurate and the result is better; the present invention performs parallax calculation on the basis of segmentation, and maintains object edges better; the present invention It can effectively reduce the matching error of the image in the lack of texture area, and compared with the existing methods, the overall accuracy is also higher.

Description

A Segmentation-Based Local Expansion Movement Method in Stereo Matching

technical field

The invention relates to a segmentation-based local expansion moving method in stereo matching.

Background technique

As an important branch of computer vision, stereo vision matching is a process of converting two-dimensional position information into three-dimensional depth information by matching matching pixels between two or more images, thereby estimating the three-dimensional model of the scene. Stereo vision matching algorithms were first applied in the field of photogrammetry to automatically construct topographic height maps from overlapping aerial images. Today, stereo vision matching is widely used in 3D navigation, 3D reconstruction, driverless cars, intelligent video surveillance, remote sensing image analysis, medical imaging, robot intelligent control and other fields.

Although stereo vision matching technology has made great progress in recent years, it still faces many problems. These problems mainly come from the matching blur caused by three reasons: occlusion, lack of texture (or texture repetition), and radioactive differences caused by lighting and noise. The following mainly discusses the problem of lack of texture. Areas lacking texture widely exist in the real world, such as white walls, grounds, or interiors of objects in large areas of solid color. The lack of texture means that the image lacks feature points in this area. Not only will there be problems in the sparse stereo vision matching algorithm that cannot find the feature points and cannot be effectively matched, but even in the dense stereo vision matching algorithm. The cost is the same as that of other pixels in regions lacking texture causing matching errors. When calculating the similarity measure, a pixel may have the same matching cost as multiple pixels in the region. At this time, the matching cost between a single pixel cannot meet the requirements.

In order to improve the effect of the algorithm in areas lacking texture, more information needs to be used. The local method can increase the window when performing cost aggregation to take more pixels into account, but the increase of the window will make the algorithm more effective for texture-rich areas. Match results drop. Therefore, relying only on the information in the local window, it is difficult to further improve the accuracy of the stereo matching algorithm in areas lacking texture. Not only that, but with the increase of the window, the computational complexity of cost aggregation will also increase. The global method will take all the pixels of the image into account in the calculation, and the result is more accurate than the local method, but the computational complexity is also higher. Moreover, with the widespread application of 3D label methods in stereo matching, the search space of disparity has changed from a discrete label space to a continuous label space, which is equivalent to an infinite number of search results, which further aggravates the computational complexity of the global method. . In the global method, the graph cut method can use the expansion movement mechanism to update the labels of multiple pixels at the same time, unlike the belief propagation that requires pixel-by-pixel update, which avoids it from falling into the local optimum. The local expansion movement algorithm adopts this graph-cut-based expansion movement mechanism, which is applied to the local area, so that it can effectively estimate the 3D label of the pixel under the global system, and avoid falling into the local optimal solution. Simultaneously updating the labels of multiple pixels also increases the speed of the algorithm. However, this method only relies on the penalty and filter function of the MRF smoothing term to maintain the edge of the object, which makes it have some problems in the calculation of the parallax of the edge of the object.

Contents of the invention

The object of the present invention is to overcome the above-mentioned shortcoming of the prior art, provide a kind of local expansion movement method based on segmentation in stereo matching, aim at the local expansion movement method based on segmentation in the lack of texture area in the image, can effectively improve the algorithm in the lack of texture area accuracy while better preserving the edges of objects.

In order to achieve the above object, the present invention adopts the following technical solutions to achieve:

A segmentation-based local expansion movement method in stereo matching, comprising the following steps:

Step 1) Initialize:

Input an image pair, and use the left image and the right image in the image pair as a reference image and a target image, respectively;

Step 2) Matching cost calculation:

Using the matching cost based on CNN, the disparity space image DSI is calculated;

Step 3) Parallax selection:

The disparity calculation is performed by adopting the segmentation-based local expansion movement method disclosed in the present invention to obtain the original disparity map;

Step 4) Parallax refinement:

The final left disparity map and right disparity map are generated by left and right consistency detection, hole filling and weighted median filtering.

The further improvement of the present invention is:

The specific method of step 3) is as follows:

Step 3-1) using SLIC to segment the image; through segmentation, each pixel of the image is assigned a Label representing the Segment number, from 1 to the maximum number of segments K;

Step 3-2), for each Segment, traverse all pixels in the smallest rectangular area containing the Segment, and only when the pixel is inside the Segment, construct an expansion area and a filter window centered on the pixel;

Step 3-3) Apply the iterative α expansion algorithm to each pixel in the expansion area established in step 3-2). The initial α is determined by the current 3D label value of the randomly selected pixel inside the segment. Compared with when the pixel parallax label is α The system energy and pixel disparity label take the current value of the energy of the entire MRF system. If the system energy is smaller when α is taken, the label of the current pixel is updated to α, otherwise the current label remains unchanged; each iteration traverses the Segment All pixels of , expand α to the expansion area.

The energy function of the system energy in step 3-3) is as follows:

The energy function definition of MRF is shown in expression (1);

In expression (1), the first item is a data item, which measures the degree of agreement between the disparity function and the input image, and the second item is a smoothing item. If the disparity between adjacent pixel pairs (p,q)∈N Continuous, the item will be punished to ensure that the item changes smoothly inside the object, and only changes drastically at the edge of the object; λ is used to balance the weight of the data item and the smooth item, and the value of λ is 10.

The data items are as follows:

Use the slope block matching item to measure the consistency between the disparity function and the input image, and use the 3D label to improve the accuracy of the algorithm on the slope. The 3D label expresses the disparity d _{p of each pixel p} with 3 parameters. These three parameters represent a plane f _p , and the definition of data items is shown in expression (2);

In expression (2), the 3D label of the disparity of point p is represented by expression (3);

In expression (3), p _x and p _y represent the x coordinate and y coordinate of p respectively, and the three parameters form a triplet In this way, estimating the disparity d _p of pixel p is transformed into a problem of estimating the three parameter values of the triplet; these three parameters are determined by the random search strategy of PatchMatch, and a value z is randomly selected in the range of continuous disparity values ₀ is used as the parallax of the point (x ₀ , y ₀ ), and the random unit vector n=(n _x , n _y , n _z ) is used as the normal vector of the point (x ₀ , y ₀ , z ₀ ), and the three parameters are converted for:

In expression (2), W _p is a square window centered on p, and the weight ω _ps refers to ASW, using the method of guided image filtering. The specific definition is shown in expression (7);

In the expression (7), I _p = I _L (p)/255, which is a normalized color vector, μ _k and Σ _k are the mean and covariance of I _p in the window W _k ′ respectively, and e is In order to avoid overfitting;

given parallax plane The function ρ(s|f _p ) in the expression (2) measures the similarity between the pixel s in the window W _p and its corresponding point s′ in the right image, and the definition of s′ is shown in the expression (8);

The definition of ρ(s|f _p ) is shown in expression (9);

ρ(s|f _p )=min(C _CNN (s,s′),τ _CNN ) (9)

In expression (9), C _CNN is a CNN-based matching cost calculation method, and the truncation value τ _CNN increases the robustness of the matching cost to occluded areas; when calculating the data items of the right image, the relative changes in the parallax of the left and right images are opposite , if both p and s represent pixels in the right image at this time, the positions of s and s′ in expression (9) should be exchanged, and the minus sign in expression (8) should be changed to a plus sign.

The smoothing term is as follows:

The smooth term measures the distance between the labels (f _p , f _q ) generated by the pixel pair {p, q} under the label function f; the specific definition of the smooth term in expression (1) is as in expression (10) Show;

E _smooth (f _p ,f _q )＝max(w _pq ,ε)min(δ _pq (f _p ,f _q ),τ _dis ) (10)

In the expression (10), ε is a constant representing the lower bound of the weight w _pq , the value is very small, which can increase the robustness of the weight to noise, and δ _pq (f _p , f _q ) punishes f _p and f _q The discontinuity of the disparity between, the truncation value τ _dis will allow the drastic change of the disparity at the edge; the definition of the weight w _pq is shown in the expression (11);

In the expression (11), the parameter γ measures the influence of the color difference;

The definition of δ _pq (f _p , f _q ) is shown in expression (12);

δ _pq (f _p ,f _q )＝|d _p (f _p )-d _p (f _q )|+|d _q (f _q )-d _q (f _p )| (12)

In expression (12), the first item represents the difference in parallax of point p under the planes represented by the two labels f _p and f _q , and the second item is the difference in parallax of point q under these two labels.

Compared with the prior art, the present invention has the following beneficial effects:

The segmentation-based local expansion and movement method in the stereo matching disclosed by the present invention adopts the image segmentation method, which can fully consider the edge information of the image during processing, and avoid the influence of parallax mutation on the result. The invention can effectively reduce the matching error of the image in the region lacking in texture, and has better parallax calculation result at the edge of the object than the local expansion moving method, and the overall accuracy is also higher. It has the following advantages:

First: the present invention applies 3D labels when calculating parallax, compared with discrete parallax labels, the obtained parallax is more accurate and the result is better;

Second: the present invention performs parallax calculation on the basis of segmentation, and better maintains object edges;

Third: the present invention can effectively reduce the matching error of the image in the lack of texture area, and compared with the existing method, the overall accuracy is also higher.

Description of drawings

Fig. 1 is flow process of the present invention;

Fig. 2 is the process that the present invention carries out window selection;

Fig. 3 is the result comparison of the present invention and the existing method.

Detailed ways

The present invention is described in further detail below in conjunction with accompanying drawing:

Referring to Figure 1, in order to improve the accuracy of stereo vision matching for images lacking texture, the present invention adopts the idea of local expansion and movement, first applies the image segmentation method to divide the image into segments, and then applies local expansion and movement on the segments to maintain stereo matching The algorithm's disparity calculation can still achieve good results at the edge of the object. Concrete steps of the present invention are as follows:

Step 1) Initialize

Input an image pair, and use the left image and the right image in the image pair as a reference image and a target image, respectively;

Step 2) Matching cost calculation

Using the matching cost based on CNN, the disparity space image DSI is calculated;

Step 3) Parallax selection:

The disparity calculation is performed by adopting the segmentation-based local expansion movement method disclosed in the present invention to obtain the original disparity map; the specific steps are as follows:

Step 3-1) Use SLIC to segment the image. Through segmentation, each pixel of the image is assigned a Label representing the Segment number, from 1 to the maximum number of segments K;

Step 3-2), as shown in Figure 2, for each Segment, traverse all pixels in the smallest rectangular area containing the Segment (the rectangle surrounded by the dotted line in Figure 2), only when the pixel is in the Segment When it is inside, the expansion area (the window with radius r in Figure 2) and the filter window (the window with radius r+R in Figure 2) are constructed with the pixel as the center. Taking a Segment as an example, for each pixel in the Segment, use it as the center and r as the radius to construct an expansion area, and then expand the expansion area to expand R as the window for guiding the filter when calculating the energy function.

Step 3-3) Apply the iterative α expansion algorithm to each pixel in the expansion area established in step 3-2. The initial α is determined by the current 3D label value of the randomly selected pixel inside the segment. Compared with the system when the pixel disparity label is α Energy and pixel disparity The size of the energy of the entire MRF system when the label takes the current value. If the system energy is smaller when α is taken, the label of the current pixel is updated to α, otherwise the current label remains unchanged. Each iteration traverses all the pixels in the Segment, and expands α to the expansion area. Generally, after 7-9 iterations, the pixel can obtain a relatively ideal disparity label.

The energy function of the above system energy is as follows:

The local expansion moving algorithm is established on the basis of graph cut-based α expansion, and is essentially a global method, so the solution process is still a process of finding the minimum overall energy of the Markov random field system. The energy function definition of MRF is shown in expression (1).

In expression (1), the first item is a data item, which measures the degree of agreement between the disparity function and the input image, and the second item is a smoothing item. If the disparity between adjacent pixel pairs (p,q)∈N If continuous, the term will be penalized to ensure that the term changes smoothly inside the object, and only changes sharply at the edge of the object. λ is the weight used to balance the data item and the smoothing item, and the value of λ is 10. The specific definitions of the data term and the smoothing term will be given below.

1) Data item

Use the slope block matching item to measure the consistency between the disparity function and the input image, and use the 3D label to improve the accuracy of the algorithm on the slope. The 3D label expresses the disparity d _{p of each pixel p} with 3 parameters. These three The parameters represent a plane f _p , and the definition of data items is shown in expression (2).

In Expression (2), the 3D label of the disparity of point p is represented by Expression (3).

In expression (3), p _x and p _y represent the x coordinate and y coordinate of p respectively, and the three parameters form a triplet In this way, estimating the disparity d _p of pixel p is transformed into a problem of estimating the three parameter values of the triplet. These three parameters can be determined by the random search strategy of PatchMatch. A value z ₀ is randomly selected as the disparity of point (x ₀ , y ₀ ) in the range of continuous disparity values, and the random unit vector n=(n _x ,n _y ,n _z ) as the normal vector of the point (x ₀ ,y ₀ ,z ₀ ), the three parameters can be transformed into:

In the expression (2), W _p is a square window centered on p, and the weight ω _ps refers to ASW, but what is used in ASW is bilateral filtering, and the present invention uses guided image filtering with better performance, specifically defined as the expression Formula (7) shown.

In the expression (7), I _p = I _L (p)/255, which is a normalized color vector, μ _k and Σ _k are the mean and covariance of I _p in the window W _k ′ respectively, and e is In order to avoid overfitting.

given parallax plane The function ρ(s|f _p ) in the expression (2) measures the similarity between the pixel s in the window W _p and its corresponding point s′ in the right image, and the definition of s′ is shown in the expression (8).

The definition of ρ(s|f _p ) is shown in Expression (9).

ρ(s|f _p )=min(C _CNN (s,s′),τ _CNN ) (9)

In expression (9), C _CNN is a CNN-based matching cost calculation method proposed by Zbontar and LeCun, and the truncation value τ _CNN increases the robustness of the matching cost to occluded areas. When calculating the data items of the right image, the relative change of the parallax of the left and right images is opposite. If p and s both represent the pixels in the right image at this time, the positions of s and s′ in the expression (9) should be exchanged, and the expression The minus sign in formula (8) is changed to plus sign.

2) smooth item

The smoothness item measures the distance (similarity, smoothness) between the labels (f _p , f _q ) generated by the pixel pair {p,q} under the label function f. The present invention uses a curvature-based second-order smoothing term, and the specific definition of the smoothing term in expression (1) is shown in expression (10).

E _smooth (f _p ,f _q )＝max(w _pq ,ε)min(δ _pq (f _p ,f _q ),τ _dis ) (10)

In the expression (10), ε is a constant representing the lower bound of the weight w _pq , the value is very small, which can increase the robustness of the weight to noise, and δ _pq (f _p , f _q ) punishes f _p and f _q The discontinuity of the disparity between, the cut-off value τ _dis will allow the disparity to change sharply at the edges. The definition of weight w _pq is shown in expression (11).

In expression (11), the parameter γ measures the influence of color differences.

The definition of δ _pq (f _p , f _q ) is shown in expression (12).

δ _pq (f _p ,f _q )＝|d _p (f _p )-d _p (f _q )|+|d _q (f _q )-d _q (f _p )| (12)

In expression (12), the first item represents the difference in parallax of point p under the planes represented by the two labels f _p and f _q , and the second item is the difference in parallax of point q under these two labels.

Step 4) Parallax refinement:

The final left disparity map and right disparity map are generated by left and right consistency detection, hole filling and weighted median filtering.

As shown in Figure 3, Figure 3 shows the result comparison between the present invention and the existing local expansion and movement method, (a) represents the input left image, and (b) represents the Ground Truth of the image. (c)-(f) respectively show the depth map and error map generated by the segmentation-based local expansion moving method and the local expansion moving method proposed by the present invention (in the error map, the black pixels are error pixels, and the black area is more The larger the number, the more errors). It can be seen from (c) and (e) that the depth map of the method of the present invention is significantly better than the local expansion moving method, especially in the area around the hanger and the wall area. From (d) and (f), it can be found that compared with the local expansion moving method, the method of the present invention has fewer error pixels and the result is better. The reason is that the method of the present invention is based on the result of image segmentation, and the segmentation will maintain the edge information of the object. Compared with the direct application of local expansion movement, the segmentation-based local expansion movement method rarely occurs cross-plane expansion, while the local expansion movement Methods have no restrictions on this. These results all prove that the method of the present invention outperforms the local dilation move algorithm in preserving edges.

The above content is only to illustrate the technical ideas of the present invention, and cannot limit the protection scope of the present invention. Any changes made on the basis of the technical solutions according to the technical ideas proposed in the present invention shall fall within the scope of the claims of the present invention. within the scope of protection.

Claims (5)

1. the differentially expanding moving method in a kind of Stereo matching based on segmentation, which is characterized in that include the following steps：

Step 1) initialization：

Input picture pair, using the left image of image pair and right image as reference picture and target image；

Step 2) matching cost calculates：

Using the matching cost based on CNN, disparity space image DSI is calculated；

The selection of step 3) parallax：

Disparity computation is carried out based on the differentially expanding moving method of segmentation using disclosed by the invention, obtains original disparity map；

Step 4) parallax is refined：

Final left disparity map and right disparity map are generated by left and right consistency detection, holes filling and Weighted median filtering.

2. the differentially expanding moving method in Stereo matching according to claim 1 based on segmentation, which is characterized in that step 3) the specific method is as follows：

Step 3-1) image is split using SLIC；By segmentation, a table is assigned in each pixel of image The Label for showing Segment number, from 1 to maximum fractionation block number K；

Step 3-2), all pictures are traversed in the smallest rectangular area comprising the Segment for each Segment Element just constructs extended region and filter window only when the pixel is inside Segment centered on the pixel；

Step 3-3) each pixel applies iteration α Extension algorithm in the extended region that step 3-2) is established, initial α by The current 3D label value of pixel of Segment internal random selection determines, compare when pixel parallax label takes α system capacity with The size of pixel parallax label energy of entire MRF system when taking current value, if take α, system capacity is smaller, then updates current The label of pixel is α, otherwise keeps current label constant；All pixels in Segment are iterated over every time, by α to expansion Region intramedullary expansion.

3. the differentially expanding moving method in Stereo matching according to claim 2 based on segmentation, which is characterized in that step The energy function of system capacity in 3-3) is as follows：

The energy function definition of MRF is as shown in expression formula (1)；

In expression formula (1), first item is data item, measures the degree of agreement between parallax function and input picture, and Section 2 is Smooth item, if parallax in adjacent pixel between discontinuous (p, q) ∈ Ν, can punish this, guarantee this in object Portion is smooth change, and acute variation can just only occur in object edge；λ is the weight for equilibrium data item and smooth item, λ Value be 10.

4. the differentially expanding moving method in Stereo matching according to claim 3 based on segmentation, which is characterized in that data Item is as follows：

The identical property between parallax function and input picture is measured using ramp blocks occurrence, improves algorithm using 3D label Accuracy on inclined-plane, 3D label is by the parallax d of each pixel p_pIt is indicated with 3 parameters, these three parameters represent one and put down Face f_p, shown in the definition of data item such as expression formula (2)；

In expression formula (2), the 3D label of the parallax of point p is indicated by expression formula (3)；

In expression formula (3), p_xAnd p_yThe x coordinate and y-coordinate of p are respectively indicated, 3 parameters constitute a tripleThe parallax d of pixel p is estimated in this way_pThe problem of being converted into three parameter values of estimation triple；These three Parameter is determined by the random searching strategy of PatchMatch, and a value z is randomly choosed in the range of continuous parallax value₀Make For point (x₀,y₀) parallax, by random unit vector n=(n_x,n_y,n_z) it is used as point (x₀,y₀,z₀) normal vector, three parameters It is converted into：

In expression formula (2), W_pIt is the square window centered on p, weight ω_psWith reference to ASW, the side filtered using navigational figure Method is specifically defined as shown in expression formula (7)；

In expression formula (7), I_p=I_LIt (p)/255, is a normalized color vectors, μ_kAnd Σ_kIt is I respectively_pIn window W_k' in Mean value and covariance, e is in order to avoid over-fitting；

Given disparity planeIn expression formula (2) function ρ (s | f_p) measure window W_pInterior pixel s and its The similarity of corresponding points s ' in right image, shown in the definition of s ' such as expression formula (8)；

ρ(s|f_p) definition such as expression formula (9) shown in；

ρ(s|f_p)=min (C_CNN(s,s′),τ_CNN) (9)

In expression formula (9), C_CNNIt is the matching cost calculation method based on CNN, cutoff value τ_CNNMatching cost is increased for hiding Keep off the robustness in region；When calculating the data item of right image, the opposite variation of left images parallax is on the contrary, if p and s are equal at this time It indicates the pixel in right image, then to exchange the position of s and s ' in expression formula (9), the minus sign in expression formula (8) is changed to add Number.

5. the differentially expanding moving method in Stereo matching according to claim 3 based on segmentation, which is characterized in that smooth Item is as follows：

Smooth item measures the label (f for indicating that pixel generates { p, q } at label function f_p,f_qThe distance between)；Expression formula (1) smooth item in is specifically defined as shown in expression formula (10)；

E_smooth(f_p,f_q)=max (w_pq,ε)min(δ_pq(f_p,f_q),τ_dis) (10)

In expression formula (10), ε is an expression weight w_pqThe constant of lower bound is worth very little, can increase weight to the robust of noise Property, δ_pq(f_p,f_q) punishment f_pAnd f_qBetween parallax discontinuity, cutoff value τ_disParallax will be allowed in the violent change of edge Change；Weight w_pqDefinition such as expression formula (11) shown in；

In expression formula (11), parameter γ measures the influence of color difference；

δ_pq(f_p,f_q) definition such as expression formula (12) shown in；

δ_pq(f_p,f_q)=| d_p(f_p)-d_p(f_q)|+|d_q(f_q)-d_q(f_p)| (12)

In expression formula (12), first item indicates p point in f_pAnd f_qThe difference of parallax, Section 2 under the plane of the two tag representations It is then the difference of point q parallax under the two labels.