CN104318576A - Super-pixel-level image global matching method - Google Patents

Abstract

A super-pixel-level image global matching method comprises the steps of obtaining an input image pair corrected by polar lines through a binocular stereo camera, calculating a self-adaption cross of each pixel of the input image pair so as to obtain a self-adaption window of the current pixel, calculating the matching cost of the pixel, adopting a replacement strategy to process a coverage area and adopting a suboptimum strategy to process an image boundary; establishing super-pixels for images, conducting planar fitting on a parallax value of each super-pixel area to determine reliable pixels and deleting obvious wrong planes so as to determine initial parallax plane set; calculating the matching cost of the super-pixels according to the obtained matching cost of the pixel, establishing a data item and a smoothing item and utilizing a Graph-Cut optimization algorithm to conduct continuous iteration on an energy equation so as to obtain a final parallax plane. The super-pixel-level image global matching method can effectively avoid the problem that image noise, distortion or pixel value abnormity and other situations easily occur in a weak texture area, a discontinuous-parallax area and the coverage area, is good in robustness and can obtain depth information more approximating to real scenes.

Description

A kind of image overall matching process of super-pixel rank

Technical field

The present invention relates to the technical field such as computer vision, image procossing, especially based on the stereo vision matching method of global optimization framework.

Background technology

Image overall matching process, also referred to as the solid matching method based on global optimization framework.Stereo matching problem is changed into an optimization problem by the mode of modeling by it, and by setting up energy function structure global optimization framework, last under this global optimization framework, utilize optimization algorithm to solve the optimum solution of energy function, obtain the Stereo matching optimum solution in a global sense.At present, solve the popular model of problems and have Markov random field model and Bayesian model and derivative model thereof; Common optimization method has figure to cut algorithm and belief propagation algorithm.Compared to local matching algorithm, based on the solid matching method of global optimization framework, often precision is higher, and result entirety is smoother; But it is abundant not also to there is detailed information, and edge zone there will be fuzzy, and lower etc. the shortcoming of execution efficiency.

Summary of the invention

Easily to produce the deficiency of noise, distortion or exceptional value etc. at weak texture region, parallax discontinuity zone and occlusion area in order to overcome existing image overall matching process, the invention provides one and can effectively avoid these situations, robustness is good, and can obtain more close to the image overall matching process of the super-pixel rank of the depth information of real scene.

The technical solution adopted for the present invention to solve the technical problems is:

An image overall matching process for super-pixel rank, described matching process comprises the steps:

1) scaling board, is utilized to demarcate binocular camera and obtain stereo pairs;

2), over-segmentation process, by input picture to resolving into color or uniform gray level region respectively, and suppose that parallax value seamlessly transits in that region, and parallax non-continuous event only appears on zone boundary;

3), for each pixel of stereo image pair calculates its self-adaptation cross, four-tuple represent that the left arm of pixel is long, right arm length, upperarm length and lower brachium.

To calculate for example, color similarity calculating is carried out, p to the one group of continuous print pixel be positioned on the left of pixel p on horizontal line _i=(x _p-i, y _p), i-th element (i is incremented to L from 1, and L is the maximum brachium preset, and it is responsible for the maximum matching window size controlling pixel p) on horizontal line on the left of expression pixel p; When $\underset{c\∈\{R,G,B\}}{\max }\left(\right|I_c\left(p\right)-I_c\left(p_i\right)\left|\right)>\mathrm{\τ}$ Time, i stops increasing progressively, order $h_p^{-}=i-1,$ Otherwise, order $h_p^{-}=L-1;$ In like manner, can obtain i _crepresent the intensity level of corresponding color component, τ is the degree of confidence controlling color similarity.

According to existing four-tuple the H (p) calculating it for pixel p represents the integration of pixel p on its horizontal line with V (p), H (p), and V (p) represents the integration of pixel p on its perpendicular line;

$\left\{\begin{array}{c}H\left(p\right)=\left\{(x,y)\right|x\∈[x_p-h_p^{-},x_p+h_p^{+}],y=y_p\}\\ V\left(\text{p}\right)=\left\{(x,y)\right|x=x_p,y=\∈[y_p-v_p^{-},y_p+v_p^{+}]\}\end{array}\right.---\left(3\right)$

H (p) and V (p) determines the self-adaptation cross of pixel p jointly;

4) be, its Adaptive matching window of each pixel assessment according to self-adaptation cross.According to the self-adaptation cross calculated, for each pixel p constructs Adaptive matching window U (p), the committed step building matching window U (p) is slided along vertical segmentation block V (p) of pixel p, does a domain integral operation to multiple horizontal segmentation block H (q);

$U\left(p\right)=\underset{q\∈V\left(p\right)}{\∪}H\left(q\right)---\left(4\right)$

Wherein, q is a pixel being positioned in vertical segmentation block V (p);

5) Adaptive matching window, is utilized to obtain Matching power flow

Calculate Adaptive matching window U (p) of left pixel p and the Adaptive matching window U'(p' of right pixel p' respectively), wherein, in left figure, have the pixel p=(x of parallax d _p, y _p) in right figure, find respective pixel p'=(x _p-d, y _p), pixel p and pixel p ' between Matching power flow computing formula as follows,

$C_d\left(p\right)=\frac{1}{\left|\right|U_d\left(p\right)\left|\right|}*\underset{t\∈U_d\left(p\right)}{\mathrm{\Σ}}{e}_d\left(t\right)*(\log \mathrm{\θ}+1)---\left(5.1\right)$

In formula, U _d(p)=and (x, y) | (x, y) ∈ U (p), (x-d, y) ∈ U'(p'), e _dt () represents the undressed Matching power flow having the pixel t of parallax d, U _dp () is the associating matching window only comprising valid pixel; || U _d(p) || represent U _dp the quantity of pixel in (), it is used to standardize the Matching power flow that cluster completes utilize respective pixel pair, calculate undressed Matching power flow, when parallax is d, the Matching power flow computing formula in left figure in t and right figure between t ' is as follows:

$e_d\left(t\right)\text{=min}\left(\underset{c\∈\{R,G,B\}}{\mathrm{\Σ}}\right|I_c\left(t\right)-I_c^{\′}\left(t^{\′}\right)|,T)---\left(5.2\right)$

Wherein, what T controlled Matching power flow blocks the upper limit, I _ct () represents the color value of t pixel corresponding color component c, I _c' (t') represent the color value of pixel t ' corresponding color component c;

6) Optimized Matching cost is carried out by process occlusion area and image boundary

D (p) represents pixel p=(x in left figure _p, y _p) parallax, simultaneously d ' (p ') represent pixel p in right figure '=(x _p-d (p), y _p) parallax, d (p) the > d'(p' if d (p), d ' (p ') and d (p ") satisfy condition simultaneously) and d'(p')≤d (p "), wherein p "=(x _p-d (p)+d'(p'), y _p), use such replacement policy: with pixel p in right figure ' Matching power flow replace the Matching power flow of pixel p in original left figure;

As (x _p-d (p)) < 1 time, respective pixel p ' will be positioned at the outside of right figure, and this just means that we can not utilize respective pixel to obtain Matching power flow C _d(p), then the Matching power flow C finding a kind of suboptimum _d^(p),

$d^=\underset{d\&Element;[d_{\min },d_{\max }],(x_d-d)>0,d\&NotEqual;d*}{\arg \min }{C}_d\left(p\right)---\left(6.1\right)$

Wherein, d ^{^}represent suboptimum mark, d ^*represent optimum mark, their computing formula is as follows:

$d*=\underset{d\&Element;[d_{\min },d_{\max }],(x_d-d)>0}{\arg \min }{C}_d\left(p\right)---(6.2)$

Finally, as (x _p-d (p)) < 1 time, use C _d^p () represents the Matching power flow of pixel p;

7), determine initial plane collection, disparity plane can characterize with three parameter a, b, c, the common parallax d=ax+by+c determining certain pixel p (x, y) in figure, and { a, b, c} represent disparity plane to adopt tlv triple;

8) Matching power flow of super-pixel, is calculated

Calculate the Matching power flow of super-pixel rank according to the Matching power flow of the pixel scale obtained, pixel scale Matching power flow refers to pixel p (x, y) the Matching power flow C corresponding when it gets parallax d _d(p), and super-pixel rank Matching power flow refers to super-pixel S gets disparity plane P{a, b, the Matching power flow C obtained during c} _s(P);

First do reliability to every block super-pixel to judge, if by its matching disparity plane formula mistake out, so just assert that this block super-pixel is insecure; Otherwise then think that it is reliable, when super-pixel is reliable, its Matching power flow computing formula is as follows:

$C_s\left(P\right)=\underset{p\&Element;S-O}{\mathrm{\&Sigma;}}{C}_{\mathrm{ax}+\mathrm{by}+c}\left(p\right)---\left(8.1\right)$

Wherein O represents the set of occluded pixels in super-pixel S, and when super-pixel is unreliable, its Matching power flow computing formula is as follows:

$C_s\left(P\right)=\underset{p\&Element;S-O+U}{\mathrm{\&Sigma;}}{C}_{\mathrm{ax}+\mathrm{by}+c}\left(p\right)---(8.2)$

Wherein U represents the pixel set through repairing;

9), compose data items

After having had the Matching power flow of super-pixel, just can find a unique tags f, to disparity plane f (S) the ∈ D that every block super-pixel S ∈ R mono-is corresponding, R refers to the super-pixel set of input picture, and D represents disparity plane set, data item E _dataf () is the set of a reliable pixel matching cost, its constructive formula is as follows:

$E_{\mathrm{data}}\left(f\right)=\underset{S\&Element;R}{\mathrm{\&Sigma;}}{C}_S\left(f\left(S\right)\right)---\left(9\right)$

10), level and smooth item is constructed

The colouring information of this block super-pixel is characterized, level and smooth item E with the mean value of all pixel color information in super-pixel _smoothf the constructive formula of () is as follows:

$E_{\mathrm{smooth}}\left(f\right)=\underset{\&ForAll;(S_i,S_j)\&Element;S_N|f\left(S_i\right)\&NotEqual;f\left(S_j\right)}{\mathrm{\&Sigma;}}\mathrm{ColorD}(S_i,S_j)*\mathrm{PlaneD}(S_i,S_j)*\mathrm{CommonB}(S_i,S_j)---\left(10\right)$

Wherein, S _nrepresent the set of all neighbouring super pixels in input figure, ColorD (S _i, S _j) represent color distortion between neighbouring super pixels, namely ask the Euclidean distance of neighbouring super pixels on RGB color space; PlaneD (S _i, S _j) represent disparity plane difference between neighbouring super pixels, namely ask { the Euclidean distance on a, b, c} plane space; CommonD (S _i, S _j) represent the length of common boundary between neighbouring super pixels;

11), utilize Graph-Cut optimization algorithm to carry out energy minimization to energy equation, energy equation comprises data item and level and smooth item, obtains final parallax information.

Further, described step 11) in, utilize minimal cut to operate and realize minimizing of global energy equation, step is as follows: the span of CurrentTagValue α is from minimum parallax d _minto maximum disparity d _max, following operation is done to each value of α, α-expansion is done to label f and processes, find label f*, make the ENERGY E of f* (f*) minimum, as E (f*) <E (f), upgrade f=f*.Accordingly, when having traveled through span [d _min, d _max], find not do renewal rewards theory to f, then minimization process terminates; Otherwise, repeat to start step.

Further again, described step 1) in, setting plane reference plate is the position being positioned at z=0 in world coordinate system, draws according to camera linear imaging model:

$s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=A\left[\begin{array}{cccc}{r}_1& r_2& r_3& t\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 0\\ 1\end{array}\right]=A\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]---\left(1.1\right)$

Wherein, s is arbitrary scale factor, and A is camera intrinsic parameter, and [R, t] is the combination of rotation and translation matrix, and its characterizes the relation of world coordinate system and camera coordinates system, r _irepresent i-th row of rotation matrix R;

Make H=A [r ₁r ₂t], according to the coordinate of monumented point on scaling board and its imaging corresponding point, we can obtain H, if h _irepresent i-th row of H, therefore

[h ₁ h ₂ h ₃]＝A[r ₁ r ₂ t] (1.2)

Due to r ₁with r ₂for two row of rotation matrix R, they are mutually orthogonal and mould length is 1, and two constraint conditions obtaining internal reference are thus:

$\begin{array}{c}{h}_1^T{A}^{-T}{A}^{-1}{h}_2=0\\ h_1^T{A}^{-T}{A}^{-1}{h}_1=h_2^T{A}^{-T}{A}^{-1}{h}_2\end{array}---\left(1.3\right)$

Order $B=A^{-T}{A}^{-1}=\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{21}& B_{22}& B_{23}\\ B_{31}& B_{32}& B_{33}\end{array}\right],$ Wherein for symmetric matrix meets B=B ^t, definition six-vector b=[B ₁₁b ₁₂b ₂₂b ₁₃b ₂₃b ₃₃], then make the i-th row h of H matrix _i=[h _i1, h _i2, h _i3] ^t, thus obtain:

$h_i^T{\mathrm{Bh}}_j=v_{\mathrm{ij}}b---\left(1.4\right)$

Wherein, vi _j=[h _i1h _j1, h _i1h _j2+ h _i2h _j1, h _i2h _j2, h _i3h _j1+ h _i1h _j3, h _i3h _j2+ h _i2h _j3, h _i3h _j3];

It is as follows that two Basic Constraint Equations (1.3) drawn from the relational matrix H provided are write as two homogeneous equations:

$\left[\begin{array}{c}{v}_{12}^T\\ {(v_{11}-v_{22})}^T\end{array}\right]b=0---\left(1.5\right)$

If observation n opens the plane reference plate of diverse location, the individual equation such such as formula (1.5) of n can be obtained:

Vb＝0 (1.6)

Here V is the matrix of 2n × 6, if n>=3, obtains a unique solution b; As b once calculate, by decomposing all inner parameters that just can calculate camera further, then these parameters are carried out to the nonlinear optimization of Levenberg-Marquardt, thus solve and obtain final camera inside and outside parameter, finally utilize binocular camera to obtain image pair.

Beneficial effect of the present invention is mainly manifested in: ask for image depth information by binocular stereo vision method, combines the method for super-pixel in the process, makes result robust and have more abundant detailed information more; And the method obtaining spatial depth information compared to traditional infrared ray, radar system etc. has the characteristic more cheap to equipment requirement, shows better practical ability.

Accompanying drawing explanation

Fig. 1 is the schematic diagram of H (p) and V (p).Horizontal black square represents H (p), and its length is vertical black square represents V (p), and its length is

Fig. 2 is the Adaptive matching window of pixel p.Vertical black square represents V (p), and horizontal black square represents H (p), empty wire frame representation adaptive region.

Fig. 3 is the composition of block.S is source point, and T is meeting point, N _irepresent super-pixel group.

Embodiment

Below in conjunction with accompanying drawing, the invention will be further described.

With reference to Fig. 1 ~ Fig. 3, a kind of image overall matching process of super-pixel rank, comprises the steps:

1) scaling board, is utilized to demarcate binocular camera and obtain stereo pairs.We suppose that plane reference plate is the position being positioned at z=0 in world coordinate system, draw according to camera linear imaging model:

$s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=A\left[\begin{array}{cccc}{r}_1& r_2& r_3& t\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 0\\ 1\end{array}\right]=A\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]---\left(1.1\right)$

Wherein, s is arbitrary scale factor, and A is camera intrinsic parameter, and [R, t] is the combination of rotation and translation matrix, and its characterizes the relation of world coordinate system and camera coordinates system, r _irepresent i-th row of rotation matrix R.

Make H=A [r ₁r ₂t], according to the coordinate of monumented point on scaling board and its imaging corresponding point, we can obtain H, if h _irepresent i-th row of H, therefore

[h ₁ h ₂ h ₃]＝A[r ₁ r ₂ t] (1.2)

Due to r ₁with r ₂for two row of rotation matrix R, they are mutually orthogonal and mould length is 1, and two constraint conditions obtaining internal reference are thus:

$\begin{array}{c}{h}_1^T{A}^{-T}{A}^{-1}{h}_2=0\\ h_1^T{A}^{-T}{A}^{-1}{h}_1=h_2^T{A}^{-T}{A}^{-1}{h}_2\end{array}---\left(1.3\right)$

Order $B=A^{-T}{A}^{-1}=\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{21}& B_{22}& B_{23}\\ B_{31}& B_{32}& B_{33}\end{array}\right],$ Wherein for symmetric matrix meets B=B ^t.Definition six-vector b=[B ₁₁b ₁₂b ₂₂b ₁₃b ₂₃b ₃₃], then make the i-th row h of H matrix _i=[h _i1, h _i2, h _i3] ^t, thus obtain:

$h_i^T{\mathrm{Bh}}_j=v_{\mathrm{ij}}b---\left(1.4\right)$

Wherein, vi _j=[h _i1h _j1, h _i1h _j2+ h _i2h _j1, h _i2h _j2, h _i3h _j1+ h _i1h _j3, h _i3h _j2+ h _i2h _j3, h _i3h _j3].

It is as follows that two Basic Constraint Equations (1.3) drawn from the relational matrix H provided can be write as two homogeneous equations:

$\left[\begin{array}{c}{v}_{12}^T\\ {(v_{11}-v_{22})}^T\end{array}\right]b=0---\left(1.5\right)$

If observation n opens the plane reference plate of diverse location, the individual equation such such as formula (1.5) of n can be obtained:

Vb＝0 (1.6)

Here V is the matrix of 2n × 6.If n>=3, we just can obtain a unique solution b; As b once calculate, by decomposing all inner parameters that just can calculate camera further.Then these parameters are carried out to the nonlinear optimization of Levenberg-Marquardt, thus solve and obtain final camera inside and outside parameter.Binocular camera is finally utilized to obtain image pair.

2), over-segmentation process.By input picture to resolving into color or uniform gray level region respectively, and suppose, parallax value seamlessly transits in that region, and parallax non-continuous event only appears on zone boundary.Why we are in conjunction with super-pixel, exactly because it contributes to this hypothesis met in real scene.We adopt the Mean Shift color segmentation algorithm proposed by Comaniciu and Meer, and this method is defined as a kind of gradient rising search strategy originally, and this strategy can maximize a density function be defined on high-dimensional feature space.This feature space defines a higher dimensional space coordinate system, and all association attributeses related to then associate each dimension of this higher dimensional space.The maximum advantage of Mean Shift method based on boundary information.

3), for each pixel of stereo image pair calculates its self-adaptation cross, four-tuple represent that the left arm of pixel is long, right arm length, upperarm length and lower brachium.

To calculate for example, we carry out color similarity calculating, p to the one group of continuous print pixel be positioned on the left of pixel p on horizontal line _i=(x _p-i, y _p), i-th element (i is incremented to L from 1, and L is the maximum brachium preset, and it is responsible for the maximum matching window size controlling pixel p) on horizontal line on the left of expression pixel p.When $\underset{c\&Element;\{R,G,B\}}{\max }\left(\right|I_c\left(p\right)-I_c\left(p_i\right)\left|\right)>\mathrm{\&tau;}$ Time, i stops increasing progressively, order $h_p^{-}=i-1,$ Otherwise, order $h_p^{-}=L-1.$ In like manner, can obtain i _crepresent the intensity level of corresponding color component, τ controls the degree of confidence of color similarity.

According to existing four-tuple we can calculate its H (p) and V (p) for pixel p.H (p) represents the integration of pixel p on its horizontal line, and V (p) represents the integration of pixel p on its perpendicular line.

$\left\{\begin{array}{c}H\left(p\right)=\left\{(x,y)\right|x\&Element;[x_p-h_p^{-},x_p+h_p^{+}],y=y_p\}\\ V\left(\text{p}\right)=\left\{(x,y)\right|x=x_p,y=\&Element;[y_p-v_p^{-},y_p+v_p^{+}]\}\end{array}\right.---\left(3\right)$

As shown in Figure 1, H (p) and V (p) determines the self-adaptation cross of pixel p jointly.

4) be, its Adaptive matching window of each pixel assessment according to self-adaptation cross.According to the self-adaptation cross calculated, we can construct Adaptive matching window U (p) for each pixel p easily.The committed step building matching window U (p) is slided along vertical segmentation block V (p) of pixel p, does a domain integral operation, as shown in Figure 2 to multiple horizontal segmentation block H (q).

$U\left(p\right)=\underset{q\&Element;V\left(p\right)}{\&cup;}H\left(q\right)---\left(4\right)$

Wherein, q is a pixel being positioned in vertical segmentation block V (p).

5) Adaptive matching window, is utilized to obtain Matching power flow.In order to obtain reliable Matching power flow set, we calculate Adaptive matching window U (p) of left pixel p and the Adaptive matching window U'(p' of right pixel p' respectively).

Wherein, the pixel p=(x of parallax d is had in left figure _p, y _p) respective pixel p'=(x can be found in right figure _p-d, y _p).Pixel p and pixel p ' between Matching power flow computing formula as follows,

$C_d\left(p\right)=\frac{1}{\left|\right|U_d\left(p\right)\left|\right|}*\underset{t\&Element;U_d\left(p\right)}{\mathrm{\&Sigma;}}{e}_d\left(t\right)*(\log \mathrm{\&theta;}+1)---\left(5.1\right)$

In formula, U _d(p)=and (x, y) | (x, y) ∈ U (p), (x-d, y) ∈ U'(p'), e _dt () represents the undressed Matching power flow having the pixel t of parallax d, U _dp () is the associating matching window only comprising valid pixel.|| U _d(p) || represent U _dp the quantity of pixel in (), it is used to standardize the Matching power flow that cluster completes utilize respective pixel pair, calculate undressed Matching power flow.Such as, when parallax is d, the Matching power flow computing formula in left figure in t and right figure between t ' is as follows:

$e_d\left(t\right)\text{=min}\left(\underset{c\&Element;\{R,G,B\}}{\mathrm{\&Sigma;}}\right|I_c\left(t\right)-I_c^{\&prime;}\left(t^{\&prime;}\right)|,T)---\left(5.2\right)$

Wherein, what T controlled Matching power flow blocks the upper limit.I _ct () represents the color value of t pixel corresponding color component c, I _c' (t') represent the color value of pixel t ' corresponding color component c.

6), Optimized Matching cost is carried out by process occlusion area and image boundary.By the inspiration of five kinds of occlusion area disposal routes that the people such as Geoffrey Egnal propose, when processing occlusion area, we have employed a kind of replacement policy.Owing to there is a common hypothesis: in one piece of close region, the pixel with similar intensity value has similar parallax value.Therefore occlusion area Matching power flow can replace by the Matching power flow of so-called " correspondence " pixel.

Such as, d (p) represents pixel p=(x in left figure _p, y _p) parallax, simultaneously d ' (p ') represent pixel p in right figure '=(x _p-d (p), y _p) parallax.D (p) the > d'(p' if d (p), d ' (p ') and d (p ") satisfy condition simultaneously) and d'(p')≤d (p "), wherein p "=(x _p-d (p)+d'(p'), y _p), we can use such replacement policy: with pixel p in right figure ' Matching power flow replace the Matching power flow of pixel p in original left figure.

In order to obtain accurate anaglyph border, we neither use left and right consistency check to assess two width disparity maps, also do not use a kind of simple border to infer step, but have employed a kind of dominant strategy.As (x _p-d (p)) < 1 time, respective pixel p ' will be positioned at the outside of right figure, and this just means that we can not utilize respective pixel to obtain Matching power flow C _d(p), then the Matching power flow C finding a kind of suboptimum _d^(p),

$d^=\underset{d\&Element;[d_{\min },d_{\max }],(x_d-d)>0,d\&NotEqual;d*}{\arg \min }{C}_d\left(p\right)---\left(6.1\right)$

Wherein d ^{^}represent the suboptimum mark that we need, d ^*represent optimum mark, their computing formula is as follows:

$d*=\underset{d\&Element;[d_{\min },d_{\max }],(x_d-d)>0}{\arg \min }{C}_d\left(p\right)---(6.2)$

Finally, as (x _p-d (p)) < 1 time, we use C _d^p () represents the Matching power flow of pixel p.

7) initial plane collection, is determined.In our algorithm, real scene structure simulates with the disparity plane set of one group of two dimension.Disparity plane can characterize with three parameter a, b, c, and they can determine the parallax d=ax+by+c of certain pixel (x, y) in figure jointly, and therefore, this chapter adopts tlv triple, and { a, b, c} represent disparity plane.

According to introduction above, the parallax value of pixel is exactly discrete positive integer in fact, and quantity is limited and is artificial setting, therefore can travel through when looking for.And disparity plane { a, b, three components in c} are all continuous print, and cannot artificially set, if will all travel through, calculated amount will be tending towards infinitely great, so need to calculate an initial plane collection in advance to reduce calculated amount, this planar set then needs the structure that fully can characterize whole scene.

We are by carrying out plane fitting to the initial parallax value of every block super-pixel, and the disparity plane set finally obtained is as initial plane collection.Because be rely on the reliable pixel in super-pixel to carry out matching, so need to determine the reliable pixel in every block super-pixel, this chapter have employed the method for Cross-Checking to determine reliable pixel.After matching, also need to delete the disparity plane that those exist apparent error, such as disparity plane (0,0,0) etc.In last iteration optimization, disparity plane rally constantly convergence, until become the most reasonable planar set, namely comes to characterize whole scene structure the most all sidedly by minimum plane.

Next introduce plane fitting process, namely calculate the tlv triple { process of a, b, c}.Although disparity plane is formed by the parallax matching of pixel reliable in every block super-pixel, in reliable pixel, still may there is different value point.A kind ofly directly determine that the method for disparity plane parameter is used to determine a minimum planes solution.As everyone knows, minimum planes solution is very responsive to different value point, and comparatively speaking, linear or intermediate value solution has more robustness.

Here we are by a kind of decomposition method solving parameters respectively of application, produce a kind of solution with robustness.First, in super-pixel, utilize one group of reliable pixel parallax set calculated level slope be positioned on same level line.By all derivatives in whole insertion list, obtain by sorting to this list and applying Gaussian convolution the horizontal tilt rate that has robustness.Then, by similar method, rely on the reliable pixel parallax set being positioned at same perpendicular line to generate vertical bank rate.Finally, at the center of super-pixel, the slope obtained is utilized to obtain the parallax value that has robustness.The corresponding center parallax of each reliable pixel obtained by slope is inserted into a list, obtains the parallax value that has robustness as previously explained.

8) Matching power flow of super-pixel, is calculated.In the global registration algorithm of super-pixel rank, what each non-directed graph node was corresponding is one piece of super-pixel, as Fig. 3.The value of pixel is lineup is the discrete integer collection set, and the value of super-pixel is then one group of disparity plane set calculated.Therefore both are when application drawing cuts algorithm, also make a big difference.And the label f of correspondence also to be mapped to parallax by original pixel and becomes super-pixel and map to disparity plane.

First yes calculates Matching power flow, before calculated pixel scale Matching power flow, need the Matching power flow calculating super-pixel rank according to the Matching power flow of pixel scale obtained here.Pixel scale Matching power flow refers to pixel p (x, y) the Matching power flow C corresponding when it gets parallax d _d(p).And super-pixel rank Matching power flow refers to super-pixel S gets disparity plane P{a, b, the Matching power flow C obtained during c} _s(P).Here, first reliability is done to every block super-pixel and judge, if by its matching disparity plane formula mistake out, so just assert that this block super-pixel is insecure, otherwise, then think that it is reliable.When super-pixel is reliable, its Matching power flow computing formula is as follows:

$C_s\left(P\right)=\underset{p\&Element;S-O}{\mathrm{\&Sigma;}}{C}_{\mathrm{ax}+\mathrm{by}+c}\left(p\right)---\left(8.1\right)$

Wherein O represents the set of occluded pixels in super-pixel S.When super-pixel is unreliable, its Matching power flow computing formula is as follows:

$C_s\left(P\right)=\underset{p\&Element;S-O+U}{\mathrm{\&Sigma;}}{C}_{\mathrm{ax}+\mathrm{by}+c}\left(p\right)---(8.2)$

Wherein U represents the pixel set through repairing.

9), compose data items.After having had the Matching power flow of super-pixel, just can find a label f, disparity plane f (S) ∈ D corresponding to every block super-pixel S ∈ R mono-can give.R refers to the super-pixel set of input picture, and D represents disparity plane set.Here data item E _dataf () is the set of a reliable pixel matching cost, its constructive formula is as follows:

$E_{\mathrm{data}}\left(f\right)=\underset{S\&Element;R}{\mathrm{\&Sigma;}}{C}_S\left(f\left(S\right)\right)---\left(9\right)$

10), level and smooth item is constructed.This chapter, when constructing level and smooth item, mainly considers three factors between neighbouring super pixels: the length of common boundary between a. neighbouring super pixels; The difference of the disparity plane b. between neighbouring super pixels; The difference of the colouring information c. between neighbouring super pixels.Here for the sake of simplicity, the colouring information of this block super-pixel is characterized with the mean value of all pixel color information in super-pixel.Level and smooth item E _smoothf the constructive formula of () is as follows:

$E_{\mathrm{smooth}}\left(f\right)=\underset{\&ForAll;(S_i,S_j)\&Element;S_N|f\left(S_i\right)\&NotEqual;f\left(S_j\right)}{\mathrm{\&Sigma;}}\mathrm{ColorD}(S_i,S_j)*\mathrm{PlaneD}(S_i,S_j)*\mathrm{CommonB}(S_i,S_j)---\left(10\right)$

Wherein S _nrepresent the set of all neighbouring super pixels in input figure, ColorD (S _i, S _j) represent color distortion between neighbouring super pixels, namely ask the Euclidean distance of neighbouring super pixels on RGB color space; PlaneD (S _i, S _j) represent disparity plane difference between neighbouring super pixels, namely ask { the Euclidean distance on a, b, c} plane space; CommonD (S _i, S _j) represent the length of common boundary between neighbouring super pixels.

11) utilize Graph-Cut optimization algorithm to carry out energy minimization to energy equation (comprising data item and level and smooth item), obtain final parallax information.If utilize Graph-Cut to complete minimizing of whole energy equation, all energy terms so in energy equation all must meet submodule state simultaneously.According to increasing progressively principle, if each energy term in energy equation meets submodule state respectively, so whole energy equation just can meet submodule state.

Finally, we utilize the minimal cut operation on figure, achieve minimizing of global energy equation, as shown in Figure 3.By α-expansion moving algorithm, we obtain minimal cut result efficiently.

Detailed to minimize step as follows: the span of CurrentTagValue α is from minimum parallax d _minto maximum disparity d _max, following operation is done to each value of α, α-expansion is done to label f and processes, find label f*, make the ENERGY E of f* (f*) minimum, as E (f*) <E (f), upgrade f=f*.Accordingly, when having traveled through span [d _min, d _max], find not do renewal rewards theory to f, then minimization process terminates; Otherwise, repeat to start step.

In the present embodiment, the input picture pair corrected through polar curve is obtained by binocular stereo camera, the adaptive windows that self-adaptation cross obtains current pixel is calculated by each pixel right for image, and calculate Matching power flow for it, cover region and one time dominant strategy process image boundary in conjunction with a kind of replacement policy process; For image creation super-pixel, plane fitting is carried out to the parallax value in every block super-pixel region, use the method for Cross-Checking to determine reliable pixel, and delete manifest error plane to determine initial parallax planar set; According to the Matching power flow obtaining pixel matching cost and calculate super-pixel, and build data item and level and smooth item, Graph-Cut optimization algorithm is utilized to carry out continuous iteration to energy equation, obtain final disparity plane, i.e. picture depth, for providing more level and smooth robust more more accurate depth information based on the three-dimensional reconstruction of stereoscopic vision.

Claims (3)

1. an image overall matching process for super-pixel rank, is characterized in that: described matching process comprises the steps:

1) scaling board, is utilized to demarcate binocular camera and obtain stereo pairs;

2), over-segmentation process, by input picture to resolving into color or uniform gray level region respectively, and suppose that parallax value seamlessly transits in that region, and parallax non-continuous event only appears on zone boundary;

3), for each pixel of stereo image pair calculates its self-adaptation cross, four-tuple represent that the left arm of pixel is long, right arm length, upperarm length and lower brachium.

To calculate for example, color similarity calculating is carried out, p to the one group of continuous print pixel be positioned on the left of pixel p on horizontal line _i=(x _p-i, y _p), i-th element (i is incremented to L from 1, and L is the maximum brachium preset, and it is responsible for the maximum matching window size controlling pixel p) on horizontal line on the left of expression pixel p; When $\underset{c\&Element;\{R,G,B\}}{\max }\left(\right|I_c\left(p\right)-I_c\left(p_i\right)\left|\right)>\mathrm{\&tau;}$ Time, i stops increasing progressively, order $h_p^{-}=i-1,$ Otherwise, order $h_p^{-}=L-1;$ In like manner, can obtain i _crepresent the intensity level of corresponding color component, τ is the degree of confidence controlling color similarity.

According to existing four-tuple the H (p) calculating it for pixel p represents the integration of pixel p on its horizontal line with V (p), H (p), and V (p) represents the integration of pixel p on its perpendicular line;

$\left\{\begin{array}{c}H\left(p\right)=\left\{(x,y)\right|x\&Element;[x_p-h_p^{-},x_p+h_p^{+}],y=y_p\}\\ V\left(p\right)=\left\{(x,y)\right|x=x_p,y\&Element;[y_p-v_p^{-},y_p+v_p^{+}]\}\end{array}\right.---\left(3.1\right)$

H (p) and V (p) determines the self-adaptation cross of pixel p jointly;

4) be, its Adaptive matching window of each pixel assessment according to self-adaptation cross.According to the self-adaptation cross calculated, for each pixel p constructs Adaptive matching window U (p), the committed step building matching window U (p) is slided along vertical segmentation block V (p) of pixel p, does a domain integral operation to multiple horizontal segmentation block H (q):

$U\left(p\right)=\underset{q\&Element;V\left(p\right)}{\&cup;}H\left(q\right)---\left(4\right)$

Wherein, q is a pixel being positioned in vertical segmentation block V (p);

5) Adaptive matching window, is utilized to obtain Matching power flow

Calculate Adaptive matching window U (p) of left pixel p and the Adaptive matching window U'(p' of right pixel p' respectively), wherein, in left figure, have the pixel p=(x of parallax d _p, y _p) in right figure, find respective pixel p'=(x _p-d, y _p), pixel p and pixel p ' between Matching power flow computing formula as follows,

$C_d\left(p\right)=\frac{1}{\left|\right|U_d\left(p\right)\left|\right|}*\underset{t\&Element;U_d\left(p\right)}{\mathrm{\&Sigma;}}{e}_d\left(t\right)*(\log \mathrm{\&theta;}+1)---\left(5\right)$

In formula, U _d(p)=and (x, y) | (x, y) ∈ U (p), (x-d, y) ∈ U'(p'), e _dt () represents the undressed Matching power flow having the pixel t of parallax d, U _dp () is the associating matching window only comprising valid pixel; || U _d(p) || represent U _dp the quantity of pixel in (), it is used to standardize the Matching power flow that cluster completes utilize respective pixel pair, calculate undressed Matching power flow, when parallax is d, the Matching power flow computing formula in left figure in t and right figure between t ' is as follows:

$e_d\left(t\right)=\min \left(\underset{c\&Element;\{R,G,B\}}{\mathrm{\&Sigma;}}\right|I_c\left(t\right)-I_c^{\&prime;}\left(t^{\&prime;}\right)|,T)---\left(5.2\right)$

Wherein, what T controlled Matching power flow blocks the upper limit, I _ct () represents the color value of t pixel corresponding color component c, I _c' (t') represent the color value of pixel t ' corresponding color component c;

6), Optimized Matching cost is carried out by process occlusion area and image boundary

D (p) represents pixel p=(x in left figure _p, y _p) parallax, simultaneously d ' (p ') represent pixel p in right figure '=(x _p-d (p), y _p) parallax, d (p) the > d'(p' if d (p), d ' (p ') and d (p ") satisfy condition simultaneously) and d'(p')≤d (p "), wherein p "=(x _p-d (p)+d'(p'), y _p), use such replacement policy: with pixel p in right figure ' Matching power flow replace the Matching power flow of pixel p in original left figure;

As (x _p-d (p)) < 1 time, respective pixel p ' will be positioned at the outside of right figure, and this just means that we can not utilize respective pixel to obtain Matching power flow C _d(p), then the Matching power flow C finding a kind of suboptimum _d^(p),

$d^=\underset{d\&Element;[d_{\min },d_{\max }],(x_d-d)>0,d\&NotEqual;d^{*}}{\arg \min }{C}_d\left(p\right)---\left(6.1\right)$

Wherein, d^ represents that suboptimum marks, d ^*represent optimum mark, their computing formula is as follows:

$d^{*}=\underset{d\&Element;[d_{\min },d_{\max }],(x_d-d)>0}{\arg \min }{C}_d\left(p\right)---\left(6.2\right)$

Finally, as (x _p-d (p)) < 1 time, use C _d^p () represents the Matching power flow of pixel p;

7), determine initial plane collection, disparity plane can characterize with three parameter a, b, c, the common parallax d=ax+by+c determining certain pixel p (x, y) in figure, and { a, b, c} represent disparity plane to adopt tlv triple;

8) Matching power flow of super-pixel, is calculated

Calculate the Matching power flow of super-pixel rank according to the Matching power flow of the pixel scale obtained, pixel scale Matching power flow refers to pixel p (x, y) the Matching power flow C corresponding when it gets parallax d _d(p), and super-pixel rank Matching power flow refers to super-pixel S gets disparity plane P{a, b, the Matching power flow C obtained during c} _s(P);

First do a reliability to every block super-pixel to judge, if be wrong by its matching disparity plane out, so just assert that this block super-pixel is insecure; Otherwise then think that it is reliable, when super-pixel is reliable, its Matching power flow computing formula is as follows:

$C_s\left(P\right)=\underset{p\&Element;S-O}{\mathrm{\&Sigma;}}{C}_{\mathrm{ax}+\mathrm{by}+c}\left(p\right)---\left(8.1\right)$

Wherein O represents in super-pixel S the set of the pixel that is blocked, and when super-pixel is unreliable, its Matching power flow computing formula is as follows:

$C_s\left(P\right)=\underset{p\&Element;S-O+U}{\mathrm{\&Sigma;}}{C}_{\mathrm{ax}+\mathrm{by}+c}\left(p\right)---\left(8.2\right)$

Wherein U represents the pixel set through repairing;

9), compose data items

After having had the Matching power flow of super-pixel, just can find a unique tags f, to disparity plane f (S) the ∈ D that every block super-pixel S ∈ R mono-is corresponding, R refers to the super-pixel set of input picture, and D represents disparity plane set, data item E _dataf () is the set of a reliable pixel matching cost, its constructive formula is as follows:

$E_{\mathrm{data}}\left(f\right)=\underset{S\&Element;R}{\mathrm{\&Sigma;}}{C}_S\left(f\left(S\right)\right)---\left(9\right)$

10), level and smooth item is constructed

The colouring information of this block super-pixel is characterized, level and smooth item E with the mean value of all pixel color information in super-pixel _smoothf the constructive formula of () is as follows:

$E_{\mathrm{smooth}}\left(f\right)=\underset{\&ForAll;(S_i,S_j)\&Element;S_N|f\left(S_i\right)\&NotEqual;f\left(S_j\right)}{\mathrm{\&Sigma;}}\mathrm{ColorD}(S_i,S_j)*\mathrm{PlaneD}(S_i,S_j)*\mathrm{CommonB}(S_i,S_j)---\left(10\right)$

Wherein, S _nrepresent the set of all neighbouring super pixels in input figure, ColorD (S _i, S _j) represent color distortion between neighbouring super pixels, namely ask the Euclidean distance of neighbouring super pixels on RGB color space; PlaneD (S _i, S _j) represent disparity plane difference between neighbouring super pixels, namely ask { the Euclidean distance on a, b, c} plane space; CommonD (S _i, S _j) represent the length of common boundary between neighbouring super pixels;

11), utilize Graph-Cut optimization algorithm to carry out energy minimization to energy equation, energy equation comprises data item and level and smooth item, obtains final parallax information.

2. the image overall matching process of a kind of super-pixel rank as claimed in claim 1, it is characterized in that: described step 11) in, utilize minimal cut to operate and realize minimizing of global energy equation, step is as follows: the span of CurrentTagValue α is from minimum parallax d _minto maximum disparity d _max, following operation is done to each value of α, α-expansion is done to label f and processes, find label f*, make the ENERGY E of f* (f*) minimum, as E (f*) <E (f), upgrade f=f*.Accordingly, when having traveled through span [d _min, d _max], find not do renewal rewards theory to f, then minimization process terminates; Otherwise, repeat to start step.

3. the image overall matching process of a kind of super-pixel rank as claimed in claim 1 or 2, is characterized in that: described step 1) in, setting plane reference plate is the position being positioned at z=0 in world coordinate system, draws according to camera linear imaging model:

$s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=A\left[\begin{array}{cccc}{r}_1& r_2& r_3& t\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 0\\ 1\end{array}\right]=A\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]---\left(1.1\right)$

Wherein, s is arbitrary scale factor, and A is camera intrinsic parameter, and [R, t] is the combination of rotation and translation matrix, and its characterizes the relation of world coordinate system and camera coordinates system, r _irepresent i-th row of rotation matrix R;

Make H=A [r ₁r ₂t], according to the coordinate of monumented point on scaling board and its imaging corresponding point, we can obtain H, if h _irepresent i-th row of H, therefore

[h ₁ h ₂ h ₃]＝A[r ₁ r ₂ t] (1.2)

Due to r ₁with r ₂for two row of rotation matrix R, they are mutually orthogonal and mould length is 1, and two constraint conditions obtaining internal reference are thus:

$\begin{array}{c}{h}_1^T{A}^{-T}{A}^{-1}{h}_2=0\\ h_1^T{A}^{-T}{A}^{-1}{h}_1=h_2^T{A}^{-T}{A}^{-1}{h}_2\end{array}---\left(1.3\right)$

Order $B=A^{-T}{A}^{-1}=\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{21}& B_{22}& B_{23}\\ B_{31}& B_{32}& B_{33}\end{array}\right],$ Wherein for symmetric matrix meets B=B ^t, definition six-vector b=[B ₁₁b ₁₂b ₂₂b ₁₃b ₂₃b ₃₃], then make the i-th row h of H matrix _i=[h _i1, h _i2, h _i3] ^t, thus obtain:

$h_i^T{\mathrm{Bh}}_j=v_{\mathrm{ij}}b---\left(1.4\right)$

Wherein, v _ij=[h _i1h _j1, h _i1h _j2+ h _i2h _j1, h _i2h _j2, h _i3h _j1+ h _i1h _j3, h _i3h _j2+ h _i2h _j3, h _i3h _j3];

Two Basic Constraint Equations (3.3) drawn from the relational matrix H provided are write as two homogeneous equations as follows:

$\left[\begin{array}{c}{v}_{12}^T\\ {(v_{11}-v_{22})}^T\end{array}\right]b=0---\left(1.5\right)$

If observation n opens the plane reference plate of diverse location, the individual equation such such as formula (1.5) of n can be obtained:

Vb＝0 (1.6)

Here V is the matrix of 2n × 6, if n>=3, obtains a unique solution b; As b once calculate, by decomposing all inner parameters that just can calculate camera further, then these parameters are carried out to the nonlinear optimization of Levenberg-Marquardt, thus solve and obtain final camera inside and outside parameter, finally utilize binocular camera to obtain image pair.