CN106447702A - Stereo image matching graph calculation method - Google Patents

Abstract

The invention belongs to the field of image processing and stereo visual technologies and provides a stereo image matching graph calculation method, for bringing a quite good pixel matching result for stereo image redirection. According to the technical scheme employed by the invention, the stereo image matching graph calculation method comprises the following steps: 1, establishment of an energy equation; 2, line selection based on dynamic planning; 3, establishment of a matching graph, i.e., after the line selection, line matching is performed according to a selected sequence, first of all, a matching line pair is obtained by searching for matched pixels for pixels in a line according to parallax relations, then, the line matching pair is removed temporarily so as to avoid repeated matching, next, a next line matching pair is sought, and the whole process lasts until each line has a matched line; and 4, matching relation establishment based on a matching graph. The method is mainly applied to image processing.

Description

A kind of Stereo image matching figure computational methods

Technical field

The invention belongs to image procossing, technical field of stereoscopic vision, specifically the present invention relates to a kind of for redirect Stereo image matching figure computational methods.

Background technology

Compared with plane picture, depth information that stereo-picture has orientation of can attaching most importance to brings valuable clue.So And, the preservation of this information also redirects for stereo-picture and brings new challenge.Stereo-picture redirects not only vision to be reduced Distortion, parallax distortion also to be reduced.The change of parallax value can affect the three-dimensional perception of image, even more so that stereoeffect disappears completely Lose.When plane picture redirection algorithm is applied directly to stereo-picture, owing to ignoring regarding between pixel in stereo-picture Difference relation, can bring serious parallax distortion.In order to reduce parallax distortion, stereo-picture redirection process should be noted that holding Matching relationship between pixel.

Existing method typically finds matched pixel by parallax relation.First the parallax of left view and right view is calculated Figure, further according to parallax relation, the pixel in left view finds its match point in right view.Such as discrete redirection side In method, the method for three-dimensional line clipping is through getting rid of the pixel matching in the view of left and right and resets stereo-picture To process.In the method, first calculate the reduction line in left view, further according to disparity map, find the sanction matching in right view Subtract line.Owing to the method does not accounts for spatial coherence, when the line needing cutting is more, it may appear that very serious distortion. Continuous print reorientation method, such as mesh transformations method, redirect place by adjusting the coordinate of grid vertex to image Reason.In this method, in left view, grid vertex or characteristic point find its coupling in right view according to parallax relation Point, more corresponding process is done to these match points, obtain final stereo-picture and redirect result.Due to only sparse to some Point carry out matching treatment, the change of other unmatched points is likely to cause serious parallax distortion.These methods are all first Pixel in left view is processed, then finds it at the matched pixel point of right view by disparity map, then do corresponding Process, preserve parallax relation, reduce distortion.Due to mistake in computation or the reason that block, have in some right views Point cannot match with the point in left view.In said method, it is not necessary to all of pixel is processed, so only finding out that Can put preferably matchingly a bit and process.If but running into the situation needing to carry out matching treatment to all pixels, directly Utilize disparity map to carry out coupling and can cause serious distortion.

The zoom factor calculating each pixel is needed to obtain based on the image reorientation method of pixel fusion final Redirection map picture.If the parallax relation mistake in computation of stereo-picture or run into and block a little, the pixel in left view can not be with Pixel in right view is mated one to one, and this results in some points in right view can not obtain match information.In order to give In right view, all of point finds the point matching, can according to the disparity map of right view find in left view with in right view The point that matches of pixel.But this method still there will be many-to-one situation, causes distortion, can not shelf space phase Guan Xing.

Content of the invention

For overcoming the deficiencies in the prior art, parallax relation is utilized to carry out the pixel in the view of left and right for simple Joining, producing multiple pixel sometimes has the situation of identical match point, it is contemplated that propose a kind of Stereo image matching figure meter Calculation method, redirects for stereo-picture and brings preferable pixel matching result.The technical solution used in the present invention is, stereo-picture Coupling figure computational methods, step is as follows：

1. the foundation of energy equation

Select lines matching to the pixel relationship representing between stereo pairs：First, carry out before On-line matching pixel Line options, sets up energy equation and carries out modelling process to line options, and this energy equation considers the selectivity characteristic of line simultaneously And matching properties, it is defined as follows：

E_total(i,j,j^±)=α E_select(i,j,j^±)+(1-α)·E_match(i,j),

Wherein i represents the abscissa of pixel, and j represents the ordinate of pixel, j^±Represent the vertical of the i-th-1 row selected pixels Coordinate.Use sensu lato line, say, that in i-1 row, selected pixels can be any one pixel of this row, and unlike Continuous print line is such, and in i-1 row, selected pixels must be the neighbor that the i-th row is chosen pixel, say, that j^±∈ {j-1,j,j+1}.α is a weighting factor, represents the importance of line options characteristic and lines matching characteristic, E_selectFor line options The energy equation of characteristic, E_matchEnergy equation for lines matching；

2. the line options based on Dynamic Programming

After establishing energy equation, set up cost matrix M according to this energy equation, and use dynamic programming method to carry out Line options；Pixel owing to being blocked does not has matched pixel point, it is impossible to is chosen, is therefore set to infinite by these cost value put (i, j)=∞, this constraint is it can be avoided that the pixel that is blocked is chosen, so cost matrix is as follows M：

(i, is j) binary map to O, represents coordinate for (i, whether pixel j) is blocked；

Selected line will remove in order to avoid repeating to select from cost matrix M, then recalculates cost matrix M and selects Select out next line, this dynamic programming process will repeat until certain a line in remaining pixel is all can not selected quilt Till blocking a little；

3. mate the foundation of figure

After line options, according to the order selecting, carry out lines matching.First, the pixel in line is found according to parallax relation Matched pixel obtains matched line pair.Then, lines matching is to being temporarily removed, to avoid repeated matching.Then next line is found It is right to mate.This process will continue until each line a matched line；

After lines matching, each line is to being allocated an ID, and ID represents the order of lines matching, and its value is from 1 to n.n Being the number of line options, in image pair, except those selected pixels, the remaining pixel that is blocked also can distribute an ID Value, from left to right, these pixels will one ID of distribution in order, ID value is from n+1 to W, and W is the width of original image, passes through This method, each pixel in every a line can have an ID value, and the matched pixel in right view also has identical ID value, and, be blocked in right view, a little also have a matched pixel, according to ID value, all of pixel be matched and And obtain coupling figure；

4. set up based on the matching relationship of coupling figure

After establishing coupling figure, mating the pixel of left and right view with coupling figure, in the view of left and right, ID value is identical Pixel be exactly the pixel matching, matched pixel is done identical process, the uniformity of left and right view can be preserved.

The coupling figure set up is in units of line；In conjunction with texture paging between pixel and parallax similitude, can Increase the accuracy of coupling；The coupling figure set up is surjection figure, can mate left and right view pixels one to one.

Wherein, the 1) energy equation of line options characteristic

Neighbor on the same line should be similar, and the pixel around line should have similar textural characteristics and deep Angle value.In view of this situation, line options energy equation E_selectByWithThis two parts form：

Represent the texture similarity between pixel；Represent the degree of depth similarity between pixel, energy equationIt is defined as follows：

Represent the similitude between pixel on the same line,It is the similitude between line surrounding pixel, It is defined as follows：

(i, j) represent coordinate in left view is (i, the textural characteristics value of pixel j) to T.The coordinate of line surrounding pixel be by I-th-1 row selected pixels coordinate determines, is divided into three kinds of situations,It is defined as follows：

Wherein

W=| T (i, j-1)-T (i, j+1) |,

Represent the energy equation of parallax similitude between pixelDefinition andSimilar, simply textural characteristics is changed Become parallax feature.

2) energy equation of lines matching

Calculate the correctness of pixel matching with coupling energy equation, this energy equation is made up of two parts：

E_match(i, j)=E_diff(i,j)+E_co_rrect(i,j).

E_diffRepresent the texture difference between matched pixel, be defined as follows：

Wherein D (i, j) represent coordinate position for (i, the parallax value of pixel j),Represent right view coordinate Position is (the textural characteristics value of the pixel of i, j+D (i, j)).Calculate the parallax distortion matching between pixel with equation the following：

Representing right view coordinate position is the (parallax value of the pixel of i, j+D (i, j)).

The feature of the present invention and providing the benefit that：

The present invention establishes a kind of coupling figure to replace original disparity correspondence relation, it is thus achieved that a pair between all pixels One matching relationship figure.This pixel matching is the coupling based on line, by increasing capacitance it is possible to increase spatial coherence, and this matching relationship is used for solid It during image redirects, is obtained in that preferable result.

Brief description：

Fig. 1 is original left and right view.

Fig. 2 is the left and right view after the redirection based on the coupling figure acquisition calculating.

Fig. 3 is flow chart.

Detailed description of the invention

For overcoming the deficiencies in the prior art, the invention provides a kind of Stereo image matching figure computational methods, described method Comprise the following steps：

1. the foundation of energy equation

Carrying out to the pixel in stereo-picture merging, insert, remove etc. and should be noted that between pixel, coupling is closed when processing The preservation of system.Lines matching to the relation that can show between the view of left and right therefore, it is possible to that avoids stereo-picture does not mates deformation. Meanwhile, lines matching is able to maintain that the spatial coherence of image.Therefore, select lines matching to the picture representing between stereo pairs Element relation.First, line options is carried out before On-line matching pixel.Set up energy equation and modelling process carried out to line options, This energy equation considers selectivity characteristic and the matching properties of line simultaneously, is defined as follows：

E_total(i,j,j^±)=α E_select(i,j,j^±)+(1-α)·E_match(i,j),

j^±Represent the ordinate of the i-th-1 row selected pixels.Use sensu lato line, say, that selected in i-1 row The pixel selected can be any one pixel of this row, and unlike continuous print line, in i-1 row, selected pixels must be I row is chosen the neighbor of pixel, say, that j^±∈{j-1,j,j+1}..α is a weighting factor, represents that line options is special Property and the importance of lines matching characteristic.

2. the line options based on Dynamic Programming

After establishing energy equation, set up cost matrix M according to this energy equation, and use dynamic programming method to carry out Line options.The low line of energy value will be selected first because the pixel on this line probably correctly coupling and with surrounding Pixel has higher similitude.Especially, left view can be seen and the invisible point of right view is referred to as being blocked a little.Due to The pixel being blocked does not has matched pixel point, it is impossible to be chosen, therefore these cost value put are set to infinite M (i, j)=∞, This constraint is it can be avoided that the pixel that is blocked is chosen.So cost matrix is as follows：

(i, is j) binary map to O, represents coordinate for (i, whether pixel j) is blocked.In the present invention, O (i, j) =1 represents coordinate position for (i, pixel j) is not blocked, and can mate；(i, j)=0 represents coordinate position is O (i, pixel j) is blocked.

Selected line will remove in order to avoid repeating to select from cost matrix M.Then, cost matrix M is recalculated Select next line.This dynamic programming process will repeat until certain a line in remaining pixel is all can not be selected Till being blocked a little.

3. mate the foundation of figure

After line options, according to the order selecting, carry out lines matching.First, the pixel in line is found according to parallax relation Matched pixel obtains matched line pair.Then, lines matching is to being temporarily removed, to avoid repeated matching.Then next line is found It is right to mate.This process will continue until each line a matched line.

After lines matching, each line is to can be allocated an ID.ID represents the order of lines matching, and its value is from 1 to n.n It is the number of line options.In image pair, except those selected pixels, the remaining pixel that is blocked also can distribute an ID Value.From left to right, these pixels will one ID of distribution in order, the ID value width of original image (W be) from n+1 to W.Pass through This method, each pixel in every a line can have an ID value, and the matched pixel in right view also has identical ID value.And, it is blocked in right view, a little also have a matched pixel.According to ID value, all of pixel be matched and And obtain coupling figure.

4. set up based on the matching relationship of coupling figure

After establishing coupling figure, with coupling figure, the pixel of left and right view is mated.In the view of left and right, ID value is identical Pixel be exactly the pixel matching.Matched pixel is done identical process, the uniformity of left and right view can be preserved.And Figure is surjection figure, solves the many-to-one mapping situation in parallax relation, thus saves spatial coherence, reduces and regard Feel distortion.

Preferred forms below by the stereo-picture redirection process explanation present invention：

1. the foundation of energy equation

Pixel in stereo-picture is being merged, is inserting, removing to wait and when processing, should be noted that between pixel coupling pass The preservation of system.Lines matching to the relation that can show between the view of left and right therefore, it is possible to that avoids stereo-picture does not mates deformation. And, lines matching is able to maintain that the spatial coherence of image.Therefore select lines matching to the picture representing between stereo pairs Element relation.First, before On-line matching pixel, need to carry out line options.The present invention is come to line options by setting up energy equation Carrying out modelling process, this energy equation considers selectivity characteristic and the matching properties of line simultaneously, is defined as follows：

E_total(i,j,j^±)=α E_select(i,j,j^±)+(1-α)·E_match(i,j),

j^±Represent the i-th-1 row selected pixels.We use sensu lato line, say, that selected in i-1 row Pixel can be any one pixel of this row, and unlike continuous print line, in i-1 row, selected pixels must be the i-th row It is chosen the neighbor of pixel, say, that j^±∈{j-1,j,j+1}..α is a weighting factor, represents that line options is special Property and the importance of lines matching characteristic.

1) energy equation of line options characteristic

Neighbor on the same line should be similar, and the pixel around line should have similar textural characteristics and deep Angle value.In view of this situation, line options energy equation E_selectByWithThis two parts form：

Represent the texture similarity between pixel；Represent the degree of depth similarity between pixel.Energy equation It is defined as follows：

Represent the similitude between pixel on the same line,It is the similitude between line surrounding pixel.'s It is defined as follows：

(i, j) represent coordinate position in left view is (i, the textural characteristics value of pixel j) to T.Line surrounding pixel is by The coordinate of i-1 row selected pixels determines, can be divided into three kinds of situations.It is defined as follows：

Wherein

W=| T (i, j-1)-T (i, j+1) |,

Represent the energy equation of parallax similitude between pixelDefinition andSimilar, simply textural characteristics is changed Become parallax feature.

2) energy equation of lines matching

In the present invention, the coupling between pixel is accomplished by parallax relation.The mistake of disparity map can result in not Correct pixel matching.Parallax relation calculates correct pixel and more should be matched.Calculate pixel with coupling energy equation The correctness joined, this energy equation is made up of two parts：

E_match(i, j)=E_diff(i,j)+E_co_rrect(i,j).

E_diffRepresent the texture difference between matched pixel, be defined as follows：

Wherein D (i, j) represent coordinate position for (i, the parallax value of pixel j),Represent right view coordinate Position is (the textural characteristics value of the pixel of i, j+D (i, j)).

Obviously, should have identical between the pixel of correct coupling is difference, therefore, it can by equation the following calculating phase Join the parallax distortion between pixel：

Representing right view coordinate position is the (parallax value of the pixel of i, j+D (i, j)).

2. utilize Dynamic Programming to carry out line options

After energy equation establishes, set up cost matrix M according to this energy equation, and utilize Dynamic Programming to select Line.The low line of energy value will be chosen, because the pixel on this line is probably correctly mated and has relatively with surrounding pixel High similitude.Left view can be seen and the invisible point of right view is referred to as being blocked a little.Because the pixel being blocked does not has Have matched pixel point, it is impossible to be chosen, therefore these cost value put are set to infinite M (i, j)=∞, this constraint it can be avoided that The pixel that is blocked is chosen.So cost matrix is as follows：

(i, is j) binary map to O, represents coordinate position for (i, whether pixel j) is blocked.In the present invention, O (i, j)=1 represents coordinate position for (i, pixel j) is not blocked, and can mate；(i, j)=0 represents coordinate bit to O It is set to that (i, pixel j) is blocked.

The present invention utilizes Dynamic Programming to select outlet.First energy value is selected in the first row or last column minimum Point as beginning of line.Then second point in the some line the most that in next line, energy value is minimum is selected.Should be noted that It is that generally, the line selected is continuous print, so except the first row, the range of choice of the pixel of other row, it should be The neighbor pixel of lastrow, say, that j^±∈{j-1,j,j+1}；But it is all to block a little if neighbor pixel, So range of choice should expand full line to.In aforementioned manners, each layer selects a pixel, obtains final line.Quilt The line selecting will be deleted in order to avoid repeating to select from cost matrix M.Then cost matrix M will be recalculated and select Next line.This dynamic programming process will repeat can not selected to be blocked until in certain a line, remaining pixel is all Till Dian.

3. mate the calculating of figure

It is said that in general, serious parallax distortion can be produced if stereo pairs is individually processed.Therefore, image pair The process that the pixel matching should match, and coupling figure should be calculated to show the matching relationship between pixel. If directly use disparity map as coupling figure, it will be apparent that vision distortion will occur.Because it cannot be guaranteed that being closed by parallax It is that each pixel has a pixel to match.The coupling figure that should set up a surjection reduces because erroneous matching is made The distortion becoming.

After choosing line, according to the order selecting, we carry out lines matching.First, the pixel in line is closed according to parallax System finds matched pixel and obtains matched line.Then, lines matching avoids repeated matching to temporarily removing.Then the next one is found Lines matching pair.This process will continue until each line a matched line.

After lines matching, each line is to can distribute an ID.ID represents the order of lines matching, and its value is from 1 to n.n It is the number of line options.In image pair, except those selected pixels, remaining pixel also can distribute an ID value.From Left-to-right, these pixels will one ID of distribution in order, the ID value width of original image (W be) from n+1 to W.By this Method, each pixel in every a line can have a unique ID value, and the matched pixel in right view also has phase Same ID value.And, it is blocked in right view, a little also have a matched pixel.According to ID value, all of pixel is matched And obtain coupling figure.

4. set up based on the matching relationship of coupling figure

After establishing coupling figure, with coupling figure, the pixel of left and right view is mated.In the view of left and right, ID value is identical Pixel be exactly the pixel matching.Matched pixel is done identical process, the uniformity of left and right view can be preserved.And Figure is surjection figure, solves the many-to-one mapping situation in parallax relation, thus saves spatial coherence, reduces and regard Feel distortion.

Claims (3)

1. Stereo image matching figure computational methods, is characterized in that, step is as follows：

1) foundation of energy equation

Select lines matching to the pixel relationship representing between stereo pairs：First, line selection is carried out before On-line matching pixel Select, set up energy equation and modelling process is carried out to line options, this energy equation consider simultaneously line selectivity characteristic and Join characteristic, be defined as follows：

E_total(i,j,j^±)=α E_select(i,j,j^±)+(1-α)·E_match(i,j),

Wherein i represents the abscissa of pixel, and j represents the ordinate of pixel, j^±Represent the vertical seat of the i-th-1 row selected pixels Mark, uses sensu lato line, say, that in i-1 row, selected pixels can be any one pixel of this row, and unlike even Continuous line is such, and in i-1 row, selected pixels must be the neighbor that the i-th row is chosen pixel, say, that j^±∈{j- 1, j, j+1}, α are weighting factors, represent line options characteristic and the importance of lines matching characteristic, E_selectSpecial for line options The energy equation of property, E_matchEnergy equation for lines matching；

2) line options based on Dynamic Programming

After establishing energy equation, set up cost matrix M according to this energy equation, and use dynamic programming method to carry out line selection Select；Pixel owing to being blocked does not has matched pixel point, it is impossible to be chosen, therefore these cost value put are set to infinite M (i, J)=∞, this constraint is it can be avoided that the pixel that is blocked is chosen, so cost matrix is as follows：

$M(i,j)=\left\{\begin{array}{cc}{E}_{total}(i,j,j^{\±}),& O(i,j)=1\\ \mathrm{\∞},& O(i,j)=0\end{array}\right.$

(i, is j) binary map to O, represents coordinate for (i, whether pixel j) is blocked；(i, j)=1 represents coordinate position to O For (i, pixel j) is not blocked, and can mate；(i, j)=0 represents coordinate position for (i, pixel j) is hidden to O Gear；

Selected line will remove in order to avoid repeating to select from cost matrix M, then, recalculates cost matrix M and selects Going out next line, this dynamic programming process will repeat can not selected to be hidden until in certain a line, remaining pixel is all Till catch point；

3) foundation of figure is mated

After line options, according to the order selecting, carry out lines matching.First, the pixel in line finds coupling according to parallax relation Pixel obtains matched line pair.Then, lines matching is to being temporarily removed, to avoid repeated matching.Then next lines matching is found Right.This process will continue until each line a matched line；

After lines matching, each line is to being allocated an ID, and ID represents the order of lines matching, and its value is from 1 to n, and n is line The number selecting, in image pair, except those selected pixels, the remaining pixel that is blocked also can distribute an ID value, From left to right, these pixels will one ID of distribution in order, ID value is from n+1 to W, and W is the width of original image, by this Method, each pixel in every a line can have an ID value, and the matched pixel in right view also has identical ID Value, and, it is blocked in right view, a little also have a matched pixel, according to ID value, all of pixel is matched and obtains Obtained coupling figure；

4) set up based on the matching relationship of coupling figure

After establishing coupling figure, with coupling figure, the pixel of left and right view is mated, in the view of left and right, the identical picture of ID value Matched pixel is done identical process, can be preserved the uniformity of left and right view by the pixel that element matches exactly.

2. Stereo image matching figure computational methods as claimed in claim 1 a kind of, is characterized in that, the coupling figure set up be with Line is unit；In conjunction with texture paging between pixel and parallax similitude, by increasing capacitance it is possible to increase the accuracy of coupling；The coupling set up Figure is surjection figure, can mate left and right view pixels one to one.

3. a kind of Stereo image matching figure computational methods as claimed in claim 1, is characterized in that, wherein, and 1) line options characteristic Energy equation：

Neighbor on the same line should be similar, and the pixel around line should have similar textural characteristics and the degree of depth Value.In view of this situation, line options energy equation E_selectByWithThis two parts form：

$E_{select}(i,j,j^{\±})=E_{select}^T(i,j,j^{\±})+E_{select}^D(i,j,j^{\±}),$

Represent the texture similarity between pixel；Represent the degree of depth similarity between pixel, energy equationDetermine Justice is as follows：

$E_{select}^T(i,j,j^{\±})=E_{seam}^T(i,j,j^{\±})+E_{round}^T(i,j,j^{\±}),$

Represent the similitude between pixel on the same line,It is the similitude between line surrounding pixel,Definition As follows：

$E_{seam}^T(i,j,j^{\±})=|T(i,j)-T(i-1,j^{\±})|,$

(i, j) represent coordinate in left view is that (line surrounding pixel coordinate is by the i-th-1 row for i, the textural characteristics value of pixel j) to T The coordinate of selected pixels determines, is divided into three kinds of situations,It is defined as follows：

$E_{round}^T(i,j,j^{\&PlusMinus;})=\left\{\begin{array}{cc}{W}_1+W& j^{\&PlusMinus;}<j,\\ W& j^{\&PlusMinus;}=j,\\ W_2+W& j^{\&PlusMinus;}>j\end{array}\right.$

Wherein

W=| T (i, j-1)-T (i, j+1) |,

$W_1=\underset{j_k=j^{\&PlusMinus;}+1}{\overset{j}{\&Sigma;}}|T(i-1,j_k)-T(i,j_k-1)|,$

$W_2=\underset{j_k=j+1}{\overset{j^{\&PlusMinus;}}{\&Sigma;}}|T(i-1,j_k-1)-T(i,j_k)|$

Represent the energy equation of parallax similitude between pixelDefinition andSimilar, simply textural characteristics is changed into and regard Difference feature；

2) energy equation of lines matching

Calculate the correctness of pixel matching with coupling energy equation, this energy equation is made up of two parts：

E_match(i, j)=E_diff(i,j)+E_correct(i,j)

E_diffRepresent the texture difference between matched pixel, be defined as follows：

$E_{deff}(i,j)=|T(i,j)-\tilde{T}(i,j+D(i,j\left)\right)|,$

Wherein D (i, j) represent coordinate position for (i, the parallax value of pixel j),Represent right view coordinate position For (the textural characteristics value of the pixel of i, j+D (i, j))；

Calculate the parallax distortion matching between pixel with equation the following：

$E_{correct}(i,j)=|D(i,j)-\tilde{D}(i,j+D(i,j\left)\right)|,$

Representing right view coordinate position is the (parallax value of the pixel of i, j+D (i, j)).