CN100586199C - Parallax acquisition method and device - Google Patents

Abstract

The invention discloses a parallax obtaining method and a device thereof, and relates to a computer vision technique to obtain more accurate parallax. The parallax obtaining method provided by the embodiment of the invention comprises the steps of: obtaining matching energy of each area in a matched image according to initial parallax parameters, wherein, the matched image comprises at least two areas and at least one operation area is determined for a current area in each of the areas, wherein, the current area refers to the area in processing; obtaining optimized parallax parameters in the current area according to the matched energy of the current area and of each operation area; obtaining the parallax of the current area according to the optimized parallax parameters of the current area.

Description

Parallax acquisition method and device

technical field

The present invention relates to computer vision technology, in particular to a parallax acquisition method and device.

Background technique

Stereo matching is a classic problem in computer vision. Stereo matching techniques are used to obtain three-dimensional (3D) images from stereo images. A stereoscopic image refers to a pair of two-dimensional (2D) images taken at different positions on the same straight line for the same object.

For example, if a stereoscopic image is a pair of two-dimensional images of the same scene captured by two cameras in front of the object, a left image and a right image, for the object at the same point in space, the coordinates in the left and right images The difference is called parallax. A 3D image is constructed from the disparity acquired from the stereo image according to the stereo matching technique.

The stereo matching technology in the prior art 1 is introduced below. The technical solution adopted by the stereo matching technology in the prior art 1, as shown in Figure 1, mainly includes the following steps: step 101, dividing the image into multiple regions; step 102, calculating the initial parallax value of the image pixels in each region ; Step 103, performing parallax plane fitting on each region to obtain the initial parallax parameters of each region; Step 104, optimizing the parallax parameters of each region; Step 105, obtaining the parallax of each region according to the optimized parallax parameters. The following mainly describes steps 102, 103 and 104 in the first prior art.

Step 102. Calculate initial disparity values of image pixels in each region.

The initial disparity range [d _min , d _max ] and the disparity step size d′ are preset. Select the current image area being processed, that is, the current area, select the first image pixel in this area, set the parallax d of this pixel as _dmin , and calculate the matching residual c(x, y, d) according to the formula (r1):

$\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 9</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; J</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Among them, (x, y) is the image coordinate value, d is the disparity

According to the disparity step size d′, continuously increase the disparity d, and calculate c(x, y, d), save the smallest c(x, y, d) and the corresponding d until d≥d _max , then the smallest c The d corresponding to (x, y, d) is the initial disparity of this pixel.

Step 103, performing parallax plane fitting on each region to obtain initial parallax parameters of each region.

得到对应图像像素(x′，y′)，计算(x′，y′)相对参考图像的初始视差

如果 $\hat{d}(x,y)\&NotEqual;\hat{d}(x^{\&prime;},y^{\&prime;}),$ 则像素(x，y)为局外点。确定初始视差平面的过程中，将所述局外点排除在外，用以加强算法的鲁棒性。Firstly, the initial disparity plane of each region is fitted according to the initial disparity value of each region. Prior Art 1. The method for determining the initial parallax planes of each region is: first judge whether each pixel is an outlier point according to the following method: according to the initial parallax of the reference image pixel (x, y)

Get the corresponding image pixel (x', y'), and calculate the initial disparity of (x', y') relative to the reference image

if $\hat{d}(x,\mathrm{the\; y})\&NotEqual;\hat{d}(x^{\&prime;},{\mathrm{the\; y}}^{\&prime;}),$ Then pixel (x, y) is an outlier point. In the process of determining the initial parallax plane, the outlier points are excluded to enhance the robustness of the algorithm.

Then, the currently processed image area is selected, and it is judged whether the number of pixels included in this area is greater than a set threshold. The prior art only processes image areas with the number of pixels greater than the threshold. If the number of pixels in the currently processed image area is not greater than the threshold, select the next image area and use the same method to judge; if the number of pixels in the currently processed image area is greater than the threshold, select the first image area in this area image pixels, and start the initial parallax plane fitting.

According to the following parallax plane model formula (r2), the parallax planes of the regions are modeled:

d＝c ₁ x+c ₂ y+c ₃ (r2)

Among them, (x, y) is the coordinate value of the image pixel, d is the initial disparity corresponding to the image pixel, (c ₁ , c ₂ , c ₃ ) is the disparity plane coefficient vector, that is, the disparity parameter.

If the pixel is an outlier, select the next pixel. If the pixel point is not an outlier point, the pixel information is added to the linear system matrix of formula (r3).

When all the points in the area are processed, according to the formula (r3), [c ₁ , c ₂ , c ₃ ] ^T can be obtained to obtain the initial parallax plane of the area:

$\left(\begin{array}{ccc}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right)\left(\begin{array}{c}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}\end{array}\right)<span\; class="notranslate">\; =</span>\left(\begin{array}{c}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\end{array}\right)<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

The same method as above is used to fit the initial parallax planes for each area, the set of initial parallax planes of each area forms a set of parallax planes, and the parallax parameters of each area are obtained according to the set of parallax planes.

Step 104, optimize the parallax parameters of each region.

Prior Art A method for optimizing the parallax parameters of each region mainly includes three steps.

The first step is to use a loop to optimize the parallax plane according to the obtained initial parallax plane.

The method of circularly optimizing the parallax plane is: set the new plane and the old plane as the initial parallax plane, set the loop count to 1, select the first pixel in the current processing area, and judge whether the pixel is an occluded pixel. If it is an occluded pixel, select the next pixel; if it is not an occluded pixel, in order to enhance the robustness of the algorithm, the weighted least square technique is used to calculate the weighting coefficient according to the formula (r4), and the weighted information of the pixel is added to the formula ( In the linear system matrix of r3):

$w\left({\mathrm{\&beta;}}_i\right)=e^{-2{\mathrm{\&beta;}}_i},$ in, ${\mathrm{\&beta;}}_i=|c_1{x}_i+c_2{\mathrm{the\; y}}_i+c_3-\hat{d}(x_i,{\mathrm{the\; y}}_i)|---\left(r4\right)$

When all the points in this area are processed, [c ₁ , c ₂ , c ₃ ] ^T can be obtained according to the formula r5, and a new parallax plane can be obtained:

$\left(\begin{array}{ccc}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right)\left(\begin{array}{c}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}\end{array}\right)<span\; class="notranslate">\; =</span>\left(\begin{array}{c}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ \underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\end{array}\right)<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

Wherein, w _i is the weighting coefficient w(β _i ) of the pixel ( _xi , y _i ).

Judging whether the difference between the new parallax plane and the old parallax plane is less than a predetermined threshold, if it is not less than, the loop count is incremented, and the old parallax plane coefficient is updated to the new parallax plane coefficient, select this area For the first pixel, repeat the above process until the difference between the new disparity plane and the old disparity plane is smaller than the threshold. If it is smaller, judge whether the new disparity plane is already in the disparity plane set, if not, add the new disparity plane into the disparity plane set. Select the next area to cycle through the above processing until all areas are processed, so a new set of disparity planes is obtained.

The second step is to cluster and stratify the acquired new disparity planes of each region.

Select the first plane from the plane set, select the first area in the plane, and determine the number of supporting pixels of the plane that satisfy the formula (r6) from the non-occluded pixels in the area:

βi<threshold (r6)

Determine whether the number of pixels in the area is greater than a given threshold, for example 40, if not greater than the threshold, then calculate the matching cost according to the formula (r7):

$\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

Among them, S is the image area, P is the parallax plane, $d=c_1^{P}x+c_2^P\mathrm{the\; y}+c_3^P,$ c ₁ ^P , c ₂ ^P , and c ₃ ^P are parameters of P.

If it is greater than the threshold, the matching cost is calculated according to the formula (r8):

$\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; o</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; e</span>}}^{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; the\; s</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; 8</span>}<span\; class="notranslate">\; )</span>$

Among them, O is the occlusion part in the area S, n is the number of non-occlusion pixels in the area S, and s is the number of pixels supporting the plane P in the area S.

Set the minimum matching cost to a large number, judge whether the calculated matching cost is less than the minimum matching cost, and if it is smaller, use the calculated matching cost as the minimum matching cost. Select the next disparity plane and repeat the above process until all disparity planes are processed. Select the next area and repeat the above process until all areas are processed.

Determine whether there is a joint area with the same parallax plane as the adjacent area. If so, use the joint area as the last area, repeat the above determination of the initial parallax plane set for each area and subsequent processing until there is no joint area.

The third step is to further optimize the parallax plane of each region by setting the label, and obtain the parallax parameter according to the parallax plane.

Prior Art 1 The method of obtaining labels from each region to the optimized parallax plane is: according to the above calculated regions and corresponding parallax planes as the initial labels of each region, select the optimal parallax plane, and calculate the labels according to formula (r9) Efficiency function:

E(f)＝E _data (f)+E _smooth (f) (r9)

Among them, f is a label, and each region S∈R is given a corresponding disparity plane f(S)∈D, R is the region set of the reference image, and D is the estimated disparity plane set. E _data is calculated according to the formula (r10):

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; data</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span>$

E _smooth is calculated according to formula (r11):

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; smooth</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; 11</span>}<span\; class="notranslate">\; )</span>$

Among them, S, S' are adjacent areas, uS, S' are proportional to the length of the common boundary between areas S and S'. When f(S)≠f(S′), δ(f(S)≠f(S′)) is 1; when f(S)=f(S′), δ(f(S)≠f (S')) is 0.

Find the label that minimizes the label efficiency function E(f') from f' and mark it as f", and judge whether E(f")≤E(f). If yes, set f=f″, go back to selecting the optimal parallax plane, and loop processing; if not, set the optimal label to f, and the process ends. Obtain the parallax parameter according to the obtained optimal label.

In the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: when optimizing the parallax parameters of each area in the prior art, the parallax of the area is only optimized according to the energy information of the area being processed parameters, and when the parallax parameters of a region change, it will also affect the energy of its adjacent regions. The existing technology considers the parallax optimization of each region in isolation, and cannot obtain more accurate parallax; and once the selection of occluded pixels is inaccurate , will lead to inaccurate clustering and stratification, so that the best label obtained is also inaccurate, resulting in a large parallax error.

Contents of the invention

On the one hand, the embodiment of the present invention provides a disparity acquisition method, which can obtain more accurate disparity.

The technical solution adopted in the embodiment of the present invention is as follows: a parallax acquisition method, the method comprising:

Acquiring the matching energy of each region in the matched image according to the initial parallax parameter, the matched image including at least two of the regions;

In each of the areas, at least one operating area is determined for the current area, and the current area is the area being processed in the various areas;

Acquiring an optimized parallax parameter of the current area according to the matching energy of the current area and the operation areas;

The disparity of the current region is acquired according to the optimized disparity parameter of the current region.

On the other hand, the embodiment of the present invention provides a disparity acquisition device, which can obtain more accurate disparity.

The technical solution adopted in the embodiment of the present invention is as follows: a parallax acquisition device, the device includes:

A matching energy acquisition unit, configured to acquire the matching energy of each region in the matched image according to the initial parallax parameter, the matched image including at least two of the regions;

An operation area determining unit, configured to determine at least one operation area for the current area in the various areas, the current area being the area being processed in the various areas;

An optimized parallax parameter acquiring unit, configured to acquire an optimized parallax parameter according to the matching energy of the current area and each operating area determined by the operating area determining unit;

The disparity value obtaining unit is configured to obtain the disparity of the current area according to the optimized disparity parameter obtained by the optimized disparity parameter obtaining unit.

In the embodiment of the present invention, when obtaining the disparity of each region in the matched image, the matching energy of each region is used as a criterion function, and when optimizing the initial disparity obtained in the current region, not only according to the matching energy of the current region , and according to the matching energy of each operation area, due to the interdependence and mutual influence of the matching energy between different areas, the optimized parallax parameter of the current area obtained is relatively accurate, so the embodiment of the present invention is based on the optimization Parallax parameter, which can get more accurate parallax.

Description of drawings

Fig. 1 is a method flow chart of the stereo matching technology provided in prior art 1;

FIG. 2 is a flow chart of a parallax acquisition method provided by Embodiment 1 of the present invention;

FIG. 3 is a flow chart of a parallax acquisition method provided by Embodiment 2 of the present invention;

Fig. 4 is the Tsukuba stereo image processing result figure provided by the embodiment of the present invention;

Fig. 5 is a comparison diagram of the plane fitting results respectively using the RANSAC method and the voting method provided by the embodiment of the present invention;

Fig. 6 is the parallax map obtained after fitting the parallax plane based on voting to the Tsukuba stereo image provided by the embodiment of the present invention;

Fig. 7 is a schematic diagram of image occlusion;

Fig. 8 is the disparity map after four collaborative optimizations of the Tsukuba image provided by the embodiment of the present invention;

Fig. 9 is a parallax result diagram of standard images Tsukuba, Venus, Teddy, and Cones processed by using the parallax acquisition method provided by the embodiment of the present invention;

FIG. 10 is a structural diagram of a parallax acquisition device provided by Embodiment 1 of the present invention;

FIG. 11 is a structural diagram of a parallax acquisition device provided by Embodiment 2 of the present invention.

Detailed ways

In order to solve the problem in the prior art that when disparity is acquired, only based on the isolated energy information of the processed current area itself, the disparity error obtained after optimization is relatively large. Embodiments of the present invention provide a disparity acquisition method and device, which can Get more accurate parallax.

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. The following descriptions are only some embodiments of the present invention. For those of ordinary skill in the art, Other implementations of the present invention can also be obtained according to these embodiments without any creative effort.

As shown in Figure 2, a parallax acquisition method, the method includes:

Step 201. Obtain the matching energy of each region in the matched image according to the initial parallax parameter, and the matched image includes at least two of the regions;

Step 202. In each area, at least one operation area is determined for the current area, and the current area is an area being processed in each area;

Step 203, according to the matching energy of the current area and the operation areas, obtain the optimized parallax parameter of the current area;

Step 204. Obtain the disparity of the current region according to the optimized disparity parameter of the current region.

In the embodiment of the present invention, when obtaining the disparity of each area in the matched image, the matching energy of each area is used as a criterion function, and when optimizing the initial disparity obtained in the current area, not only according to the matching energy of the current area, Also according to the matching energy of each operation area, due to the interdependence and mutual influence of the matching energy between different areas, the optimized parallax parameter of the current area obtained is relatively accurate. Therefore, according to the optimized parallax parameters to obtain more accurate disparity.

As shown in Figure 3, the technical solution provided by the embodiment of the present invention obtains the matching energy of each region in the matched image according to the initial parallax parameter in step 201, and before the matched image includes at least two of the regions, it also includes:

Step 201a, acquiring the initial parallax of the pixels in each area;

Step 201b. Obtain initial parallax parameters of each region according to the initial parallax. Each step is described in detail below.

The stereogram pair used in the embodiment of the present invention is a left image and a right image taken on the same object in the same scene by using two standard cameras. In the embodiment of the present invention, the matched image is the left image, and the reference image is the right image .

In the embodiment of the present invention, a mean-shift algorithm (Mean-Shift) is used to divide the matched image into at least two regions.

Step 201a, acquiring the initial parallax of the pixels in each area.

There are many ways to obtain the initial disparity of pixels in each region. In the embodiment of the present invention, the method described in the background technology can be used to obtain the initial disparity of pixels in each region, and the adaptive correlation window algorithm can also be used to obtain the pixel in each region. The initial disparity of the point, which is not limited here. The adaptive correlation window algorithm is a prior art, and those skilled in the art can obtain the result of step 201a through related content disclosed in the prior art.

In the embodiment of the present invention, the left and right stereogram pairs captured by two standard cameras are described. As shown in Fig. 4, Fig. 4 shows the result of processing the Tsukuba stereo image pair according to the embodiment of the present invention, where Fig. 4(a) and (b) are the segmented image and the initial disparity map corresponding to the left image respectively.

Step 201b. Obtain initial parallax parameters of each region according to the initial parallax.

The embodiment of the present invention adopts a robust voting-based parallax plane fitting method to obtain the initial parallax parameters of each region. The obtained initial parallax often has a certain error, and the voting-based parallax plane fitting method can effectively remove error points, that is, outlier data, so that the fitting result is more accurate, thereby further ensuring that more accurate Parallax, as shown in Figure 5, gives a comparison chart of the plane fitting results of the RANSAC method and the voting method, where the horizontal line represents the fitting error rate of the voting method, and the broken line represents the fitting error rate of the RANSAC method, which can be It can be seen that the proposed voting method not only has a low fitting error rate, but also has very stable results, while the RANSAC method has a low fitting error rate, but its performance is unstable. As shown in Figure 6, it is the disparity map obtained after fitting the disparity plane based on voting. Experimental results show that the noise in the initial disparity map can be effectively suppressed. Step 201b will be described in detail below.

According to the parallax plane model formula d=c ₁ x+c ₂ y+c ₃ , model the parallax planes of the regions, where c ₁ , c ₂ , and c ₃ are parallax plane parameters, and c ₁ is the first parameter, c ₂ is the second parameter and c ₃ is the third parameter.

The step of obtaining the initial parallax parameters of each region includes:

S1. Obtaining the first parameter c ₁ , this step specifically includes:

S11. In each of the regions, select at least one row of pixel point pairs in each same pixel row;

The selection of pixel point pairs in one row may be a pair of adjacent points, or a pair of points apart.

S12. In each of the regions, each candidate first parameter is obtained according to the coordinate value of each row of pixel point pairs and the initial parallax;

Perform correlation calculations on the selected row pixel pairs, and perform correlation calculations on any row pixel pair as follows: calculate the ratio of the parallax difference and the coordinate value difference of this row pixel pair, Δd/Δx, Obtain one of the candidate first parameters of the plane parameter _c1 , perform the above-mentioned same calculation on all selected row pixel pairs, and obtain the candidate first parameters of the first parameter _c1 ;

S13. Obtain the first parameters of the regions by voting according to the candidate first parameters;

The candidate first parameters are voted in the one-dimensional parameter space of the first parameter c ₁ to generate a histogram of frequency relative to c ₁ , and filter the histogram. The embodiment of the present invention adopts Gaussian smoothing filter, and the value of c ₁ corresponding to the peak point of Gaussian smoothing filter, that is, the point with the most votes in the voting, is used as the first parameter c ₁ .

S2. Acquiring the second parameter c ₂ , this step specifically includes:

S21. In each of the regions, select at least one column pixel point pair in each same pixel column;

The selection of pixel point pairs in a column may be a pair of adjacent points, or a pair of points apart from each other.

S22. In each of the regions, each candidate second parameter is obtained according to the coordinate value of each column of pixel point pairs and the initial disparity;

Perform correlation calculations on the selected column pixel pairs, and perform correlation calculations on any column pixel point pair as follows: calculate the ratio of the parallax difference and the coordinate value difference of this column pixel point pair, Δd/Δy, One of the candidate second parameters of the plane parameter _c2 is obtained, and the same calculation as above is performed on all selected column pixel point pairs to obtain the candidate second parameters of the second parameter _c2 .

S23. In each area, obtain the second parameter by voting according to each candidate second parameter;

The candidate second parameters are voted in the parameter space of the one-dimensional second parameter c ₂ to generate a histogram of the frequency relative to c ₂ , and filter the histogram. The embodiment of the present invention adopts Gaussian smoothing filter, and the peak point of Gaussian smoothing filter, that is, the value of c ₂ corresponding to the point with the most votes in the voting, is used as the second parameter c ₂ .

S3. Acquiring the third parameter c ₃ , this step specifically includes:

S31. Fitting corresponding candidate third parameters with a parallax plane according to each candidate first parameter and each corresponding candidate second parameter in each region;

Substituting the candidate first parameters, corresponding candidate second parameters, pixel coordinates and disparity into the parallax plane model formula d=c ₁ x+c ₂ y+c ₃ to obtain corresponding candidate third parameters .

S32. Obtain the third parameter of each area by voting according to each candidate third parameter.

The candidate third parameters are voted in the one-dimensional parameter space of the third parameter to generate a histogram of frequency relative to c ₃ , and filter processing is performed on the histogram. The embodiment of the present invention adopts Gaussian smoothing filter, and uses the peak point of Gaussian smoothing filter, that is, the value of c ₃ corresponding to the point with the most votes in the voting, as the third parameter c ₃ .

Step 201. Acquire the matching energy of each region in the matched image according to the initial disparity parameter, and the matched image includes at least two of the regions.

The matching energy includes data energy, occlusion energy and smoothing energy. The matching energy acquired during parallax optimization includes the data energy and a combination of at least one of the occlusion energy and smoothing energy.

The matching energy obtained in the embodiment of the present invention includes the data energy, occlusion energy and smoothing energy, that is, the matching energy of any image area is E _i =E _data +E _occlude +E _smooth , where E _i is area i The subscript i represents the i-th image region, E _data is the data energy, E _occlude is the occlusion energy, and E _smooth is the smoothing energy.

The specific method for obtaining the matching energy of each region provided by the embodiment of the present invention is as follows:

T1. Obtain the data energy of each area.

The data energy E _data is calculated by the following formula (r12):

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; data</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}\underset{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\mathrm{<span\; class="notranslate">\; and\; q</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; )</span>$

Among them, V ₁ and V _r respectively represent the visible pixel sets of the current area on the left and right images, p∈V ₁ and q∈V _r are two corresponding pixels matched on the left and right images, and r, g, b represent the pixel values of the corresponding pixels RGB color value. The RGB color value of each pixel is obtained as follows: Calculate the pixel q on the reference image (right image) corresponding to the pixel p on the matched image (left image) according to the estimated parallax plane parameters of the current area, and obtain its RGB color value. Since the calculated disparity is a floating-point type, the pixel q is generally not exactly at an integer pixel position, so its RGB color value generally cannot be obtained directly from the image. In the embodiment of the present invention, its value is obtained by interpolating the RGB color values of four neighboring pixels of q on the reference image. The calculation of data energy takes into account the visibility criterion, which requires that the pixels used to calculate data energy are visible in both left and right image pairs.

The formula for calculating the data energy provided by the technical solution of the embodiment of the present invention is not limited to the formula (r12), and all formulas that make approximate changes based on the formula (r12) belong to the technical solution provided by the embodiment of the present invention, for example, using two The sum of the absolute values of the differences between the color components of a pixel, or the sum of the squares of the differences between the color components of two pixels.

T2. Obtain the shading energy of each region.

In order to facilitate understanding, a brief introduction to image occlusion is given first. Occlusion is divided into two cases: left occlusion and right occlusion. As shown in Figure 7, consider two adjacent regions A and B on the matched image (left image), A is on the right of B. When the disparity of A area at the adjacent boundary is greater than that of its left neighbor B, then when A and B are mapped to the reference image (right image) according to the current disparity calculation result, a part of B' (D in the figure region) will be occluded by A' (left occlusion), and when the disparity of A region at the adjacent boundary is smaller than the disparity of its left neighbor B, it is mapped between the two regions A' and B' after the right image There will be a gap (area C in the figure), which is equivalent to a part of area A' on the right image being blocked by B on the left image (right occlusion). In the embodiment of the present invention, the shading energy is set as follows: when left shading occurs, the left shading energy is added to the corresponding left shading area; when right shading occurs, the right shading energy is added to the corresponding right shading area. In practice, the detection of the occluded area can be carried out as follows: first, map the relevant areas on the left image to the right image according to the current disparity calculation result, and then check the mapping situation on the right image; if a pixel is repeatedly mapped, then Record the pixel as left occlusion, and add the left occlusion energy; if a pixel is not mapped, record the pixel as right occlusion, and add the right occlusion energy.

To sum up, according to the following formula (r13), the occlusion energy E _occlude can be obtained as:

E _occlude ＝(|Occ _L |+|Occ _R |)λ _occ (r13)

Among them, Occ _L , Occ _R represent the number of left occlusion and right occlusion pixels respectively, and λ _occ represents the set occlusion penalty constant.

The formula for calculating the shading energy provided by the technical solution of the embodiment of the present invention is not limited to the formula (r13), and all formulas that make approximate changes based on the formula (r13) belong to the technical solution provided by the embodiment of the present invention, such as for left occlusion and Set different occlusion penalty constants for right occlusion, or only deal with left occlusion, or only deal with right occlusion.

T3. Obtain the smoothed energy of each region.

According to the following formula (r14), the smoothing energy E _smooth can be obtained as:

Here, _BC represents the boundary point set of the current area on the matched image, N represents the boundary point set on other areas adjacent to _BC , and p∈BC _and q∈N are two adjacent pixel points in the sense of four-connectivity , d(p), d(q) are the parallax of pixels p and q, and λ _S is the set smoothing penalty, which can be a constant or a function of the difference between the two pixel colors, such as When the difference is small, it is set to 2 times the difference, and when the difference is large, it is set to 1 times the difference. The difference calculation can be the maximum value of the absolute values of the differences of the color components of the two pixels, or the sum of the absolute values of the differences of the color components of the two pixels, or the sum of the squares of the differences of the color components of the two pixels. The discriminant |d(p)-d(q)|≥threshold 1 is used to judge whether a boundary point on the current region is a point with discontinuous parallax. Threshold 1 can take a number greater than 0, such as 0.5 or 1. As an additional condition, it is required that p is not an occluded boundary point. Equation (r14) gives the sum of smoothing penalty energies applied to boundary points on the current region that have discontinuous disparity and do not belong to occlusion boundaries.

Step 202, in each of the areas, determine at least one operation area for the current area, where the current area is the area being processed in the various areas.

The operation area of the current area may be an image area adjacent to the current area, or an image area not adjacent to the current area, and the number of the operation areas selected shall be appropriate in consideration of the amount of calculation and the optimization effect. .

Step 203. Obtain the optimized disparity parameter of the current area according to the matching energy of the current area and the matching energy of each operation area, and this step specifically includes:

U1. Using the disparity parameters of the operation areas, or the disparity parameters generated by the weighted combination of the disparity parameters of the operation areas and the disparity parameters of the current area, as the new disparity parameters of the current area, according to the new The disparity parameter acquires each candidate matching energy corresponding to the current region;

In the embodiment of the present invention, the image region adjacent to the current region (matched as the energy E _i (x), where i represents the i-th region) is taken as the optimal operation region. U1 is described by taking the processing of each adjacent area and the current area as an example.

The disparity parameters of each adjacent area of the current area, or a new disparity parameter generated by a weighted combination of the disparity parameters of each adjacent area and the disparity parameter of the current area is used as the disparity parameter of the current area. And, according to the new parallax parameter, reacquire the data points of the current area and each operation area, the occlusion point and smooth point pixels, update the matching energy of the current area according to the pixel points, and obtain the updated matching energy E _i ' (x), that is, each candidate matching energy of the current region.

U2. Obtain updated matching energies of the operation regions according to the new parallax parameters;

According to the new parallax parameter, reacquire the data points of the current region and each operation region, the occlusion point and smooth point pixels, update the matching energy of each operation region according to the pixel points, and obtain the updated matching energy E′ _j (x), j is the label of the selected adjacent area.

U3. Obtain the collaborative optimization energies of the matching energies to be selected and the corresponding updated matching energies of the operating areas

In the embodiment of the present invention, weighting coefficients λ _k , w _ij , 0≤λ _k ≤1, 0≤w _ij ≤1 are set to construct the collaborative optimization energy. The various collaborative optimization energies described are:

U4. Acquire optimized parallax parameters of the current region according to the collaborative optimization energies.

The smaller the collaborative optimization energy is, the more reasonable the parallax parameter corresponding to the collaborative optimization energy is. In the embodiment of the present invention, a minimum value is selected among the collaborative optimization energies, and the parallax parameter used in the current region when the minimum collaborative optimization energy is obtained is used as the optimized parallax parameter of the current region.

Step 204. Obtain the disparity of the current region according to the optimized disparity parameter of the current region.

The parallax of the current area is obtained from the parallax plane model formula according to the optimized parallax parameter of the current area.

In step 203, the processing method for the current region is introduced. The embodiment of the present invention uses the same method to iteratively process all the regions in an image using a collaborative optimization method to complete the processing of the entire image. In addition, the same method can be used to repeat multiple iterations on the entire image to obtain better parallax results. The embodiment of the present invention carried out four iterations. As shown in FIG. 8, the disparity of the Tsukuba image was corrected after four times of collaborative optimization, wherein (a) to (b) are the disparity maps obtained in the first to fourth iterations , e in the figure is the sum of the collaborative energy of each region of the entire image.

The embodiment of the present invention takes the processing of the Tsukuba image as an example to describe, but the embodiment of the present invention is not limited to only applicable to a specific image, as shown in FIG. The processing results of Tsukuba, Venus, Teddy, and Cones are very close to the real parallax by using the method provided by the embodiment of the present invention, and an ideal parallax result can be obtained.

In the embodiment of the present invention, when obtaining the disparity of each area in the matched image, the matching energy of each area is used as a criterion function, and when optimizing the initial disparity obtained in the current area, not only according to the matching energy of the current area, Also according to the matching energy of each operation area, due to the interdependence and mutual influence of the matching energy between different areas, the optimized parallax parameter of the current area obtained is relatively accurate. Therefore, according to the optimized parallax parameters to obtain more accurate disparity.

The embodiment of the present invention also provides a parallax acquisition device, which can obtain more accurate parallax, as shown in Figure 10, the device includes:

A matching energy acquisition unit, configured to acquire the matching energy of each region in the matched image according to the initial parallax parameter, the matched image including at least two of the regions;

An operation area determining unit, configured to determine at least one operation area for the current area in the various areas, the current area being the area being processed in the various areas;

An optimized parallax parameter acquiring unit, configured to acquire an optimized parallax parameter according to the matching energy of the current area and each operating area determined by the operating area determining unit;

The disparity value obtaining unit is configured to obtain the disparity of the current area according to the optimized disparity parameter obtained by the optimized disparity parameter obtaining unit.

In the embodiment of the present invention, when obtaining the disparity of each region in the matched image, the matching energy of each region is used as a criterion function, and when optimizing the initial disparity obtained in the current region, not only according to the matching energy of the current region , and according to the matching energy of each operation area, due to the interdependence and mutual influence of the matching energy between different areas, the optimized parallax parameter of the current area obtained is relatively accurate, so the embodiment of the present invention is based on the optimization Parallax parameter, which can get more accurate parallax.

As shown in Figure 11, the device provided by the embodiment of the present invention also includes:

an initial parallax acquisition unit, configured to acquire the initial parallax of pixels in each region;

The disparity parameter acquisition unit is configured to acquire the initial disparity parameters of the regions according to the initial disparity acquired by the initial disparity acquisition unit.

The parallax parameter includes a first parameter, a second parameter and a third parameter, and the parallax parameter acquisition unit includes:

The first parameter acquisition module is used to first select at least one row pixel point pair in each same pixel row in each of the regions; then in each region, according to the pixels of each row of pixel point pairs value and initial disparity to obtain each candidate first parameter; finally obtain the first parameter of each region by voting according to each candidate first parameter;

The second parameter acquisition module is used to first select at least one column pixel point pair in each same pixel column in each of the regions; then in each of the regions, according to the coordinates of each of the column pixel point pairs value and initial disparity to obtain each candidate second parameter; finally obtain the second parameter of each region by voting according to each candidate second parameter;

The third parameter acquisition module is used to use the parallax plane to fit the corresponding candidate first parameters obtained by the first parameter acquisition module and the corresponding candidate second parameters obtained by the second parameter acquisition module. Each candidate third parameter of each candidate third parameter is voted according to each candidate third parameter to obtain the third parameter of each region.

In the embodiment of the present invention, the parallax parameter acquiring unit adopts a robust voting-based parallax plane fitting method to acquire the initial parallax parameters of each region. Due to the obtained initial parallax, there is often a certain error, and the parallax parameter acquisition unit can effectively remove error points, that is, outlier data, through the parallax plane fitting method based on voting, so that the fitting result is more accurate, thus, Further ensure that more accurate parallax can be obtained,

The optimized parallax parameter acquisition unit includes:

The matching energy processing module is configured to use the disparity parameters of the operation areas, or the disparity parameters generated by the weighted combination of the disparity parameters of the operation areas and the disparity parameters of the current area, as the new disparity parameters of the current area, Acquiring corresponding candidate matching energies of the current region according to the new parallax parameter,

An operation area energy updating module, configured to obtain updated matching energy of each operation area according to the new parallax parameter;

A collaborative optimization energy acquisition module, configured to calculate the collaborative optimization energies of the matching energies to be selected and the corresponding matching energies of the operating regions according to preset weighting coefficients;

An optimized parallax parameter acquiring module, configured to acquire the optimized parallax parameter of the current region according to the collaborative optimization energies.

The embodiment of the present invention utilizes the optimized parallax parameter acquisition unit to iteratively process all areas in an image using a collaborative optimization method to complete the processing of the entire image, and can repeat multiple times for the entire image in the same way. times of iterative processing, thus, better parallax results can be obtained.

The matching energy in the embodiment of the present invention includes data energy, occlusion energy and smoothing energy. The matching energy obtained during parallax optimization includes the data energy, and a combination of at least one of occlusion energy and smoothing energy, and the matching energy acquisition unit has three situations:

In the first case, when the matching energy includes data energy and occlusion energy, the matching energy acquisition unit includes:

A first pixel acquisition module, configured to acquire data point pixels and occlusion point pixels in each area;

A data energy acquisition module, configured to acquire the data energy according to the data point pixels acquired by the first pixel point acquisition module;

An occlusion energy acquisition module, configured to acquire occlusion energy according to the occlusion point pixels acquired by the first pixel point acquisition module.

In the second case, when the matching energy includes data energy and smoothing energy, the matching energy acquisition unit includes:

The second pixel point acquisition module is used to acquire data point pixels and smooth point pixels in each area;

A data energy acquisition module, configured to acquire data energy according to the data point pixels acquired by the second pixel point acquisition module;

A smoothing energy acquiring module, configured to acquire smoothing energy according to the smoothing point pixels acquired by the second pixel point acquiring module.

In the third case, when the matching energy includes data energy, occlusion energy and smoothing energy, the matching energy acquisition unit includes:

The third pixel point acquisition module is used to acquire data point pixels, occlusion point pixels and smooth point pixels in each area;

A data energy acquisition module, configured to acquire the data energy according to the data point pixels acquired by the third pixel point acquisition module;

An occlusion energy acquisition module, configured to acquire the occlusion energy according to the occlusion point pixels acquired by the third pixel point acquisition module;

A smoothing energy acquisition module, configured to acquire the smoothed energy according to the smooth point pixels acquired by the third pixel point acquisition module.

The third case is adopted in the embodiment of the present invention, the matching energy includes data energy, occlusion energy and smoothing energy, and the disparity obtained during disparity optimization is more accurate.

For the specific working methods of each unit and module in the device embodiment of the present invention, reference may be made to the method embodiment of the present invention, which will not be repeated here.

In the embodiment of the present invention, when obtaining the disparity of each region in the matched image, the matching energy of each region is used as a criterion function, and when optimizing the initial disparity obtained in the current region, not only according to the matching energy of the current region , and according to the matching energy of each operation area, due to the interdependence and mutual influence of the matching energy between different areas, the optimized parallax parameter of the current area obtained is relatively accurate, so the embodiment of the present invention is based on the optimization Parallax parameter, which can get more accurate parallax.

Those of ordinary skill in the art can understand that all or part of the steps in the above embodiments can be implemented through program instructions and related hardware. The software corresponding to the embodiments may be stored in a computer-readable medium.

Of course, there are many other embodiments of the present invention, and those skilled in the art can make various corresponding changes and modifications according to the embodiments of the present invention without departing from the spirit and essence of the embodiments of the present invention. , but these corresponding changes and modifications should all belong to the protection scope of the appended claims of the embodiments of the present invention.

Claims (19)

1. A parallax acquisition method, characterized in that the method comprises:

acquiring matching energy of each region in a matched image according to an initial parallax parameter, wherein the matched image comprises at least two regions;

determining at least one operation area for a current area in each area, wherein the current area is an area being processed in each area;

obtaining an optimized parallax parameter of the current area according to the matching energy of the current area and each operation area;

and acquiring the parallax of the current area according to the optimized parallax parameter of the current area.

2. The parallax acquisition method according to claim 1, wherein the step of acquiring matching energy of each region in a matched image according to the initial parallax parameter further comprises, before the step of acquiring matching energy of each region in the matched image, the matched image including at least two regions:

acquiring initial parallax of pixel points in each region;

and acquiring the initial parallax parameters of the regions according to the initial parallax.

3. The parallax acquisition method according to claim 2, wherein the parallax parameters include a first parameter, a second parameter and a third parameter, and the step of acquiring the initial parallax parameters of the regions according to the initial parallax includes:

in each region, at least one row pixel point pair is selected in each same pixel row;

in each region, obtaining each candidate first parameter according to the coordinate value and the initial parallax of each row of pixel point pairs;

voting according to the candidate first parameters to acquire the first parameters of the regions;

in each region, at least one column pixel point pair is selected in each same pixel column;

in each region, obtaining each candidate second parameter according to the coordinate value and the initial parallax of each row of pixel point pairs;

voting according to the candidate second parameters to obtain the second parameters of the regions;

fitting out corresponding candidate third parameters by using a parallax plane according to the candidate first parameters and the corresponding candidate second parameters in the regions;

and voting according to the candidate third parameters to acquire the third parameters of the regions.

4. The parallax acquisition method according to any one of claims 1, 2 or 3, wherein the step of acquiring the optimized parallax parameters of the current region according to the matching energy of the current region and each operation region comprises:

taking the parallax parameter of each operation area as a new parallax parameter of the current area, and acquiring each to-be-selected matching energy of the corresponding current area according to the new parallax parameter of the current area;

acquiring updated matching energy of each operation area according to the new parallax parameter of the current area;

acquiring each to-be-selected matching energy and each collaborative optimization energy of the corresponding updating matching energy of each operation area;

and obtaining the optimized parallax parameter of the current area according to the collaborative optimization energies.

5. The parallax acquisition method according to any one of claims 1, 2 or 3, wherein the step of acquiring the optimized parallax parameters of the current region according to the matching energy of the current region and each operation region comprises:

the parallax parameter generated by the weighted combination of the parallax parameter of each operation area and the parallax parameter of the current area is used as a new parallax parameter of the current area, and corresponding to-be-selected matching energy of the current area is obtained according to the new parallax parameter of the current area;

acquiring updated matching energy of each operation area according to the new parallax parameter of the current area;

acquiring each to-be-selected matching energy and each collaborative optimization energy of the corresponding updating matching energy of each operation area;

and obtaining the optimized parallax parameter of the current area according to the collaborative optimization energies.

6. The parallax acquisition method according to claim 1, wherein the matching energy includes data energy and occlusion energy, and the step of acquiring the matching energy of each region includes:

acquiring data point pixels and shielding point pixels in each region;

acquiring the data energy according to the data point pixel;

acquiring the shielding energy according to the shielding point pixels;

and the acquired matching energy of each region is the sum of the data energy and the shielding energy.

7. The parallax acquisition method according to claim 1, wherein the matching energy includes data energy and smoothing energy, and the step of acquiring the matching energy of each region includes:

acquiring data point pixels and smooth point pixels of each region;

acquiring the data energy according to the data point pixel;

acquiring the smoothing energy according to the smoothing point pixels;

and the acquired matching energy of each region is the sum of the data energy and the smoothing energy.

8. The parallax acquisition method according to claim 1, wherein the matching energy includes data energy, occlusion energy, and smoothing energy, and the step of acquiring the matching energy of each region includes:

acquiring data point pixels, shielding point pixels and smooth point pixels in each region;

acquiring the data energy according to the data point pixel;

acquiring the shielding energy according to the shielding point pixels;

acquiring the smoothing energy according to the smoothing point pixels;

and the acquired matching energy of each region is the sum of the data energy, the shielding energy and the smoothing energy.

9. The parallax acquisition method according to any one of claims 6, 7 or 8, wherein the step of acquiring the data energy from the data point pixels comprises:

acquiring a corresponding pixel of a current data point pixel in a reference image according to the parallax parameter of the current area;

respectively acquiring RGB color values of the data point pixel and the corresponding pixel;

and acquiring the data energy according to the RGB color values.

10. The parallax acquisition method according to claim 6 or 8, wherein the step of acquiring the occlusion energy from the occlusion point pixel comprises:

setting an occlusion penalty constant;

acquiring a left occlusion pixel and a right occlusion pixel in the current region;

and acquiring the shielding energy according to the shielding punishment constant and the left shielding pixel and the right shielding pixel.

11. The parallax acquisition method according to claim 7 or 8, wherein the step of acquiring the smoothing energy from the smoothing point pixels comprises:

setting a smooth penalty amount;

acquiring non-shielding boundary points of the current area;

and obtaining the smoothing energy according to the smoothing penalty amount and the parallax value of each non-shielding boundary point.

12. A parallax acquisition apparatus, characterized by comprising:

the matching energy acquisition unit is used for acquiring the matching energy of each region in a matched image according to an initial parallax parameter, and the matched image comprises at least two regions;

an operation area determining unit, configured to determine at least one operation area for a current area in each of the areas, where the current area is an area being processed in each of the areas;

an optimized parallax parameter obtaining unit, configured to obtain an optimized parallax parameter according to the current region and the matching energy of each operation region determined by the operation region determining unit;

and the parallax value acquisition unit is used for acquiring the parallax of the current area according to the optimized parallax parameter acquired by the optimized parallax parameter acquisition unit.

13. The parallax acquisition apparatus according to claim 12, further comprising:

an initial parallax acquiring unit, configured to acquire an initial parallax of a pixel point in each region;

and a parallax parameter acquiring unit, configured to acquire the initial parallax parameter of each region according to the initial parallax acquired by the initial parallax acquiring unit.

14. The parallax acquisition apparatus according to claim 13, wherein the parallax parameters include a first parameter, a second parameter, and a third parameter, and the parallax parameter acquisition unit includes:

a first parameter obtaining module, configured to optionally select at least one row pixel point pair in each same pixel row in each of the regions; in each area, obtaining each candidate first parameter according to the pixel value and the initial parallax of each row of pixel point pairs; voting according to the candidate first parameters to acquire the first parameters of the regions;

a second parameter obtaining module, configured to optionally select at least one column pixel point pair in each same pixel column in each region; in each region, obtaining each candidate second parameter according to the coordinate value and the initial parallax of each row of pixel point pairs; voting according to the candidate second parameters to obtain the second parameters of the regions;

and a third parameter obtaining module, configured to fit, according to each candidate first parameter obtained by the first parameter obtaining module and each corresponding candidate second parameter obtained by the second parameter obtaining module, each corresponding candidate third parameter by using a disparity plane, and obtain a third parameter of each region according to each candidate third parameter vote.

15. The parallax acquisition apparatus according to claim 12, wherein the optimized parallax parameter acquisition unit comprises:

the matching energy processing module is used for taking the parallax parameter of each operation area as a new parallax parameter of the current area and acquiring each matching energy to be selected of the corresponding current area according to the new parallax parameter of the current area;

the operation area energy updating module is used for acquiring updated matching energy of each operation area according to the new parallax parameter of the current area;

the collaborative optimization energy acquisition module is used for calculating each piece of collaborative optimization energy of each to-be-selected matching energy and the corresponding matching energy of each operation area according to a preset weighting coefficient;

and the optimized parallax parameter acquisition module is used for acquiring the optimized parallax parameters of the current area according to the collaborative optimization energies.

16. The parallax acquisition apparatus according to claim 12, wherein the optimized parallax parameter acquisition unit comprises:

the matching energy processing module is used for performing weighted combination on the parallax parameter of each operation area and the parallax parameter of the current area to generate a parallax parameter which is used as a new parallax parameter of the current area, and acquiring each matching energy to be selected of the corresponding current area according to the new parallax parameter of the current area;

the operation area energy updating module is used for acquiring updated matching energy of each operation area according to the new parallax parameter of the current area;

the collaborative optimization energy acquisition module is used for calculating each piece of collaborative optimization energy of each to-be-selected matching energy and the corresponding matching energy of each operation area according to a preset weighting coefficient;

and the optimized parallax parameter acquisition module is used for acquiring the optimized parallax parameters of the current area according to the collaborative optimization energies.

17. The parallax acquiring apparatus according to any one of claims 12 to 16, wherein the matching energy includes data energy and occlusion energy, and the matching energy acquiring unit includes:

the first pixel point acquisition module is used for acquiring data point pixels and shielding point pixels in each region;

the data energy acquisition module is used for acquiring the data energy according to the data point pixel acquired by the first pixel point acquisition module;

and the shielding energy acquisition module is used for acquiring shielding energy according to the shielding point pixels acquired by the first pixel point acquisition module.

18. The parallax acquiring apparatus according to any one of claims 12 to 16, wherein the matching energy includes data energy and smoothing energy, and the matching energy acquiring unit includes:

the second pixel point acquisition module is used for acquiring data point pixels and smooth point pixels in each region;

the data energy acquisition module is used for acquiring data energy according to the data point pixels acquired by the second pixel point acquisition module;

and the smooth energy acquisition module is used for acquiring smooth energy according to the smooth point pixels acquired by the second pixel point acquisition module.

19. The parallax acquisition apparatus according to claim 12, wherein the matching energy includes data energy, occlusion energy, and smoothing energy, and the matching energy acquisition unit includes:

a third pixel point obtaining module, configured to obtain a data point pixel, a blocking point pixel, and a smooth point pixel in each region;

the data energy acquisition module is used for acquiring the data energy according to the data point pixel acquired by the third pixel point acquisition module;

the shielding energy acquisition module is used for acquiring the shielding energy according to the shielding point pixel acquired by the third pixel point acquisition module;

and the smooth energy acquisition module is used for acquiring the smooth energy according to the smooth point pixel acquired by the third pixel point acquisition module.

Images