WO2020125637A1 - 一种立体匹配方法、装置和电子设备 - Google Patents

一种立体匹配方法、装置和电子设备 Download PDF

Info

Publication number
WO2020125637A1
WO2020125637A1 PCT/CN2019/126040 CN2019126040W WO2020125637A1 WO 2020125637 A1 WO2020125637 A1 WO 2020125637A1 CN 2019126040 W CN2019126040 W CN 2019126040W WO 2020125637 A1 WO2020125637 A1 WO 2020125637A1
Authority
WO
WIPO (PCT)
Prior art keywords
seed point
disparity
point
seed
pixel
Prior art date
Application number
PCT/CN2019/126040
Other languages
English (en)
French (fr)
Inventor
谢青青
Original Assignee
深圳市道通智能航空技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市道通智能航空技术有限公司 filed Critical 深圳市道通智能航空技术有限公司
Publication of WO2020125637A1 publication Critical patent/WO2020125637A1/zh
Priority to US17/350,714 priority Critical patent/US11995858B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20024—Filtering details
    • G06T2207/20028—Bilateral filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20156—Automatic seed setting

Definitions

  • Embodiments of the present invention relate to the field of computer vision technology, and in particular, to a stereo matching method, device, and electronic equipment.
  • Stereo matching is a technique for calculating disparity maps from binocular views of the same scene, and is used to restore depth information from planar images.
  • 3D matching technology has been widely used in unmanned driving (distance measurement, navigation), 3D display, security monitoring, remote sensing image analysis, robot intelligent control, etc. .
  • the mainstream stereo matching algorithms can be divided into two categories: local stereo matching algorithms and global stereo matching algorithms.
  • the local stereo matching algorithm takes less time but the obtained stereo matching effect is fuzzy, while the global stereo matching algorithm can obtain better stereo matching effect but cannot achieve real-time due to the slow convergence optimization process. Therefore, in order to balance speed and accuracy, some researchers have proposed a stereo matching method that uses the disparity values of seed pixels in the segmentation area for disparity propagation, but the previous segmentation work of this method is very time-consuming and it is still difficult to balance speed at the same time. And precision.
  • embodiments of the present invention provide a stereo matching method, device, and electronic device, which can obtain good results in both the speed and accuracy of stereo matching.
  • the embodiments of the present invention provide the following technical solutions:
  • an embodiment of the present invention provides a stereo matching method, including:
  • the preset condition includes: passing left and right consistency detection;
  • a credibility propagation region corresponding to the non-seed point is constructed in the horizontal direction;
  • a dense disparity map is formed based on the disparity values of all the seed points and the disparity values of all the non-seed points.
  • selecting a pixel point satisfying a preset condition as a seed point includes:
  • a line segmentation to the right is performed based on the color difference and the pixel spatial distance to obtain multiple support regions, and, in each of the support regions Inside, the leftmost pixel is used as the starting point, and the pixel that satisfies the preset condition is used as the seed point.
  • the seed point When a seed point is found in the support area, it jumps to the next support area to find the next seed point.
  • the preset condition further includes:
  • the difference between the second smallest cost corresponding to the pixel and the minimum cost is greater than or equal to the first preset threshold.
  • the constructing the credibility propagation region corresponding to the non-seed point along the horizontal direction based on the color difference and the pixel spatial distance includes:
  • the non-seed point As the center, detect the left and right split points of the non-seed point in the horizontal left and horizontal right directions, respectively, and use the left and right split points
  • the area between the split points serves as the credibility propagation area corresponding to the non-seed points
  • left split point and the right split point satisfy the following conditions:
  • the color difference from the non-seed point is greater than the second preset threshold; or,
  • the pixel spatial distance from the non-seed point is greater than a third preset threshold.
  • the determining the disparity value of the non-seed point based on the disparity value of the left seed point or the right seed point of the nearest neighbor in the credibility propagation area of the non-seed point includes: :
  • the disparity value of the left seed point is used as the disparity value of the non-seed point
  • the disparity value of the right seed point is used as the disparity value of the non-seed point
  • the non-seed point has the nearest neighbor left seed point and the right seed point in the credibility propagation area at the same time, based on the characteristics of the area where the non-seed point is located, based on the characteristics of the left seed point
  • the disparity value and the disparity value of the right seed point determine the disparity value of the non-seed point.
  • the disparity value of the non-seed point is determined based on the disparity value of the left seed point and the disparity value of the right seed point according to the characteristics of the area where the non-seed point is located ,include:
  • the non-seed point is in an occlusion area or a depth discontinuity area, the smaller of the disparity value of the left seed point and the disparity value of the right seed point is used as the non-seed point Parallax value
  • the linear interpolation between the disparity value of the left seed point and the disparity value of the right seed point is used as the disparity value of the non-seed point.
  • the method further includes:
  • the streak effect in the dense disparity map is corrected based on the vertical voting method.
  • the method further includes:
  • an embodiment of the present invention provides a stereo matching device, including: a matching unit, a seed point unit, a disparity propagation unit, and a disparity map generating unit;
  • the matching unit is used to calculate the initial disparity value of each pixel in the left view and the right view through a local stereo matching algorithm to obtain the initial left disparity map of the left view and the initial right disparity map of the right view ;
  • the seed point unit includes a seed point selection module and a seed point parallax determination module:
  • the seed point selection module is used to select a pixel point satisfying a preset condition as a seed point in the initial left disparity map or the initial right disparity map, where the preset condition includes: Detection
  • the seed point parallax determination module is configured to use the initial parallax value of the pixel point as the parallax value of the seed point;
  • the parallax propagation unit includes a segmentation module and a parallax propagation module:
  • the segmentation module is used to construct, for each non-seed point, a credibility propagation region corresponding to the non-seed point along the horizontal direction based on the color difference and the pixel spatial distance;
  • the disparity propagation module is used to determine the disparity value of the non-seed point based on the disparity value of the left seed point or the right seed point of the nearest neighbor of the non-seed point in the credibility propagation area;
  • the disparity map generating unit is configured to form a dense disparity map based on the disparity values of all the seed points and the disparity values of all the non-seed points.
  • the seed point selection module is specifically used for:
  • line segmentation is performed based on the color difference and pixel spatial distance to the right to obtain multiple support regions, and, in each of the support regions Inside, the leftmost pixel is used as the starting point, and the pixel that satisfies the preset condition is used as the seed point.
  • the seed point When a seed point is found in the support area, it jumps to the next support area to find the next seed point.
  • the preset condition further includes:
  • the difference between the second smallest cost corresponding to the pixel and the minimum cost is greater than or equal to the first preset threshold.
  • the segmentation module is specifically used for:
  • the non-seed point As the center, detect the left and right split points of the non-seed point in the horizontal left and horizontal right directions, respectively, and use the left and right split points
  • the area between the split points serves as the credibility propagation area corresponding to the non-seed points
  • left split point and the right split point satisfy the following conditions:
  • the color difference from the non-seed point is greater than the second preset threshold; or,
  • the pixel spatial distance from the non-seed point is greater than a third preset threshold.
  • the disparity propagation module is specifically used to:
  • the disparity value of the left seed point is used as the disparity value of the non-seed point
  • the disparity value of the right seed point is used as the disparity value of the non-seed point
  • the non-seed point has both the left seed point and the right seed point of the nearest neighbor in the credibility propagation area, based on the characteristics of the area where the non-seed point is located, based on the characteristics of the left seed point
  • the disparity value and the disparity value of the right seed point determine the disparity value of the non-seed point.
  • the parallax propagation module is specifically configured to:
  • the non-seed point is in an occlusion area or a depth discontinuity area, the smaller of the disparity value of the left seed point and the disparity value of the right seed point is used as the non-seed point Parallax value
  • the linear interpolation between the disparity value of the left seed point and the disparity value of the right seed point is used as the disparity value of the non-seed point.
  • the device further includes: a correction unit;
  • the correction unit is used to correct the streak effect in the dense disparity map based on the vertical voting method.
  • correction unit is also used for:
  • an embodiment of the present invention provides an electronic device, including:
  • At least one processor and,
  • a memory communicatively connected to the at least one processor; wherein,
  • the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the stereo matching method as described above.
  • an embodiment of the present invention further provides a non-transitory computer-readable storage medium that stores computer-executable instructions, and the computer-executable instructions are used to enable an electronic device The stereo matching method as described above is performed.
  • an embodiment of the present invention also provides a computer program product.
  • the computer program product includes a computer program stored on a non-transitory computer-readable storage medium.
  • the computer program includes program instructions. When the program When the instruction is executed by the electronic device, the electronic device is caused to perform the stereo matching method as described above.
  • the stereo matching method, device and electronic device provided by the embodiments of the present invention first calculate the initial view of each pixel in the binocular view based on the local stereo matching algorithm Difference values to obtain an initial left disparity map and an initial right disparity map, and then select pixels that satisfy preset conditions (including: through left and right consistency detection) in the initial left disparity map or the initial right disparity map As a seed point, and use the initial disparity value of the pixel point as the disparity value of the seed point; and for each non-seed point, the non-seed is constructed along the horizontal direction based on the color difference and the pixel spatial distance
  • the credibility propagation area corresponding to the point and based on the disparity value of the left seed point or the right seed point of the nearest neighbor of the non-seed point in the credibility propagation area, determine the visual angle of the non-seed point
  • the difference value is finally based on the disparity value of all the seed points and the disparity value of
  • FIG. 1 is a schematic flowchart of a stereo matching method provided by an embodiment of the present invention.
  • FIG. 2 is a schematic diagram of an example of selecting a seed point provided by an embodiment of the present invention.
  • FIG. 3 is an exemplary schematic diagram of constructing a credibility propagation region of non-seed points provided by an embodiment of the present invention
  • FIG. 4 is a schematic flowchart of another stereo matching method provided by an embodiment of the present invention.
  • FIG. 5 is a schematic structural diagram of a stereo matching device provided by an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present invention.
  • Embodiments of the present invention provide a stereo matching method, device, and electronic equipment, which can be applied to any application scenario that requires stereo matching to obtain a parallax map, such as three-dimensional reconstruction, binocular visual ranging, unmanned driving, and three-dimensional display. , Robot intelligent control, etc.
  • the stereo matching method provided by the embodiment of the present invention can be executed by any electronic device, and is a method of growing a dense disparity map based on seed point propagation. Specifically, it firstly calculates the left view and the right separately through a local stereo matching algorithm The initial disparity value of each pixel in the view to obtain the initial left disparity map of the left view and the initial right disparity map of the right view; then, in the initial left disparity map or the initial right disparity map , Select a pixel point that satisfies a preset condition as a seed point, and use the initial disparity value of the pixel point as a disparity value of the seed point, where the preset condition includes: passing left and right consistency detection; For each non-seed point, based on the color difference and pixel spatial distance, a credibility propagation region corresponding to the non-seed point is constructed in the horizontal direction; based on the nearest neighbor of the non-seed point in the credibility propagation region The disparity value of the left seed point or the right seed point
  • the stereo matching method provided by the embodiment of the present invention can form a dense parallax map quickly because only local stereo matching and parallax propagation are needed, on the other hand, because the pixel points with higher reliability in the local stereo matching are selected as Seed points, and carry out parallax propagation in the confidence propagation area formed based on the color difference and pixel spatial distance, which can improve the accuracy of the parallax value of each pixel in the formed dense parallax map, thus, the speed of stereo matching Both accuracy and accuracy have obtained good results.
  • the stereo matching device provided by the embodiment of the present invention may be a virtual device composed of a software program capable of implementing the stereo matching method provided by the embodiment of the present invention, which is based on the same inventive concept and has the same concept as the stereo matching method provided by the embodiment of the present invention.
  • the electronic device provided by the embodiment of the present invention may be any hardware device that has a certain logic operation capability and can provide one or more functions that can meet the user's intention.
  • it may include, but is not limited to: drones, robots, and automobile MCUs. , Smart phones, tablets, servers, etc.
  • the electronic device can execute the stereo matching method provided by the embodiment of the present invention or run the stereo matching device provided by the embodiment of the present invention, which is also based on the same inventive concept and has the same technical features as the stereo matching method provided by the embodiment of the present invention Beneficial effect.
  • FIG. 1 is a schematic flowchart of a stereo matching method provided by an embodiment of the present invention.
  • the method may be executed by electronic devices such as drones, robots, and automobile MCUs.
  • the stereo matching method may include but not limited to the following steps:
  • Step 110 Calculate the initial disparity value of each pixel in the left view and the right view through a local stereo matching algorithm to obtain the initial left disparity map of the left view and the initial right disparity map of the right view.
  • the "left view” and the “right view” refer to the binocular view corresponding to the same shooting scene.
  • the binocular view may be pre-stored on the electronic device, or may be from a certain Received from a data source, or it can be captured in real time through a binocular or monocular camera.
  • the "initial left disparity map” includes the initial disparity value of each pixel in the left view, wherein, in the initial left disparity map, the initial disparity value of a pixel specifically refers to the position of the pixel and The distance d1 between the position of the corresponding point/matching point of the pixel in the right view.
  • the "initial right parallax map” includes the initial parallax value of each pixel in the right view, wherein, in the initial right parallax map, the initial parallax value of a pixel specifically refers to the position of the pixel and The distance d2 between the position of the corresponding point/matching point of the pixel in the left view.
  • any local stereo matching algorithm such as the absolute value of pixel difference (SAD), adaptive window algorithm, adaptive weight algorithm, etc.
  • SAD absolute value of pixel difference
  • adaptive window algorithm adaptive window algorithm
  • adaptive weight algorithm etc.
  • the color difference and the structure code of the pixel p and its corresponding point q in the right view constitutes the cost equation of the pixel p, and then the initial disparity value of the pixel p is determined by optimizing the cost equation; similarly, for each pixel q in the right view, it can also be determined by the The sum of the color difference between the pixel point q and its corresponding point p in the left view and the difference in structural encoding constitute the cost equation of the pixel point q, and then the initial parallax of the pixel point q is determined by optimizing the cost equation value.
  • the initial disparity value of each pixel in the left view and the right view is calculated by a simple local stereo matching algorithm, which can reduce the calculation amount of the stereo matching process and increase the speed of stereo matching.
  • Step 120 In the initial left disparity map or the initial right disparity map, select a pixel point satisfying a preset condition as a seed point, and use the initial disparity value of the pixel point as the seed point of view Difference.
  • the "preset condition" includes: through left and right consistency detection, that is, the initial parallax value of the pixel point p in the initial left disparity map and the pixel point p at the initial The initial disparity value of the corresponding point q in the right disparity map is equal; or, the initial disparity value of the pixel point q in the initial right disparity map and the corresponding point of the pixel point q in the initial left disparity map The initial parallax value of p is equal.
  • the "seed point” is the starting point of the disparity propagation process.
  • the initial left disparity map may be used as a reference to perform disparity propagation to form a final dense disparity map; or the initial right disparity map may be used as a reference to perform disparity propagation to form a final dense disparity map.
  • pixels that satisfy the preset condition can be selected as seed points in the initial left disparity map, and pixels that do not satisfy the preset condition are non-seed points, and finally all The disparity values of the seed points and the disparity values of all non-seed points form a dense disparity map.
  • the pixel points that satisfy the preset condition can be selected as seed points in the initial right disparity map, and the pixel points that do not satisfy the preset condition are non-seed points, and finally the disparity of all seed points is combined The values and the disparity values of all non-seed points form a dense disparity map.
  • a specific implementation manner of selecting pixels that satisfy preset conditions as seed points may be: performing a one-by-one judgment on each pixel in the initial left disparity map, If a certain pixel meets the preset condition, the pixel is marked as a seed point.
  • a specific implementation manner of selecting a pixel point satisfying a preset condition as a seed point may also be:
  • line segmentation to the right based on the color difference and the pixel spatial distance may be performed to obtain multiple support regions.
  • the pixel point at the leftmost end is used as a starting point to find a pixel point that satisfies the preset condition as a seed point.
  • the "color difference” is used to describe the color difference between two pixels, which may specifically be: the absolute value of the difference between the two pixels on the color channel with the largest difference among the color channels, for example, In RGB space, if the absolute value of the difference between pixel p0 and pixel p1 on the R channel is I1, the absolute value of the difference on the G channel is I2, and the absolute value of the difference on the B channel is I3, and I1>I2>I3, then the color difference between pixel points p0 and p1 is I1.
  • the absolute value of the difference on the color channel with the largest difference is used to measure the color difference between the two pixels, which can avoid a large change in each color channel due to changes in lighting and other conditions. The resulting misjudgments more accurately reflect the true edge of the scene.
  • pixel spatial distance refers to the pixel distance between two pixels, for example, the pixel spatial distance between pixel p0 and pixel p1 is x pixels.
  • the "from the leftmost pixel point of the scan line, based on the color difference and the pixel space distance to the right line segmentation to obtain multiple support areas" can specifically be: as shown in Figure 2, the scan line
  • the leftmost pixel point p0 is the starting point. Determine whether the color difference and pixel spatial distance between each pixel point and the starting point p0 are within the corresponding predetermined range to the right one by one. If so, return the pixel point and the starting point p0 It is the same support area; if not, the pixel is used as the starting point p1 of the next support area, and the linear segmentation to the right is continued; and so on, multiple support areas on the scan line can be obtained.
  • the seed point is a pixel point that has passed at least the left-right consistency detection, and passing the left-right consistency detection indicates that the initial disparity value of the pixel is highly reliable, and it can be considered as the correct disparity value, thus ,
  • the initial disparity value of the pixel point can be directly used as the disparity value of the seed point/pixel point.
  • the "preset condition" may further include: when performing local stereo matching, the difference between the second smallest cost corresponding to the pixel and the minimum cost is greater than or equal to the first preset threshold.
  • the first threshold may be a fixed value or a value greater than 50% of the second-smallest cost, as long as the minimum cost corresponding to the pixel is significantly smaller than the second-smallest cost.
  • Step 130 For each non-seed point, based on the color difference and the pixel spatial distance, construct a credibility propagation region corresponding to the non-seed point in the horizontal direction.
  • non-seed points are pixels other than the seed points in the initial left disparity map (or the initial right disparity map).
  • the "credibility propagation area” is a linear segmentation area of non-seed points, which is used for parallax propagation to determine the parallax value of the non-seed points. The parallax value in each credibility propagation area changes slowly.
  • each non-seed point can be linearly segmented along the scan line direction to obtain the credibility propagation area corresponding to each non-seed point.
  • the left seed point P1 and the right finger point of the non-seed point P can be detected in the horizontal left and horizontal right directions with the non-seed point P as the center, respectively.
  • Pr, and the area A between the left division point Pl and the right division point Pr (excluding the left division point Pl and the right division point Pr) corresponds to the non-seed point Credibility propagation area; where the left split point Pl and the right split point Pr satisfy one or all of the following conditions:
  • the second preset threshold and the third preset threshold may be set according to actual application scenarios, as long as the area A between the left split point Pl and the right split point Pr satisfies the characteristic that the parallax value changes slowly That's it.
  • Step 140 Determine the disparity value of the non-seed point based on the disparity value of the left seed point or the right seed point of the nearest neighbor of the non-seed point in the confidence propagation area.
  • the non-seed point is determined based on the disparity value of the left seed point or the right seed point of the nearest neighbor in the corresponding confidence propagation area of the non-seed point The parallax value of the point.
  • the seed point parallax can be propagated along the direction of the scan line. For each non-seed point P, the left seed point Sl and the right seed point Sr of the nearest neighbor can be found first in its corresponding credibility propagation area.
  • the non-seed point P only has the nearest left seed point Sl in the credibility propagation area, then each pixel in the same credibility propagation area has a similar parallax value , Using the disparity value of the left seed point Sl as the disparity value of the non-seed point P; in the same way, if the non-seed point P only exists in the credibility propagation area, there is only the nearest neighbor's right For the side seed point Sr, the disparity value of the right seed point Sr is used as the disparity value of the non-seed point P.
  • the non-seed point P has both the nearest left seed point Sl and the right seed point Sr in the credibility propagation area, in some embodiments, the left seed point Sl
  • the linear interpolation between the disparity value and the disparity value of the right seed point Sr is used as the disparity value of the non-seed point P.
  • the non-seed point P may also be based on The characteristics of the region where P is located, based on the disparity value of the left seed point Sl and the disparity value of the right seed point Sr, determine the disparity value of the non-seed point P.
  • the non-seed point P is in an occlusion area (where, if the initial disparity value of the non-seed point P and the initial disparity value of the corresponding point Q of the non-seed point P in the initial right disparity map are If the absolute value of the difference is greater than or equal to the fourth preset threshold (for example, 1 or 2 pixels), it can be considered that the non-seed point P is in the occlusion area), indicating that the non-seed point P is in the switching area between the foreground and the background
  • the fourth preset threshold for example, 1 or 2 pixels
  • the non-seed point P is in a depth discontinuity region (where, if the difference between the disparity value of the left seed point Sl of the nearest neighbor of the non-seed point P and the disparity value of the right seed point Sr is absolute If the value is greater than the fifth preset threshold, the non-seed point P can be considered to be in a depth discontinuity area), which means that the non-seed point P is on the edge of some rapid changes. At this time, the non-seed point P can also be considered to be located In the distance, the smaller of the disparity value of the left seed point S1 and the disparity value of the right seed point Sr is used as the disparity value of the non-seed point P.
  • the non-seed point P is neither in the occlusion area nor the depth discontinuity area, it can be considered that the non-seed point P is in the smooth area where the parallax slowly changes.
  • the The linear interpolation between the disparity value and the disparity value of the right seed point may be used as the disparity value of the non-seed point.
  • the The disparity value can further enhance the credibility of the disparity value of each non-seed point and improve the accuracy of stereo matching.
  • Step 150 Form a dense disparity map based on the disparity values of all the seed points and the disparity values of all the non-seed points.
  • all pixel points in the initial left disparity map are divided into two categories, namely, seed points and non-seed points. Therefore, the disparity values of all seed points and the disparity values of all non-seed points are integrated according to the corresponding coordinate positions to form a dense disparity map.
  • the stereo matching method provided by the embodiment of the present invention first calculates the initial disparity value of each pixel in the binocular view based on the local stereo matching algorithm to obtain the initial left Disparity map and initial right disparity map, and then in the initial left disparity map or the initial right disparity map, select the pixel points that satisfy the preset conditions (including: through left and right consistency detection) as seed points, and use the The initial disparity value of the pixel point is used as the disparity value of the seed point; and for each non-seed point, based on the color difference and the pixel spatial distance, the credibility propagation corresponding to the non-seed point is constructed along the horizontal direction Area, and based on the disparity value of the left seed point or the right seed point of the nearest neighbor of the non-seed point in the confidence propagation area, determine the disparity value of the non-seed point, and finally based on all The disparity value of the seed point and the disparity value of
  • the second embodiment of the present invention provides another stereo matching method based on the foregoing first embodiment.
  • the method may include but not limited to the following steps:
  • Step 210 Calculate the initial disparity value of each pixel in the left view and the right view through the local stereo matching algorithm to obtain the initial left disparity map of the left view and the initial right disparity map of the right view.
  • Step 220 In the initial left disparity map or the initial right disparity map, select a pixel point satisfying a preset condition as a seed point, and use the initial disparity value of the pixel point as the seed point of view Difference.
  • Step 230 For each non-seed point, based on the color difference and the pixel spatial distance, construct a credibility propagation region corresponding to the non-seed point in the horizontal direction.
  • Step 240 Determine the disparity value of the non-seed point based on the disparity value of the left seed point or the right seed point of the nearest neighbor of the non-seed point in the confidence propagation area.
  • Step 250 Form a dense disparity map based on the disparity values of all the seed points and the disparity values of all the non-seed points.
  • Step 260 Correct the streak effect in the dense disparity map based on the vertical voting method.
  • the reliability propagation area is constructed along the horizontal direction for parallax propagation, and horizontal stripes may be generated. Therefore, in this embodiment, The streak effect in the dense disparity map is corrected based on the vertical voting method.
  • a histogram with (0, dmax) as the horizontal axis range is first constructed, where dmax is a given maximum disparity value. Then, using the pixel point p as the center, select a pixel point in a certain neighborhood along the vertical direction. If the color difference between a pixel point p1 and the pixel point p in the neighborhood is less than the threshold, then in the histogram
  • the horizontal axis of the figure marks the parallax value corresponding to the pixel point p1, and adds 1 to the vertical axis corresponding to the parallax value.
  • the parallax value corresponding to the maximum value on the vertical axis in the histogram (that is, the parallax value with the largest number of votes) is selected as the parallax value of the pixel point p. Therefore, the adverse effects in the horizontal direction can be offset, and the real-time performance is better.
  • the method may further include: using bilateral filtering to update the disparity value of each pixel in the four-neighborhood.
  • the "four-neighborhood" refers to four pixels on the top, bottom, left and right of the nearest neighbor of the pixel.
  • steps 210 to 250 have the same technical features as the steps 110 to 150 in the stereo matching method shown in FIG. 1 respectively. Therefore, for the specific implementation, refer to steps 110 to 150 in the above embodiment The corresponding description in 150 will not be repeated in this embodiment.
  • the beneficial effect of the embodiments of the present invention is that after obtaining the dense disparity map, the streak effect in the dense disparity map is corrected based on the vertical voting method, which can further improve the accuracy of the stereo matching.
  • the device 50 includes a matching unit 51, a seed point unit 52, a disparity propagation unit 53, and a disparity map generating unit 54.
  • the matching unit 51 is used to calculate the initial disparity value of each pixel in the left view and the right view through a local stereo matching algorithm to obtain the initial left disparity map of the left view and the initial right disparity of the right view Figure;
  • the seed point unit 52 includes: a seed point selection module 521 and a seed point disparity determination module 522; the seed point selection module 521 is used to select a pre-satisfaction in the initial left disparity map or the initial right disparity map Set a conditional pixel point as a seed point, wherein the preset condition includes: through left and right consistency detection; the seed point parallax determination module 522 is used to use the initial parallax value of the pixel point as the seed point Parallax value
  • the disparity propagation unit 53 includes: a segmentation module 531 and a parallax propagation module 532; the segmentation module 531 is used to construct, for each non-seed point, based on the color difference and the pixel spatial distance, the non-seed point corresponding to the non-seed point in the horizontal direction Credibility propagation area; the disparity propagation module 532 is used to determine the non-seed point based on the disparity value of the left seed point or the right seed point of the nearest neighbor in the credibility propagation area The parallax value of the seed point;
  • the disparity map generating unit 54 is configured to form a dense disparity map based on the disparity values of all the seed points and the disparity values of all the non-seed points.
  • the matching unit 51 may first be used to calculate the initial disparity value of each pixel in the left view and the right view through a local stereo matching algorithm to obtain the initial left disparity map and the An initial right disparity map of the right view; then the seed point unit 52 selects a pixel point satisfying a preset condition as a seed point in the initial left disparity map or the initial right disparity map, and uses the The initial disparity value of the pixel is used as the disparity value of the seed point; further, for each non-seed point by the disparity propagation unit 53, based on the color difference and the pixel spatial distance, the non-seed point correspondence is constructed in the horizontal direction The credibility propagation area of, and based on the disparity value of the left seed point or the right seed point of the nearest neighbor of the non-seed point in the credibility propagation area, determine the disparity value of the non-seed point Finally, the disparity map generation unit 54 forms a dense disparity map based on the disparity values of all the seed points and the disparity values of
  • the seed point selection module 521 is specifically used to:
  • the preset condition further includes: when performing local stereo matching, the difference between the second smallest cost corresponding to the pixel and the minimum cost is greater than or equal to the first preset threshold.
  • the segmentation module 531 is specifically used to:
  • the non-seed point As the center, detect the left and right split points of the non-seed point in the horizontal left and horizontal right directions, respectively, and use the left and right split points
  • the area between the division points serves as the credibility propagation area corresponding to the non-seed point; wherein the left division point and the right division point satisfy the following conditions:
  • the color difference from the non-seed point is greater than the second preset threshold; or,
  • the pixel spatial distance from the non-seed point is greater than a third preset threshold.
  • the disparity propagation module 532 is specifically used to:
  • the disparity value of the left seed point is used as the disparity value of the non-seed point
  • the disparity value of the right seed point is used as the disparity value of the non-seed point
  • the non-seed point has the nearest neighbor left seed point and the right seed point in the credibility propagation area at the same time, based on the characteristics of the area where the non-seed point is located, based on the characteristics of the left seed point
  • the disparity value and the disparity value of the right seed point determine the disparity value of the non-seed point.
  • the parallax propagation module 532 is specifically configured to:
  • the non-seed point is in an occlusion area or a depth discontinuity area, the smaller of the disparity value of the left seed point and the disparity value of the right seed point is used as the non-seed point The disparity value; otherwise, the linear interpolation between the disparity value of the left seed point and the disparity value of the right seed point is used as the disparity value of the non-seed point.
  • the device 50 further includes: a correction unit 55; the correction unit 55 is used to correct the streak effect in the dense parallax map based on the vertical voting method.
  • the correction unit 55 is further configured to: use bilateral filtering to update its disparity value in the four neighborhoods of each pixel.
  • the stereo matching device uses the matching unit 51 to first calculate the pixel points in the left view and the right view through the local stereo matching algorithm, respectively An initial disparity value to obtain the initial left disparity map of the left view and the initial right disparity map of the right view; and then the seed point unit 52 in the initial left disparity map or the initial right disparity map Select a pixel point that satisfies a preset condition as a seed point, and use the initial disparity value of the pixel point as the disparity value of the seed point; and then pass the disparity propagation unit 53 for each non-seed point, Based on the color difference and the pixel spatial distance, a credibility propagation region corresponding to the non-seed point is constructed in the horizontal direction, and based on the non-seed point in the credibility propagation region, the nearest left seed point or right The disparity value of the side seed points, to determine the disparity value of the non-seed
  • the electronic device 600 may be any type of hardware device, such as: drones, automotive MCUs, smart phones, robots, personal computers, and wearable intelligence.
  • the device and the like can execute the stereo matching method provided by the above method embodiment, or run the stereo matching device provided by the above device embodiment.
  • the electronic device 600 includes:
  • One or more processors 601 and a memory 602, one processor 601 is taken as an example in FIG. 6.
  • the processor 601 and the memory 602 may be connected through a bus or in other ways.
  • the connection through a bus is used as an example.
  • the memory 602 can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions/modules corresponding to the stereo matching method in the embodiments of the present invention (For example, the matching unit 51, the seed point unit 52, the parallax propagation unit 53, the parallax map generation unit 54 and the correction unit 55 shown in FIG. 5).
  • the processor 601 executes non-transitory software programs, instructions, and modules stored in the memory 602 to execute various functional applications and data processing of the stereo matching device 50, that is, to implement the stereo matching method of any of the foregoing method embodiments.
  • the memory 602 may include a storage program area and a storage data area, where the storage program area may store an operating system and application programs required for at least one function; the storage data area may store data created according to the use of the stereo matching device 50 and the like.
  • the memory 602 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices.
  • the memory 602 may optionally include memories remotely set with respect to the processor 601, and these remote memories may be connected to the processor 601 through a network. Examples of the aforementioned network include, but are not limited to, the Internet, intranet, local area network, mobile communication network, and combinations thereof.
  • the one or more modules are stored in the memory 602, and when executed by the one or more processors 601, execute the stereo matching method in any of the above method embodiments, for example, execute the above-described FIG. 1
  • the method steps 110 to 150 and the method steps 210 to 260 in FIG. 4 implement the functions of the units 51-55 in FIG. 5.
  • Embodiments of the present invention also provide a non-transitory computer-readable storage medium.
  • the non-transitory computer-readable storage medium stores computer-executable instructions, which are executed by one or more processors, for example Is executed by a processor 601 in FIG. 6, which may cause the one or more processors to perform the stereo matching method in any of the above method embodiments, for example, performing the above-described method steps 110 to 150 in FIG.
  • the method steps 210 to 260 in 4 implement the functions of the units 51-55 in FIG.
  • each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware.
  • a person of ordinary skill in the art may understand that all or part of the processes in the method of the above embodiments may be completed by a computer program in a computer program product instructing relevant hardware.
  • the computer program may be stored in a non-transitory computer.
  • the computer program includes program instructions.
  • the program instructions When the program instructions are executed by the electronic device, the electronic device may be allowed to execute the processes of the foregoing method embodiments.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) or a random storage memory (Random Access Memory, RAM), etc.
  • the above products can execute the stereo matching method provided by the embodiments of the present invention, and have corresponding functional modules and beneficial effects for performing the stereo matching method.
  • the stereo matching method provided in this embodiment of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

一种立体匹配方法、装置和电子设备，所述方法包括：通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，获得初始左视差图和初始右视差图（110）；在初始左视差图或初始右视差图中，选择出种子点，并确定该种子点的视差值为其初始视差值（120）；对每一个非种子点，基于颜色差以及像素空间距离沿水平方向构造其可信度传播区域（130）；基于该非种子点在该可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定该非种子点的视差值（140）；基于所有种子点的视差值和所有非种子点的视差值形成稠密视差图（150）。通过上述技术方案，能够在立体匹配的速度与精度这两个方面均获得较好的结果。

Description

一种立体匹配方法、装置和电子设备 技术领域
本发明实施例涉及计算机视觉技术领域，尤其涉及一种立体匹配方法、装置和电子设备。
背景技术
立体匹配是一种从相同场景的双目视图中计算视差图的技术，用于从平面图像中恢复深度信息。作为三维重建、立体导航、非接触测距等技术中的关键步骤，立体匹配技术已广泛应用于无人驾驶(测距，导航)、3D显示、安防监控、遥感图像分析、机器人智能控制等领域。
当前，立体匹配的主流算法可以分为两大类：局部立体匹配算法和全局立体匹配算法。其中，局部立体匹配算法耗时较短但获取到的立体匹配效果较模糊，而全局立体匹配算法能够获取到较佳的立体匹配效果但由于收敛缓慢的优化过程而无法达到实时。从而，为了兼顾速度与精度，一些学者提出了一种在分割区域中利用种子像素点的视差值进行视差传播的立体匹配方法，但该方法前期的分割工作十分耗时，依然难以同时兼顾速度和精度。
因此，现有的立体匹配方法还有待改进和发展。
发明内容
有鉴于此，本发明实施例提供一种立体匹配方法、装置和电子设备，能够在立体匹配的速度与精度这两个方面均获得较好的结果。
为解决上述技术问题，本发明实施例提供了如下技术方案：
第一方面，本发明实施例提供一种立体匹配方法，包括：
通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图；
在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，并将所述像素点的初始视差值作为所述种子点的视差值，其中，所述预设条件包括：通过左右一致性检测；
对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述 非种子点对应的可信度传播区域；
基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值；
基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图。
可选地，所述在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，包括：
对所述初始左视差图或所述初始右视差图进行逐行扫描，以选择出满足预设条件的像素点作为种子点；
其中，对于每一扫描行，从所述扫描行的最左端像素点开始，基于颜色差和像素空间距离进行向右传输的线分割，获得多个支持区域，并且，在每一个所述支持区域内，以最左端的像素点为起始点，寻找满足预设条件的像素点作为种子点，当在所述支持区域内寻找到一个种子点时，跳至下一支持区域寻找下一种子点。
可选地，所述预设条件还包括：
在进行局部立体匹配时，所述像素点对应的次小代价与最小代价之差大于或等于第一预设阈值。
可选地，所述基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域，包括：
以所述非种子点为中心，分别沿水平向左和水平向右的方向检测所述非种子点的左侧分割点和右侧分割点，并以所述左侧分割点和所述右侧分割点之间的区域作为所述非种子点对应的可信度传播区域；
其中，所述左侧分割点和所述右侧分割点满足下述条件：
与所述非种子点的颜色差大于第二预设阈值；或者，
与所述非种子点的像素空间距离大于第三预设阈值。
可选地，所述基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值，包括：
如果所述非种子点在所述可信度传播区域内只存在最近邻的左侧种子点，则以所述左侧种子点的视差值作为所述非种子点的视差值；
如果所述非种子点在所述可信度传播区域内只存在最近邻的右侧种子点， 则以所述右侧种子点的视差值作为所述非种子点的视差值；
如果所述非种子点在所述可信度传播区域内同时存在最近邻的左侧种子点和右侧种子点，则根据所述非种子点所在区域的特性，基于所述左侧种子点的视差值和所述右侧种子点的视差值，确定所述非种子点的视差值。
可选地，所述根据所述非种子点所在区域的特性，基于所述左侧种子点的视差值和所述右侧种子点的视差值，确定所述非种子点的视差值，包括：
如果所述非种子点处于闭塞区域或深度不连续区域，则以所述左侧种子点的视差值和所述右侧种子点的视差值中的较小值作为所述非种子点的视差值；
否则，以所述左侧种子点的视差值和所述右侧种子点的视差值之间的线性插值作为所述非种子点的视差值。
可选地，在所述基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图的步骤之后，所述方法还包括：
基于竖直方向的投票法修正所述稠密视差图中的条纹效应。
可选地，在所述基于竖直方向的投票法修正所述稠密视差图中的条纹效应的步骤之后，所述方法还包括：
在每一像素点的四邻域内利用双边滤波来更新其视差值。
第二方面，本发明实施例提供一种立体匹配装置，包括：匹配单元、种子点单元、视差传播单元以及视差图生成单元；
所述匹配单元用于通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图；
所述种子点单元包括种子点选取模块和种子点视差确定模块：
所述种子点选取模块用于在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，其中，所述预设条件包括：通过左右一致性检测；
所述种子点视差确定模块用于将所述像素点的初始视差值作为所述种子点的视差值；
所述视差传播单元包括分割模块和视差传播模块：
所述分割模块用于对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域；
所述视差传播模块用于基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值；
所述视差图生成单元用于基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图。
可选地，所述种子点选取模块具体用于：
对所述初始左视差图或所述初始右视差图进行逐行扫描，以选择出满足预设条件的像素点作为种子点；
其中，对于每一扫描行，从所述扫描行的最左端像素点开始，基于颜色差和像素空间距离进行向右传输的线分割，获得多个支持区域，并且，在每一个所述支持区域内，以最左端的像素点为起始点，寻找满足预设条件的像素点作为种子点，当在所述支持区域内寻找到一个种子点时，跳至下一支持区域寻找下一种子点。
可选地，所述预设条件还包括：
在进行局部立体匹配时，所述像素点对应的次小代价与最小代价之差大于或等于第一预设阈值。
可选地，所述分割模块具体用于：
以所述非种子点为中心，分别沿水平向左和水平向右的方向检测所述非种子点的左侧分割点和右侧分割点，并以所述左侧分割点和所述右侧分割点之间的区域作为所述非种子点对应的可信度传播区域；
其中，所述左侧分割点和所述右侧分割点满足下述条件：
与所述非种子点的颜色差大于第二预设阈值；或者，
与所述非种子点的像素空间距离大于第三预设阈值。
可选地，所述视差传播模块具体用于：
如果所述非种子点在所述可信度传播区域内只存在最近邻的左侧种子点，则以所述左侧种子点的视差值作为所述非种子点的视差值；
如果所述非种子点在所述可信度传播区域内只存在最近邻的右侧种子点，则以所述右侧种子点的视差值作为所述非种子点的视差值；
如果所述非种子点在所述可信度传播区域内同时存在最近邻的左侧种子点和右侧种子点，则根据所述非种子点所在区域的特性，基于所述左侧种子点的视差值和所述右侧种子点的视差值，确定所述非种子点的视差值。
可选地，如果所述非种子点在所述可信度传播区域内同时存在最近邻的左侧种子点和右侧种子点，所述视差传播模块具体用于：
如果所述非种子点处于闭塞区域或深度不连续区域，则以所述左侧种子点的视差值和所述右侧种子点的视差值中的较小值作为所述非种子点的视差值；
否则，以所述左侧种子点的视差值和所述右侧种子点的视差值之间的线性插值作为所述非种子点的视差值。
可选地，所述装置还包括：修正单元；
所述修正单元用于基于竖直方向的投票法修正所述稠密视差图中的条纹效应。
可选地，所述修正单元还用于：
在每一像素点的四邻域内利用双边滤波来更新其视差值。
第三方面，本发明实施例提供一种电子设备，包括：
至少一个处理器；以及，
与所述至少一个处理器通信连接的存储器；其中，
所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如上所述的立体匹配方法。
第四方面，本发明实施例还提供了一种非暂态计算机可读存储介质，所述非暂态计算机可读存储介质存储有计算机可执行指令，所述计算机可执行指令用于使电子设备执行如上所述的立体匹配方法。
第五方面，本发明实施例还提供了一种计算机程序产品，所述计算机程序产品包括存储在非暂态计算机可读存储介质上的计算机程序，所述计算机程序包括程序指令，当所述程序指令被电子设备执行时，使所述电子设备执行如上所述的立体匹配方法。
本发明实施例的有益效果是：区别于现有技术的情况，本发明实施例提供的立体匹配方法、装置和电子设备通过首先基于局部立体匹配算法计算出双目视图中各个像素点的初始视差值，以获得初始左视差图和初始右视差图，然后在所述初始左视差图或者所述初始右视差图中，选择出满足预设条件(包括：通过左右一致性检测)的像素点作为种子点，并将所述像素点的初始视差值作为所述种子点的视差值；而对于每一个非种子点，则基于颜色差以及像素空间 距离，沿水平方向构造所述非种子点对应的可信度传播区域，并基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值，最终基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图，能够快速形成精度较高的稠密视差图，在立体匹配的速度与精度这两个方面均获取了较好的结果。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对本发明实施例中所需要使用的附图作简单地介绍。显而易见地，下面所描述的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本发明实施例提供的一种立体匹配方法的流程示意图；
图2是本发明实施例提供的一种选取种子点的示例示意图；
图3是本发明实施例提供的一种构造非种子点的可信度传播区域的示例示意图；
图4是本发明实施例提供的另一种立体匹配方法的流程示意图；
图5是本发明实施例提供的一种立体匹配装置的结构示意图；
图6是本发明实施例提供的一种电子设备的硬件结构示意图。
具体实施方式
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本发明，并不用于限定本发明。
需要说明的是，如果不冲突，本发明实施例中的各个特征可以相互结合，均在本发明的保护范围之内。另外，虽然在装置示意图中进行了功能模块划分，在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于装置中的模块划分，或流程图中的顺序执行所示出或描述的步骤。再者，本发明所采用的“第一”“第二”等字样并不对数据和执行次序进行限定，仅是对功能和作用基本相同的相同项或相似项进行区分。
本发明实施例提供了一种立体匹配方法、装置和电子设备，能够应用于任 意需要进行立体匹配以获取视差图的应用场景，比如，三维重建、双目视觉测距、无人驾驶、三维显示、机器人智能控制等等。
具体地，本发明实施例提供的立体匹配方法可以由任意电子设备执行，是一种基于种子点传播生长成稠密视差图的方法，具体为：首先通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图；然后，在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，并将所述像素点的初始视差值作为所述种子点的视差值，其中，所述预设条件包括：通过左右一致性检测；对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域；基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值；最后，基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图。本发明实施例提供的立体匹配方法一方面由于只需进行局部立体匹配和视差传播，能够快速形成稠密视差图，另一方面，由于选取了在局部立体匹配中可信度较高的像素点作为种子点，并在基于颜色差和像素空间距离形成的可信度传播区域内进行视差传播，能够提升所形成的稠密视差图中各个像素点的视差值的精度，从而，在立体匹配的速度与精度这两个方面均获取了较好的结果。
本发明实施例提供的立体匹配装置可以是由软件程序构成的能够实现本发明实施例提供的立体匹配方法的虚拟装置，其与本发明实施例提供的立体匹配方法基于相同的发明构思，具有相同的技术特征以及有益效果。
本发明实施例提供的电子设备可以是任意具有一定逻辑运算能力，能够提供一个或者多个可以满足用户意图的功能的硬件设备，比如，其可以包括但不限于：无人机、机器人、汽车MCU、智能手机、平板电脑、服务器等等。该电子设备能够执行本发明实施例提供的立体匹配方法或者运行本发明实施例提供的立体匹配装置，其同样与本发明实施例提供的立体匹配方法基于相同的发明构思，具有相同的技术特征以及有益效果。
下面结合各实施例和附图，对本发明提供的立体匹配方法、装置和电子设备作进一步阐述。
实施例一
图1是本发明实施例提供的一种立体匹配方法的流程示意图，该方法可以由无人机、机器人、汽车MCU等电子设备执行。
具体地，请参阅图1，该立体匹配方法可以包括但不限于如下步骤：
步骤110：通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图。
在本实施例中，所述“左视图”和所述“右视图”是指同一拍摄场景对应的双目视图，该双目视图可以是预先存储在该电子设备上的，也可以是从某个数据源处接收到的，或者，还可以是通过双目或者单目摄像头实时拍摄得到的。
所述“初始左视差图”包括所述左视图中各个像素点的初始视差值，其中，在所述初始左视差图中，像素点的初始视差值具体是指该像素点所在位置与该像素点在所述右视图中的对应点/匹配点所在位置之间的距离d1。所述“初始右视差图”包括所述右视图中各个像素点的初始视差值，其中，在所述初始右视差图中，像素点的初始视差值具体是指该像素点所在位置与该像素点在所述左视图中的对应点/匹配点所在位置之间的距离d2。
在本实施例中，可以采用任意局部立体匹配算法，比如，像素差的绝对值(SAD)、自适应窗口算法、自适应权重算法等，分别计算给定的左视图和右视图中各个像素点的初始视差值，以获取到所述左视图的初始左视差图和所述右视图的初始右视差图。
其中，在一些实施例中，在进行局部立体匹配的过程中，对于每一左视图中的像素点p，可以由该像素点p与其在右视图中的对应点q的颜色差以及结构编码的差这两项的和构成该像素点p的代价方程，进而通过优化该代价方程来确定该像素点p的初始视差值；同理，对于每一个右视图中的像素q，也可以由该像素点q与其在左视图中的对应点p的颜色差以及结构编码的差这两项的和构成该像素点q的代价方程，进而通过优化该代价方程来确定该像素点q的初始视差值。
在本发明实施例中，通过简单的局部立体匹配算法计算出左视图和右视图中各个像素点的初始视差值，能够减少立体匹配过程的计算量，提高立体匹配的速度。
步骤120：在所述初始左视差图或所述初始右视差图中，选择出满足预设条 件的像素点作为种子点，并将所述像素点的初始视差值作为所述种子点的视差值。
在本实施例中，所述“预设条件”包括：通过左右一致性检测，亦即，所述初始左视差图中的像素点p的初始视差值与所述像素点p在所述初始右视差图中的对应点q的初始视差值相等；或者，所述初始右视差图中的像素点q的初始视差值与所述像素点q在所述初始左视差图中的对应点p的初始视差值相等。所述“种子点”即视差传播过程的起始点。
在实际应用中，可以以所述初始左视差图为基准进行视差传播以形成最终的稠密视差图；也可以以所述初始右视差图为基准进行视差传播以形成最终的稠密视差图。
因此，在本实施例中，可以在所述初始左视差图中选择出满足预设条件的像素点作为种子点，而不满足所述预设条件的像素点则为非种子点，最后结合所有种子点的视差值和所有非种子点的视差值形成稠密的视差图。或者，也可以在所述初始右视差图中选择出满足预设条件的像素点作为种子点，而不满足所述预设条件的像素点则为非种子点，最后结合所有种子点的视差值和所有非种子点的视差值形成稠密的视差图。
下文以在所述初始左视差图中选择出满足预设条件的像素点作为种子点为例进行详细说明。
具体地，在所述初始左视差图中，选择出满足预设条件的像素点作为种子点的具体实施方式可以是：对所述在所述初始左视差图中的各个像素点进行逐一判断，如果某一像素点满足所述预设条件，则将该像素点标注为种子点。
或者，在另一些实施例中，在所述初始左视差图中，选择出满足预设条件的像素点作为种子点的具体实施方式也可以是：
对所述初始左视差图进行逐行扫描，以选择出满足预设条件的像素点作为种子点。具体地，在进行逐行扫描时，对于每一扫描行，可以从所述扫描行的最左端像素点开始，基于颜色差和像素空间距离进行向右传输的线分割，获得多个支持区域，并且，在每一个所述支持区域内，以最左端的像素点为起始点，寻找满足所述预设条件的像素点作为种子点，当在所述支持区域内寻找到一个种子点时，跳至下一支持区域寻找下一种子点。
其中，所述“颜色差”用于描述两个像素点之间的颜色差异，其具体可以为： 两个像素点在各个颜色通道中差异最大的颜色通道上的差值的绝对值，比如，在RGB空间中，如果像素点p0和像素点p1在R通道上的差值的绝对值为I1，在G通道上的差值的绝对值为I2，在B通道上的差值的绝对值为I3，且I1＞I2＞I3，那么，像素点p0和p1之间的颜色差即I1。其中，在该实施例中，以差异最大的颜色通道上的差值的绝对值来衡量两个像素点之间的颜色差异，能够避免因光照等条件变化引起各个颜色通道同时发生较大改变而产生的误判，更加准确地反映真实的场景边缘。
所述“像素空间距离”即两个像素点之间的像素距离，比如，像素点p0和像素点p1之间的像素空间距离为x个像素点。
所述“从所述扫描行的最左端像素点开始，基于颜色差和像素空间距离进行向右传输的线分割，获得多个支持区域”具体可以为：如图2所示，以扫描行的最左端像素点p0为起始点，向右逐一判断各个像素点与该起始点p0的颜色差和像素空间距离是否均在对应的预定范围以内，若是，则将该像素点与该起始点p0归为同一支持区域；若否，则以该像素点作为下一支持区域的起始点p1，继续进行向右传输的线性分割；以此类推，即可获得该扫描行上的多个支持区域。
进一步地，由于具有相似颜色的相邻像素点具有相似的视差，因此，可以认为，在同一支持区域内的各个像素点之间的视差变化是连续且缓慢的。从而，当进行种子点的选取时，只要在某一支持区域(比如，支持区域1)内寻找到一个种子点S0，即可跳至下一支持区域(比如，支持区域2)的起始点p1，以在下一支持区域(比如，支持区域2)内寻找下一种子点S1，依此类推。如此，便无需对全部的像素点进行逐一判断(比如，进行左右一致性检测)，能够在无损立体匹配精度的情况下，减少立体匹配的计算量，提高立体匹配的速度。
再者，由于种子点是至少通过左右一致性检测的像素点，而通过左右一致性检测则说明该像素点的初始视差值的可信度较高，可以认为是正确的视差值，从而，可以直接将该像素点的初始视差值作为所述种子点/像素点的视差值。
当然，为了进一步提升种子点的视差值的可信度，以提高后续计算的精度，也可以在所述“预设条件”中增加一些额外的限定。比如，在一些实施例中，所述“预设条件”还可以包括：在进行局部立体匹配时，所述像素点对应的次小代价与最小代价之差大于或等于第一预设阈值。其中，所述第一阈值可以为一个固定 值，也可以是大于所述次小代价的50％的值，只要所述像素点对应的最小代价显著小于次小代价即可。
步骤130：对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域。
在本实施例中，所述“非种子点”即所述初始左视差图(或所述初始右视差图)中除种子点之外的其他像素点。所述“可信度传播区域”即非种子点的线性分割区域，用于进行视差传播，以确定该非种子点的视差值。每一个可信度传播区域中的视差值是缓慢变化的。
在本实施例中，可以沿着扫描线方向分别针对每一个非种子点进行线性分割，以获取每一个非种子点对应的可信度传播区域。
在进行线性分割时，如图3所示，可以以非种子点P为中心，分别沿水平向左和水平向右的方向检测所述非种子点P的左侧分割点Pl和右侧分割点Pr，并以所述左侧分割点Pl和所述右侧分割点Pr之间的区域A(不包括所述左侧分割点Pl和所述右侧分割点Pr)作为所述非种子点对应的可信度传播区域；其中，所述左侧分割点Pl和所述右侧分割点Pr满足下述条件之一或者全部：
(1)与所述非种子点P的颜色差大于第二预设阈值；
(2)与所述非种子点P的像素空间距离大于第三预设阈值。
其中，所述第二预设阈值和所述第三预设阈值可以根据实际应用场景而设置，只要左侧分割点Pl和右侧分割点Pr之间的区域A满足视差值缓慢变化的特性即可。
在本实施例中，通过基于颜色差和像素空间距离，确定每一个非种子点的可信度传播区域，并在该可信度传播区域内进行视差传播，能够避免在视差传播过程中，因跨越前景和后景而引起的模糊，提高立体匹配的精度。
步骤140：基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值。
在本实施例中，对于每一个非种子点，均基于该非种子点在其对应的可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值来确定该非种子点的视差值。
在具体实施时，可以沿着扫描线的方向进行种子点视差的传播。对于每一个非种子点P，可以首先在其对应的可信度传播区域内找出最近邻的左侧种子点 Sl和右侧种子点Sr。
如果所述非种子点P在所述可信度传播区域内只存在最近邻的左侧种子点Sl，则可以根据同一可信度传播区域内的各个像素点具有相似的视差值这一特性，以所述左侧种子点Sl的视差值作为所述非种子点P的视差值；同理，如果所述非种子点P在所述可信度传播区域内只存在最近邻的右侧种子点Sr，则以所述右侧种子点Sr的视差值作为所述非种子点P的视差值。
如果所述非种子点P在所述可信度传播区域内同时存在最近邻的左侧种子点Sl和右侧种子点Sr，在一些实施例中，可以直接以所述左侧种子点Sl的视差值和所述右侧种子点Sr的视差值之间的线性插值作为所述非种子点P的视差值。
或者，在另一些实施例中，如果所述非种子点P在所述可信度传播区域内同时存在最近邻的左侧种子点Sl和右侧种子点Sr，也可以根据所述非种子点P所在区域的特性，基于所述左侧种子点Sl的视差值和所述右侧种子点Sr的视差值，确定所述非种子点P的视差值。
比如，如果所述非种子点P处于闭塞区域(其中，若所述非种子点P的初始视差值与所述非种子点P在初始右视差图中的对应点Q的初始视差值之差的绝对值大于或等于第四预设阈值(比如，1或2个像素点)，可以认为该非种子点P处于闭塞区域)，则说明该非种子点P处于前景和后景的切换区域，而这种切换区域的特点为：其中的某些部分对一张视图(比如，左视图)可见而对另一张视图(比如，右视图)不可见，故而不存在准确的视差计算。此时，可以认为该非种子点P位于远处，以所述左侧种子点Sl的视差值和所述右侧种子点Sr的视差值中的较小值作为所述非种子点P的视差值。
又如，如果所述非种子点P处于深度不连续区域(其中，若该非种子点P最近邻的左侧种子点Sl的视差值和右侧种子点Sr的视差值之差的绝对值大于第五预设阈值，则可以认为该非种子点P处于深度不连续区域)，则说明该非种子点P处于某些变化迅速的边缘，此时，也可以认为该非种子点P位于远处，以所述左侧种子点Sl的视差值和所述右侧种子点Sr的视差值中的较小值作为所述非种子点P的视差值。
此外，如果所述非种子点P既不处于闭塞区域，也不处于深度不连续区域，则可以认为该非种子点P处于视差缓慢变化的平滑区域，此时，以所述左侧种子点的视差值和所述右侧种子点的视差值之间的线性插值作为所述非种子点的 视差值即可。
在该实施例中，通过根据该非种子点P所在区域的特性，基于所述左侧种子点Sl的视差值和所述右侧种子点Sr的视差值，确定该非种子点P的视差值，能够进一步提升各个非种子点的视差值的可信度，提高立体匹配精度。
步骤150：基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图。
在本实施例中，将初始左视差图(或者，初始右视差图)中所有的像素点分为两类，即，种子点和非种子点。从而，将所有种子点的视差值和所有非种子点的视差值按照对应的坐标位置整合到一起，即可形成稠密视差图。
通过上述技术方案可知，本发明实施例的有益效果在于：本发明实施例提供的立体匹配方法通过首先基于局部立体匹配算法计算出双目视图中各个像素点的初始视差值，以获得初始左视差图和初始右视差图，然后在所述初始左视差图或者所述初始右视差图中，选择出满足预设条件(包括：通过左右一致性检测)的像素点作为种子点，并将所述像素点的初始视差值作为所述种子点的视差值；而对于每一个非种子点，则基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域，并基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值，最终基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图，能够快速形成精度较高的稠密视差图，在立体匹配的速度与精度这两个方面均获取了较好的结果。
实施例二
进一步地，本发明实施例二在上述实施例一的基础上，还提供了另一种立体匹配方法。
具体地，请参阅图4，该方法可以包括但不限于如下步骤：
步骤210：通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图。
步骤220：在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，并将所述像素点的初始视差值作为所述种子点的视差值。
步骤230：对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域。
步骤240：基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值。
步骤250：基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图。
步骤260：基于竖直方向的投票法修正所述稠密视差图中的条纹效应。
由于上述步骤230和240在确定非种子点的视差时，是沿着水平方向构造可信度传播区域以进行视差传播的，有可能会产生水平方向的条纹，因此，在本实施例中，还基于竖直方向的投票法修正所述稠密视差图中的条纹效应。
具体地，可以针对所述稠密视差图中的每一个像素点p，首先构造一个以(0，dmax)为横轴范围的直方图，其中，dmax为给定的最大视差值。然后，以该像素点p为中心，沿着竖直方向选出一定邻域内的像素点，若邻域内某一像素点p1与该像素点p之间的颜色差小于阈值，则在所述直方图的横轴上标记该像素点p1对应的视差值，并在该视差值对应的纵轴上加1。最后，选取直方图中纵轴上的最大值对应的视差值(即，投票数量最多的视差值)作为该像素点p的视差值。从而，可以抵消水平方向上的不利影响，并且实时性较好。
进一步地，在一些实施例中，为了进一步提升立体匹配的精度，所述方法还可以包括：在每一像素点的四邻域内利用双边滤波来更新其视差值。其中，所述“四邻域”是指该像素的最近邻的上、下、左、右四个像素点。通过在每一像素点的四邻域内利用双边滤波来更新其视差值，能够过滤掉上述视差传播和/或垂直投票过程产生的噪点。
此外，需说明的是，上述步骤210至250分别与如图1所示的立体匹配方法中的步骤110至150具有相同的技术特征，因此，其具体实施方式可以参考上述实施例的步骤110至150中相应的描述，在本实施例中便不再赘述。
通过上述技术方案可知，本发明实施例的有益效果在于：通过在获取到稠密视差图之后，基于竖直方向的投票法修正所述稠密视差图中的条纹效应，能够进一步提升立体匹配的精度。
实施例三
图5是本发明实施例提供的一种立体匹配装置的结构示意图，请参阅图5，该装置50包括：匹配单元51、种子点单元52、视差传播单元53以及视差图生成单元54。
所述匹配单元51用于通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图；
所述种子点单元52包括：种子点选取模块521和种子点视差确定模块522；所述种子点选取模块521用于在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，其中，所述预设条件包括：通过左右一致性检测；所述种子点视差确定模块522用于将所述像素点的初始视差值作为所述种子点的视差值；
所述视差传播单元53包括：分割模块531和视差传播模块532；所述分割模块531用于对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域；所述视差传播模块532用于基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值；
所述视差图生成单元54用于基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图。
在本实施例中，可以首先利用所述匹配单元51通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图；然后由所述种子点单元52在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，并将所述像素点的初始视差值作为所述种子点的视差值；进而通过所述视差传播单元53对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域，并基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值；最后由所述视差图生成单元54基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图。
其中，在一些实施例中，所述种子点选取模块521具体用于：
对所述初始左视差图或所述初始右视差图进行逐行扫描，以选择出满足预 设条件的像素点作为种子点；其中，对于每一扫描行，从所述扫描行的最左端像素点开始，基于颜色差和像素空间距离进行向右传输的线分割，获得多个支持区域，并且，在每一个所述支持区域内，以最左端的像素点为起始点，寻找满足预设条件的像素点作为种子点，当在所述支持区域内寻找到一个种子点时，跳至下一支持区域寻找下一种子点。
其中，在一些实施例中，所述预设条件还包括：在进行局部立体匹配时，所述像素点对应的次小代价与最小代价之差大于或等于第一预设阈值。
其中，在一些实施例中，所述分割模块531具体用于：
以所述非种子点为中心，分别沿水平向左和水平向右的方向检测所述非种子点的左侧分割点和右侧分割点，并以所述左侧分割点和所述右侧分割点之间的区域作为所述非种子点对应的可信度传播区域；其中，所述左侧分割点和所述右侧分割点满足下述条件：
与所述非种子点的颜色差大于第二预设阈值；或者，
与所述非种子点的像素空间距离大于第三预设阈值。
其中，在一些实施例中，所述视差传播模块532具体用于：
如果所述非种子点在所述可信度传播区域内只存在最近邻的左侧种子点，则以所述左侧种子点的视差值作为所述非种子点的视差值；
如果所述非种子点在所述可信度传播区域内只存在最近邻的右侧种子点，则以所述右侧种子点的视差值作为所述非种子点的视差值；
如果所述非种子点在所述可信度传播区域内同时存在最近邻的左侧种子点和右侧种子点，则根据所述非种子点所在区域的特性，基于所述左侧种子点的视差值和所述右侧种子点的视差值，确定所述非种子点的视差值。
进一步地，在另一些实施例中，如果所述非种子点在所述可信度传播区域内同时存在最近邻的左侧种子点和右侧种子点，所述视差传播模块532具体用于：
如果所述非种子点处于闭塞区域或深度不连续区域，则以所述左侧种子点的视差值和所述右侧种子点的视差值中的较小值作为所述非种子点的视差值；否则，以所述左侧种子点的视差值和所述右侧种子点的视差值之间的线性插值作为所述非种子点的视差值。
此外，在一些实施例中，所述装置50还包括：修正单元55；所述修正单元 55用于基于竖直方向的投票法修正所述稠密视差图中的条纹效应。
其中，在又一些实施例中，所述修正单元55还用于：在每一像素点的四邻域内利用双边滤波来更新其视差值。
需要说明的是，由于所述立体匹配装置与上述方法实施例中的立体匹配方法基于相同的发明构思，因此，上述方法实施例的相应内容同样适用于本装置实施例，此处不再详述。
通过上述技术方案可知，本发明实施例的有益效果在于：本发明实施例提供的立体匹配装置通过首先利用所述匹配单元51通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图；然后由所述种子点单元52在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，并将所述像素点的初始视差值作为所述种子点的视差值；进而通过所述视差传播单元53对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域，并基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值；最后由所述视差图生成单元54基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图，能够快速形成精度较高的稠密视差图，在立体匹配的速度与精度这两个方面均获取了较好的结果。
应当理解的是，以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
实施例四
图6是本发明实施例提供的一种电子设备的硬件结构示意图，该电子设备600可以是任意类型的硬件设备，如：无人机、汽车MCU、智能手机、机器人、个人电脑、可穿戴智能设备等，能够执行上述方法实施例提供的立体匹配方法，或者，运行上述装置实施例提供的立体匹配装置。
具体地，请参阅图6，该电子设备600包括：
一个或多个处理器601以及存储器602，图6中以一个处理器601为例。
处理器601和存储器602可以通过总线或者其他方式连接，图6中以通过总线连接为例。
存储器602作为一种非暂态计算机可读存储介质，可用于存储非暂态软件程序、非暂态性计算机可执行程序以及模块，如本发明实施例中的立体匹配方法对应的程序指令/模块(例如，附图5所示的匹配单元51、种子点单元52、视差传播单元53、视差图生成单元54以及修正单元55)。处理器601通过运行存储在存储器602中的非暂态软件程序、指令以及模块，从而执行立体匹配装置50的各种功能应用以及数据处理，即实现上述任一方法实施例的立体匹配方法。
存储器602可以包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需要的应用程序；存储数据区可存储根据立体匹配装置50的使用所创建的数据等。此外，存储器602可以包括高速随机存取存储器，还可以包括非暂态存储器，例如至少一个磁盘存储器件、闪存器件、或其他非暂态固态存储器件。在一些实施例中，存储器602可选包括相对于处理器601远程设置的存储器，这些远程存储器可以通过网络连接至处理器601。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
所述一个或者多个模块存储在所述存储器602中，当被所述一个或者多个处理器601执行时，执行上述任意方法实施例中的立体匹配方法，例如，执行以上描述的图1中的方法步骤110至150，图4中的方法步骤210至260，实现图5中的单元51-55的功能。
实施例五
本发明实施例还提供了一种非暂态计算机可读存储介质，所述非暂态计算机可读存储介质存储有计算机可执行指令，该计算机可执行指令被一个或多个处理器执行，例如，被图6中的一个处理器601执行，可使得上述一个或多个处理器执行上述任意方法实施例中的立体匹配方法，例如，执行以上描述的图1中的方法步骤110至150，图4中的方法步骤210至260，实现图5中的单元51-55的功能。
通过以上的实施方式的描述，本领域普通技术人员可以清楚地了解到各实施方式可借助软件加通用硬件平台的方式来实现，当然也可以通过硬件。本领 域普通技术人员可以理解实现上述实施例方法中的全部或部分流程是可以通过计算机程序产品中的计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非暂态计算机可读取存储介质中，该计算机程序包括程序指令，当所述程序指令被电子设备执行时，可使所述电子设备执行上述各方法实施例的流程。其中，所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)或随机存储记忆体(Random Access Memory,RAM)等。
上述产品(包括：电子设备、非暂态计算机可读存储介质以及计算机程序产品)可执行本发明实施例所提供的立体匹配方法，具备执行立体匹配方法相应的功能模块和有益效果。未在本实施例中详尽描述的技术细节，可参见本发明实施例所提供的立体匹配方法。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；在本发明的思路下，以上实施例或者不同实施例中的技术特征之间也可以进行组合，步骤可以以任意顺序实现，并存在如上所述的本发明的不同方面的许多其它变化，为了简明，它们没有在细节中提供；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。

Claims (18)

  1. 一种立体匹配方法，其特征在于，包括：
    通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图；
    在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，并将所述像素点的初始视差值作为所述种子点的视差值，其中，所述预设条件包括：通过左右一致性检测；
    对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域；
    基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值；
    基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图。
  2. 根据权利要求1所述的方法，其特征在于，所述在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，包括：
    对所述初始左视差图或所述初始右视差图进行逐行扫描，以选择出满足预设条件的像素点作为种子点；
    其中，对于每一扫描行，从所述扫描行的最左端像素点开始，基于颜色差和像素空间距离进行向右传输的线分割，获得多个支持区域，并且，在每一个所述支持区域内，以最左端的像素点为起始点，寻找满足预设条件的像素点作为种子点，当在所述支持区域内寻找到一个种子点时，跳至下一支持区域寻找下一种子点。
  3. 根据权利要求1或2所述的方法，其特征在于，所述预设条件还包括：
    在进行局部立体匹配时，所述像素点对应的次小代价与最小代价之差大于或等于第一预设阈值。
  4. 根据权利要求1-3任一项所述的方法，其特征在于，所述基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域，包括：
    以所述非种子点为中心，分别沿水平向左和水平向右的方向检测所述非种 子点的左侧分割点和右侧分割点，并以所述左侧分割点和所述右侧分割点之间的区域作为所述非种子点对应的可信度传播区域；
    其中，所述左侧分割点和所述右侧分割点满足下述条件：
    与所述非种子点的颜色差大于第二预设阈值；或者，
    与所述非种子点的像素空间距离大于第三预设阈值。
  5. 根据权利要求1-4任一项所述的方法，其特征在于，所述基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值，包括：
    如果所述非种子点在所述可信度传播区域内只存在最近邻的左侧种子点，则以所述左侧种子点的视差值作为所述非种子点的视差值；
    如果所述非种子点在所述可信度传播区域内只存在最近邻的右侧种子点，则以所述右侧种子点的视差值作为所述非种子点的视差值；
    如果所述非种子点在所述可信度传播区域内同时存在最近邻的左侧种子点和右侧种子点，则根据所述非种子点所在区域的特性，基于所述左侧种子点的视差值和所述右侧种子点的视差值，确定所述非种子点的视差值。
  6. 根据权利要求5所述的方法，其特征在于，所述根据所述非种子点所在区域的特性，基于所述左侧种子点的视差值和所述右侧种子点的视差值，确定所述非种子点的视差值，包括：
    如果所述非种子点处于闭塞区域或深度不连续区域，则以所述左侧种子点的视差值和所述右侧种子点的视差值中的较小值作为所述非种子点的视差值；
    否则，以所述左侧种子点的视差值和所述右侧种子点的视差值之间的线性插值作为所述非种子点的视差值。
  7. 根据权利要求1-6任一项所述的方法，其特征在于，在所述基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图的步骤之后，所述方法还包括：
    基于竖直方向的投票法修正所述稠密视差图中的条纹效应。
  8. 根据权利要求7所述的方法，其特征在于，在所述基于竖直方向的投票法修正所述稠密视差图中的条纹效应的步骤之后，所述方法还包括：
    在每一像素点的四邻域内利用双边滤波来更新其视差值。
  9. 一种立体匹配装置，其特征在于，包括：匹配单元、种子点单元、视差传播单元以及视差图生成单元；
    所述匹配单元用于通过局部立体匹配算法，分别计算左视图和右视图中各像素点的初始视差值，以获得所述左视图的初始左视差图和所述右视图的初始右视差图；
    所述种子点单元包括种子点选取模块和种子点视差确定模块：
    所述种子点选取模块用于在所述初始左视差图或所述初始右视差图中，选择出满足预设条件的像素点作为种子点，其中，所述预设条件包括：通过左右一致性检测；
    所述种子点视差确定模块用于将所述像素点的初始视差值作为所述种子点的视差值；
    所述视差传播单元包括分割模块和视差传播模块：
    所述分割模块用于对每一个非种子点，基于颜色差以及像素空间距离，沿水平方向构造所述非种子点对应的可信度传播区域；
    所述视差传播模块用于基于所述非种子点在所述可信度传播区域内最近邻的左侧种子点或右侧种子点的视差值，确定所述非种子点的视差值；
    所述视差图生成单元用于基于所有所述种子点的视差值和所有所述非种子点的视差值形成稠密视差图。
  10. 根据权利要求9所述的装置，其特征在于，所述种子点选取模块具体用于：
    对所述初始左视差图或所述初始右视差图进行逐行扫描，以选择出满足预设条件的像素点作为种子点；
    其中，对于每一扫描行，从所述扫描行的最左端像素点开始，基于颜色差和像素空间距离进行向右传输的线分割，获得多个支持区域，并且，在每一个所述支持区域内，以最左端的像素点为起始点，寻找满足预设条件的像素点作为种子点，当在所述支持区域内寻找到一个种子点时，跳至下一支持区域寻找下一种子点。
  11. 根据权利要求9或10所述的装置，其特征在于，所述预设条件还包括：
    在进行局部立体匹配时，所述像素点对应的次小代价与最小代价之差大于 或等于第一预设阈值。
  12. 根据权利要求9-11任一项所述的装置，其特征在于，所述分割模块具体用于：
    以所述非种子点为中心，分别沿水平向左和水平向右的方向检测所述非种子点的左侧分割点和右侧分割点，并以所述左侧分割点和所述右侧分割点之间的区域作为所述非种子点对应的可信度传播区域；
    其中，所述左侧分割点和所述右侧分割点满足下述条件：
    与所述非种子点的颜色差大于第二预设阈值；或者，
    与所述非种子点的像素空间距离大于第三预设阈值。
  13. 根据权利要求9-12任一项所述的装置，其特征在于，所述视差传播模块具体用于：
    如果所述非种子点在所述可信度传播区域内只存在最近邻的左侧种子点，则以所述左侧种子点的视差值作为所述非种子点的视差值；
    如果所述非种子点在所述可信度传播区域内只存在最近邻的右侧种子点，则以所述右侧种子点的视差值作为所述非种子点的视差值；
    如果所述非种子点在所述可信度传播区域内同时存在最近邻的左侧种子点和右侧种子点，则根据所述非种子点所在区域的特性，基于所述左侧种子点的视差值和所述右侧种子点的视差值，确定所述非种子点的视差值。
  14. 根据权利要求13所述的装置，其特征在于，如果所述非种子点在所述可信度传播区域内同时存在最近邻的左侧种子点和右侧种子点，所述视差传播模块具体用于：
    如果所述非种子点处于闭塞区域或深度不连续区域，则以所述左侧种子点的视差值和所述右侧种子点的视差值中的较小值作为所述非种子点的视差值；
    否则，以所述左侧种子点的视差值和所述右侧种子点的视差值之间的线性插值作为所述非种子点的视差值。
  15. 根据权利要求9-14任一项所述的装置，其特征在于，所述装置还包括：修正单元；
    所述修正单元用于基于竖直方向的投票法修正所述稠密视差图中的条纹效应。
  16. 根据权利要求15所述的装置，其特征在于，所述修正单元还用于：
    在每一像素点的四邻域内利用双边滤波来更新其视差值。
  17. 一种电子设备，其特征在于，包括：
    至少一个处理器；以及，
    与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如权利要求1-8任一项所述的方法。
  18. 一种非暂态计算机可读存储介质，所述非暂态计算机可读存储介质存储有计算机可执行指令，所述计算机可执行指令用于使电子设备执行如权利要求1-8任一项所述的方法。
PCT/CN2019/126040 2018-12-17 2019-12-17 一种立体匹配方法、装置和电子设备 WO2020125637A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/350,714 US11995858B2 (en) 2018-12-17 2021-06-17 Method, apparatus and electronic device for stereo matching

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811544155.6A CN109859253A (zh) 2018-12-17 2018-12-17 一种立体匹配方法、装置和电子设备
CN201811544155.6 2018-12-17

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/350,714 Continuation US11995858B2 (en) 2018-12-17 2021-06-17 Method, apparatus and electronic device for stereo matching

Publications (1)

Publication Number Publication Date
WO2020125637A1 true WO2020125637A1 (zh) 2020-06-25

Family

ID=66891388

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/126040 WO2020125637A1 (zh) 2018-12-17 2019-12-17 一种立体匹配方法、装置和电子设备

Country Status (3)

Country Link
US (1) US11995858B2 (zh)
CN (1) CN109859253A (zh)
WO (1) WO2020125637A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114331919A (zh) * 2022-03-09 2022-04-12 合肥的卢深视科技有限公司 深度恢复方法、电子设备及存储介质

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109859253A (zh) * 2018-12-17 2019-06-07 深圳市道通智能航空技术有限公司 一种立体匹配方法、装置和电子设备
CN112116660B (zh) * 2019-06-19 2024-03-29 京东方科技集团股份有限公司 视差图校正方法、装置、终端及计算机可读介质
CN110322518B (zh) * 2019-07-05 2021-12-17 深圳市道通智能航空技术股份有限公司 立体匹配算法的评价方法、评价系统及测试设备
CN110825080B (zh) * 2019-10-31 2020-11-13 华南农业大学 基于模糊控制算法的果园路径视觉导航方法、系统和介质
CN113965697B (zh) * 2021-10-21 2022-12-16 合肥的卢深视科技有限公司 基于连续帧信息的视差成像方法、电子设备及存储介质
CN114155289A (zh) * 2021-12-08 2022-03-08 电子科技大学 基于双目视觉的电点火系统电火花轮廓尺寸测量方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103489183A (zh) * 2012-10-17 2014-01-01 北京大学深圳研究生院 一种基于边缘分割和种子点的局部立体匹配方法
CN105513064A (zh) * 2015-12-03 2016-04-20 浙江万里学院 一种基于图像分割和自适应权重的立体匹配方法
US20180211400A1 (en) * 2017-01-26 2018-07-26 Samsung Electronics Co., Ltd. Stereo matching method and apparatus
CN109859253A (zh) * 2018-12-17 2019-06-07 深圳市道通智能航空技术有限公司 一种立体匹配方法、装置和电子设备

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7388973B2 (en) * 2004-06-01 2008-06-17 General Electric Company Systems and methods for segmenting an organ in a plurality of images
CN101625768B (zh) * 2009-07-23 2011-11-09 东南大学 一种基于立体视觉的三维人脸重建方法
CN103226821B (zh) * 2013-04-27 2015-07-01 山西大学 基于视差图像素分类校正优化的立体匹配方法
CN104167000B (zh) * 2014-08-25 2017-02-22 同济大学 一种仿射不变的宽基线图像密集匹配方法
TWI658720B (zh) * 2016-12-30 2019-05-01 創研光電股份有限公司 運用變焦影像資訊以改善原視差影像匹配錯誤之方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103489183A (zh) * 2012-10-17 2014-01-01 北京大学深圳研究生院 一种基于边缘分割和种子点的局部立体匹配方法
CN105513064A (zh) * 2015-12-03 2016-04-20 浙江万里学院 一种基于图像分割和自适应权重的立体匹配方法
US20180211400A1 (en) * 2017-01-26 2018-07-26 Samsung Electronics Co., Ltd. Stereo matching method and apparatus
CN109859253A (zh) * 2018-12-17 2019-06-07 深圳市道通智能航空技术有限公司 一种立体匹配方法、装置和电子设备

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114331919A (zh) * 2022-03-09 2022-04-12 合肥的卢深视科技有限公司 深度恢复方法、电子设备及存储介质
CN114331919B (zh) * 2022-03-09 2022-05-31 合肥的卢深视科技有限公司 深度恢复方法、电子设备及存储介质

Also Published As

Publication number Publication date
US20210319580A1 (en) 2021-10-14
CN109859253A (zh) 2019-06-07
US11995858B2 (en) 2024-05-28

Similar Documents

Publication Publication Date Title
WO2020125637A1 (zh) 一种立体匹配方法、装置和电子设备
US10244164B1 (en) Systems and methods for image stitching
US10621783B2 (en) Image processing method and apparatus using depth value estimation
KR102464523B1 (ko) 이미지 속성 맵을 프로세싱하기 위한 방법 및 장치
US10853960B2 (en) Stereo matching method and apparatus
US9349073B2 (en) Apparatus and method for image matching between multiview cameras
JP2016071846A (ja) 単眼カメラに基づく障害物検知方法及び装置
CN110223222B (zh) 图像拼接方法、图像拼接装置和计算机可读存储介质
JP2019526847A (ja) 動的オクルージョン処理のためのシステム及び方法
US20130002827A1 (en) Apparatus and method for capturing light field geometry using multi-view camera
CN103582802A (zh) 立体照相机装置和计算机可读记录介质
CN110443186B (zh) 立体匹配方法、图像处理芯片及移动载具
WO2020187339A1 (zh) 一种裸眼3d的虚拟视点图像生成方法及便携式终端
CN111295667A (zh) 图像立体匹配的方法和辅助驾驶装置
JP2017142613A (ja) 情報処理装置、情報処理システム、情報処理方法及び情報処理プログラム
CN105005988A (zh) 一种基于交互式图像分割的立体匹配方法
Shivakumar et al. Real time dense depth estimation by fusing stereo with sparse depth measurements
KR20160098012A (ko) 영상 매칭 방법 및 장치
US20130308826A1 (en) Image processing apparatus, non-transitory computer readable recording medium, and image processing method
JP5178538B2 (ja) 深度マップを画像から決定する方法、深度マップを決定する装置
JP2013058848A (ja) 立体画像処理装置、立体画像処理方法、及びプログラム
JP2009530701A5 (zh)
CN105335934A (zh) 视差图计算方法和装置
Chang et al. Robust stereo matching with trinary cross color census and triple image-based refinements
JP6991700B2 (ja) 情報処理装置、情報処理方法、プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19901099

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19901099

Country of ref document: EP

Kind code of ref document: A1