CN108564604B - Binocular vision stereo matching method and device based on plane constraint and triangulation - Google Patents

Abstract

The embodiment of the invention provides a binocular vision stereo matching method and device based on plane constraint and triangulation. The method comprises the following steps: acquiring left and right images acquired by two cameras, and taking any one image as a reference image; determining a support point of a reference image and calculating a parallax; segmenting the reference image according to a preset mode; dividing each divided area into a first type and/or a second type divided area; secondarily dividing each first-class divided area; determining the parallax of the non-support point in each sub-segmentation region according to the support point and the parallax of each sub-segmentation region; triangulating each second type of segmented region; determining a parallax searching range of a non-support point in each triangular area according to the vertex and the parallax of the triangular area; and matching the point to be matched and the non-support point corresponding to the range pixel by pixel to determine the parallax of the non-support point. The invention can more accurately match the shielded area and the large-area non-texture area. And the parallax searching range of the small area segmentation area is reduced, and the matching efficiency is improved.

Description

Stereo matching method and device for binocular vision based on plane constraint and triangulation

technical field

The invention relates to the technical field of computer vision, in particular to a method and device for stereo matching of binocular vision based on plane constraints and triangulation.

Background technique

Binocular stereo vision mainly includes four steps: binocular camera calibration, acquisition of two-dimensional image pairs, image correction, and binocular stereo matching. Among them, the stereo matching of binocular vision includes: first, use a binocular camera or two cameras placed in parallel to shoot the same physical scene at the same time to obtain left and right images; then find the projection point of the same object in the scene in the left and right images, which is called the corresponding point ; Finally, a disparity map is generated according to the disparity of the corresponding point, that is, the offset of the corresponding point in the u-axis direction of the pixel coordinate system. According to the generated disparity map and the principle of similar triangles, the actual distance between the target and the camera in the captured physical scene can be further calculated, and this process is also called stereoscopic depth calculation. Stereo vision depth calculation is widely used in practical applications such as 3D scene reconstruction, autonomous navigation of mobile robots, and more and more applications in medical imaging, industrial inspection and other fields.

At present, the commonly used stereo matching algorithms for binocular vision include: local algorithm based on support window, global algorithm based on image cutting or dynamic programming, and non-local matching algorithm based on minimum spanning tree.

Since the left and right images are obtained from different shooting angles, it is inevitable that there will be an occluded area or a large area without texture. Among them, the occlusion area refers to the area that is only visible in one image and has no corresponding pixel information in the other image, which is usually caused by different shooting angles. Large untextured areas usually refer to those areas without obvious feature points, such as white walls in indoor areas, etc. The above stereo matching algorithms are all matched pixel by pixel. Therefore, for the lack of pixel information in occluded areas and the ambiguity in matching of large areas without texture, the above stereo matching algorithms can not solve the problem of occlusion areas or large areas. The matching of the area without texture area will result in a high matching error rate and a low matching efficiency.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a method and device for stereo matching of binocular vision based on plane constraints and triangulation, so as to reduce the matching error rate of stereo matching of binocular vision and improve the matching efficiency. The specific technical solutions are as follows:

In order to achieve the above object, in the first aspect, the present invention provides a stereo matching method for binocular vision based on plane constraints and triangulation, the method comprising:

Acquire the left image and the right image of the same shooting scene collected by two cameras placed in parallel at the same time, and use any one of them as a reference image and the other as an image to be matched;

Determine the support points in the reference image, and calculate the parallax of each support point; wherein, the support points are pixels in the reference image that have unique and correct matching points in the image to be matched ;

According to a preset image segmentation method, the reference image is segmented to obtain a plurality of segmented regions; each of the segmented regions is divided into a first-type segmented region with a segmented area greater than a first threshold, and/or a segmented area not greater than the second type of segmentation region of the first threshold;

For each first-type segmentation area, perform: perform a second segmentation on the first-type segmentation area, and divide the first-type segmentation area into a plurality of first-type sub-segmentation areas; according to each first-type sub-segmentation The disparity of the support point and the support point in the area, fit the first plane equation of the first type of sub-segmented area; according to the first plane equation, determine the disparity of the non-supported point in the first type of sub-segmented area; wherein , the plurality of first-type sub-segmentation regions respectively correspond to different physical planes in the shooting scene;

For each segmented region of the second type, perform: triangulate the segmented region of the second type, and divide the segmented region of the second type into a plurality of triangular regions; for each non-support point in each triangular region, Determine the first parallax search range corresponding to the unsupported point according to the parallax of each vertex and each vertex of the triangle area; perform pixel-by-pixel matching between the to-be-matched point corresponding to each parallax within the first parallax search range and the non-supported point , determine the parallax of the non-supporting point; wherein, each vertex is a supporting point.

Optionally, the determining of the support points in the reference image includes:

Taking the pixel in the upper left corner of the reference image as the starting point, along the u-axis and the v-axis of the pixel coordinate system corresponding to the reference image, construct a grid with a preset step size in the reference image; In the grid, except for the intersection points located at the edge of the reference image as candidate support points; wherein, the u axis is to the right, and the v axis is downward;

For each candidate support point, calculate the distance between the vector of the to-be-matched point corresponding to each parallax within the second parallax search range corresponding to the candidate support point and the vector of the candidate support point; if the calculated distance is small Among the distances of the second threshold, if the number of the smallest distance is 1, the candidate support point is regarded as a support point; wherein, the vector of the points to be matched corresponding to each parallax within the second parallax search range, and the The vectors of the candidate support points are all calculated according to the preset size of the sobel operator.

Optionally, according to the preset image segmentation method, the reference image is segmented to obtain a plurality of segmented regions, including:

The reference image is segmented according to the mean shift image segmentation method to obtain at least one segmented region.

Optionally, performing the second segmentation on the first-type segmentation area, and dividing the first-type segmentation area into a plurality of first-type sub-segmentation areas, including:

According to the preset plane fitting rule, fit the plane equation of the physical plane corresponding to the plurality of support point subsets in the first type of segmentation area; wherein, the multiple support point subsets correspond to different the physical plane;

In each of the physical planes obtained by fitting, if multiple physical planes meet the preset merging conditions, then according to the support points in the multiple physical planes and the disparity of the support points, the fitting of the merged physical planes is obtained. Plane equation; according to the plane equation of the merged physical plane and the plane equation of the unmerged physical plane, calculate the line of intersection between the merged physical planes and the unmerged physical planes, and determine the plurality of first-type subtypes dividing a region; wherein, the preset merging condition is: among the plurality of physical planes, the distance between any physical plane and at least one other physical plane is less than a third threshold;

In each of the physical planes obtained by fitting, if any two of the physical planes do not satisfy the preset merging condition, then according to the plane equation of each of the physical planes, the intersection line between the physical planes is calculated, The plurality of sub-segmented regions of the first type are determined.

Optionally, according to a preset plane fitting rule, fitting the plane equation of the physical plane corresponding to the multiple support point subsets in the first type of segmented region, including:

S1. Based on the random sampling consistency RANSAC algorithm, select a target support point subset O from the current support point set Y in the first type of segmentation area; wherein, the O is the support point in the Y with the largest number of support points point subset;

S2, fit the plane equation of the physical plane corresponding to the O according to the support point of the O and the parallax of the support point;

S3, judging whether the number of support points of the current unfitted support point subset Y-O is less than the number of support points in the first type of segmentation area of a preset ratio;

If the number of support points of the Y-O is not less than the number of support points in the first type of segmented area at a preset ratio, then the Y-O is used as the current set of support points in the first type of segmented area, and the execution returns to execute step S1;

If the number of support points of the Y-O is less than the number of support points in the first-type segmented region at a preset ratio, the plane equation of the currently fitted physical plane is used as the final fitting result.

Optionally, fitting the first plane equation of the first type of sub-segmented region according to the support points and the disparity of the support points in each of the first-type sub-segmented regions, including:

The first plane equation of the first type of sub-segmented region is set as:

d(u, v)=au+bv+c

Among them, d(u, v) represents the disparity of the pixel point (u, v) in the first type of sub-segmented region, a, b, c are the parameters to be solved for the first plane equation, and the a is solved by the following formula , b, c:

Among them, _ui represents the coordinate of the i-th support point in the first type of sub-segmentation area on the u-axis of the pixel coordinate system corresponding to the reference image, and v _i represents the i-th support point in the first type of sub-segmentation area in the The coordinates of the v-axis of the pixel coordinate system corresponding to the reference image, the u-axis is to the right, the v-axis is down, i=1, . _i represents the disparity of the i-th support point in the first type of sub-segmented region.

Optionally, for each non-supporting point in each triangular area, determine the first parallax search range corresponding to the non-supporting point according to the parallax of each vertex and each vertex of the triangular area, including:

For each triangular area, according to each vertex in the triangular area and the parallax of each vertex, determine the second plane equation of the plane where the triangular area is located;

determining the disparity estimate of the unsupported point according to the second plane equation;

The first parallax search range includes: the parallax of each vertex, the parallax obtained by adding and subtracting the parallax of each vertex by a preset value, the parallax estimated value, and the non-support point within the preset neighborhood The parallax of the target support point; wherein, the absolute value of the difference between the parallax of the target support point and the disparity estimated value of the target support point determined according to the second plane equation does not exceed a fourth threshold.

Optionally, performing pixel-by-pixel matching between the to-be-matched point corresponding to each parallax within the first parallax search range and the non-supported point, and determining the parallax of the non-supported point, comprising:

的视差：The non-support point is determined by the following formula

Parallax of:

为该非支持点

的视差，d_n表示所述该非支持点的视差估计值，

表示所述第一视差搜索范围内的各视差，N表示所述第一视差搜索范围内的视差数，S表示所述参考图像的支持点集；

表示

的后验概率，通过以下公式计算：in,

for the non-support point

The disparity of , d _n represents the disparity estimation value of the non-supported point,

represents each parallax within the first parallax search range, N represents the number of parallaxes within the first parallax search range, and S represents the support point set of the reference image;

express

The posterior probability of , is calculated by the following formula:

表示d_n对应的待匹配点为所述非支持点

的正确匹配点的概率，

表示所述第一视差搜索范围内的各视差对应的匹配点为所述非支持点

的正确匹配点的概率。in,

Indicates that the to-be-matched point corresponding to d _n is the non-supported point

The probability of the correct matching point of ,

Indicates that the matching point corresponding to each parallax within the first parallax search range is the unsupported point

The probability of the correct matching point.

In a second aspect, an embodiment of the present invention provides a stereo matching device for binocular vision based on plane constraints and triangulation, and the device includes:

an acquisition module, configured to acquire the left image and the right image of the same shooting scene respectively collected by two cameras placed in parallel at the same time, and use any one of them as a reference image and the other as an image to be matched;

A determination module, configured to determine the support points in the reference image, and calculate the disparity of each support point; wherein, the support points are unique and correct in the reference image and in the to-be-matched image Pixel points of matching points;

a segmentation module, configured to segment the reference image according to a preset image segmentation method to obtain a plurality of segmented regions; to divide each of the segmented regions into a first-type segmented region with a segmented area greater than a first threshold, and/ Or the second type of segmentation area whose segmentation area is not greater than the first threshold;

The first execution module is configured to, for each first-type segmented region, execute: perform a second segmentation on the first-type segmented region, and divide the first-type segmented region into a plurality of first-type sub-segmented regions; according to For the support points and the disparity of the support points in each first-type sub-segmentation area, fit the first plane equation of the first-type sub-segmentation area; according to the first plane equation, determine the first-type sub-segmentation area Disparity of non-supported points; wherein, the plurality of first-type sub-segmentation regions respectively correspond to different physical planes in the shooting scene;

The second execution module is configured to perform, for each segmented region of the second type: triangulate the segmented region of the second type, and divide the segmented region of the second type into a plurality of triangular regions; For each non-supporting point, determine the first parallax search range corresponding to the non-supporting point according to the parallax of each vertex and each vertex of the triangle area; compare the point to be matched corresponding to each parallax in the first parallax search range The unsupported point is matched pixel by pixel to determine the parallax of the unsupported point; wherein, each of the vertices is a support point.

In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

memory for storing computer programs;

The processor is configured to implement the method steps for stereo matching of binocular vision based on plane constraints and triangulation as described in the first aspect when executing the program stored in the memory.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium runs on a computer, the computer causes the computer to execute the plane-based method described in the first aspect above. Constrained and triangulated method steps for binocular stereo matching.

In a fifth aspect, an embodiment of the present invention provides a computer program product including instructions, which, when run on a computer, enables the computer to perform the stereo matching of binocular vision based on plane constraints and triangulation as described in the first aspect above method steps.

In the binocular vision stereo matching method and device based on plane constraint and triangulation provided by the embodiment of the present invention, after dividing the reference image, the segmented area with a larger area is segmented a second time, so as to determine the corresponding images corresponding to different physical planes. The sub-segmented area is further fitted according to the support points of the sub-segmented area and the disparity of the support points, and the plane equation of the sub-segmented area is determined, and the disparity of the non-supported points in the sub-segmented area is determined according to the plane equation. At the same time, triangulate the segmented area with a small area and divide it into multiple triangles; further determine the first parallax search range of the non-support points in the triangle according to the parallax of the vertices of the triangles and the vertexes, and assign the corresponding parallaxes within the range to the first parallax search range. The to-be-matched point and the unsupported point are matched pixel by pixel, so as to determine the parallax of the unsupported point.

It can be seen that, in this embodiment of the present invention, for each sub-segmentation area of a larger segmented area, based on the idea of plane constraints, the parallax of all non-supporting points in each sub-segmented area is determined by the plane equation of each sub-segmented area, which can realize the Occlusion areas with missing pixel information and large textureless areas with matching ambiguity are more accurately and robustly matched, thereby reducing the matching error rate. In addition, in this embodiment of the present invention, triangulation is performed for a divided area with a small area, and the parallax search range of the non-supported points in the triangle is determined according to the vertex of the triangle and the parallax of the vertex, which greatly reduces the parallax search range of the non-supported points and improves the matching efficiency.

Of course, it is not necessary for any product or method to implement the present invention to simultaneously achieve all of the advantages described above.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that are required in the description of the embodiments or the prior art.

1 is a flowchart of a method for stereo matching of binocular vision based on plane constraints and triangulation provided by an embodiment of the present invention;

FIG. 2 is a specific flow chart of implementing the second division of the first type of segmentation area and dividing the first type of segmentation area into a plurality of first type of sub-segmented areas in step S104 of the embodiment shown in FIG. 1 ;

Fig. 3 is a specific flow chart of step S201 in the embodiment shown in Fig. 2;

Fig. 4 is a simulation experiment result diagram provided by an embodiment of the present invention;

5 is a structural diagram of a stereo matching device for binocular vision based on plane constraints and triangulation provided by an embodiment of the present invention;

FIG. 6 is a structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the stereo matching of binocular vision, in order to reduce the matching error rate of occluded areas and large areas without texture, and improve the matching efficiency, as shown in FIG. 1, the present invention provides a binocular based on plane constraints and triangulation. A visual stereo matching method, the method includes:

S101: Acquire a left image and a right image of the same shooting scene respectively collected by two cameras placed in parallel at the same time, and use any one of them as a reference image and the other as an image to be matched.

In order to imitate the 3D stereo vision produced by human eyes when viewing a scene, in computer vision, the left and right images of the same shooting scene collected by two parallel cameras at the same time can be acquired to further find the shooting scene The projected points of the target in the left and right images, also called corresponding points, to generate the disparity map. Then, through the similar triangle principle, the disparity map can be converted into a depth map containing the depth information of the camera and each target in the shooting scene, that is, the actual distance between the camera and each target in the shooting scene, for further 3D reconstruction.

In this embodiment, the left and right images of a shooting scene can also be simultaneously collected by two camera units of a binocular camera.

Either the left image or the right image captured by the camera can be used as the reference image, and if one of them is used as the reference image, the other is the image to be matched. For example, taking the left image as the reference image, the purpose of the present invention is to determine the disparity of each pixel in the left image and generate a disparity map of the left image. In this embodiment, each step is described in detail by taking the left image as a reference image exemplarily.

S102: Determine the support points in the reference image, and calculate the disparity of each support point; wherein, the support points are those in the reference image that have unique and correct matching points in the to-be-matched image pixel.

The present invention is based on the observation that a certain point in the actual scene is usually not isolated, but coexists with some other points on the same physical plane. Correspondingly, a certain pixel in the reference image is usually not isolated, but It corresponds to the same physical plane in the shooting scene together with some other pixels. Then, based on the idea of plane constraints, if the physical plane corresponding to each pixel in the reference image can be accurately determined, the disparity value can be filled with the plane as the minimum calculation unit, that is, the disparity value can be filled according to the physical plane corresponding to the pixel. The plane equation to determine the disparity of the pixel. Therefore, the disparity determination of pixels in occluded areas and large textureless areas is robust.

Based on the above observation, from another perspective, the reference image must correspond to at least one physical plane in the shooting scene. In order to determine the plane equation of the physical plane corresponding to the reference image, in this embodiment, some support points in the reference image may be pre-determined, and these support points are pixels with unique and correct matching points in the image to be matched. The difference has good robustness and can be used to fit a plane equation in subsequent calculations.

It can be understood that there must be many pixels in the reference image that have unique and correct matching points in the image to be matched, but in this embodiment, it is not necessary to determine all such pixels, but can choose Determining a part of such pixel points as support points.

In an implementation manner, determining the support points in the reference image may include the following steps:

S11, taking the upper left pixel point of the reference image as a starting point, along the u-axis and the v-axis of the pixel coordinate system corresponding to the reference image, constructing a grid with a step size of a preset step size in the reference image; The intersections in the grid except at the edge of the reference image are taken as candidate support points; wherein, the u-axis is rightward, and the v-axis is downward.

It can be understood that by constructing a grid in the reference image, taking the intersections in the grid except at the edge of the reference image as candidate support points, and further determining the support points from the candidate support points, the support points can be uniformly determined. distributed in the reference image. When the plane equation is fitted by the parallax of support points and support points, the distribution of support points in each plane is relatively uniform, and the density of support points in different planes is relatively similar, so that the fitted plane equation can be more accurate. .

For example, the size of the reference image is 256×256, the upper left corner pixel is (0, 0), the preset step size is 5 pixels, and the grid is constructed along the u-axis and v-axis of the pixel coordinate system corresponding to the reference image, Then the pixels at the intersection in the grid include: the first row of the grid: (0, 0), (0, 5), (0, 10), ..., (0, 255), the second row of the grid: ( 5, 0), (5, 5), (5, 10), ..., (5, 255), ..., last row of grid: (255, 0), (255, 5), (255, 10), ..., (255, 255). In the above intersections, after removing the intersections at the edge of the reference image, the pixels at the remaining intersections are candidate support points: The second row of the grid: (5, 5), (5, 10), ..., (5, 255) , third row of grid: (10, 5), (10, 10), …, (10, 255), …, last row of grid: (255, 5), (255, 10), …, (255 , 255).

The preset step size can be determined according to actual needs or experience.

S12, for each candidate support point, calculate the distance between the vector of the to-be-matched point corresponding to each parallax within the second parallax search range corresponding to the candidate support point and the vector of the candidate support point; if the calculated Among the distances that are less than the second threshold, the number of the smallest distance is 1, and the candidate support point is used as a support point; wherein, the vector of the points to be matched corresponding to each parallax within the second parallax search range, And the vector of the candidate support point is calculated according to the sobel operator of the preset size.

Parallax is the difference between the abscissas of the two projection points of the target in the shooting scene on the left and right images. For example, the projection point x _l of the target in the left image is (5, 10), and the projection point x _r in the right image is (5, 8), then the disparity d(x _l )=2 of x _l .

The second parallax search range includes multiple parallax values. For a candidate support point, each parallax value in the corresponding second parallax search range is the same as a pixel to be matched in the image to be matched (referred to as corresponds to the point to be matched). For example, for the candidate support point (5, 255), a parallax corresponding to the second parallax search range is 2, and the point to be matched corresponding to the parallax 2 is (5, 253).

In this embodiment, the minimum value of the second parallax search range may be 0, and the maximum value may be set as required, for example, set to be half the width of the reference image. For example, if the width of the reference image is 256, the second parallax search range may be set to 0-128. Then for the candidate support point (5, 255), the corresponding points to be matched for each parallax in the corresponding second parallax search range are: pixel points (5, 137), ..., (5, 255) in the image to be matched ). Of course, it can be understood that the maximum value of the second parallax search range corresponding to different candidate support points may be different.

The preset size of the sobel operator can be set according to actual needs, for example, it is set to 3×3. The second threshold can also be set empirically, for example, set to 2. In step S12, after the vector of candidate support points and the vector of each point to be matched are obtained by calculating the sobel operator, the distance between the vector of candidate support point and the vector of each point to be matched can be calculated to determine the candidate support point and the vector of each point to be matched. The distance between the points to be matched, for example, the distance can be 0, 1, 2, etc. The smaller the distance, the more similar the candidate support point is to the point to be matched. Further, if the number of the smallest distances among the calculated distances is 1, it means that the candidate support point has a unique correct matching point, and the candidate support point can be used as a support point. Conversely, if the number of the smallest distances among the calculated distances is greater than 1, it means that there are multiple points to be matched that are similar to the candidate support points, that is, there is ambiguity in the matching of the candidate support points, then the candidate support point Will not be used as a support point.

For example, the candidate support point L1 has 5 corresponding points to be matched, which are R1-R5 respectively, and the second threshold is 2. If the distances between the candidate support point L1 and the points to be matched R1-R5 are: 0, 1, 3, 2, 1, then among the distances smaller than the second threshold (0, 1, 1 in sequence), the smallest If the number of distances from 0 is 1, then the candidate support point L1 has a unique correct matching point R1, and the candidate support point can be used as the support point. If the distances between the candidate support point L1 and the points to be matched R1-R5 are: 1, 1, 2, 2, 2, then among the distances smaller than the second threshold (1, 1 in sequence), the smallest distance is 1 If the number of L1 is 2, then the matching of the candidate support point L1 is ambiguous, and the candidate support point will not be used as a support point.

If a candidate support point is used as the support point, the disparity of the support point is the difference between the abscissa of the candidate support point and the abscissa of the matching point.

S103: Divide the reference image according to a preset image segmentation method to obtain a plurality of divided areas; divide each of the divided areas into a first type of divided area with a divided area larger than a first threshold, and/or divided areas A segmented region of the second type that is not greater than the first threshold.

The purpose of segmenting the reference image is to segment the segmented regions corresponding to different physical planes in the reference image as accurately as possible. In an implementation manner, the reference image may be segmented according to the mean shift image segmentation method to obtain a plurality of segmented regions. Specifically, in the segmentation process, according to the size of the reference image and/or other attributes of the image, the color radius and the spatial radius in the Meanshift segmentation can be adjusted according to experience, so that the segmentation result is as accurate as possible. Normally, when adjusting the color radius and space radius for the image size property, if the image is relatively large, the color radius is relatively small and the space radius is relatively large. For example, for an image of size 450×375, you can set the color radius to 30 and the space radius to 30; for an image of size 128×128, you can set the color radius to 40 and the space radius to Set at 20.

However, in actual segmentation, segmentation errors may also occur, that is, different pixels that do not correspond to the same plane in the shooting scene may be segmented into the same segmentation area. The larger the size of the segmented area, the more likely this error will occur. Therefore, in this embodiment, a first threshold can be set based on experience, so as to perform secondary segmentation of the first type of segmentation region with an area greater than the first threshold, and consider that the second type of segmentation region whose area is not greater than the first threshold The pixels of all correspond to the same plane. Specifically, when using Meanshift to segment the reference image, the setting of the first threshold is related to the color radius and the space radius in the Meanshift segmentation. Take the above example as an example: when the image size is 450×375, the color radius is 30, and the space radius is set to 30, the first threshold can be set to 800 pixels, that is, the number of pixels in a divided area. If there are more than 800, the segmented area is the first type of segmented area.

Generally, when the space radius is constant, if the color radius is relatively large, the first threshold is relatively small; when the color radius is constant, if the space radius is relatively large, the first threshold is relatively large.

In this embodiment, other image segmentation methods may also be used to segment the reference image, which is not limited in the present invention.

S104, for each first-type segmented area, perform: perform a second segmentation on the first-type segmented area, and divide the first-type segmented area into a plurality of first-type sub-segmented areas; according to each first-type segmented area The parallax of the support point and the support point in the sub-segmented area is fitted to the first plane equation of the first type of sub-segmented area; according to the first plane equation, the parallax of the non-supported point in the first type of sub-segmented area is determined ; wherein, the plurality of sub-segmented regions of the first type respectively correspond to different physical planes in the shooting scene.

Since there is a segmentation error in setting the first-type segmentation region, that is, different segmentation regions corresponding to multiple planes are divided into the same first-type segmentation region. Therefore, the first-type segmented region can be segmented a second time to obtain a plurality of first-type sub-segmented regions, so that the pixels in each of the first-type sub-segmented regions correspond to the same plane.

In an implementation manner, in step S104, fitting the first plane equation of the first type of sub-segmented region according to the support point and the disparity of the support point in each of the first-type sub-segmented regions may include:

The first plane equation of the first type of sub-segmented region is set as:

d(u, v)=au+bv+c

Among them, d(u, v) represents the disparity of the pixel point (u, v) in the first type of sub-segmented region, a, b, c are the parameters to be solved for the first plane equation, and the a is solved by the following formula , b, c:

Among them, _ui represents the coordinate of the i-th support point in the first type of sub-segmentation area on the u-axis of the pixel coordinate system corresponding to the reference image, and v _i represents the i-th support point in the first type of sub-segmentation area in the The coordinates of the v-axis of the pixel coordinate system corresponding to the reference image, the u-axis is to the right, the v-axis is down, i=1, . _i represents the disparity of the i-th support point in the first type of sub-segmented region.

Based on the first plane equation of each sub-segmented region of the first type, the disparity of the unsupported points in the sub-segmented region of the first type can be determined. The non-support point here is a concept relative to the support point, and specifically refers to the pixel point whose disparity is to be determined.

S105, for each second-type segmentation area, perform: triangulate the second-type segmentation area, and divide the second-type segmentation area into a plurality of triangular areas; for each non-supporting area in each triangular area point, determine the first parallax search range corresponding to the non-supported point according to the parallax of each vertex and each vertex of the triangle area; carry out a step-by-step process between the to-be-matched point corresponding to each parallax in the first parallax search range and the non-supported point Pixel matching is performed to determine the parallax of the unsupported point; wherein, each of the vertices is a support point.

In this embodiment, although it is considered that the pixels in the second type of segmented regions whose area is not larger than the first threshold value all correspond to the same plane. However, due to the small area of the second type of segmentation area, the number of supporting points contained in it is small, and there may even be insufficient supporting points for plane fitting. Therefore, for the second type of segmented region, the present invention adopts the triangulation method to calculate the parallax.

Triangulation, the construction of Delaunay triangles, is a fundamental research method in algebraic topology. After triangulating an image area, any two triangles either do not intersect, or intersect on exactly one edge. In this embodiment, since each vertex of the triangle is a support point, the parallax of the pixel point in the triangle is similar to the parallax of each vertex of the triangle. Moreover, the disparity of the pixels in the triangle is similar to the disparity of the surrounding support points in a certain range. Matching the to-be-matched point corresponding to the parallax of a pixel point with the pixel point by pixel can greatly reduce the parallax search range of the pixel point, and improve the matching accuracy on the basis of ensuring the matching accuracy. Efficiency, in the specific computer implementation process, achieves faster matching calculations.

In an implementation manner of the embodiment shown in FIG. 1 , as shown in FIG. 2 , in step S104 , the first type of segmentation area is divided a second time, and the first type of segmentation area is divided into a plurality of first type segmentation areas. Class sub-division area, which can include:

S201, according to a preset plane fitting rule, fit a plane equation of a physical plane corresponding to a plurality of support point subsets in the first-type segmented region; wherein, the multiple support point subsets respectively correspond to the shooting scene different physical planes.

S202, in each of the physical planes obtained by fitting, if a plurality of the physical planes satisfy the preset merging condition, then according to the support points in the plurality of physical planes and the parallax of the support points, a combined physical plane is obtained by fitting The plane equation of the plane; according to the plane equation of the merged physical plane and the plane equation of the unmerged physical plane, calculate the intersection between the merged physical planes and the unmerged physical planes, and determine the plurality of first Class sub-division area; wherein, the preset merging condition is: in the plurality of physical planes, the distance between any physical plane and at least one other physical plane is less than a third threshold;

In this embodiment, with respect to the multiple planes of the first-type segmented region obtained in step S201, if the distance between the two planes is small, the two planes may be combined and regarded as one plane. It can be understood that the number of support points of the merged plane is more than the number of support points of a single unmerged plane, so fitting the plane equation by the support points of the merged plane is more accurate than the plane equation of a single plane. The above-mentioned third threshold can be set according to experience, for example, set to 2.

Specifically, calculating the distance between two planes can be implemented in the following ways:

Set the plane equation of plane 1 as: A ₁ *x+B ₁ *y+C ₁ *d+D ₁ =0, its center point is (x ₁ , y ₁ , d ₁ ), and the plane equation of plane 2 is : A ₂ *x+B ₂ *y+C ₂ *d+D ₂ =0, and its center point is (x ₂ , y ₂ , d ₂ ), then the distance between plane 1 and plane 2 D=dis ₁ + dis ₂ , where,

Specifically, for the implementation process of "according to the support points in the plurality of physical planes and the parallax of the support points, the plane equation of the merged physical plane is obtained by fitting" in step S202, and reference may be made to the implementation process of "according to each th The implementation process of the step of fitting the first plane equation of the first type of sub-segmented area to the support point and the disparity of the support point in the first type of sub-segmented area.

In each merged plane and each unmerged plane, the pixel points on the intersection between the two adjacent planes satisfy the respective plane equations of the two adjacent planes. After the intersections between all adjacent planes are determined, a plurality of first-type sub-segmentation regions of the first-type segmented region are also determined.

S203, in each of the physical planes obtained by fitting, if any two of the physical planes do not meet the preset merging condition, calculate the intersection between the physical planes according to the plane equation of the physical planes line to determine the plurality of first-type sub-segmentation regions.

After the above steps S201-S203, the first type of segmentation area is divided twice, and each obtained first type of sub-segmented area corresponds to a different physical plane in the shooting scene. Therefore, in this embodiment, the plane equation of the first type of sub-segmented area can be fitted based on the determined support points in the first type of sub-segmented area.

In a specific implementation manner of the implementation manner shown in FIG. 2 , as shown in FIG. 3 , in step S201 , according to a preset plane fitting rule, a plurality of support point subsets corresponding to the first type of segmented regions are fitted. The plane equation of the physical plane can include:

S301, based on the random sampling consistency RANSAC algorithm, select a target support point subset O from the current support point set Y in the first type of segmentation area; wherein, the O is the support point in the Y with the largest number of support points point subset;

The RANSAC algorithm can more accurately select the pixels corresponding to the same plane in the image. Since the present invention specifically fits the equation of a plane through the support points corresponding to the plane and the parallax of the support points. Therefore, in this embodiment, it is only necessary to select the support points corresponding to each plane in the first type of segmented region through the RANSAC algorithm. Specifically, the plane equations of the planes corresponding to each support point subset may be fitted one by one according to the descending order of the number of support points in the support point subset.

S302, fit the plane equation of the physical plane corresponding to the O according to the support point of the O and the parallax of the support point;

Specifically, for the implementation process of step S302, reference may be made to the step S104, "Fit the first plane equation of the first type of sub-segmented region according to the support point and the disparity of the support point in each of the first-type sub-segmented regions". One-step implementation process.

S303, judging whether the number of support points in the currently unfitted support point subset Y-O is less than the number of support points in the first type of segmentation area of a preset ratio;

It can be understood that if the number of support points in the current unfitted support point subset Y-O is too small, or even the current unfitted support point subset Y-O does not have enough support points for plane fitting, then in this case, the The support points of the support point subset Y-O are not currently fitted to fit their plane equations. Specifically, the above-mentioned preset ratio can be set according to experience, for example, set to 10%.

S304, if the number of support points of the Y-O is not less than the number of support points in the first type of segmentation area at a preset ratio, then the Y-O is used as the current set of support points in the first type of segmentation area, Return to step S301;

S305 , if the number of support points of the Y-O is less than the number of support points in the first-type segmented region at a preset ratio, use the plane equation of the currently fitted physical plane as the final fitting result.

The support points of the remaining unfitted support point subset Y-O will be included in a certain first-type sub-division area during the process of determining the plane intersection in step S202 or step S203 in the embodiment shown in FIG. 2 . Inside. Specifically, these unfitted support points may be included in different first-type sub-segmentation regions, or may be included in the same first-type sub-segmentation region.

After the iteration of the above steps S301-S305, the plane equations of a plurality of different planes in the segmented region of the first type can be obtained. Moreover, in a specific computer implementation process, the iterative processing of the above steps S301-S305 can be performed on a plurality of first-type segmented regions at the same time, thus improving the matching efficiency.

In an implementation of the embodiment shown in FIG. 1 , for each non-supporting point in each triangular area in step S105, the corresponding non-supporting point is determined according to each vertex of the triangular area and the parallax of each vertex. The first parallax search range may include:

For each triangular area, according to each vertex in the triangular area and the parallax of each vertex, determine the second plane equation of the plane where the triangular area is located;

determining the disparity estimate of the unsupported point according to the second plane equation;

The first parallax search range includes: the parallax of each vertex, the parallax obtained by adding and subtracting the parallax of each vertex by a preset value, the parallax estimated value, and the parallax within the preset neighborhood to which the non-supported point belongs. The parallax of the target support point; wherein, the absolute value of the difference between the parallax of the target support point and the disparity estimated value of the target support point determined according to the second plane equation does not exceed a fourth threshold.

In the above-mentioned implementation manner, for the implementation process of "for each triangle area, according to each vertex in the triangle area and the parallax of each vertex, determine the second plane equation of the plane where the triangle area is located", please refer to step S104 "According to The implementation process of the step of fitting the first plane equation of the first type of sub-segmented area to the support point and the disparity of the support point in each first-type sub-segmented area.

Exemplarily, assuming that the determined second plane equation is: A*x+B*y+C*d+D=0, then the estimated disparity of an unsupported point (x ₀ , y ₀ ) in the triangle is

The above-mentioned preset value may be one value or multiple values, if it is one value, it may be 1, and if it is multiple values, it may be 1, 2, and so on. For example, if the parallax value of one vertex of a triangle is 2, and the default value is 1, the parallax value after adding and subtracting the default value to the vertex parallax is: 1 and 3.

The preset neighborhood to which the unsupported point belongs may be: after dividing the reference image into a plurality of rectangular areas of preset size, the rectangular area to which the unsupported point belongs is determined according to the coordinates of the unsupported point. More specifically, each rectangular area can be marked with a position mark, and the position mark of the rectangular area to which the non-support point belongs is determined according to the coordinates of the non-support point, that is, the rectangular area described by the non-support point is determined. For example: the reference image is a 40×40 image, which is divided into 4 small areas of 20×20, and the 4 rectangular areas are marked as (0, 0), (0, 1), (1, 0) , (1, 1). The coordinates of an unsupported point are (3, 5), then 3/20=0, 5/20=0, so it belongs to the area of (0, 0); the coordinates of another unsupported point are (35, 2) , similarly 35/20=1, 2/20=0, then it belongs to the rectangular area of (1, 0). The above preset size can be set according to experience, such as 20×20. Similarly, the above-mentioned fourth threshold can also be set according to experience, for example, set to 2.

After determining the first parallax search range of an unsupported point in the triangle, each to-be-matched point corresponding to each parallax of the first parallax search range can be matched with the non-supported point one by one to determine from the to-be-matched points. Find the matching point of the support point.

In an implementation manner of the embodiment shown in FIG. 1 , in step S105, the points to be matched corresponding to each parallax in the first parallax search range are matched with the non-support point pixel by pixel, and the unsupported point is determined by pixel-by-pixel matching Parallax, which can include:

的视差：The non-support point is determined by the following formula

Parallax of:

为该非支持点

的视差，d_n表示所述该非支持点的视差估计值，

表示所述第一视差搜索范围内的各视差，N表示所述第一视差搜索范围内的视差数，S表示所述参考图像的支持点集；

表示

的后验概率，通过以下公式计算：in,

for the non-support point

The disparity of , d _n represents the disparity estimation value of the non-supported point,

represents each parallax within the first parallax search range, N represents the number of parallaxes within the first parallax search range, and S represents the support point set of the reference image;

express

The posterior probability of , is calculated by the following formula:

表示d_n对应的待匹配点为所述非支持点

的正确匹配点的概率，

表示所述第一视差搜索范围内的各视差对应的匹配点为所述非支持点

的正确匹配点的概率。in,

Indicates that the to-be-matched point corresponding to d _n is the non-supported point

The probability of the correct matching point of ,

Indicates that the matching point corresponding to each parallax within the first parallax search range is the unsupported point

The probability of the correct matching point.

以及

是一种概率化的表示。具体的，本实施例中，可以计算非支持点

与各待匹配点的相似度，该相似度可以用非支持点

的像素值与待匹配点的像素值之差的绝对值表示。如果一个待匹配点的像素值与非支持点

的像素值之差的绝对值越小，表示这个待匹配点与非支持点

越相似，则这个待匹配点是非支持点

的正确匹配点的概率就越大。the above

as well as

is a probabilistic representation. Specifically, in this embodiment, the non-support point can be calculated

The similarity with each point to be matched, the similarity can use the non-support point

The absolute value of the difference between the pixel value of and the pixel value of the point to be matched. If the pixel value of a point to be matched is the same as the non-support point

The smaller the absolute value of the difference between the pixel values, the smaller the point to be matched and the non-support point

The more similar, the point to be matched is a non-support point

The greater the probability of correct matching points.

The binocular vision stereo matching method based on plane constraint and triangulation provided by the embodiment of the present invention, for each sub-segmentation region of a larger-area segmented region, based on the idea of plane constraint, the plane equation of each sub-segmentation region determines the The disparity of all non-supported points in the sub-segmentation area can achieve more accurate and robust matching of occluded areas with missing pixel information and large-area textureless areas with matching ambiguity, thereby reducing matching errors. Rate. In addition, in this embodiment of the present invention, triangulation is performed for a divided area with a small area, and the parallax search range of the non-supported points in the triangle is determined according to the vertex of the triangle and the parallax of the vertex, which greatly reduces the parallax search range of the non-supported points and improves the matching efficiency.

In order to verify the beneficial effects of the stereo matching method for binocular vision based on plane constraints and triangulation provided by the embodiment of the present invention, the present invention conducts specific verification through simulation experiments, as shown in FIG. 4 . Among them, (a) is the left image in the Cones test set; (b) is the right image in the Cones test set; (c) is the real disparity map provided by the Middlebury platform with (a) as the reference image; (d) is the pair ( a) The result diagram after Meanshift segmentation; (e) the black box represents the first type of segmentation region that is wrongly segmented in the Meanshift segmentation result diagram; (f) is a certain segment extracted from (e) to be subjected to secondary segmentation The outline of the first type of segmentation area; (g) is the second segmentation result obtained by fitting the first type of segmentation area in (f) to the RANSAC plane; (h) is obtained by triangulating (a) (i) is the disparity map obtained based on the matching method proposed by the present invention; (j) is the disparity map obtained by the non-local method based on the minimum spanning tree; (k) is based on LIBELAS (Library for Efficient Large-scale The disparity map obtained by the Stereo Matching algorithm; (1) is the disparity map obtained by the local method based on the support window.

In the simulation experiment, in order to improve the matching efficiency, the existing computing resources are used to perform two-way operations on the reference image at the same time, one way to perform Meanshift segmentation on the reference image, and the other way to directly triangulate the entire reference image, such as (h) shown.

It can be seen from the simulation results in FIG. 4 that the effect of the disparity map obtained by the binocular stereo matching method based on plane constraints and triangulation proposed by the present invention is obviously better than that obtained by the other three methods.

In order to further verify the beneficial effects of the binocular vision stereo matching method based on plane constraints and triangulation provided in the embodiment of the invention, as shown in Table 1, the present invention is directed to four sets of standard test charts provided by the Middlebury platform, specifically: Venus , Cones, Tsukuba, and Teddy, compared the matching error rate of the stereo matching method for binocular vision based on plane constraints and triangulation provided in this application with other three stereo matching algorithms when matching each set of standard test images. Specifically, for each set of standard test maps, the matching error rates in three specific cases are compared. The other three stereo matching algorithms include: a Guided filter (GF for short) algorithm; a Non-Local (NL for short) algorithm; and a LIBELAS algorithm. The three specific cases include: Nonocc: the case where the occluded area is not considered; All: the case where all areas of the whole image are included; Disc: the case where only the depth discontinuity area is considered.

Table 1 Matching error rate between the method of the present invention and other three stereo matching algorithms

It can be seen from Table 1 that in most cases, the matching error rate of the binocular vision stereo matching method based on plane constraints and triangulation proposed in the present invention is lower than that of the other three stereo matching algorithms. Therefore, the stereo matching effect of the method of the present invention is better than the other three stereo matching algorithms.

5 is a structural diagram of a binocular vision stereo matching device based on plane constraints and triangulation provided by an embodiment of the present invention, and the device includes:

The acquisition module 501 is used to acquire the left image and the right image of the same shooting scene respectively collected by two cameras placed in parallel at the same time, and use any one of them as a reference image and the other as an image to be matched;

A determination module 502, configured to determine the support points in the reference image, and calculate the disparity of each support point; wherein, the support points are unique and unique in the image to be matched in the reference image. Correctly match the pixel point of the point;

A segmentation module 503, configured to segment the reference image according to a preset image segmentation method to obtain a plurality of segmented regions; to divide each of the segmented regions into a first-type segmented region with a segmented area greater than a first threshold, and /or a second type of segmented area whose segmented area is not greater than the first threshold;

The first execution module 504 is configured to perform, for each first-type segmented region, performing a second segmentation on the first-type segmented region, and dividing the first-type segmented region into a plurality of first-type sub-segmented regions; According to the support points in each first-type sub-segmentation region and the disparity of the support points, the first plane equation of the first-type sub-segmentation region is fitted; according to the first plane equation, the first-type sub-segmentation region is determined Parallax of non-supported points within; wherein, the plurality of first-type sub-segmentation regions respectively correspond to different physical planes in the shooting scene;

The second execution module 505 is configured to perform, for each segmented region of the second type: triangulate the segmented region of the second type, and divide the segmented region of the second type into a plurality of triangular regions; for each triangular region For each non-support point in the triangle area, determine the first parallax search range corresponding to the non-support point according to the parallax of each vertex and each vertex of the triangle area; The unsupported point is matched pixel by pixel to determine the parallax of the unsupported point; wherein, each of the vertices is a support point.

The beneficial effects of the stereo matching device for binocular vision based on plane constraints and triangulation provided by the embodiments of the present invention are the same as those of the method embodiments. Aiming at each sub-segmentation area of a segmented area with a larger area, the device determines the parallax of all non-supported points in each sub-segmented area through the plane equation of each sub-segmented area based on the idea of plane constraint, and can realize the occlusion of missing pixel information Regions and large textureless regions with matching ambiguity are more accurately and robustly matched, thereby reducing the matching error rate. In addition, in the example of the present invention, triangulation is performed for the divided area with a small area, and the parallax search range of the non-support points in the triangle is determined according to the vertex of the triangle and the vertex parallax, which greatly reduces the parallax search range of the non-support points and improves the matching. efficiency.

Optionally, the determining module 502 includes: a grid construction sub-module and a determination sub-module.

The grid construction sub-module is used to take the upper left pixel of the reference image as the starting point, along the u-axis and the v-axis of the pixel coordinate system corresponding to the reference image, the construction step in the reference image is a preset A grid of steps; the intersections in the grid except at the edge of the reference image are used as candidate support points; where the u-axis is to the right, and the v-axis is downward.

The first determination sub-module is used to calculate, for each candidate support point, the distance between the vector of the to-be-matched point corresponding to each parallax within the second parallax search range corresponding to the candidate support point and the vector of the candidate support point; If among the calculated distances smaller than the second threshold, the number of the smallest distances is 1, the candidate support point is used as a support point; wherein, the corresponding parallax in the second parallax search range The vector of the point to be matched and the vector of the candidate support point are calculated according to the sobel operator of the preset size.

Optionally, the segmentation module 503 is specifically configured to segment the reference image according to the mean shift image segmentation method to obtain at least one segmented region.

Optionally, the first execution module 504 includes: a first fitting submodule, a second fitting submodule, a second determination submodule, and a third determination submodule.

a first fitting sub-module, configured to fit a plane equation of a physical plane corresponding to a plurality of support point subsets in the first type of segmented region according to a preset plane fitting rule; wherein the plurality of support point subsets respectively correspond to different physical planes in the shooting scene; in each of the physical planes obtained by fitting, if a plurality of the physical planes meet the preset merging condition, the second fitting sub-module is triggered; after the fitting is obtained In each of the physical planes, if any two of the physical planes do not meet the preset merging condition, the third determination sub-module is triggered; wherein, the preset merging condition is: among the plurality of physical planes, The distance between any physical plane and at least one other physical plane is less than a third threshold.

a second fitting sub-module, configured to obtain a plane equation of the merged physical plane by fitting according to the support points in the plurality of physical planes and the parallax of the support points, and trigger the second determination sub-module;

The second determination sub-module is configured to calculate the intersection between the merged physical planes and the unmerged physical planes according to the plane equation of the merged physical plane and the plane equation of the unmerged physical plane, and determine the multiple physical planes. A first-class sub-segmentation region.

The third determination sub-module is configured to calculate the intersection lines between the physical planes according to the plane equations of the physical planes, and determine the plurality of first-type sub-division regions.

Optionally, the first fitting submodule includes: a selection subunit, a fitting subunit, and a judgment subunit.

The selection subunit is used to select the target support point subset O from the current support point set Y in the first type of segmentation area based on the random sampling consistency RANSAC algorithm; wherein, the O is the number of support points in the Y The largest subset of support points.

The fitting subunit is used for fitting the plane equation of the physical plane corresponding to the O according to the support point of the O and the parallax of the support point.

The judging subunit is used for judging whether the number of support points of the current unfitted support point subset Y-O is less than the number of support points in the first-class segmented area of the preset ratio; if the number of support points of the Y-O is not less than the number of support points in the first type of segmentation area of the preset ratio, then the Y-O is used as the current support point set in the first type of segmentation area, triggering the selection of subunits; if the support points of the Y-O The number of support points in the first-type segmented region is less than the preset ratio, and the plane equation of the currently fitted physical plane is used as the final fitting result.

Optionally, the first execution module 504 is specifically configured to set the first plane equation of the first type of sub-segmented region as:

d(u, v)=au+bv+c

Among them, d(u, v) represents the disparity of the pixel point (u, v) in the first type of sub-segmented region, a, b, c are the parameters to be solved for the first plane equation, and the a is solved by the following formula , b, c:

Among them, _ui represents the coordinate of the i-th support point in the first type of sub-segmentation area on the u-axis of the pixel coordinate system corresponding to the reference image, and v _i represents the i-th support point in the first type of sub-segmentation area in the The coordinates of the v-axis of the pixel coordinate system corresponding to the reference image, the u-axis is to the right, the v-axis is down, i=1, . _i represents the disparity of the i-th support point in the first type of sub-segmented region.

Optionally, the second execution module 505 includes: a fourth determination sub-module and a fifth determination sub-module.

The fourth determination sub-module is configured to, for each triangular area, determine the second plane equation of the plane where the triangular area is located according to each vertex in the triangular area and the parallax of each vertex.

The fifth determination sub-module is configured to determine the disparity estimation value of the non-support point according to the second plane equation.

The first parallax search range includes: the parallax of each vertex, the parallax obtained by adding and subtracting the parallax of each vertex by a preset value, the parallax estimated value determined by the fifth determination sub-module, and the non-supported parallax value. The parallax of the target support point in the preset neighborhood with the point as the center; wherein, the absolute value of the difference between the parallax of the target support point and the disparity estimated value of the target support point determined according to the second plane equation does not exceed the first Four thresholds.

的视差：Optionally, the second execution module 505 is specifically configured to determine the non-support point through the following formula

Parallax of:

为该非支持点

的视差，d_n表示所述该非支持点的视差估计值，

表示所述第一视差搜索范围内的各视差，N表示所述第一视差搜索范围内的视差数，S表示所述参考图像的支持点集；

表示

的后验概率，通过以下公式计算：in,

for the non-support point

The disparity of , d _n represents the disparity estimation value of the non-supported point,

represents each parallax within the first parallax search range, N represents the number of parallaxes within the first parallax search range, and S represents the support point set of the reference image;

express

The posterior probability of , is calculated by the following formula:

表示d_n的先验概率，

表示所述第一视差搜索范围内的各视差对应的匹配点为所述非支持点

的正确匹配点的概率。in,

represents the prior probability of d _n ,

Indicates that the matching point corresponding to each parallax within the first parallax search range is the unsupported point

The probability of the correct matching point.

An embodiment of the present invention further provides an electronic device, as shown in FIG. 6 , including a processor 601 , a communication interface 602 , a memory 603 and a communication bus 604 , wherein the processor 601 , the communication interface 602 , and the memory 603 pass through the communication bus 604 complete communication with each other,

a memory 603 for storing computer programs;

When the processor 601 is used to execute the program stored in the memory 603, the following steps are implemented:

Acquire the left image and the right image of the same shooting scene collected by two cameras placed in parallel at the same time, and use any one of them as a reference image and the other as an image to be matched;

Determine the support points in the reference image, and calculate the parallax of each support point; wherein, the support points are pixels in the reference image that have unique and correct matching points in the image to be matched ;

According to a preset image segmentation method, the reference image is segmented to obtain a plurality of segmented regions; each of the segmented regions is divided into a first-type segmented region with a segmented area greater than a first threshold, and/or a segmented area not greater than the second type of segmentation region of the first threshold;

For each first-type segmentation area, perform: perform a second segmentation on the first-type segmentation area, and divide the first-type segmentation area into a plurality of first-type sub-segmentation areas; according to each first-type sub-segmentation The disparity of the support point and the support point in the area, fit the first plane equation of the first type of sub-segmented area; according to the first plane equation, determine the disparity of the non-supported point in the first type of sub-segmented area; wherein , the plurality of first-type sub-segmentation regions respectively correspond to different physical planes in the shooting scene;

For each segmented region of the second type, perform: triangulate the segmented region of the second type, and divide the segmented region of the second type into a plurality of triangular regions; for each non-support point in each triangular region, Determine the first parallax search range corresponding to the unsupported point according to the parallax of each vertex and each vertex of the triangle area; perform pixel-by-pixel matching between the to-be-matched point corresponding to each parallax within the first parallax search range and the non-supported point , determine the parallax of the non-supporting point; wherein, each vertex is a supporting point.

In the electronic device provided by the embodiment of the present invention, when the processor executes the program stored in the memory, for each sub-division area of the divided area with a larger area, based on the idea of plane constraints, the plane equation of each sub-division area determines each sub-division area. The disparity of all non-supported points in the sub-segmentation area can achieve more accurate and robust matching of occluded areas with missing pixel information and large-area textureless areas with matching ambiguity, thereby reducing matching errors. Rate. In addition, in the example of the present invention, triangulation is performed for the divided area with a small area, and the parallax search range of the non-support points in the triangle is determined according to the vertex of the triangle and the vertex parallax, which greatly reduces the parallax search range of the non-support points and improves the matching. efficiency.

The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI for short) bus or an Extended Industry Standard Architecture (EISA for short) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above electronic device and other devices.

The memory may include random access memory (Random Access Memory, RAM for short), and may also include non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; may also be a digital signal processor (Digital Signal Processing, referred to as DSP) , Application Specific Integrated Circuit (ASIC for short), Field-Programmable Gate Array (FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

In yet another embodiment provided by the present invention, a computer-readable storage medium is also provided, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium is run on a computer, the computer is made to execute any one of the above-mentioned embodiments. A stereo matching method for binocular vision based on planar constraints and triangulation.

When the instructions stored in the computer-readable storage medium provided by the embodiments of the present invention are run on a computer, for each sub-segmentation area of the segmented area with a larger area, based on the idea of plane constraint, the plane of each sub-segmented area is The equation determines the disparity of all non-supported points in each sub-segmented area, which can achieve more accurate and robust matching of occluded areas with missing pixel information and large-area textureless areas with matching ambiguity, thereby reducing match error rate. In addition, in the example of the present invention, triangulation is performed for the divided area with a small area, and the parallax search range of the non-support points in the triangle is determined according to the vertex of the triangle and the vertex parallax, which greatly reduces the parallax search range of the non-support points and improves the matching. efficiency.

In yet another embodiment provided by the present invention, there is also provided a computer program product comprising instructions, which, when run on a computer, causes the computer to execute any one of the above-mentioned embodiments based on the dual plane constraint and triangulation. A visual stereo matching method.

When the computing program product including the instructions provided by the embodiment of the present invention is run on a computer, for each sub-segmented region of the larger-area segmented region, based on the idea of plane constraints, the plane equations of the sub-segmented regions are used to determine each sub-segmented region. The disparity of all non-supported points in the sub-segmentation area can achieve more accurate and robust matching of occluded areas with missing pixel information and large-area textureless areas with matching ambiguity, thereby reducing matching errors. Rate. In addition, in the example of the present invention, triangulation is performed for the divided area with a small area, and the parallax search range of the non-support points in the triangle is determined according to the vertex of the triangle and the vertex parallax, which greatly reduces the parallax search range of the non-support points and improves the matching. efficiency.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions according to the embodiments of the present invention result in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. Computer instructions may be stored on or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server, or data center over a wire (e.g. coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.) to another website site, computer, server, or data center. A computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, or the like that includes an integration of one or more available media. Useful media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), among others.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article, or device that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. Especially, for the apparatus/electronic device/storage medium/computer program product embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

The above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (9)

1. a binocular vision stereo matching method based on plane constraint and triangulation, is characterized in that, comprises: Acquire the left image and the right image of the same shooting scene collected by two cameras placed in parallel at the same time, and use any one of them as a reference image and the other as an image to be matched; Determine the support points in the reference image, and calculate the parallax of each support point; wherein, the support points are pixels in the reference image that have unique and correct matching points in the image to be matched ; According to a preset image segmentation method, the reference image is segmented to obtain a plurality of segmented regions; the second type of segmentation region of the first threshold; For each first-type segmentation area, perform: perform a second segmentation on the first-type segmentation area, and divide the first-type segmentation area into a plurality of first-type sub-segmentation areas; according to each first-type sub-segmentation The disparity of the support point and the support point in the area, fit the first plane equation of the first type of sub-segmented area; according to the first plane equation, determine the disparity of the non-supported point in the first type of sub-segmented area; wherein , the plurality of first-type sub-segmentation regions respectively correspond to different physical planes in the shooting scene; For each segmented region of the second type, perform: triangulate the segmented region of the second type, and divide the segmented region of the second type into a plurality of triangular regions; for each non-support point in each triangular region, Determine the first parallax search range corresponding to the unsupported point according to the parallax of each vertex and each vertex of the triangle area; perform pixel-by-pixel matching between the to-be-matched point corresponding to each parallax within the first parallax search range and the non-supported point , determine the parallax of the non-supporting point; wherein, each vertex is a supporting point; Wherein, the second division of the first type of segmentation area is performed, and the first type of segmentation area is divided into a plurality of first type of sub-segmentation areas, including: According to the preset plane fitting rule, fit the plane equation of the physical plane corresponding to the plurality of support point subsets in the first type of segmentation area; wherein, the multiple support point subsets correspond to different the physical plane; In each of the physical planes obtained by fitting, if multiple physical planes meet the preset merging conditions, then according to the support points in the multiple physical planes and the disparity of the support points, the fitting of the merged physical planes is obtained. Plane equation; according to the plane equation of the merged physical plane and the plane equation of the unmerged physical plane, calculate the line of intersection between the merged physical planes and the unmerged physical planes, and determine the plurality of first-type subtypes dividing a region; wherein, the preset merging condition is: among the plurality of physical planes, the distance between any physical plane and at least one other physical plane is less than a third threshold; In each of the physical planes obtained by fitting, if any two of the physical planes do not satisfy the preset merging condition, then according to the plane equation of each of the physical planes, the intersection line between the physical planes is calculated, The plurality of sub-segmented regions of the first type are determined. 2. The method according to claim 1, wherein the determining a support point in the reference image comprises: Taking the pixel in the upper left corner of the reference image as the starting point, along the u-axis and the v-axis of the pixel coordinate system corresponding to the reference image, construct a grid with a preset step size in the reference image; In the grid, except for the intersection points located at the edge of the reference image as candidate support points; wherein, the u axis is to the right, and the v axis is downward; For each candidate support point, calculate the distance between the vector of the to-be-matched point corresponding to each parallax within the second parallax search range corresponding to the candidate support point and the vector of the candidate support point; if the calculated distance is small Among the distances of the second threshold, if the number of the smallest distance is 1, the candidate support point is regarded as a support point; wherein, the vector of the points to be matched corresponding to each parallax within the second parallax search range, and the The vectors of the candidate support points are all calculated according to the preset size of the sobel operator. 3. The method according to claim 1, wherein the reference image is segmented according to a preset image segmentation method to obtain a plurality of segmented regions, comprising: The reference image is segmented according to the mean shift image segmentation method to obtain a plurality of segmented regions. 4. The method according to claim 1, wherein according to a preset plane fitting rule, fitting the plane equation of the physical plane corresponding to a plurality of support point subsets in the first-type segmented region, comprising: S1. Based on the random sampling consistency RANSAC algorithm, select a target support point subset O from the current support point set Y in the first type of segmentation area; wherein, the O is the support point in the Y with the largest number of support points point subset; S2, fit the plane equation of the physical plane corresponding to the O according to the support point of the O and the parallax of the support point; S3, judging whether the number of support points of the current unfitted support point subset Y-O is less than the number of support points in the first type of segmentation area of a preset ratio; If the number of support points of the Y-O is not less than the number of support points in the first type of segmented area at a preset ratio, then the Y-O is used as the current set of support points in the first type of segmented area, and the execution returns to execute step S1; If the number of support points of the Y-O is less than the number of support points in the first-type segmented region at a preset ratio, the plane equation of the currently fitted physical plane is used as the final fitting result. 5 . The method according to claim 1 , wherein the first plane equation of the first type of sub-segmented region is fitted according to the support point and the disparity of the support point in each of the first-type sub-segmented regions. 6 . ,include: The first plane equation of the first type of sub-segmented region is set as: d(u,v)=au+bv+c Among them, d(u, v) represents the disparity of the pixel (u, v) in the first type of sub-segmentation area, a, b, c are the parameters to be solved for the first plane equation, and the a is solved by the following formula ,b,c:

Among them, _ui represents the coordinate of the i-th support point in the first type of sub-segmentation area on the u-axis of the pixel coordinate system corresponding to the reference image, and v _i represents the i-th support point in the first type of sub-segmentation area in the The coordinates of the v-axis of the pixel coordinate system corresponding to the reference image, the u-axis is to the right, the v-axis is down, i=1,...,m, m represents the number of support points in the first type of sub-segmentation area, d _i represents the disparity of the i-th support point in the first type of sub-segmented region. 6 . The method according to claim 1 , wherein, for each non-supporting point in each triangular area, the third corresponding to the non-supporting point is determined according to each vertex of the triangular area and the parallax of each vertex. 7 . A parallax search range, including: For each triangular area, according to each vertex in the triangular area and the parallax of each vertex, determine the second plane equation of the plane where the triangular area is located; determining the disparity estimate of the unsupported point according to the second plane equation; The first parallax search range includes: the parallax of each vertex, the parallax obtained by adding and subtracting the parallax of each vertex by a preset value, the parallax estimated value, and the parallax within the preset neighborhood to which the non-supported point belongs. The parallax of the target support point; wherein, the absolute value of the difference between the parallax of the target support point and the disparity estimated value of the target support point determined according to the second plane equation does not exceed a fourth threshold. 7 . The method according to claim 6 , wherein the point to be matched corresponding to each parallax in the first parallax search range is matched with the unsupported point pixel by pixel, and the parallax of the unsupported point is determined. 8 . ,include: The non-support point is determined by the following formula

Parallax of:

in,

for the non-support point

The disparity of , d _n represents the disparity estimation value of the non-supported point,

represents each parallax within the first parallax search range, N represents the number of parallaxes within the first parallax search range, and S represents the support point set of the reference image;

express

The posterior probability of , is calculated by the following formula:

in,

Indicates that the to-be-matched point corresponding to d _n is the non-supported point

The probability of the correct matching point of ,

Indicates that the to-be-matched point corresponding to each parallax within the first parallax search range is the non-supported point

The probability of the correct matching point. 8. A binocular vision stereo matching device based on plane constraint and triangulation, is characterized in that, comprises: an acquisition module, configured to acquire the left image and the right image of the same shooting scene respectively collected by two cameras placed in parallel at the same time, and use any one of them as a reference image and the other as an image to be matched; A determination module, configured to determine the support points in the reference image, and calculate the disparity of each support point; wherein, the support points are unique and correct in the reference image and in the to-be-matched image Pixel points of matching points; A segmentation module, configured to segment the reference image according to a preset image segmentation method to obtain a plurality of segmented regions; divide each of the segmented regions into a first-type segmented region with a segmented area greater than a first threshold, and segment a second type of segmentation region whose area is not greater than the first threshold; The first execution module is configured to, for each first-type segmented region, execute: perform a second segmentation on the first-type segmented region, and divide the first-type segmented region into a plurality of first-type sub-segmented regions; according to For the support points and the disparity of the support points in each first-type sub-segmentation area, fit the first plane equation of the first-type sub-segmentation area; according to the first plane equation, determine the first-type sub-segmentation area Disparity of non-supported points; wherein, the plurality of first-type sub-segmentation regions respectively correspond to different physical planes in the shooting scene; The second execution module is configured to perform, for each segmented region of the second type: triangulate the segmented region of the second type, and divide the segmented region of the second type into a plurality of triangular regions; For each non-supporting point, determine the first parallax search range corresponding to the non-supporting point according to the parallax of each vertex and each vertex of the triangle area; compare the point to be matched corresponding to each parallax in the first parallax search range The unsupported point is matched pixel by pixel, and the parallax of the unsupported point is determined; wherein, each of the vertices is a support point; Wherein, the first execution module includes: a first fitting sub-module, a second fitting sub-module, a second determination sub-module and a third determination sub-module; The first fitting sub-module is configured to fit the plane equation of the physical plane corresponding to the plurality of support point subsets in the first type of segmented region according to the preset plane fitting rule; wherein, the plurality of support points The point subsets respectively correspond to different physical planes in the shooting scene; in each of the physical planes obtained by fitting, if a plurality of the physical planes satisfy a preset merging condition, the second fitting submodule is triggered; In each of the physical planes obtained by fitting, if any two of the physical planes do not meet the preset merging condition, the third determination sub-module is triggered; wherein, the preset merging condition is: the Among the multiple physical planes, the distance between any physical plane and at least one other physical plane is less than a third threshold; The second fitting sub-module is configured to obtain the plane equation of the merged physical plane by fitting according to the support points in the multiple physical planes and the parallax of the support points, and trigger the second determination sub-module; The second determination submodule is used to calculate the intersection between the merged physical planes and the unmerged physical planes according to the plane equation of the merged physical plane and the plane equation of the unmerged physical plane, and determine the the plurality of first-type sub-segmentation regions; The third determination sub-module is configured to calculate the intersection lines between the physical planes according to the plane equations of the physical planes, and determine the plurality of first-type sub-division regions. 9. An electronic device, characterized in that it comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus; memory for storing computer programs; The processor is configured to implement the method steps of any one of claims 1-7 when executing the program stored in the memory.

Images