CN116030085A - Stereoscopic matching method and system for LiDAR point cloud and image line feature guidance - Google Patents
Stereoscopic matching method and system for LiDAR point cloud and image line feature guidance Download PDFInfo
- Publication number
- CN116030085A CN116030085A CN202310098953.5A CN202310098953A CN116030085A CN 116030085 A CN116030085 A CN 116030085A CN 202310098953 A CN202310098953 A CN 202310098953A CN 116030085 A CN116030085 A CN 116030085A
- Authority
- CN
- China
- Prior art keywords
- matching
- line
- pixel
- image
- depth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a stereoscopic matching method and a stereoscopic matching system for LiDAR point cloud and image line feature guidance, and belongs to the technical field of digital photogrammetry. Wherein the method comprises the steps of: step S1, acquiring a stereoscopic image and LiDAR point clouds located in the same region, and extracting depth discontinuous lines based on the LiDAR point clouds and image line features; step S2, based on a space domain, an intensity domain and a depth discontinuous line, identifying homogeneous pixels around LiDAR projection points, and updating matching cost of the homogeneous pixels by using a Gaussian function; and S3, in the cost aggregation process of stereo matching, a semi-global matching method of depth discontinuous line perception is utilized to realize high-precision stereo image dense matching. The invention provides and applies a three-edge updating strategy of a cost matrix and a depth discontinuous line sensing semi-global matching method so as to integrate LiDAR data and depth discontinuous lines into a dense matching algorithm.
Description
Technical Field
The invention belongs to the technical field of digital photogrammetry, and particularly relates to a stereoscopic matching method and system for LiDAR point cloud and image line feature guidance.
Background
Stereo matching, also called disparity estimation, has an input of a pair of two images captured at the same time and corrected by epipolar lines. And its output is a disparity map consisting of disparity values corresponding to each pixel in the reference image. Stereo matching has many interesting applications in photogrammetry and computer vision. However, conventional dense matching methods have poor resolution at low texture, depth discontinuity areas due to texture sensitivity. In contrast, liDAR point clouds have higher geometric accuracy and are not affected by the characteristic spectrum. However, unlike pixel-by-pixel measurements of cameras, liDAR data is sparse in most cases, which may result in depth discontinuity areas that are not well reconstructed. Thus, complementary fusion of LiDAR data and image data is a promising solution to produce an accurate, finely structured three-dimensional point cloud.
LiDAR data constrained dense image matching integrates LiDAR data into an advanced dense matching framework. It is a reliable method to fuse LiDAR data with images. LiDAR data can be used in many aspects of dense matching, such as reducing disparity search range, optimizing matching costs, and adjusting penalty parameters. The above method produces better matching results than the original dense matching algorithm, but due to smoothness constraints of dense matching, an extended region tends to be produced at the edge of depth discontinuity. Reducing the penalty constraint on gradient or texture edges may improve the matching results for edge regions. However, the above strategy does not distinguish texture edges from geometric edges, which conflicts with the penalty constraints of geometric edges tending to vary.
Disclosure of Invention
With the introduction of LiDAR data, it is possible to distinguish texture edges from geometric edges. Therefore, we propose stereo matching for edge protection using LiDAR data and geometric edge knowledge to recover the exact three-dimensional structure of low-texture, depth discontinuity areas.
The input of the invention is a stereoscopic image and LiDAR point cloud which are positioned in the same area. Accurate registration is a prerequisite for most LiDAR and image fusion methods, including our proposed method. In preprocessing, stereo images need to be corrected in epipolar image space, and outliers of LiDAR need to be filtered out by position uncertainty. First, we triangulate a triangle mesh model from LiDAR point cloud and project it onto the stereonuclear line image by collinearly equation to generate an initial disparity map. Next, we extract depth discontinuity lines located in the depth discontinuity areas by combining the image line features and the initial disparity map. Finally, we propose a cost matrix for trilateral updates to improve low texture regions, and semi-global matching (Semi Global Matching, SGM) of depth discontinuity line perception to preserve depth discontinuities.
The invention provides a stereoscopic matching method for LiDAR point cloud and image line feature guidance, which comprises the following steps:
a stereoscopic matching method for LiDAR point cloud and image line feature guidance is characterized by comprising the following steps: the method comprises the following steps:
step S1, acquiring a stereoscopic image and LiDAR point clouds located in the same region, and extracting depth discontinuous lines based on the LiDAR point clouds and image line features;
step S2, based on a space domain, an intensity domain and a depth discontinuous line, identifying homogeneous pixels around LiDAR projection points, and updating matching cost of the homogeneous pixels by using a Gaussian function;
and S3, in the cost aggregation process of stereo matching, a semi-global matching method of depth discontinuous line perception is utilized to realize high-precision stereo image dense matching.
Further, the specific implementation manner of the step S1 is as follows;
step S1.1, firstly, carrying out accurate registration between LiDAR point clouds and images, then correcting the stereoscopic images in a epipolar image space, and filtering outliers of the LiDAR through position uncertainty;
step S1.2, generating a triangular mesh model by using a LiDAR point cloud, then projecting the triangular mesh model from the LiDAR onto a three-dimensional epipolar line image to generate an initial parallax map and LiDAR projection points, wherein the LiDAR projection points display sparse laser points with known depth or parallax in an image grid, and the shielded LiDAR points are eliminated when the LiDAR triangular mesh model is reversely projected onto the original image;
s1.3, extracting line features on a reference image in the three-dimensional nuclear line image by using an existing linear feature detection algorithm;
and S1.4, establishing buffer rectangles on two sides of a straight line, wherein the two buffer rectangles are symmetrical to the straight line in the image, the lengths of the two buffer rectangles are parallel to the straight line, the widths of the two buffer rectangles are perpendicular to the straight line, firstly calculating the parallax median value of the two buffer rectangles in the initial parallax map, and selecting the straight line with obvious parallax change, namely a parallax change value larger than 1 pixel, as a depth discontinuous line, wherein the parallax median value refers to the middle value of the effective parallax value in the buffer rectangles.
Further, in step S2, the homogeneous pixels refer to pixels with no obvious intensity or color change in the spatial neighborhood of the LiDAR projection point, and the homogeneous pixels around the LiDAR projection point are determined based on the three-edge filtering method of the spatial domain, the intensity domain and the depth discontinuous line, which defines the similarity measurement between the central pixel q and each pixel p around, and the formula for calculating the similarity is as follows:
wherein I is p And I q The intensities of the current and center pixels, respectively; the weighting functions f (-) and g (-) are functions based on Gaussian distribution and correspond to a space domain and an intensity domain respectively; whether p is a homogeneous pixel of q may be determined by a fixed threshold truncation; t { - } is an indicator function that verifies whether the line between the center pixel and the current pixel passes through the depth discontinuity line; m represents a binarized set of lines between the center pixel and the current pixel in image space; DL represents a binarized set of depth discontinuities in image space;
the three-sided based pixel similarity ensures that pixels on both sides of the depth discontinuity line are not judged to be homogenous pixels.
Further, in step S2, the matching costs of the homogeneous pixels are updated by using a gaussian function, the matching costs refer to the feature description distance between the matching points and the points to be matched, the matching costs of the pixels near the depth discontinuous line are set to be the minimum value which is not equal to 0 in consideration of the ambiguity of the matching costs of the pixels near the depth discontinuous line, finally, the matching costs of the homogeneous pixels are updated by using a gaussian function, the matching costs close to the correct parallax are reduced, and the matching costs far from the correct parallax are increased according to the gaussian function.
Further, the specific implementation manner of the semi-global matching method for perception of depth discontinuous lines in step S3 is as follows;
in step S3.1, the semi-global matching method is along a one-dimensional path L r Traversing the image in eight directions minimizes the global energy function along each path L r Recursively calculating the minimum cost of all disparities to a pixel p on the path;
wherein C (p, d) represents the matching cost of pixel p at disparity d, L r (P, d) represents the matching cost of the pixel P along the path r, k and i are the symbols used for counting in the matching process, and have no practical meaning, P 1 And P 2 The parallax difference between the current pixel p and the previous pixel in the path r direction is 1 and the parallax difference is greater than 1; finally, the cost paths are added to all paths on each pixel:
s (p, d) denotes that the optimal disparity is obtained from the multipath aggregate matching costs, and thus, the disparity of each pixel corresponds to the minimum multipath aggregate matching cost;
step S3.2, the semi-global matching method for depth discontinuity line perception is to use smaller penalty parameter P at depth discontinuity line 2 To allow for significant changes in parallax of adjacent pixels;
according to whether adjacent pixels on the cost aggregation path are located on depth discontinuous lines or not, if at least one of the current pixel p and the previous pixel p-r in the aggregation direction has depth discontinuous lines, the parallax information transmitted to the current pixel is not reliable any more, a smaller penalty parameter should be selected to allow parallax mutation to occur, and if neither p nor p-r has depth discontinuous lines, the penalty parameter is not adjusted, and the specific formula is as follows:
where DL represents the binarized set of depth discontinuities in image space.
In a second aspect, the present invention further provides a stereo matching system guided by LiDAR point cloud and image line features, including the following modules:
the preprocessing module is used for acquiring stereoscopic images and LiDAR point clouds which are positioned in the same area, and extracting depth discontinuous lines based on the LiDAR point clouds and image line features;
the matching cost updating module is used for identifying homogeneous pixels around LiDAR projection points based on the space domain, the intensity domain and the depth discontinuous lines and updating the matching cost of the homogeneous pixels by using a Gaussian function;
and the matching module is used for realizing high-precision dense matching of the stereoscopic images by using a semi-global matching method perceived by depth discontinuous lines in the cost aggregation process of the stereoscopic matching.
Further, the specific implementation manner of the preprocessing module is as follows;
step S1.1, firstly, carrying out accurate registration between LiDAR point clouds and images, then correcting the stereoscopic images in a epipolar image space, and filtering outliers of the LiDAR through position uncertainty;
step S1.2, generating a triangular mesh model by using a LiDAR point cloud, then projecting the triangular mesh model from the LiDAR onto a three-dimensional epipolar line image to generate an initial parallax map and LiDAR projection points, wherein the LiDAR projection points display sparse laser points with known depth or parallax in an image grid, and the shielded LiDAR points are eliminated when the LiDAR triangular mesh model is reversely projected onto the original image;
s1.3, extracting line features on a reference image in the three-dimensional nuclear line image by using an existing linear feature detection algorithm;
and S1.4, establishing buffer rectangles on two sides of a straight line, wherein the two buffer rectangles are symmetrical to the straight line in the image, the lengths of the two buffer rectangles are parallel to the straight line, the widths of the two buffer rectangles are perpendicular to the straight line, firstly calculating the parallax median value of the two buffer rectangles in the initial parallax map, and selecting the straight line with obvious parallax change, namely a parallax change value larger than 1 pixel, as a depth discontinuous line, wherein the parallax median value refers to the middle value of the effective parallax value in the buffer rectangles.
Further, the homogeneous pixels in the matching cost updating module refer to pixels with no obvious intensity or color change in the spatial neighborhood of the LiDAR projection point, and the homogeneous pixels around the LiDAR projection point are determined based on a three-side filtering method of a spatial domain, an intensity domain and a depth discontinuous line, wherein the similarity measurement between a central pixel q and each surrounding pixel p is defined by the method, and a formula for calculating the similarity is as follows:
wherein I is p And I q The intensities of the current and center pixels, respectively; the weighting functions f (-) and g (-) are functions based on Gaussian distribution and correspond to a space domain and an intensity domain respectively; whether p is a homogeneous pixel of q may be determined by a fixed threshold truncation; t { - } is to verify whether the line between the center pixel and the current pixel passes through the depth discontinuityAn indication function of the line; m represents a binarized set of lines between the center pixel and the current pixel in image space; DL represents a binarized set of depth discontinuities in image space;
the three-sided based pixel similarity ensures that pixels on both sides of the depth discontinuity line are not judged to be homogenous pixels.
Further, the matching cost updating module updates the matching cost of the homogeneous pixels by using a Gaussian function, the matching cost refers to the feature description distance between the matching point and the point to be matched, the matching cost of the pixels near the depth discontinuous line is set to be a minimum value which is not equal to 0 in consideration of the ambiguity of the matching cost of the pixels near the depth discontinuous line, finally, the matching cost of the homogeneous pixels is updated by using a Gaussian function, the matching cost updated by the Gaussian function refers to the fact that the matching cost close to the correct parallax is reduced, and the matching cost far away from the correct parallax is increased according to the Gaussian function.
Further, the implementation mode of the semi-global matching method for perception of the depth discontinuous lines in the matching module is as follows;
in step S3.1, the semi-global matching method is along a one-dimensional path L r Traversing the image in eight directions minimizes the global energy function along each path L r Recursively calculating the minimum cost of all disparities to a pixel p on the path;
wherein C (p, d) represents the matching cost of pixel p at disparity d, L r (P, d) represents the matching cost of the pixel P along the path r, k and i are the symbols used for counting in the matching process, and have no practical meaning, P 1 And P 2 Penalty parameter for disparity difference of 1 and penalty for disparity difference greater than 1 for the current pixel p and the previous pixel in the path r direction, respectivelyParameters; finally, the cost paths are added to all paths on each pixel:
s (p, d) denotes that the optimal disparity is obtained from the multipath aggregate matching costs, and thus, the disparity of each pixel corresponds to the minimum multipath aggregate matching cost;
step S3.2, the semi-global matching method for depth discontinuity line perception is to use smaller penalty parameter P at depth discontinuity line 2 To allow for significant changes in parallax of adjacent pixels;
according to whether adjacent pixels on the cost aggregation path are located on depth discontinuous lines or not, if at least one of the current pixel p and the previous pixel p-r in the aggregation direction has depth discontinuous lines, the parallax information transmitted to the current pixel is not reliable any more, a smaller penalty parameter should be selected to allow parallax mutation to occur, and if neither p nor p-r has depth discontinuous lines, the penalty parameter is not adjusted, and the specific formula is as follows:
where DL represents the binarized set of depth discontinuities in image space.
Compared with the prior art, the invention has the following beneficial effects:
1. according to the stereoscopic matching method for the LiDAR point cloud and the image line feature guidance, sparse but high-precision LiDAR points are combined with sharp object edges of images, and a three-dimensional point cloud with an accurate and fine structure is generated. After extracting depth discontinuous lines on an image by using LiDAR depth information, we propose a three-edge update strategy of a cost matrix and semi-global matching of depth discontinuous line perception to integrate LiDAR data and depth discontinuous lines into a dense matching algorithm. Our method significantly improves the results of dense matching, especially in restoring the three-dimensional structure of weak texture and deep discontinuities.
2. The invention discloses a method for updating three sides of a cost matrix. Homogeneous pixels refer to pixels that have no significant intensity or color change within the spatial neighborhood of the LiDAR projection point. We propose a trilateral filtering method based on spatial, intensity and depth discontinuities to determine homogenous pixels around LiDAR proxels, which defines a similarity measure between the center pixel and every pixel around. The three-sided based pixel similarity ensures that pixels on both sides of the depth discontinuity line are not judged to be homogenous pixels.
3. The invention discloses a semi-global matching method for perception of depth discontinuous lines. The method is compatible with the previous cost aggregation penalty algorithm to tolerate significant parallax changes on both sides of the depth discontinuity line in the cost aggregation process. After adaptively adjusting the penalty parameters using gradients, canny, or texture edges (optional), we update the penalty parameters based on whether neighboring pixels on the cost-aggregated path lie on a depth discontinuity line. If at least one of the current pixel p and the previous pixel p-r in the aggregation direction has a depth discontinuity line, the disparity information delivered to the current pixel is no longer reliable, and a smaller penalty parameter should be chosen to allow the occurrence of disparity abrupt changes. If neither p nor p-r have a depth discontinuity line, the penalty parameter is not adjusted.
Drawings
FIG. 1 is a flow chart of a stereoscopic matching method for LiDAR point cloud and image line feature guidance provided by an embodiment of the invention;
fig. 2 is a disparity map of a semi-global match and generated by the present invention.
Detailed Description
The invention provides a stereoscopic matching method for LiDAR point cloud and image line feature guidance, which combines sparse but high-precision LiDAR points with sharp object edges of images to generate a three-dimensional point cloud with precise and fine structures. After extracting depth discontinuous lines on an image by using LiDAR depth information, we propose a three-edge update strategy of a cost matrix and semi-global matching of depth discontinuous line perception to integrate LiDAR data and depth discontinuous lines into a dense matching algorithm. The invention will be further illustrated with reference to examples.
In this embodiment, the method includes the following steps:
s1: and projecting a triangular mesh model from the LiDAR onto the stereoscopic nuclear line image to generate an initial parallax map and LiDAR projection points, and judging whether the straight line is positioned in a depth discontinuous region according to parallax difference values of buffer areas at two sides of the straight line.
In this embodiment, for the extraction of the depth discontinuous line, the data preprocessing and the screening of the depth discontinuous line from the image line feature are preferably performed sequentially, specifically as follows:
data preprocessing:
s1.1: accurate registration is a prerequisite for most LiDAR and stereoscopic image fusion methods, including our proposed method. In order to realize accurate registration between LiDAR point cloud and stereo image, the method generally comprises two steps of coarse registration and fine registration. Then, the stereoscopic image is corrected by the epipolar line to generate a reference epipolar line image and a epipolar line image to be matched, and outliers of LiDAR are filtered out through position uncertainty. Because the invention is the prior art, the implementation process is not specifically described;
s1.2: the method comprises the steps of firstly generating a triangular mesh model by using a LiDAR point cloud, and then projecting the triangular mesh model from the LiDAR onto a reference epipolar line image to generate an initial parallax map and LiDAR projection points. LiDAR proxels exhibit sparse laser points of known depth or parallax in an image grid. And then, when the LiDAR triangle network model is back projected onto the original image, the blocked LiDAR points are eliminated.
And (3) selecting depth discontinuous lines from the image line characteristics:
s1.3: the straight line provides important information about the man-made object, which is usually located at the boundary of the object. Therefore, the line characteristics on the reference epipolar line image are extracted by using the existing linear characteristic detection algorithm;
s1.4: depth discontinuities can be clearly identified by the initial disparity map. The buffer rectangles are built on two sides of the straight line, the two buffer rectangles are symmetrical to the straight line in the image, the length of the buffer rectangles is parallel to the straight line, and the width (10 pixels are taken in the example) of the buffer rectangles is perpendicular to the straight line. The median disparity value of the two buffered rectangles in the initial disparity map can be calculated. The median disparity value refers to the median value of the effective disparity values within the buffered rectangle. Only straight lines with significant parallax variations (> 1 pixel) are selected as depth discontinuity lines.
S2: and carrying out three-side updating on the cost matrix of the three-dimensional matching based on the space domain, the intensity domain and the depth discontinuous line. The method specifically comprises the following steps: s2.1: identifying homogenous pixels around LiDAR proxels based on the spatial domain, the intensity domain, and the depth discontinuity line; s2.2: and updating the matching cost of the homogeneous pixels by using a Gaussian function.
S2.1: the cost matrix stores the matching cost of each pixel in the reference image and the image to be matched in the parallax range. When LiDAR data is available, cost optimization to update the matching cost of homogenous pixels is an efficient strategy for closely matching constrained by LiDAR data. Homogeneous pixels refer to pixels that have no significant intensity or color change within the spatial neighborhood of the LiDAR projection point. We propose a trilateral filtering method based on spatial, intensity and depth discontinuities to determine homogenous pixels around LiDAR proxels, which defines the measure of similarity between the center pixel q and every pixel p around, and the method of computing similarity is shown in equation (1).
Wherein I is p And I q The intensities of the current pixel and the center pixel, respectively; the weighting functions f (-) and g (-) are functions based on Gaussian distribution and correspond to a space domain and an intensity domain respectively; whether p is a homogeneous pixel of q can be determined by a fixed threshold cutoff (empirical value of 0.7); t { - } is an indicator function that verifies whether the line between the center pixel and the current pixel passes through the depth discontinuity line; m represents a binarized set of lines between the center pixel and the current pixel in image space; DL represents a binarized set of depth discontinuities in image space.
The three-sided based pixel similarity ensures that pixels on both sides of the depth discontinuity line are not judged to be homogenous pixels.
S2.2: and updating the matching cost of the homogeneous pixels by using a Gaussian function. The matching cost refers to a feature description distance between the matching point and the point to be matched (in this example, census transformation is used for feature description, and hamming distance is used for distance). Considering the ambiguity of the matching cost of pixels near the depth discontinuity line, we set the matching cost of these pixels to a minimum value that is not equal to 0. Finally, the matching costs of these homogeneous pixels are updated using a gaussian function, which may promote continuous parallax or depth changes in the homogeneous region of the image. The Gaussian function updating matching cost means that the matching cost close to the correct parallax is reduced, and the matching cost far from the correct parallax is increased in a Gaussian function mode. Thus, the gaussian function ensures that most pixels, including those close to depth discontinuities, are more prone to correct parallax than incorrect parallax.
S3: in the cost aggregation process of stereo matching, a depth discontinuous line perception semi-global matching method is provided, and finally high-precision stereo image dense matching with reserved edges is realized. The specific implementation process is as follows:
s3.1: because SGM is a widely accepted dense matching algorithm in the industry, exhibiting excellent matching accuracy and computational efficiency, the approach presented herein is based on improvements of SGM. SGM follows one-dimensional path L r Traversing the image in eight directions minimizes the global energy function along each path L r The minimum cost of all disparities to a pixel p on the path is computed recursively.
Wherein C (p, d) represents the matching cost of pixel p at disparity d, L r (p, d) represents the pixel p edgeThe matching cost of the path r, k and i are symbols used for counting in the matching process, have no practical meaning and P 1 And P 2 The penalty parameter that the parallax difference of the current pixel p and the previous pixel in the path r direction is 1 and the penalty parameter that the parallax difference is larger than 1 are respectively, and the final subtraction ensures L r (p,d)<Cmax+P 2 To prevent excessive matching costs. SGM accumulates the cost paths onto all paths on each pixel:
s (p, d) denotes that the optimal disparity is obtained from the multipath aggregate matching costs, and thus, the disparity of each pixel corresponds to the minimum multipath aggregate matching cost.
S3.2: using smaller penalty parameters P at depth discontinuities 2 To allow for significant changes in the parallax of adjacent pixels.
After adaptively adjusting the penalty parameters using gradients, canny, or texture edges (optional), we update the penalty parameters based on whether neighboring pixels on the cost-aggregated path lie on a depth discontinuity line. If at least one of the current pixel p and the previous pixel p-r in the aggregate direction (p-r referring to the previous pixel of the pixel p in the r direction) has a depth discontinuity line, the disparity information delivered to the current pixel is no longer reliable and a smaller penalty parameter should be chosen to allow the disparity mutation to occur.
If neither p nor p-r have a depth discontinuity line, the penalty parameter is not adjusted. The specific formula is as follows.
Finally, in combination with the cost matrix trilateral update and punishment adjustment, pixels around the depth discontinuity line will be more likely to obtain the correct disparity.
The invention also provides a stereoscopic matching system guided by LiDAR point cloud and image line characteristics, which comprises the following modules:
the preprocessing module is used for acquiring stereoscopic images and LiDAR point clouds which are positioned in the same area, and extracting depth discontinuous lines based on the LiDAR point clouds and image line features;
the matching cost updating module is used for identifying homogeneous pixels around LiDAR projection points based on the space domain, the intensity domain and the depth discontinuous lines and updating the matching cost of the homogeneous pixels by using a Gaussian function;
and the matching module is used for realizing high-precision dense matching of the stereoscopic images by using a semi-global matching method perceived by depth discontinuous lines in the cost aggregation process of the stereoscopic matching.
It should be understood that the system provided in the present invention corresponds to the above method, and therefore, specific details are set forth in the above method, and the disclosure is not repeated.
In addition, the above-described division of the functional module units is merely a division of logic functions, and there may be another division manner when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted or not performed. Meanwhile, the integrated units can be realized in a hardware form or a software functional unit form.
Claims (10)
- A stereoscopic matching method for LiDAR point cloud and image line feature guidance is characterized by comprising the following steps: the method comprises the following steps:step S1, acquiring a stereoscopic image and LiDAR point clouds located in the same region, and extracting depth discontinuous lines based on the LiDAR point clouds and image line features;step S2, based on a space domain, an intensity domain and a depth discontinuous line, identifying homogeneous pixels around LiDAR projection points, and updating matching cost of the homogeneous pixels by using a Gaussian function;and S3, in the cost aggregation process of stereo matching, a semi-global matching method of depth discontinuous line perception is utilized to realize high-precision stereo image dense matching.
- 2. The method according to claim 1, characterized in that: the specific implementation manner of the step S1 is as follows;step S1.1, firstly, carrying out accurate registration between LiDAR point clouds and a stereoscopic image, then, correcting the stereoscopic image through epipolar lines to generate a reference epipolar line image and a epipolar line image to be matched, and filtering outliers of the LiDAR through position uncertainty;step S1.2, generating a triangular mesh model by using a LiDAR point cloud, then projecting the triangular mesh model from the LiDAR onto a reference epipolar line image to generate an initial parallax map and LiDAR projection points, wherein the LiDAR projection points display sparse laser points with known depth or parallax in an image grid, and the shielded LiDAR points are eliminated when the LiDAR triangular mesh model is reversely projected onto the original image;s1.3, extracting line features on a reference epipolar line image by using an existing linear feature detection algorithm;and S1.4, establishing buffer rectangles on two sides of a straight line, wherein the two buffer rectangles are symmetrical to the straight line in the image, the lengths of the two buffer rectangles are parallel to the straight line, the widths of the two buffer rectangles are perpendicular to the straight line, firstly calculating the parallax median value of the two buffer rectangles in the initial parallax map, and selecting the straight line with obvious parallax change, namely a parallax change value larger than 1 pixel, as a depth discontinuous line, wherein the parallax median value refers to the middle value of the effective parallax value in the buffer rectangles.
- 3. The method according to claim 1, characterized in that: the homogeneous pixels in step S2 refer to pixels with no obvious intensity or color change in the spatial neighborhood of the LiDAR projection point, and the homogeneous pixels around the LiDAR projection point are determined based on a three-side filtering method of spatial domain, intensity domain and depth discontinuous line, which defines a similarity measure between the central pixel q and each pixel p around, and the formula for calculating the similarity is as follows:
wherein I is p And I q Respectively, current andintensity of the center pixel; the weighting functions f (-) and g (-) are functions based on Gaussian distribution and correspond to a space domain and an intensity domain respectively; whether p is a homogeneous pixel of q may be determined by a fixed threshold truncation; t { - } is an indicator function that verifies whether the line between the center pixel and the current pixel passes through the depth discontinuity line; m represents a binarized set of lines between the center pixel and the current pixel in image space; DL represents a binarized set of depth discontinuities in image space;
the three-sided based pixel similarity ensures that pixels on both sides of the depth discontinuity line are not judged to be homogenous pixels. - 4. The method according to claim 1, characterized in that: in step S2, the matching costs of the homogeneous pixels are updated by using a gaussian function, the matching costs refer to feature description distances between the matching points and the points to be matched, the matching costs of the pixels near the depth discontinuous line are set to be minimum values different from 0 in consideration of ambiguity of the matching costs of the pixels near the depth discontinuous line, finally, the matching costs of the homogeneous pixels are updated by using a gaussian function, the matching costs of the gaussian function update refer to that the matching costs close to correct parallax are reduced, and the matching costs far away from the correct parallax are increased in a gaussian function form.
- 5. The method according to claim 1, characterized in that: the specific implementation manner of the semi-global matching method for perception of the depth discontinuous line in the step S3 is as follows;in step S3.1, the semi-global matching method is along a one-dimensional path L r Traversing the image in eight directions minimizes the global energy function along each path L r Recursively calculating the minimum cost of all disparities to a pixel p on the path;
wherein C (p, d) represents the matching cost of pixel p at disparity d, L r (P, d) represents the matching cost of the pixel P along the path r, k and i are the symbols used for counting in the matching process, and have no practical meaning, P 1 And P 2 The parallax difference between the current pixel p and the previous pixel in the path r direction is 1 and the parallax difference is greater than 1; finally, the cost paths are added to all paths on each pixel:
s (p, d) denotes that the optimal disparity is obtained from the multipath aggregate matching costs, and thus, the disparity of each pixel corresponds to the minimum multipath aggregate matching cost;step S3.2, the semi-global matching method for depth discontinuity line perception is to use smaller penalty parameter P at depth discontinuity line 2 To allow for significant changes in parallax of adjacent pixels;according to whether adjacent pixels on the cost aggregation path are located on depth discontinuous lines or not, if at least one of the current pixel p and the previous pixel p-r in the aggregation direction has depth discontinuous lines, the parallax information transmitted to the current pixel is not reliable any more, a smaller penalty parameter should be selected to allow parallax mutation to occur, and if neither p nor p-r has depth discontinuous lines, the penalty parameter is not adjusted, and the specific formula is as follows:
where DL represents the binarized set of depth discontinuities in image space. - LiDAR point cloud and image line feature guided stereo matching system, characterized by comprising the following modules:the preprocessing module is used for acquiring stereoscopic images and LiDAR point clouds which are positioned in the same area, and extracting depth discontinuous lines based on the LiDAR point clouds and image line features;the matching cost updating module is used for identifying homogeneous pixels around LiDAR projection points based on the space domain, the intensity domain and the depth discontinuous lines and updating the matching cost of the homogeneous pixels by using a Gaussian function;and the matching module is used for realizing high-precision dense matching of the stereoscopic images by using a semi-global matching method perceived by depth discontinuous lines in the cost aggregation process of the stereoscopic matching.
- 7. The system according to claim 6, wherein: the specific implementation mode of the preprocessing module is as follows;step S1.1, firstly, carrying out accurate registration between LiDAR point clouds and a stereoscopic image, then, correcting the stereoscopic image through epipolar lines to generate a reference epipolar line image and a epipolar line image to be matched, and filtering outliers of the LiDAR through position uncertainty;step S1.2, generating a triangular mesh model by using a LiDAR point cloud, then projecting the triangular mesh model from the LiDAR onto a three-dimensional epipolar line image to generate an initial parallax map and LiDAR projection points, wherein the LiDAR projection points display sparse laser points with known depth or parallax in an image grid, and the shielded LiDAR points are eliminated when the LiDAR triangular mesh model is reversely projected onto an original image;s1.3, extracting line features on a reference image in the three-dimensional nuclear line image by using an existing linear feature detection algorithm;and S1.4, establishing buffer rectangles on two sides of a straight line, wherein the two buffer rectangles are symmetrical to the straight line in the image, the lengths of the two buffer rectangles are parallel to the straight line, the widths of the two buffer rectangles are perpendicular to the straight line, firstly calculating the parallax median value of the two buffer rectangles in the initial parallax map, and selecting the straight line with obvious parallax change, namely a parallax change value larger than 1 pixel, as a depth discontinuous line, wherein the parallax median value refers to the middle value of the effective parallax value in the buffer rectangles.
- 8. The system according to claim 6, wherein: the homogeneous pixels in the matching cost updating module refer to pixels with no obvious intensity or color change in the spatial neighborhood of the LiDAR projection point, and the homogeneous pixels around the LiDAR projection point are determined based on a three-side filtering method of a spatial domain, an intensity domain and a depth discontinuous line, wherein the similarity measurement between a central pixel q and each surrounding pixel p is defined by the method, and the formula for calculating the similarity is as follows:
wherein I is p And I q The intensities of the current and center pixels, respectively; the weighting functions f (-) and g (-) are functions based on Gaussian distribution and correspond to a space domain and an intensity domain respectively; whether p is a homogeneous pixel of q may be determined by a fixed threshold truncation; t { - } is an indicator function that verifies whether the line between the center pixel and the current pixel passes through the depth discontinuity line; m represents a binarized set of lines between the center pixel and the current pixel in image space; DL represents a binarized set of depth discontinuities in image space;
the three-sided based pixel similarity ensures that pixels on both sides of the depth discontinuity line are not judged to be homogenous pixels. - 9. The system according to claim 6, wherein: and updating the matching cost of the homogeneous pixels by using a Gaussian function in a matching cost updating module, wherein the matching cost refers to the feature description distance between a matching point and a point to be matched, the matching cost of the pixels near the depth discontinuous line is set to be a minimum value which is not equal to 0 in consideration of the ambiguity of the matching cost of the pixels near the depth discontinuous line, finally, the matching cost of the homogeneous pixels is updated by using the Gaussian function, the matching cost updated by the Gaussian function refers to the reduction of the matching cost close to the correct parallax, and the matching cost far away from the correct parallax is increased in the form of the Gaussian function.
- 10. The system according to claim 6, wherein: the specific implementation mode of the semi-global matching method for perception of the depth discontinuous lines in the matching module is as follows;in step S3.1, the semi-global matching method is along a one-dimensional path L r Traversing the image in eight directions minimizes the global energy function along each path L r Recursively calculating the minimum cost of all disparities to a pixel p on the path;
wherein C (p, d) represents the matching cost of pixel p at disparity d, L r (P, d) represents the matching cost of the pixel P along the path r, k and i are the symbols used for counting in the matching process, and have no practical meaning, P 1 And P 2 The parallax difference between the current pixel p and the previous pixel in the path r direction is 1 and the parallax difference is greater than 1; finally, the cost paths are added to all paths on each pixel:
s (p, d) denotes that the optimal disparity is obtained from the multipath aggregate matching costs, and thus, the disparity of each pixel corresponds to the minimum multipath aggregate matching cost;step S3.2, the semi-global matching method for depth discontinuity line perception is to use smaller penalty parameter P at depth discontinuity line 2 To allow for significant changes in parallax of adjacent pixels;according to whether adjacent pixels on the cost aggregation path are located on depth discontinuous lines or not, if at least one of the current pixel p and the previous pixel p-r in the aggregation direction has depth discontinuous lines, the parallax information transmitted to the current pixel is not reliable any more, a smaller penalty parameter should be selected to allow parallax mutation to occur, and if neither p nor p-r has depth discontinuous lines, the penalty parameter is not adjusted, and the specific formula is as follows:
where DL represents the binarized set of depth discontinuities in image space.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310098953.5A CN116030085A (en) | 2023-01-31 | 2023-01-31 | Stereoscopic matching method and system for LiDAR point cloud and image line feature guidance |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310098953.5A CN116030085A (en) | 2023-01-31 | 2023-01-31 | Stereoscopic matching method and system for LiDAR point cloud and image line feature guidance |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116030085A true CN116030085A (en) | 2023-04-28 |
Family
ID=86075868
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310098953.5A Pending CN116030085A (en) | 2023-01-31 | 2023-01-31 | Stereoscopic matching method and system for LiDAR point cloud and image line feature guidance |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116030085A (en) |
-
2023
- 2023-01-31 CN CN202310098953.5A patent/CN116030085A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Cox et al. | A maximum likelihood stereo algorithm | |
| US10477178B2 (en) | High-speed and tunable scene reconstruction systems and methods using stereo imagery | |
| Liu et al. | Joint geodesic upsampling of depth images | |
| CN101512601B (en) | Method for determining a depth map from images, device for determining a depth map | |
| EP3869797B1 (en) | Method for depth detection in images captured using array cameras | |
| Zhou et al. | Seamless fusion of LiDAR and aerial imagery for building extraction | |
| US9070042B2 (en) | Image processing apparatus, image processing method, and program thereof | |
| CN108596975B (en) | Stereo matching algorithm for weak texture region | |
| Pantilie et al. | SORT-SGM: Subpixel optimized real-time semiglobal matching for intelligent vehicles | |
| EP2806396A1 (en) | Sparse light field representation | |
| KR100953076B1 (en) | Multi-view matching method and device using foreground/background separation | |
| US20140049612A1 (en) | Image processing device, imaging device, and image processing method | |
| CN108682039B (en) | Binocular stereo vision measuring method | |
| US20170270680A1 (en) | Method for Determining Depth Maps from Stereo Images with Improved Depth Resolution in a Range | |
| CN106257537B (en) | A kind of spatial depth extracting method based on field information | |
| CN104851089A (en) | Static scene foreground segmentation method and device based on three-dimensional light field | |
| CN102447917A (en) | Three-dimensional image matching method and equipment thereof | |
| Oliveira et al. | Selective hole-filling for depth-image based rendering | |
| CN114445473B (en) | Stereo matching method and system based on deep learning operator | |
| Patil et al. | A comparative evaluation of SGM variants (including a new variant, tMGM) for dense stereo matching | |
| Zhang et al. | Automatic terrain extraction using multiple image pair and back matching | |
| Zhu et al. | Triangulation of well-defined points as a constraint for reliable image matching | |
| CN116030085A (en) | Stereoscopic matching method and system for LiDAR point cloud and image line feature guidance | |
| CN116189140A (en) | Binocular vision-based vehicle three-dimensional target detection algorithm | |
| Mandici et al. | Aggregate road surface based environment representation using digital elevation maps |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |