CN117876984A - Road sign detection and recognition method based on MLS point cloud - Google Patents

Abstract

The invention discloses a road marking line detection and identification method based on MLS point clouds, which aims at solving the problem that road boundaries cannot be extracted effectively, adopts a progressive road point cloud extraction method to extract road point clouds, aims at solving the problems of over-segmentation and under-segmentation, jaggy of a point cloud projection intensity image and natural incomplete road marking lines generated by road marking line extraction, designs a road marking line extraction method based on self-adaptive threshold segmentation and morphological optimization, and finally solves the problem of lower classification precision caused by feature similarity by realizing a road marking line classification method of multi-scale clustering and PointNet++, and effectively improves the detection and identification precision of the road marking lines.

Description

Road sign detection and recognition method based on MLS point cloud

Technical Field

The present invention belongs to the technical field of computer vision and artificial intelligence, and relates to a road sign line detection and recognition method based on MLS point cloud.

Background technique

Road sign detection is one of the important perception modules for high-precision maps and unmanned driving. Generally, road sign detection is mainly divided into the following three categories: image-based methods, image-based methods, and point cloud-based methods. The specific advantages and disadvantages are as follows: There are many related studies on image-based road sign detection, but this method has high requirements for the lighting conditions of the data set and is too dependent on the texture characteristics of the road surface; image-based road sign detection is also affected by lighting, and due to the problem of image splicing, errors will occur if it is projected into the same coordinate system, which will affect the accuracy of subsequent detection. Some researchers have also overcome the error problem caused by splicing by using remote sensing images instead of traditional images and images, but the resolution of remote sensing images is insufficient, so it is still difficult to accurately and comprehensively extract road features. Compared with the above methods, the point cloud-based method shows its unique advantages in road sign detection: 1) It is not affected by lighting; 2) The data acquisition efficiency is higher; 3) The acquired data has better continuity; 4) It has higher accuracy and acquisition density; 5) It has good echo reflection intensity for special materials of road signs. However, due to the problems of excessively large outdoor scene data and uneven point density distribution, this research is still challenging.

Summary of the invention

The purpose of the present invention is to provide a road sign line detection and recognition method based on MLS point cloud, which effectively improves the detection and recognition accuracy of road sign lines.

The technical solution adopted by the present invention is a road sign line detection and recognition method based on MLS point cloud, which is specifically implemented according to the following steps:

Step 1, progressive road point cloud extraction;

Step 2: Road sign line extraction based on adaptive threshold segmentation and morphological method optimization;

Step 3: Road sign line classification based on improved PointNet++.

The present invention is also characterized in that:

Step 1 is as follows:

Step 1.1, rough road extraction using the improved CSF algorithm;

Step 1.2, using the pseudo-scan line road point cloud extraction algorithm to perform fine extraction of road point cloud;

Step 1.3, road point cloud boundary optimization: cluster the road points based on the DBSCAN clustering algorithm, set constraint rules, and use the RANSAC method to refit the boundary;

Step 2 is as follows:

Step 2.1, road sign line segmentation;

Step 2.2, road sign line segmentation optimization;

Step 2.3, generating a point cloud by reversely projecting the depth image saved during point cloud projection and the binary image after adaptive threshold segmentation, thereby realizing the extraction of the road sign line point cloud;

Step 3 is as follows:

Step 3.1, road sign line clustering segmentation;

Step 3.2, lane line and zebra crossing classification;

Step 3.3, small-size road sign line classification based on improved PointNet++.

Step 1.1 is implemented according to the following steps:

Step 1.1.1, scene preprocessing: Use the discrete point removal method based on statistical filters to preprocess the scene. Index all point clouds in the scene. First, calculate the average distance _dn between any point p _i and its n nearest neighboring points. Assume that the average distance _dn of any point p _i follows a Gaussian distribution. Calculate the mean μ and standard deviation σ of the Gaussian distribution, and establish a standard range based on the obtained mean and standard deviation. When the average distance _dn is greater than the set standard range, the point is defined as a discrete point and removed.

The formula for _dn is:

Step 1.1.2, rough extraction of road point cloud: The improved CSF algorithm is used to filter the road to preliminarily extract the road point cloud.

Step 1.1.2.1, input and invert the denoised road point cloud data;

Step 1.1.2.2, establish a simulated cloth grid of the road point cloud, customize the size of the grid, and at the initial moment, the grid is located above all road point clouds;

Step 1.1.2.3, project all point clouds and grid particles in the scene onto the same horizontal plane, find the nearest neighbor of each particle, and record its elevation value IHV before projection;

Step 1.1.2.4, for each movable grid particle, calculate the displacement caused by its internal factors and compare it with the elevation value of the current nearest neighbor. If the elevation value of the particle is lower than or equal to IHV, reset the elevation value of the particle to IHV, and the particle becomes an immovable point;

Step 1.1.2.5, for each grid particle, calculate the displacement caused by internal factors;

Step 1.1.2.6, repeat steps 1.1.2.4 and 1.1.2.5. When the number of iterations is greater than the user-set value, stop the simulation process;

Step 1.1.2.7, calculate the elevation difference between the point cloud and the grid particles;

In step 1.1.2.8, if the distance between the point cloud and the grid particle is less than the preset threshold HCC, the point is considered to be a ground point, otherwise it is considered to be a non-ground point. The extracted ground point is the road point cloud.

Step 1.2 is implemented as follows:

Step 1.2.1, use the road point cloud scan line direction instead of the trajectory direction, and establish a pseudo scan line along the scan line direction, that is, slice the road point cloud, assuming that any point on the scan line trajectory The coordinates of are (X _i ,Y _i ), so/> The formula for the mileage value _Si corresponding to the point is:

Let c _n (X' _n ,Y' _n ) represent the position of any slice point, where the formula for the coordinate value of c _n is:

In formula (3), Indicates/> The coordinate azimuth of the slice position, _Δn represents the distance between the slice position and the nearest neighbor point on the trajectory, /> Indicates/> The coordinate value of

The position, direction and length of the slice are determined by c _n (X' _n ,Y′ _n ), α' _n and D respectively; the distance d from the point cloud to the slice segment is calculated. When the distance is less than half the slice length, the point cloud is located on the same slice. L represents the projection of the slice segment on the xoy plane of the coordinate system, T represents the normal of the slice segment, P (X _p ,Y _p ) represents the coordinates of the point cloud, and d represents the distance from the slice segment to the point cloud. Then the vector The formula for the relationship between and the normal vector T is expressed as:

d＝|(Xp- _{X'n )} cos( _α'n )+( _Yp - _Y'n )sin( _α'n )| (4);

Step 1.2.2, road cross-section slice projection, reconstruct the local coordinate system of the pseudo scan line to replace the geodetic coordinate system of the radar system, transform the scene point cloud into the local coordinate system of the corresponding pseudo scan line, define the coordinate origin as the center of each road segment, the X-axis is the trajectory direction, the Y-axis is perpendicular to the X-axis, _Pi ( _Xi , _Yi , _Zi ) represents any point on the slice segment, and the formula of _Pi coordinates is:

In formula (5), (X _C , Y _C ) represents the coordinates on the slice, and α represents the azimuth of the trajectory point;

Project the 3D point to the YOZ plane, where Represents the origin in the two-dimensional coordinate system, the point on the pseudo scan line after projection/> The coordinate formula is:

Step 1.2.3, use the sliding window filter to extract the road point cloud, divide the points on the projected cross-sectional slice into several small intervals, set the interval threshold to M, and then divide the point cloud on the pseudo scan line into k intervals. Assume that the last point is Then/> The coordinates of the pseudo scanning points generated for the last interval are used for straight line detection using the RANSAC algorithm. A portion of points are randomly selected for fitting, and the data deviations from other points are counted to screen out points that meet the threshold. A buffer is established for the screened points, and a sliding window filter is used to remove non-road point clouds in the buffer.

Step 2.1 is as follows:

Step 2.1.1, road point cloud projection: select the XOY plane as the horizontal plane, project the road point cloud data onto the plane along the normal vector direction of the XOY plane, assume that the original road point cloud set is _Pn = { _P1 , _P2 , ..., _Pn }, the projected road point cloud set is _Pn ′ = { _P1 ', _P2 ', ..., _Pn '}, and arbitrarily select a point t( _x0 , _y0 , _z0 ) on plane π, where the normal vector of plane π is At this time/> In/> The calculation formula for the projection height H in the direction is:

at this time Yes/> In/> The projection vector in the direction, due to/> With/> The direction is the same, so /> Then the coordinate calculation formula of the projection point _Pi ' is:

Step 2.1.2, rasterization of two-dimensional projection image: convert the projected point cloud obtained in step 2.1.1 into a two-dimensional raster image, use the inverse distance weighted interpolation method IDW, and calculate the inverse distance weighted weight influence of the control point on the surrounding pixels according to the given control point and the displacement vector of the control point pair, so as to calculate the intensity value of each pixel in the image. Assume that _Pi ( _Xi , _Yi , _Zi , _Ii ) represents the coordinates of any road point and the reflection intensity _Ii of the point, _Rpixel represents the resolution of the generated raster image, _RIDW represents the radius of interpolation, when point _Pi is projected onto plane π, _Pi changes to _pj ( _Xj , _Yj , _Ij ) after projection, and the calculation formula of the size of the raster image is:

In formula (9), maxX represents the maximum value of the X axis in the road point cloud; minX represents the minimum value of the X axis in the road point cloud; maxY represents the maximum value of the Y axis in the road point cloud; minY represents the minimum value of the Y axis in the road point cloud; Width and Height represent the size of the grating;

The intensity value of each point is obtained by the IDW algorithm. The point P _k inside the interpolation radius is determined by the position of the interpolation point Pixel _(m,n) and the interpolation radius. d _(m,n,k) represents the distance from P _k to the pixel center Pixel _(m,n) . The calculation formula is:

In formula (10), _Xk represents the X coordinate of the kth point; _Yk represents the Y coordinate of the kth point;

At this time, the intensity value I _(m,n) of the interpolation point is expressed as:

By determining the intensity value in each grid and mapping it to the grayscale interval of the image, the road point cloud is finally converted into an intensity image of the road point cloud;

Step 2.1.3, eliminate abnormal areas in the intensity image: use the IDW algorithm to remove most of the abnormal intensity points in the intensity image of the road point cloud. For the small amount of salt and pepper noise that still exists, use the improved adaptive median filter to process the intensity image of the road point cloud;

Step 2.1.4, adaptive threshold segmentation based on integral image: The integral image means that each pixel value is the sum of the rectangular area in the upper left corner of the original image. The sum of the pixel values of any rectangular area of the original image is calculated by the integral image, and the adaptive threshold of each pixel is determined by the average grayscale value of the surrounding pixels. The original pixel point is p _i ( _xi , _yi ), and the corresponding grayscale value of the point is I ( _xi , _yi ). The integral image is expressed as intI ( _xi , _yi ). By defining the background radius s and the segmentation coefficient t, the sum of the grayscale values of the pixels in the rectangular area is calculated. The calculation is performed by taking p _i as the center and the sum of the grayscale values of the length 2s+1. The formula for the sum of the grayscale values of the pixels in the rectangular area is expressed as:

When the rectangular area exceeds the boundary of the image, the image boundary is used as the standard. At this time, the grayscale value calculation formula of the binary image is:

Step 2.2 is as follows:

Step 2.2.1: Use morphological closing operations to optimize the road marking lines, repair and fill the holes in the middle of the road marking lines, and establish the original image Mask and the marked image Maker. Mask is used as a reconstruction reference. Maker uses morphological methods to continuously expand the image obtained in step 2.1.4, and then converges according to the constraints of Mask, that is, Mask ≥ Maker. Finally, the Maker is complemented and then subtracted from the Mask to complete the filling of the white holes. The expression of Mask is:

Step 2.2.2, morphological method to optimize road marking lines: The morphological MLAA method is used to process the jagged phenomenon of the binary image boundary, specifically:

Step 2.2.2.1, first find the part with obvious pixel discontinuity in the image that has been filled with white holes in step 2.2.1 as the edge that needs to be processed;

Step 2.2.2.2, classify the edges to be processed into different modes and re-vectorize the marked edges;

Step 2.2.2.3, keep the smooth part, calculate the weight through edge vectorization, mix the edge pixels with the surrounding pixels according to the weight, and achieve edge smoothing.

Step 3.1 uses Euclidean clustering to divide the road sign point cloud into independent objects, specifically:

Step 3.1.1, assuming that the road sign line point set in the scene is P = {P _i (X _i ,Y _i ), i = 1, 2, 3...}, parameter D represents the clustering radius, C represents the clustering circle with D as the radius, select any point p from the point set P, and put point p into the point set cluster;

Step 3.1.2, draw a circle with D as radius and p as center, and merge all points in the circle into the previously defined cluster;

Step 3.1.3, use other points in the point set cluster as the new center of the circle, and repeat step 3.1.2 until no new points are added to the cluster;

Step 3.1.4, repeat steps 3.1.1 and 3.1.3, and use other points in the point set P to generate new clusters, each of which is an independent object.

Step 3.2 is as follows:

Step 3.2.1, road sign feature extraction: establish MBR for the independent objects separated in step 3.1;

Step 3.2.2, lane line classification: define the aspect ratio β of the MBR obtained in 3.2.1 as the geometric feature of the object, extract the real lane lines and virtual lane lines from the cluster according to the aspect ratio, and when the conditions of the real lane lines and the virtual lane lines are not met, the sign is considered to be a small-sized road sign arrow sign or zebra crossing;

Step 3.2.3, zebra crossing classification: Use a PCA-based method to determine the arrangement of point cloud clusters, thereby extracting rectangles arranged side by side and dividing them into zebra crossings. For any rectangular sign R, calculate the coordinates of its center of mass on the xoy plane, and then calculate its covariance matrix _CR for R. By performing eigenvalue decomposition on _CR , select the eigenvector associated with the maximum eigenvalue as the distribution feature of the sign R, and set it to _vr . The parameter code represents the direction of the rectangular sign R. In order to determine the arrangement of the sign R, first search along both sides of the sign R with a search width of w. If a rectangular sign Q with a center of mass of q and a distribution feature of _vq is found in the search area, and the formula ( _vr // _vq )∧( _vr // _vrq )∧( _vq //vrq) is _satisfied , then the sign R is classified as a zebra crossing.

Step 3.3 is as follows:

Step 3.3.1, data set preparation, specifically: for the unclassified small-sized road sign lines in the scene, obtain their data, manually select the data, manually select the point set category, annotate the data, and divide the data set;

In step 3.3.2, a top-down network is built next to the main branch of the original PointNet++ network, and the PointNet network is used locally and iteratively to continuously extract features and classify the data set in step 3.3.1.

The beneficial effects of the present invention are:

The invention discloses a road sign line detection and recognition method based on MLS point cloud. Aiming at the problem that road boundaries cannot be effectively extracted, a progressive road point cloud extraction method is used to extract road point cloud. Aiming at the problems of over-segmentation and under-segmentation caused by road sign line extraction, jagged point cloud projection intensity image and natural incompleteness of road sign lines, a road sign line extraction method based on adaptive threshold segmentation and morphological method optimization is designed. Finally, by implementing multi-scale clustering and PointNet++ road sign line classification method, the problem of low classification accuracy caused by feature similarity is solved, and the detection and recognition accuracy of road sign lines is effectively improved.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a road point cloud extraction result diagram obtained after the original scene data is processed by step 1 of the present invention, wherein Figure 1(a) is the original scene diagram, Figure 1(b) is the preprocessed scene diagram, Figure 1(c) is the scene diagram after ground point filtering, Figure 1(d) is the scene road point cloud coarse extraction result diagram, Figure 1(e) is the scene non-road point cloud extraction result diagram, and Figure 1(f) is the scene road point cloud fine extraction result diagram.

Figure 2 is a diagram of the final road sign line point cloud extraction results of the three selected scenes after optimization by step 2 of the present invention, wherein Figure 2(a) is a diagram of the final road sign line point cloud extraction results after optimization of a selected scene, Figure 2(b) is a diagram of the final road sign line point cloud extraction results after optimization of a selected scene, and Figure 2(c) is a diagram of the final road sign line point cloud extraction results after optimization of another selected scene.

FIG. 3 is a diagram showing the classification result of road sign lines obtained after processing in step 3 of the present invention.

Detailed ways

The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

Example 1

This embodiment provides a road sign line detection and recognition method based on MLS (Moving Least Square) point cloud, which is specifically implemented according to the following steps:

Step 1, progressive road point cloud extraction;

Step 1.1, rough road extraction is performed using an improved Cloth Simulation Filter (CSF) algorithm;

Step 1.2, using the pseudo-scan line road point cloud extraction algorithm to perform fine extraction of road point cloud;

Step 1.3, road point cloud boundary optimization: Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering algorithm is used to cluster road points, set constraint rules, and use the Random Sample Consensus (RANSAC) method to refit the boundary, so that the extracted road boundary is closer to the real boundary and the extraction accuracy of the road point cloud is improved;

Step 2: Road sign line extraction based on adaptive threshold segmentation and morphological method optimization;

Step 2.1, road sign line segmentation;

Step 2.2, road sign line segmentation optimization;

Step 2.3, generating a point cloud by reversely projecting the depth image saved during point cloud projection and the binary image after adaptive threshold segmentation, thereby realizing the extraction of the road sign line point cloud;

Step 3: Road sign line classification based on improved PointNet++;

Step 3.1, road sign line clustering segmentation;

Step 3.2, lane line and zebra crossing classification;

Step 3.3, small-size road sign line classification based on improved PointNet++.

Example 2

This embodiment provides a method for detecting and recognizing road signs based on MLS point cloud. Based on Embodiment 1, step 1 of progressive road point cloud extraction is specifically as follows:

Step 1.1 is implemented according to the following steps:

Step 1.1.1, scene preprocessing: The original scene shown in Figure 1(a) is preprocessed by using a discrete point removal method based on a statistical filter. The preprocessed scene is shown in Figure 1(b), which specifically indexes all point clouds in the scene. First, the average distance _dn between any point p _i and its n nearest neighboring points is calculated. Assuming that the average distance _dn of any point p _i follows a Gaussian distribution, the mean μ and standard deviation σ of the Gaussian distribution are calculated, and a standard range is established based on the obtained mean and standard deviation. When the average distance _dn is greater than the set standard range, the point is defined as a discrete point and removed.

The formula for _dn is:

Step 1.1.2, rough extraction of road point cloud: The improved CSF algorithm is used to filter the road to preliminarily extract the road point cloud. The scene after ground point filtering is shown in Figure 1(c).

Step 1.1.2.1, input and invert the denoised road point cloud data;

Step 1.1.2.2, establish a simulated cloth grid of the road point cloud, customize the size of the grid, and at the initial moment, the grid is located above all road point clouds;

Step 1.1.2.3, project all point clouds and grid particles in the scene onto the same horizontal plane, find the nearest neighbor of each particle, and record its elevation value IHV before projection;

Step 1.1.2.4, for each movable grid particle, since the improved model ignores its external factors, the displacement caused by its internal factors is calculated and compared with the elevation value of the current nearest neighbor point. If the elevation value of the particle is lower than or equal to IHV at this time, the elevation value of the particle is reset to IHV, and the particle becomes an immovable point;

Step 1.1.2.5, for each grid particle, calculate the displacement caused by internal factors;

Step 1.1.2.6, repeat steps 1.1.2.4 and 1.1.2.5. When the number of iterations is greater than the user-set value, stop the simulation process;

Step 1.1.2.7, calculate the elevation difference between the point cloud and the grid particles;

In step 1.1.2.8, if the distance between the point cloud and the grid particle is less than the preset threshold HCC, the point is considered to be a ground point, otherwise it is considered to be a non-ground point. The extracted ground point is the road point cloud, and the rough extraction of the road is realized. The rough extraction result of the scene road point cloud is shown in Figure 1(d), and the extraction result of the scene non-road point cloud is shown in Figure 1(e).

Step 1.2 is implemented as follows:

Step 1.2.1, establish a pseudo scan line of the road point cloud. To extract the road point cloud, we only need to consider the plane information of the driving trajectory, that is, the x and y coordinates of the point cloud. The road point cloud scan line direction is used instead of the trajectory direction, and a pseudo scan line is established along the scan line direction, that is, the road point cloud is sliced. Assuming that the coordinates of any point P _ri on the scan line trajectory are (X _i ,Y _i ), the formula for the mileage value S _i corresponding to the P _ri point is:

Let c _n (X' _n ,Y′ _n ) represent the position of any slice point, where the formula for the coordinate value of c _n is:

In formula (3), Indicates/> The coordinate azimuth of the slice position, _Δn represents the distance between the slice position and the nearest neighbor point on the trajectory, /> Indicates/> The coordinate value of

The position, direction and length of the slice are determined by c _n (X' _n ,Y′ _n ), α' _n and D respectively; the slicing process of the point cloud also requires the calculation of the distance d between the point cloud and the slice segment. When the distance is less than half the length of the slice, the point cloud is located on the same slice. L represents the projection of the slice segment on the xoy plane of the coordinate system, T represents the normal of the slice segment, P (X _p ,Y _p ) represents the coordinates of the point cloud, and d represents the distance between the slice segment and the point cloud. Then the vector The formula for the relationship between and the normal vector T is expressed as:

d＝|( _Xp - _X'n )cos( _α'n )+( _Yp - _Y'n )sin( _α'n )| (4)

Step 1.2.2, road cross-section slice projection, reconstruct the local coordinate system of the pseudo scan line to replace the geodetic coordinate system of the radar system, transform the scene point cloud into the local coordinate system of the corresponding pseudo scan line, define the coordinate origin as the center of each road segment, the X-axis is the trajectory direction, the Y-axis is perpendicular to the X-axis, _Pi ( _Xi , _Yi , _Zi ) represents any point on the slice segment, and the formula of _Pi coordinates is:

In formula (5), (X _C , Y _C ) represents the coordinates on the slice, and α represents the azimuth of the trajectory point;

Project the 3D point to the YOZ plane, where Represents the origin in the two-dimensional coordinate system, the point on the pseudo scan line after projection/> The coordinate formula is:

Step 1.2.3, use the sliding window filter to extract the road point cloud, divide the points on the projected cross-sectional slice into several small intervals, set the interval threshold to M, and then divide the point cloud on the pseudo scan line into k intervals. Assume that the last point is Then/> The coordinates of the pseudo scanning points generated for the last interval are used for line detection using the RANSAC algorithm. A portion of points are randomly selected for fitting, and the data deviations from other points are counted to screen out points that meet the threshold. A buffer is established for the screened points. After the buffer is established, a sliding window filter is used to remove the non-road point cloud in the buffer to achieve fine extraction of the road point cloud. The extraction result is shown in Figure 1(f).

The principle of the sliding window filter is to remove non-road point clouds by establishing several small windows based on the elevation difference between the road point cloud and the curb. First, several rectangular windows are established in k intervals. Each window is the minimum enclosing window of the point cloud in the interval. The height of the window is dh _i . At this time, the road threshold is set This threshold represents the highest elevation value of the road point. When , it indicates that there is a noise point with an elevation higher than the road surface in the window, and the window is marked as a key window;

Step 1.3, road point cloud boundary optimization: cluster the road points based on the DBSCAN clustering algorithm, set constraint rules, and then use the RANSAC method to re-fit the boundary, so that the extracted road boundary is closer to the real boundary and the extraction accuracy of the road point cloud is improved.

Example 3

This embodiment provides a method for detecting and recognizing road sign lines based on MLS point clouds. On the basis of Embodiments 1-2, step 2 is based on the extraction of road sign lines optimized by adaptive threshold segmentation and morphological methods as follows:

Step 2.1 is implemented as follows:

Step 2.1.1, road point cloud projection: select the XOY plane as the horizontal plane, project the road point cloud data onto the plane along the normal vector direction of the XOY plane, assume that the original road point cloud set is _Pn = { _P1 , _P2 , ..., _Pn }, the projected road point cloud set is _Pn ' = { _P1 ', _P2 ', ..., _Pn '}, and arbitrarily select a point t( _x0 , _y0 , _z0 ) on plane π, where the normal vector of plane π is At this time/> In/> The calculation formula for the projection height H in the direction is:

at this time Yes/> In/> The projection vector in the direction, due to/> With/> The direction is the same, so /> Then the coordinate calculation formula of the projection point _Pi ' is:

Step 2.1.2, rasterization of two-dimensional projection image: convert the projected point cloud obtained in step 2.1.1 into a two-dimensional raster image, use the inverse distance weighting (IDW) interpolation method, and calculate the inverse distance weighted weight influence of the control point on the surrounding pixels according to the given control point and the displacement vector of the control point pair, so as to calculate the intensity value of each pixel in the image. Assume that _Pi ( _Xi , _Yi , _Zi , _Ii ) represents the coordinates of any road point and the reflection intensity _Ii of the point, _Rpixel represents the resolution of the generated raster image, and _RIDW represents the radius of interpolation. When point _Pi is projected onto plane π, _Pi changes to _Pi ( _Xj , _Yj , _Ij ) after projection. At this time, the calculation formula of the size of the raster image is:

In formula (9), maxX represents the maximum value of the X axis in the road point cloud; minX represents the minimum value of the X axis in the road point cloud; maxY represents the maximum value of the Y axis in the road point cloud; minY represents the minimum value of the Y axis in the road point cloud; Width and Height represent the size of the grating;

The intensity value of each point is obtained by the IDW algorithm. The point P _k inside the interpolation radius is determined by the position of the interpolation point Pixel _(m,n) and the interpolation radius. d _(m,n,k) represents the distance from P _k to the pixel center Pixel _(m,n) . The calculation formula is:

In formula (10), _Xk represents the X coordinate of the kth point; _Yk represents the Y coordinate of the kth point;

At this time, the intensity value I _(m,n) of the interpolation point is expressed as:

By determining the intensity value in each grid and mapping it to the grayscale interval of the image, the road point cloud is finally converted into an intensity image of the road point cloud;

Step 2.1.3, eliminate abnormal areas in the intensity image: use the IDW algorithm to remove most of the abnormal intensity points in the intensity image of the road point cloud. For the small amount of salt and pepper noise that still exists, use the improved adaptive median filter to process the intensity image of the road point cloud;

The improved adaptive median filter is used to process the intensity image of the road point cloud as follows: 1) Assuming that the pixel point of the intensity image is (x, y), then f(x, y) represents the pixel value of the intensity image, _Wij is a moving window, and four indicators of the moving window are added: _Wmax , _fmax , _fmin , and _fmid , which respectively represent the maximum value of the moving window, the maximum value of the grayscale value, the minimum value of the grayscale value, and the median value of the grayscale value; 2) When _fmin ＜f(x, y)＜ _fmax , the pixel point is determined to be not a noise point and is output, otherwise the point is determined to be a noise point; 3) For the determined noise point, determine whether its window _fmid is noise. If _fmin ＜ _fmid ＜ _fmax , the median value in the window is not noise, and the median value in the window is output; if _fmid is noise, the window is expanded and _Wij ＝ _Wij +1. When _Wij < _Wmax , repeat the above operation 2); 4) if _{Wij ≥Wmax} , remove all the maximum and minimum points in the window, and output the remaining pixels in the intensity image after weighted average;

Step 2.1.4, adaptive threshold segmentation based on integral image: The integral image means that each pixel value is the sum of the rectangular area in the upper left corner of the original image. The sum of the pixel values of any rectangular area of the original image is calculated by the integral image, and the adaptive threshold of each pixel is determined by the average grayscale value of the surrounding pixels. The original pixel point is p _i ( _xi , _yi ), and the corresponding grayscale value of the point is I ( _xi , _yi ). The integral image is expressed as intI ( _xi , _yi ). By defining the background radius s and the segmentation coefficient t, the sum of the grayscale values of the pixels in the rectangular area is calculated. The calculation is performed by taking p _i as the center and the sum of the grayscale values of the length 2s+1. The formula for the sum of the grayscale values of the pixels in the rectangular area is expressed as:

When the rectangular area exceeds the boundary of the image, the image boundary should be used as the standard. At this time, the grayscale value calculation formula of the binary image is:

Step 2.2 is as follows:

Step 2.2.1: Use morphological closing operations to optimize the road marking lines, repair and fill the holes in the middle of the road marking lines, and establish the original image Mask and the marked image Maker. Mask is used as a reconstruction reference. Maker uses morphological methods to continuously expand the image obtained in step 2.1.4, and then converges according to the constraints of Mask, that is, Mask ≥ Maker. Finally, the Maker is complemented and then subtracted from the Mask to complete the filling of the white holes. The expression of Mask is:

Step 2.2.2, morphological method to optimize road marking lines: the aliasing phenomenon of the binary image boundary is processed by the morphological antialiasing method (MLAA), specifically:

Step 2.2.2.1, first find the part with obvious pixel discontinuity in the image that has been filled with white holes in step 2.2.1 as the edge that needs to be processed;

Step 2.2.2.2, classify the edges to be processed into different modes and re-vectorize the marked edges;

Step 2.2.2.3, keep the smooth part, calculate the weight through edge vectorization, mix the edge pixels with the surrounding pixels according to the weight, and achieve edge smoothing.

The road sign line extraction result obtained in this step is shown in Figure 2, wherein Figure 2(a) is a final road sign line point cloud extraction result diagram after optimization of a selected scene, Figure 2(b) is a final road sign line point cloud extraction result diagram after optimization of a selected scene, and Figure 2(c) is a final road sign line point cloud extraction result diagram after optimization of another selected scene.

Example 4

This embodiment provides a method for detecting and recognizing road sign lines based on MLS point clouds. On the basis of Embodiments 1-3, step 3 is based on the road sign line classification of improved PointNet++ and is specifically implemented according to the following steps:

Step 3.1 uses Euclidean clustering to divide the road sign point cloud into independent objects, specifically:

Step 3.1.1, assuming that the road sign line point set in the scene is P = {P _i (X _i ,Y _i ), i = 1, 2, 3...}, parameter D represents the clustering radius, C represents the clustering circle with D as the radius, select any point p from the point set P, and put point p into the point set cluster;

Step 3.1.2, draw a circle with D as radius and p as center, and merge all points in the circle into the previously defined cluster;

Step 3.1.3, use other points in the point set cluster as the new center of the circle, and repeat step 3.1.2 until no new points are added to the cluster;

Step 3.1.4, repeat steps 3.1.1 and 3.1.3, and use other points in the point set P to generate new clusters, each of which is an independent object.

Step 3.2 is as follows:

Step 3.2.1, road sign line feature extraction: establish a minimum bounding rectangle (MBR) for the independent object separated in step 3.1;

The specific steps of establishing MBR are as follows: 1) Establish a simple circumscribed rectangle of the object and find the center point of the object. Assume that the point set in the object is p = { _pi ( _xi , _yi ), i = 1, 2, 3...}, and the center point coordinate formula is:

2) Rotate the circumscribed rectangle with the center point as the origin, set the rotation threshold to 1, calculate the area of the circumscribed rectangle at each rotation of 1 degree, and record the rotation angle and vertex coordinates of the rectangle at this time;

3) Compare the areas of all the circumscribed rectangles obtained by the rotation in 2) above, and the circumscribed rectangle with the smallest area is the MBR of the object;

Step 3.2.2, lane line classification: define the aspect ratio β of the MBR obtained in 3.2.1 as the geometric feature of the object, extract the real lane lines and virtual lane lines from the cluster according to the aspect ratio, when β>37, the point cloud cluster is considered to be a real lane line, when 13<β<27, the point cloud cluster is considered to be a virtual lane line, when the two conditions of real lane lines and virtual lane lines are not met, the sign is considered to be a small-sized road sign arrow sign or zebra crossing;

Step 3.2.3, zebra crossing classification: The arrangement of point cloud clusters is determined by the principal component analysis (PCA) method, so as to extract the rectangles arranged side by side and divide them into zebra crossings. For any rectangular sign R, the coordinates of its centroid on the xoy plane are calculated, and then the covariance matrix _CR is calculated for R. By performing eigenvalue decomposition on _CR , the eigenvector associated with the maximum eigenvalue is selected as the distribution feature of the sign R and set it to _vr . The parameter code represents the direction of the rectangular sign R. In order to determine the arrangement of the sign R, first search along both sides of the sign R with a search width of w. If a rectangular sign Q with a centroid of q and a distribution feature of _vq is found in the search area and satisfies the formula ( _vr // _vq )∧( _vr // _vrq )∧( _vq // _vrq ), the sign R is classified as a zebra crossing.

Step 3.3 is as follows:

Step 3.3.1, data set preparation, specifically: for the unclassified small-sized road sign lines in the scene, obtain their data, manually select the data, manually select the point set category, annotate the data, and divide the data set;

In step 3.3.2, a top-down network is built next to the main branch of the original PointNet++ network. The PointNet network is used locally and iteratively to continuously extract features and classify the data set in step 3.3.1 to obtain the road sign line classification result diagram shown in Figure 3.

Claims (9)

1. The road sign line detection and identification method based on the MLS point cloud is characterized by comprising the following steps of:

step 1, extracting a progressive road point cloud;

step 2, extracting road sign lines optimized based on a self-adaptive threshold segmentation and morphology method;

and 3, classifying road sign lines based on the improved PointNet++.

2. The road sign line detection and identification method based on the MLS point cloud according to claim 1, wherein the step 1 specifically comprises:

step 1.1, performing rough road extraction through an improved CSF algorithm;

step 1.2, carrying out fine extraction of road point clouds by using a road point cloud extraction algorithm of the pseudo scanning line;

step 1.3, road point cloud boundary optimization: clustering road points based on a DBSCAN clustering algorithm, setting constraint rules, and re-fitting boundaries by adopting a RANSAC method;

the step 2 specifically comprises the following steps:

step 2.1, road sign line segmentation;

step 2.2, road sign line segmentation optimization;

step 2.3, generating point cloud through the back projection of the depth image stored during the point cloud projection and the binary image segmented by the self-adaptive threshold value, and realizing the extraction of the point cloud of the road sign line;

the step 3 specifically comprises the following steps:

Step 3.1, clustering and dividing the road sign lines;

step 3.2, classifying lane lines and zebra crossings;

step 3.3, small-sized road sign line classification based on modified PointNet++.

3. The road sign line detection and identification method based on the MLS point cloud according to claim 2, wherein the step 1.1 is specifically implemented according to the following steps:

step 1.1.1, scene preprocessing: preprocessing a scene by adopting a discrete point removal method based on a statistical filter, indexing all point clouds in the scene, and firstly calculating any point p _i Average distance d from n nearest neighbors around it _n Assuming arbitrary p _i Average distance d of (2) _n Obeying Gaussian distribution, calculating the mean value mu and standard deviation sigma of the Gaussian distribution, establishing a standard range according to the obtained mean value and standard deviation, and taking the mean distance d as _n If the point is larger than the set standard range, defining the point as a discrete point and removing the discrete point;

d _n the formula is:

step 1.1.2, coarse extraction of road point cloud: filtering the road by adopting an improved CSF algorithm, thereby preliminarily extracting the road point cloud, in particular

Step 1.1.2.1, inputting and reversing denoised road point cloud data;

step 1.1.2.2, establishing a simulated distribution grid of the road point cloud, and customizing the size of the grid, wherein the grid is positioned above all the road point clouds at the initial moment;

Step 1.1.2.3, projecting all point clouds and grid particles in a scene onto the same horizontal plane, searching nearest neighbors of each particle at the same time, and recording elevation values IHV before projection;

step 1.1.2.4, for each movable grid particle, calculating the displacement generated by the influence of the internal factors, comparing with the elevation value of the current nearest neighbor, and resetting the elevation value of the particle to IHV and changing the particle to an immovable point if the elevation value of the particle is lower than or equal to IHV at this time;

step 1.1.2.5, for each mesh particle, calculating the displacement thereof due to the internal factors;

step 1.1.2.6, repeating step 1.1.2.4 and step 1.1.2.5, and stopping the simulation process when the iteration number is greater than the user set value;

step 1.1.2.7, calculating the elevation difference between the point cloud and the grid particles;

in step 1.1.2.8, if the distance between the point cloud and the grid particles is smaller than the preset threshold HCC, the point is considered to be a ground point, otherwise, the point is considered to be a non-ground point, and the extracted ground point is the road point cloud.

4. The road sign line detection and identification method based on the MLS point cloud according to claim 2, wherein the step 1.2 is specifically implemented according to the following steps:

Step 1.2.1, using the scanning line direction of the road point cloud to replace the track direction, and establishing a pseudo scanning line along the scanning line direction, namely slicing the road point cloud, and assuming any point on the track of the scanning lineIs (X) _i ,Y _i ) Therefore->Mileage value S corresponding to point _i The formula of (2) is:

let c _n (X′ _n ,Y′ _n ) Represents arbitrary slice point position, where c _n The formula of the coordinate value of (a) is:

in the formula (3), the amino acid sequence of the compound,representation->Coordinate azimuth, delta _n Representing the distance between the slice position and the nearest neighbor of the slice position on the track,/o>Representation->Coordinate values of (2);

from c _n (X′ _n ,Y′ _n )、α′ _n And D, determining the position of the slice, the direction of the slice and the length of the slice respectively; calculating the distance d between the point cloud and the slice segment, wherein when the distance is smaller than half of the slice length, the point cloud is positioned on the same slice, L represents the projection of the slice segment on the xoy plane of the coordinate system, T represents the normal line of the slice segment, and P (X) _p ,Y _p ) Representing the coordinates of the point cloud, and d representing the distance between the slice segment and the point cloud, then the vectorThe relationship between the normal vector T is formulated as:

d＝|(X _p -X′ _n )cos(α′ _n )+(Y _p -Y′ _n )sin(α′ _n )| (4)；

step 1.2.2, reconstructing a local coordinate system of a pseudo-scanning line to replace a geodetic coordinate system of a radar system by the projection of the road cross section slice, converting scene point clouds into the local coordinate system of the corresponding pseudo-scanning line, defining a coordinate origin as the center of each road segment, taking an X axis as a track direction, and taking a Y axis perpendicular to the X axis and P _i (X _i ,Y _i ,Z _i ) Represents any point on the segment, P _i The formula of the coordinates is:

in the formula (5), (X) _C ,Y _C ) Representing coordinates on the slice, α representing an azimuth angle of the track point;

projecting the three-dimensional point to the YOZ plane, whereinRepresenting the origin in the two-dimensional coordinate system, points on the pseudo-scan line after projection +.>The coordinate formula of (2) is:

step 1.2.3, extracting the road point cloud by using a sliding window filter, dividing the points on the projected cross section slice into a plurality of cells, setting the threshold value of the cells as M, dividing the point cloud on the pseudo scan line into k sections at the moment, and assuming that the last point isThen->And (3) for the pseudo scanning point coordinates generated in the last interval, adopting a RANSAC algorithm to perform straight line detection, randomly selecting a part of points to perform fitting, counting data deviation with other points, screening out points meeting a threshold value, establishing a buffer area for the screened points, and removing non-road point clouds in the buffer area by using a sliding window filter.

5. The method for detecting and identifying road sign lines based on MLS point clouds according to claim 4, wherein said step 2.1 is specifically,

step 2.1.1, road point cloud projection: selecting an XOY plane as a horizontal plane, projecting road point cloud data to the plane along the normal vector direction of the XOY plane, and setting an original road point cloud set as P _n ＝{P ₁ ,P ₂ ,...,P _n The projected road point cloud set isThe point t (x) is arbitrarily selected on the plane pi ₀ ,y ₀ ,z ₀ ) Wherein the normal vector of the plane pi is +.>At this time->At->The calculation formula of the projected height H in the direction is:

at this timeIs->At->Projection vector in direction due to +.>And->Is the same direction, thus->Then the point of projection P _i The' coordinate calculation formula is:

step 2.1.2, two-dimensional projection image rasterization: converting the projected point cloud obtained in the step 2.1.1 into a two-dimensional grating image, calculating the inverse distance weighting influence of the control points on surrounding pixels according to the given control points and the displacement vector of the control point pairs by using an inverse distance weighting interpolation method IDW, thereby calculating the intensity value of each pixel point in the image, and assuming P _i (X _i ,Y _i ,Z _i ,I _i ) Representing coordinates of an arbitrary road point and reflection intensity I of the point _i ，R _pixel Representing the resolution of the generated raster image, R _IDW Representing the radius of the interpolation, point P _i When projected onto the plane pi, P is the same as P _i Change to p _i (X _j ,Y _j ,I _j ) The calculation formula of the size of the raster image at this time is:

in the formula (9), maxX represents the maximum value of the X axis in the road point cloud; minX represents the minimum value of the X axis in the road point cloud; maxY represents the maximum value of the Y axis in the road point cloud; minY represents the minimum value of the Y axis in the road point cloud; width, height represent the size of the grating;

The intensity value of each point is obtained through an IDW algorithm, and the point P is positioned in the interpolation radius _k From interpolation points Pixel _(m,n) Position and interpolation radius determination, d _(m,n,k) Then represent P _k To the Pixel center Pixel _(m,n) The calculation formula is as follows:

in the formula (10), X _k An X coordinate representing a kth point; y is Y _k A Y coordinate representing a kth point;

intensity value I of interpolation point at this time _(m,n) Expressed as:

the intensity value in each grid is judged and mapped to the gray scale interval of the image, and finally the road point cloud is converted into an intensity image of the road point cloud;

step 2.1.3, eliminating abnormal areas of the intensity image: removing most of abnormal points of intensity in the intensity image of the road point cloud by adopting an IDW algorithm, and processing the intensity image of the road point cloud by adopting improved self-adaptive median filtering aiming at the condition of small amount of salt and pepper noise still existing;

step 2.1.4, based on integral image adaptive threshold segmentation: the integral image refers to the sum of the pixel values of the rectangular area of the left upper corner of the original image, the sum of the pixel values of any rectangular area of the original image is calculated through the integral image, the self-adaptive threshold value of each pixel is determined by the average value of the gray values of the surrounding pixels, and the original pixel point is p _i (x _i ,y _i ) The corresponding gray value of the point is I (x _i ,y _i ) The integral image is expressed as an intI (x _i ,y _i ) Calculating the sum of pixel gray values in a rectangular region by defining a background radius s and a partition coefficient t to p _i For the center, the sum of gray values of 2s+1 length is calculated, and the sum of pixel gray values in the rectangular area is expressed as:

when the rectangular area exceeds the boundary of the image, taking the boundary of the image as the reference, the gray value calculation formula of the binary image is as follows:

6. the method for detecting and identifying road sign lines based on MLS point clouds according to claim 5, wherein said step 2.2 is specifically,

step 2.2.1: optimizing a road marking line by using a morphological-based closing operation, repairing and filling hole defects in the middle of the road marking line, establishing an original image Mask and a marked image Mask as reconstruction references, continuously expanding the image obtained in the step 2.1.4 by using a morphological method, converging according to the constraint of the Mask, namely, the Mask is more than or equal to the Mask, and finally performing a compensation operation on the Mask and subtracting the Mask to finish filling the white holes, wherein the expression of the Mask is as follows:

step 2.2.2, optimizing the road marking line by a morphological method: the aliasing phenomenon of the binary image boundary is processed by a morphological MLAA method, specifically:

Step 2.2.2.1, firstly searching a part with obvious pixel discontinuity in the picture filled with the white holes in step 2.2.1 as an edge to be processed;

step 2.2.2.2, classifying the edges to be processed into different modes, and re-vectorizing the marked edges;

and 2.2.2.3, reserving a smooth part, calculating weights through edge vectorization, and mixing edge pixels with surrounding pixels according to the weights to realize edge smoothing.

7. The method for detecting and identifying the road marking based on the MLS point cloud according to claim 6, wherein the step 3.1 adopts European clustering to divide the road marking point cloud into individual objects, specifically:

step 3.1.1, assume that the set of road marking points in the scene is p= { P _i (X _i ,Y _i ) I=1, 2,3. }, the parameter D represents a cluster radius, C represents a cluster circle with D as a radius, taking any point P from the point set P, and placing the P points into the point set cluster;

step 3.1.2, taking D as a radius, taking p as a circle center to make a circle, and merging points in the circle into a cluster defined before;

step 3.1.3, using other points in the point cluster as new circle centers, repeating the step 3.1.2 until no new points are added into the cluster;

And 3.1.4, repeating the step 3.1.1 and the step 3.1.3, and generating new cluster clusters by using other points in the point set P, wherein each cluster is an independent object.

8. The method for detecting and identifying road sign lines based on MLS point clouds according to claim 3, wherein said step 3.2 is specifically,

step 3.2.1, extracting road sign line characteristics: establishing MBR for the independent object obtained by the separation in the step 3.1;

step 3.2.2, lane line classification: defining the aspect ratio beta of the MBR obtained in 3.2.1 as the geometric feature of the object, extracting a real lane line and a virtual lane line from the cluster according to the aspect ratio, and considering the mark as a small-size road mark line arrow mark or a zebra mark when the conditions of the real lane line and the virtual lane line are not satisfied;

step 3.2.3, zebra crossing classification: judging the arrangement mode of the point cloud clusters by adopting a PCA-based method, extracting rectangles arranged side by side, dividing the rectangles into zebra stripes, calculating the coordinates of the mass centers of any rectangular mark R on an xoy plane, and then calculating a covariance matrix C of any rectangular mark R _R By the method of C _R Performing eigenvalue decomposition, selecting eigenvector related to maximum eigenvalue as distribution characteristic of sign R, and setting it as v _r In order to judge the arrangement mode of the marks R, the direction of the rectangular marks R of the parameter code table is firstly searched along the two sides of the marks R, the searching width is w, if the centroid is q and the distribution characteristic is v in the searching area _q And satisfies the formula (v) _r //v _q )∧(v _r //v _rq )∧(v _q //v _rq The marker R is classified as a zebra crossing.

9. The method for detecting and identifying road sign lines based on MLS point clouds according to claim 2, wherein the step 3.3 is specifically,

step 3.3.1, data set preparation, specifically: for unclassified small-size road mark lines in a scene, acquiring data of the unclassified small-size road mark lines, manually framing the data, manually selecting point set categories, labeling the data, and dividing a data set;

and 3.3.2, constructing a top-down network beside a main branch of the original PointNet++ network, continuously extracting features by using the PointNet network in a local iteration mode, and classifying the data set in the step 3.3.1.