CN113569778A - Pavement slippery area detection and early warning method based on multi-mode data fusion - Google Patents
Pavement slippery area detection and early warning method based on multi-mode data fusion Download PDFInfo
- Publication number
- CN113569778A CN113569778A CN202110885654.7A CN202110885654A CN113569778A CN 113569778 A CN113569778 A CN 113569778A CN 202110885654 A CN202110885654 A CN 202110885654A CN 113569778 A CN113569778 A CN 113569778A
- Authority
- CN
- China
- Prior art keywords
- image
- road surface
- vehicle
- pavement
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a pavement slippery area detection and early warning method and device based on multi-mode data fusion, wherein image features are extracted based on an image segmentation algorithm to obtain a pavement detection result; obtaining a point cloud pavement detection result based on a point cloud segmentation algorithm; fusing the two detection results to obtain a pavement detection area; fusing the reflection intensity information of the laser point cloud in the pavement area with the image characteristics, increasing the correlation degree of the similar pixel point characteristics by using a pixel point bidirectional matching method, and obtaining a pavement dry-wet state detection result by using a pavement dry-wet state segmentation network; constructing a vehicle motion equation according to the vehicle motion state information and predicting the vehicle motion track and speed based on a particle filter algorithm; and providing early warning for the driver according to the road surface dry and wet state and the vehicle track prediction result. Therefore, the road surface state in front of the vehicle can be detected in real time, the vehicle instability risk is evaluated, early warning is provided for a driver when the vehicle has the instability risk, driving safety is improved, and traffic accidents are reduced.
Description
Technical Field
The invention relates to the technical field of unmanned driving, in particular to a pavement slippery area detection and early warning method based on multi-mode data fusion.
Background
The wet and slippery state of the road surface caused by severe weather can reduce the tire road adhesion coefficient of the vehicle and the road, thereby increasing the vehicle control difficulty and increasing the driving risk. The road surface slippery state is detected, and the driver is provided with early warning before the vehicle enters a slippery area, so that traffic accidents can be reduced, and the road surface slippery state early warning method has important significance in improving driving safety.
Aiming at the problem of detection of a slippery area of a road surface, related researches are carried out by a plurality of organizations at home and abroad at present. Almazan E J et al (2016) propose a road wet and slippery state detection algorithm based on a vehicle-mounted camera, the algorithm estimates road vanishing points and spatial horizontal lines through a geometric method, then segments roads through road form prior knowledge, and finally classifies road dry and wet states through a Bayesian classifier. Asuzu P and the like (2018) provide a road surface slippery state detection algorithm based on a vehicle-mounted millimeter wave radar, the method researches the echo intensity of millimeter wave signals within 0-1 meter, and classification of road surface slippery state is completed through the mean value of a distribution histogram of the echo intensity. Shin J et al (2019) propose a road surface slippery state detection algorithm based on a laser radar, which collects road surface laser point cloud data, projects point cloud in a three-dimensional space to a two-dimensional plane, and completes detection of a road surface slippery area by comparing and analyzing a reflection intensity value of the point cloud.
Most of the existing algorithms detect the slippery state of the road surface based on a single sensor, the slippery area of the road surface cannot be detected specifically, and the single sensor is susceptible to the influence of conditions such as illumination and the like, so that the detection result lacks robustness. The existing method has no mechanism which is designed to provide early warning for a driver in a targeted manner after the wet and slippery state of the road surface is detected. Therefore, the research and design of the road surface slippery area detection method based on multi-sensor data fusion and the design of a mechanism for providing early warning for a driver have important theoretical and practical significance.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.
Therefore, the invention aims to provide a pavement slippery area detection and early warning method based on multi-mode data fusion, which can detect the pavement slippery area and provide early warning before a vehicle is in danger.
The invention also aims to provide a road surface slippery area detection and early warning device based on multi-mode data fusion.
In order to achieve the above object, an embodiment of the invention provides a road surface slippery area detection and early warning method based on multi-mode data fusion, which includes the following steps:
collecting road surface images and laser point clouds through a calibrated vehicle-mounted camera and a laser radar;
detecting the road surface image based on an image segmentation algorithm to obtain an image characteristic image and an image road surface detection result;
screening the laser point cloud based on a point cloud segmentation algorithm to obtain a point cloud pavement detection result;
fusing the point cloud pavement detection result with the image pavement detection result to obtain a fused pavement detection area;
fusing the reflection intensity information of the laser point cloud in the pavement detection area with the image characteristics, increasing the correlation degree of similar pixel point characteristics by using a pixel point bidirectional matching method, and obtaining a pavement dry-wet state detection result by using a pavement dry-wet state segmentation network;
collecting vehicle motion state information, constructing a vehicle motion equation and predicting vehicle motion track and speed based on a particle filter algorithm;
and early warning the vehicle according to the detection result of the dry and wet state of the road surface and the motion track and speed of the vehicle.
In order to achieve the above object, an embodiment of the present invention provides a road surface slippery region detecting and warning device based on multi-modal data fusion, including:
the acquisition module is used for acquiring a road surface image and laser point cloud through a calibrated vehicle-mounted camera and a calibrated laser radar;
the first detection module is used for detecting the road surface image based on an image segmentation algorithm to obtain an image characteristic diagram and an image road surface detection result;
the second detection module is used for screening the laser point cloud based on a point cloud segmentation algorithm to obtain a point cloud pavement detection result;
the fusion module is used for fusing the point cloud pavement detection result with the image pavement detection result to obtain a fused pavement detection area;
the third detection module is used for fusing the laser point cloud reflection intensity information of the pavement detection area with the image characteristics, increasing the correlation degree of similar pixel point characteristics by using a pixel point bidirectional matching method, and obtaining a pavement dry-wet state detection result by using a pavement dry-wet state segmentation network;
the prediction module is used for acquiring vehicle motion state information, constructing a vehicle motion equation and predicting vehicle motion track and speed based on a particle filter algorithm;
and the early warning module is used for early warning the vehicle according to the road surface dry and wet state detection result and the vehicle motion track and speed.
The pavement slippery area detection and early warning method and device based on multi-mode data fusion in the embodiment of the invention have the following beneficial effects:
1) and enhancing the laser point cloud in the pixel coordinate system to ensure that the density of the laser point cloud in the pavement area is consistent with the pixel density of the image, and assisting the target-level fusion of the subsequent laser point cloud and the image.
2) And in the stage of road surface detection and the stage of road surface slippery area detection, data fusion methods are respectively used, so that the accuracy of road surface detection and road surface slippery area detection is improved.
3) And performing bidirectional matching of pixel point characteristics on the characteristic graph after data fusion, increasing the relevance of the same type of characteristics, and expanding the discrimination of different types of characteristics, so that the detection result of the road surface slippery area has higher robustness.
4) And evaluating the risk of the vehicle falling into the unstable state based on the road surface slippery area detection result and the vehicle track prediction result, and providing early warning for the driver before the vehicle risks.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow chart of a road slippery region detection and early warning method based on multi-modal data fusion according to an embodiment of the present invention;
FIG. 2 is a block diagram of a flow chart of a road surface slippery region detection and early warning method based on multi-modal data fusion according to an embodiment of the present invention;
FIG. 3 is a diagram of an Encoder structure of a road segmentation algorithm based on a deep neural network according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a Decoder structure of a road segmentation algorithm based on a deep neural network according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a deep neural network-based road surface segmentation algorithm Prediction Head architecture according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of point cloud bi-directional matching according to one embodiment of the present invention;
FIG. 7 is a diagram of a road wet and dry segmentation network according to one embodiment of the present invention;
FIG. 8 is a schematic illustration of vehicle trajectory prediction according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a road slippery region detection and early warning device based on multi-modal data fusion according to an embodiment of the invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
The following describes a road surface slippery region detection and early warning method and device based on multi-mode data fusion, which are provided by the embodiment of the invention, with reference to the accompanying drawings.
Firstly, a pavement slippery area detection and early warning method based on multi-mode data fusion provided by the embodiment of the invention will be described with reference to the attached drawings.
As shown in fig. 1 and fig. 2, the method for detecting and warning a slippery area on a road surface based on multi-modal data fusion comprises the following steps:
in step S1, a road surface image and a laser point cloud are acquired by the calibrated vehicle-mounted camera and the laser radar.
Before collecting road surface images and laser point clouds, firstly, a vehicle-mounted camera and a laser radar are subjected to combined calibration, and an internal reference matrix and an external reference matrix of the vehicle-mounted camera are obtained.
Specifically, the calibration steps are as follows:
in a software environment in a vehicle, an autoware-based calibration toolkit is configured. After the calibration tool box is started, a calibration worker places the chessboard pattern calibration board in front of the vehicle at three positions far away from the vehicle, near the vehicle and in the middle of the vehicle;
using a calibration tool box to collect an image containing a calibration plate in a visual field and laser point cloud data;
obtaining a camera internal reference matrix P by using a checkerboard calibration method built in a calibration tool box; obtaining an extrinsic parameter matrix T by correlating the position of a calibration plate in an image and a laser point cloudr。
In step S2, the road surface image is detected based on the image segmentation algorithm, and an image feature map and an image road surface detection result are obtained.
Optionally, in an embodiment of the present invention, detecting a road surface image based on an image segmentation algorithm to obtain an image feature and an image road surface detection result, includes: constructing an image segmentation algorithm model; carrying out data enhancement and training an image segmentation algorithm model through a training set; and obtaining an image characteristic diagram of the road surface image and an image road surface detection result through the trained image segmentation algorithm model.
In one embodiment of the invention, the image segmentation algorithm model comprises a feature extraction unit, a fusion unit and a prediction unit; the feature extraction unit comprises a plurality of convolution layers, pooling layers and activation functions which are connected in series and is used for calculating the features of the road surface image, down-sampling the image and outputting image feature maps with different scales; the fusion unit comprises a plurality of convolution layers, a pooling layer and an activation function which are connected in series and is used for fusing the image characteristic diagrams of different scales output by the characteristic extraction unit and recovering the resolution of the characteristic diagrams; the prediction unit is used for predicting the road surface area in the road surface image according to the image characteristic diagram output by the fusion unit to obtain an image road surface detection result.
Specifically, an image segmentation algorithm model based on a deep neural network is designed, and the model comprises an Encoder (shown in fig. 3), a Decoder (shown in fig. 4) and a Prediction Head (shown in fig. 5). The Encoder comprises a plurality of convolution layers, pooling layers and activation functions which are connected in series, calculates the characteristics of an input image, down-samples the image and outputs image characteristic graphs with different scales; the Decoder also comprises a plurality of convolution layers, a pooling layer and an activation function which are connected in series, and the feature graphs of different scales output by the Endecoder are fused and the resolution of the feature graphs is gradually restored; the Prediction Head takes a characteristic graph output by the Decoder as input and predicts a road surface area in the image.
And performing data enhancement on the training set and training the model constructed in the step. The data enhancement mode comprises the following steps: random scaling of images, random flipping of images, random cropping of images, random adjustment of image brightness, random adjustment of image contrast, and random adjustment of image saturation.
In the detection process, aiming at each input image, an algorithm simultaneously outputs an image feature map obtained by Decoder calculation and a road surface Prediction result obtained by Prediction Head.
In step S3, the laser point cloud is screened based on the point cloud segmentation algorithm to obtain a point cloud road surface detection result.
Using a point cloud segmentation algorithm based on RANSAC, the points belonging to the road surface in the laser point cloud captured in step S1 are screened.
Optionally, in an embodiment of the present invention, the screening the laser point cloud based on the point cloud segmentation algorithm to obtain a point cloud road surface detection result includes: determining a plane model through a plurality of laser points aiming at each frame of laser point cloud; determining the number of laser points in the plane model according to the distance between the laser points and the plane model; and determining an optimal model according to the number of the laser points, and determining a point cloud pavement detection result by using the optimal model.
Specifically, as a specific embodiment, for each frame of the laser point cloud, a plurality of laser points, for example, 3 laser points, are randomly selected, and the plane model determined by the 3 points is calculated.
And calculating whether other laser points meet the plane model or not according to a set threshold, if so, recording the total number of the points in the model, such as calculating the distance between the other laser points and the plane model, if the distance is less than 0.2, recording the distance as the inner point of the model, and recording the total number of the points in the model.
And repeating iteration for N times, such as 100 times, recording the generated model as an optimal model if the total number of the inner points is higher than that of the existing model, and recording the inner points of the optimal model, wherein the inner points are the detected road surface.
In step S4, the point cloud road surface detection result and the image road surface detection result are fused to obtain a fused road surface detection area.
Optionally, in an embodiment of the present invention, fusing the point cloud road surface detection result and the image road surface detection result to obtain a fused road surface detection area, including: mapping the point cloud pavement detection result to an image through an internal reference matrix and an external reference matrix of the vehicle-mounted camera, and obtaining dense point cloud in a pixel coordinate system by using a point cloud enhancement method; and fusing the dense point cloud and the image pavement detection result to obtain a data-fused pavement detection area.
Specifically, the road surface laser point cloud obtained in step S3 is mapped into an image pixel coordinate system, and the mapping formula is:
y＝PTrx
wherein x is the coordinate of the laser point cloud in the world coordinate system, P is the camera reference matrix, TrAnd y is the pixel coordinate of the point cloud in the image.
Performing point cloud enhancement on the information (x coordinate in a world coordinate system, y coordinate in the world coordinate system and reflection intensity) of the laser point cloud except for the height in a pixel coordinate system to form dense point cloud, wherein the formula of the point cloud enhancement is as follows:
wherein x0，x1Is the abscissa of two adjacent points in the x-axis direction, x0＜x1，x∈(x0,x1)；y0，y1Is the ordinate of two adjacent points in the y-axis direction, y0＜y1，y∈(y0,y1)。
And (3) the enhanced point cloud is a dense road surface detection result, the enhanced point cloud is fused with the road surface detection result obtained in the step (2), and a union of the detection results is obtained to obtain a road surface detection result after data fusion. And outputting the reflection intensity information of the point cloud at the corresponding position according to the fused road surface detection result.
In step S5, the laser point cloud reflection intensity information of the road surface detection area is fused with the image features, the correlation degree of similar pixel point features is increased by using a pixel point bidirectional matching method, and the road surface dry-wet state detection result is obtained by using a road surface dry-wet state segmentation network.
It can be understood that according to the road surface detection result obtained in the step S4, the reflection intensity information of the laser point cloud in the road surface area is obtained and is subjected to feature fusion with the image features obtained in the step S2, the correlation degree of similar pixel point features is increased by using a pixel point bidirectional matching method for the feature graph obtained after fusion, and then the road surface dry-wet state detection result is obtained by using a road surface dry-wet state segmentation network.
Specifically, the image feature map obtained by the Decoder in step S2 and the point cloud reflection intensity information output in step S4 are subjected to feature splicing, and the obtained feature map is a feature map after feature fusion.
And performing bidirectional matching on each pixel point of the fused feature graph, and increasing the correlation degree of the features of the similar pixel points, wherein the specific process is as follows:
(1) setting N pixel points in the feature map: { a1,…,aNThe feature vector at each pixel point is: { b1,…,bN}. And calculating the similarity between each pixel point and all other pixel points in a characteristic vector point multiplication mode.
(2) For each pixel point ak,k∈[1,N]And taking n points with the highest similarity, and recording as:
wherein n is one half of the total number of the pixel points of the characteristic diagram.
(3) For all pixel points
i∈[1,n]If the n points with the highest similarity contain the pixel point akThen a iskAnd
matching, fig. 6 shows in schematic form that when the feature map has 4 pixels in total, the pixel point a1The matching relationship of (1).
(4) Take all ofkThe matched pixel points are subjected to weighted summation by taking the normalized similarity as weight to obtain a new feature vector for updating the pixel point akThe characteristic value of (2).
And inputting the characteristic graph obtained after the pixel points are subjected to bidirectional matching to the road surface dry and wet state segmentation network to obtain a road surface dry and wet state detection result. As shown in fig. 7, the road surface wet-dry state segmentation network is composed of several convolution layers, a pooling layer and an activation function.
In step S6, vehicle motion state information is collected, a vehicle motion equation is constructed, and a vehicle motion trajectory and speed are predicted based on a particle filter algorithm.
The motion state information of vehicle speed, steering wheel turning angle and the like is collected through an automobile bus, a vehicle motion equation is constructed, and the motion track and the motion speed of the vehicle are predicted based on a particle filter algorithm.
Specifically, a ground two-dimensional plane is set as a global coordinate system, a current own vehicle position is set as an origin, and the directions of x and y axes are the same as a vehicle coordinate system.
Collecting vehicle motion state information: { dx,dyV, θ }, where dx,dyThe abscissa and the ordinate of the vehicle in the global coordinate system are shown, v is the vehicle speed value, and theta is the vehicle direction.
Establishing a vehicle motion equation:
6k+1＝θk+δθ
vk+1＝vk+δv
The position and speed of the vehicle within 3-4 seconds are predicted according to the vehicle motion equation and the particle filter algorithm and the probability distribution thereof is output, as shown in fig. 8.
In step S7, a vehicle is warned according to the road surface dry and wet state detection result and the vehicle motion trajectory and speed.
Optionally, in an embodiment of the present invention, the early warning of the vehicle according to the detection result of the dry-wet state of the road surface and the motion track and speed of the vehicle includes: and calculating the probability of the vehicle passing through the road surface slippery area and the probability that the speed of the vehicle is greater than the threshold speed when the vehicle passes through the slippery area, and early warning the vehicle according to the probability and the probability threshold.
Specifically, the road surface dry-wet state detection result obtained in step S5 is mapped to the global coordinate system using the 3D coordinates of the point cloud. The threshold speed v is set according to the wet state of the wet area. The greater the degree of road surface hydroplaning, the lower the threshold speed.
Estimating the probability p that the vehicle passes through the wet skid region based on the vehicle trajectory prediction result obtained in step S61And the probability p that the vehicle speed is greater than the threshold speed when passing through the wet and slippery region2If p is1p2And if the speed is more than or equal to 50%, early warning is provided for the driver.
According to the pavement slippery area detection and early warning method based on multi-mode data fusion, provided by the embodiment of the invention, a pavement image and laser point cloud are collected through a calibrated vehicle-mounted camera and a laser radar; detecting the road surface image based on an image segmentation algorithm to obtain an image characteristic image and an image road surface detection result; screening laser point clouds based on a point cloud segmentation algorithm to obtain a point cloud pavement detection result; fusing the point cloud pavement detection result with the image pavement detection result to obtain a fused pavement detection area; fusing laser point cloud reflection intensity information and image features in a pavement detection area, increasing the correlation degree of similar pixel point features by using a pixel point bidirectional matching method, and obtaining a pavement dry-wet state detection result by using a pavement dry-wet state segmentation network; collecting vehicle motion state information, constructing a vehicle motion equation and predicting vehicle motion track and speed based on a particle filter algorithm; and early warning is carried out on the vehicle according to the detection result of the dry and wet state of the road surface and the motion track and speed of the vehicle. The road surface state in front of the vehicle can be detected in real time, the vehicle instability risk is evaluated, early warning is provided for a driver when the vehicle has the instability risk, and the method and the device have great significance for improving driving safety and reducing traffic accidents.
Next, a road surface slippery region detection and early warning device based on multi-modal data fusion according to an embodiment of the present invention will be described with reference to the drawings.
Fig. 9 is a schematic structural diagram of a road slippery region detection and early warning device based on multi-modal data fusion according to an embodiment of the invention.
As shown in fig. 9, the road surface slippery area detection and early warning device based on multi-modal data fusion comprises: the system comprises an acquisition module 100, a first detection module 200, a second detection module 300, a fusion module 400, a third detection module 500, a prediction module 600 and an early warning module 700.
The acquisition module 100 is used for acquiring road surface images and laser point clouds through a calibrated vehicle-mounted camera and a calibrated laser radar. The first detection module 200 is configured to detect a road surface image based on an image segmentation algorithm, so as to obtain an image feature map and an image road surface detection result. And the second detection module 300 is used for screening the laser point cloud based on a point cloud segmentation algorithm to obtain a point cloud pavement detection result. And the fusion module 400 is configured to fuse the point cloud road surface detection result and the image road surface detection result to obtain a fused road surface detection area. The third detection module 500 is configured to fuse the laser point cloud reflection intensity information of the road surface detection area with the image features, increase the degree of association of similar pixel point features by using a pixel point bidirectional matching method, and obtain a road surface dry-wet state detection result by using a road surface dry-wet state segmentation network. The prediction module 600 is configured to collect vehicle motion state information, construct a vehicle motion equation, and predict a vehicle motion trajectory and speed based on a particle filtering algorithm. And the early warning module 700 is used for early warning the vehicle according to the detection result of the dry and wet state of the road surface and the motion track and speed of the vehicle.
Optionally, in an embodiment of the present invention, the method further includes: and the calibration module is used for carrying out combined calibration on the vehicle-mounted camera and the laser radar to obtain an internal parameter matrix and an external parameter matrix of the vehicle-mounted camera.
It should be noted that the foregoing explanation of the method embodiment is also applicable to the apparatus of this embodiment, and is not repeated herein.
According to the pavement slippery area detection and early warning device based on multi-mode data fusion, a calibrated vehicle-mounted camera and a calibrated laser radar are used for collecting pavement images and laser point clouds; detecting the road surface image based on an image segmentation algorithm to obtain an image characteristic image and an image road surface detection result; screening laser point clouds based on a point cloud segmentation algorithm to obtain a point cloud pavement detection result; fusing the point cloud pavement detection result with the image pavement detection result to obtain a fused pavement detection area; fusing laser point cloud reflection intensity information and image features in a pavement detection area, increasing the correlation degree of similar pixel point features by using a pixel point bidirectional matching method, and obtaining a pavement dry-wet state detection result by using a pavement dry-wet state segmentation network; collecting vehicle motion state information, constructing a vehicle motion equation and predicting vehicle motion track and speed based on a particle filter algorithm; and early warning is carried out on the vehicle according to the detection result of the dry and wet state of the road surface and the motion track and speed of the vehicle. The road surface state in front of the vehicle can be detected in real time, the vehicle instability risk is evaluated, early warning is provided for a driver when the vehicle has the instability risk, and the method and the device have great significance for improving driving safety and reducing traffic accidents.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (10)
1. A pavement slippery area detection and early warning method based on multi-mode data fusion is characterized by comprising the following steps:
collecting road surface images and laser point clouds through a calibrated vehicle-mounted camera and a laser radar;
detecting the road surface image based on an image segmentation algorithm to obtain an image characteristic image and an image road surface detection result;
screening the laser point cloud based on a point cloud segmentation algorithm to obtain a point cloud pavement detection result;
fusing the point cloud pavement detection result with the image pavement detection result to obtain a fused pavement detection area;
fusing the reflection intensity information of the laser point cloud in the pavement detection area with the image characteristics, increasing the correlation degree of similar pixel point characteristics by using a pixel point bidirectional matching method, and obtaining a pavement dry-wet state detection result by using a pavement dry-wet state segmentation network;
collecting vehicle motion state information, constructing a vehicle motion equation and predicting vehicle motion track and speed based on a particle filter algorithm;
and early warning the vehicle according to the detection result of the dry and wet state of the road surface and the motion track and speed of the vehicle.
2. The method of claim 1, further comprising:
and carrying out combined calibration on the vehicle-mounted camera and the laser radar to obtain an internal reference matrix and an external reference matrix of the vehicle-mounted camera.
3. The method according to claim 1, wherein the detecting the road surface image based on the image segmentation algorithm to obtain image features and an image road surface detection result comprises:
constructing an image segmentation algorithm model;
performing data enhancement through a training set and training the image segmentation algorithm model;
and obtaining the image characteristic graph and the image pavement detection result of the pavement image through the trained image segmentation algorithm model.
4. The method of claim 3, wherein the image segmentation algorithm model comprises a feature extraction unit, a fusion unit, and a prediction unit;
the feature extraction unit comprises a plurality of convolution layers, pooling layers and activation functions which are connected in series and is used for calculating the features of the road surface image, down-sampling the image and outputting image feature maps with different scales;
the fusion unit comprises a plurality of convolution layers, a pooling layer and an activation function which are connected in series and is used for fusing the image feature maps with different scales output by the feature extraction unit and recovering the resolution of the feature maps;
the prediction unit is used for predicting the road surface area in the road surface image according to the image feature map output by the fusion unit to obtain the image road surface detection result.
5. The method of claim 4, wherein the data enhancement mode comprises: random scaling of images, random flipping of images, random cropping of images, random adjustment of image brightness, random adjustment of image contrast, and random adjustment of image saturation.
6. The method of claim 1, wherein the screening the laser point cloud based on a point cloud segmentation algorithm to obtain a point cloud road surface detection result comprises:
determining a plane model through a plurality of laser points aiming at each frame of laser point cloud;
determining the number of laser points in the plane model according to the distance between the laser points and the plane model;
and determining an optimal model according to the number of the laser points, and determining the point cloud pavement detection result by using the optimal model.
7. The method of claim 2, wherein the fusing the point cloud road surface detection result with the image road surface detection result to obtain a fused road surface detection area comprises:
mapping the point cloud pavement detection result to an image through an internal reference matrix and an external reference matrix of the vehicle-mounted camera, and obtaining dense point cloud in a pixel coordinate system by using a point cloud enhancement method;
and fusing the dense point cloud and the image pavement detection result to obtain a data-fused pavement detection area.
8. The method according to claim 1, wherein the early warning of the vehicle according to the detection result of the dry and wet state of the road surface and the motion track and speed of the vehicle comprises:
and calculating the probability of the vehicle passing through a road surface slippery area and the probability of the vehicle passing through the slippery area and the speed being greater than the threshold speed, and early warning the vehicle according to the probability and the probability threshold.
9. The utility model provides a road surface slippery area detects and early warning device based on multimode data fusion which characterized in that includes:
the acquisition module is used for acquiring a road surface image and laser point cloud through a calibrated vehicle-mounted camera and a calibrated laser radar;
the first detection module is used for detecting the road surface image based on an image segmentation algorithm to obtain an image characteristic diagram and an image road surface detection result;
the second detection module is used for screening the laser point cloud based on a point cloud segmentation algorithm to obtain a point cloud pavement detection result;
the fusion module is used for fusing the point cloud pavement detection result with the image pavement detection result to obtain a fused pavement detection area;
the third detection module is used for fusing the laser point cloud reflection intensity information of the pavement detection area with the image characteristics, increasing the correlation degree of similar pixel point characteristics by using a pixel point bidirectional matching method, and obtaining a pavement dry-wet state detection result by using a pavement dry-wet state segmentation network;
the prediction module is used for acquiring vehicle motion state information, constructing a vehicle motion equation and predicting vehicle motion track and speed based on a particle filter algorithm;
and the early warning module is used for early warning the vehicle according to the road surface dry and wet state detection result and the vehicle motion track and speed.
10. The apparatus of claim 9, further comprising: and the calibration module is used for carrying out combined calibration on the vehicle-mounted camera and the laser radar to obtain an internal parameter matrix and an external parameter matrix of the vehicle-mounted camera.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110885654.7A CN113569778A (en) | 2021-08-03 | 2021-08-03 | Pavement slippery area detection and early warning method based on multi-mode data fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110885654.7A CN113569778A (en) | 2021-08-03 | 2021-08-03 | Pavement slippery area detection and early warning method based on multi-mode data fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113569778A true CN113569778A (en) | 2021-10-29 |
Family
ID=78170207
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110885654.7A Pending CN113569778A (en) | 2021-08-03 | 2021-08-03 | Pavement slippery area detection and early warning method based on multi-mode data fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113569778A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114030355A (en) * | 2021-11-15 | 2022-02-11 | 智己汽车科技有限公司 | Vehicle control method and device, vehicle and medium |
| CN114155415A (en) * | 2021-12-07 | 2022-03-08 | 华东交通大学 | Multi-data fusion vehicle detection method, system, equipment and storage medium |
| CN114842438A (en) * | 2022-05-26 | 2022-08-02 | 重庆长安汽车股份有限公司 | Terrain detection method, system and readable storage medium for autonomous driving vehicle |
-
2021
- 2021-08-03 CN CN202110885654.7A patent/CN113569778A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114030355A (en) * | 2021-11-15 | 2022-02-11 | 智己汽车科技有限公司 | Vehicle control method and device, vehicle and medium |
| CN114155415A (en) * | 2021-12-07 | 2022-03-08 | 华东交通大学 | Multi-data fusion vehicle detection method, system, equipment and storage medium |
| CN114155415B (en) * | 2021-12-07 | 2024-05-03 | 华东交通大学 | Multi-data fusion vehicle detection method, system, equipment and storage medium |
| CN114842438A (en) * | 2022-05-26 | 2022-08-02 | 重庆长安汽车股份有限公司 | Terrain detection method, system and readable storage medium for autonomous driving vehicle |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10949684B2 (en) | Vehicle image verification | |
| US10853670B2 (en) | Road surface characterization using pose observations of adjacent vehicles | |
| CN111442776B (en) | Method and equipment for sequential ground scene image projection synthesis and complex scene reconstruction | |
| CN103176185B (en) | Method and system for detecting road barrier | |
| US8670592B2 (en) | Clear path detection using segmentation-based method | |
| EP2574958B1 (en) | Road-terrain detection method and system for driver assistance systems | |
| CN113569778A (en) | Pavement slippery area detection and early warning method based on multi-mode data fusion | |
| US8699754B2 (en) | Clear path detection through road modeling | |
| EP1780675B1 (en) | Object detector | |
| US8332134B2 (en) | Three-dimensional LIDAR-based clear path detection | |
| US8605947B2 (en) | Method for detecting a clear path of travel for a vehicle enhanced by object detection | |
| US8452053B2 (en) | Pixel-based texture-rich clear path detection | |
| Labayrade et al. | In-vehicle obstacles detection and characterization by stereovision | |
| CN110531376A (en) | Detection of obstacles and tracking for harbour automatic driving vehicle | |
| CN104573646A (en) | Detection method and system, based on laser radar and binocular camera, for pedestrian in front of vehicle | |
| CN108638999A (en) | A kind of collision early warning system and method for looking around input based on 360 degree | |
| WO2005086078A1 (en) | Method and apparatus for classifying an object | |
| CN108960083B (en) | Automatic driving target classification method and system based on multi-sensor information fusion | |
| CN110659548B (en) | Vehicle and target detection method and device thereof | |
| CN107209998A (en) | Lane detection device | |
| CN114291011A (en) | Vehicle neural network | |
| CN113119977A (en) | Method and device for evaluating a vehicle driving surface | |
| JP4956099B2 (en) | Wall detector | |
| KR20180075985A (en) | Apparatus for autonomous steering prediction considering driving environment and method thereof | |
| CN116587978A (en) | Collision early warning method and system based on vehicle-mounted display screen |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |