CN116990776A - Laser radar point cloud compensation method and device, electronic equipment and storage medium - Google Patents
Laser radar point cloud compensation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116990776A CN116990776A CN202310993146.XA CN202310993146A CN116990776A CN 116990776 A CN116990776 A CN 116990776A CN 202310993146 A CN202310993146 A CN 202310993146A CN 116990776 A CN116990776 A CN 116990776A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- cloud data
- data
- laser
- dynamic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 77
- 230000003068 static effect Effects 0.000 claims abstract description 83
- 238000005259 measurement Methods 0.000 claims abstract description 50
- 238000012545 processing Methods 0.000 claims abstract description 35
- 238000001514 detection method Methods 0.000 claims abstract description 23
- 230000008569 process Effects 0.000 claims description 19
- 239000011159 matrix material Substances 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 13
- 238000013519 translation Methods 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 6
- 230000008447 perception Effects 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000001360 synchronised effect Effects 0.000 claims description 4
- 238000004590 computer program Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 10
- 230000000694 effects Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 4
- 230000004927 fusion Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/4802—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/497—Means for monitoring or calibrating
Abstract
The application discloses a laser radar point cloud compensation method, a device, electronic equipment and a storage medium, wherein the method comprises the steps of acquiring laser point cloud data to be processed, wherein the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar, a millimeter wave radar and an IMU inertial measurement unit which are also carried on the vehicle are subjected to joint calibration and time synchronization; performing point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data; and adopting detection results in the millimeter wave radar or the IMU inertial measurement unit to respectively carry out compensation processing on the static point cloud data and the dynamic point cloud data, and taking the result as a point cloud compensation result of the laser radar. The method and the device can realize that different compensation strategies are adopted for the dynamic target point cloud and the static target point cloud in the laser radar point cloud data, so that the point cloud in the whole scene can be compensated.
Description
Technical Field
The application relates to the technical field of automatic driving, in particular to a laser radar point cloud compensation method and device, electronic equipment and a storage medium.
Background
The intelligent driving vehicle is added with advanced sensors (such as radar, camera shooting), controllers, actuators and other devices on the basis of a common vehicle, and integrates network technology and communication technology, and the vehicle-vehicle, vehicle-road and vehicle-person information sharing and intelligent information exchange are realized through a vehicle-mounted sensing system and an information terminal, so that the vehicle can have the functions of intelligent perception, intelligent decision and the like.
In the related art, for dynamic compensation of laser radar point cloud data, two technical paths are generally as follows:
a. the method comprises the steps of recording a plurality of groups of self-vehicle speeds in the process of scanning the point cloud, and performing point cloud motion compensation according to the self-vehicle speeds;
b. and recording a plurality of groups of measurement speeds of the dynamic target while recording the point cloud, and performing point cloud motion compensation according to the corresponding measurement speeds of the point cloud.
The two technical paths can be used better corresponding to respective application scenes, but motion compensation for all point clouds in a laser radar view field under the full scene cannot be completed. Therefore, a motion compensation solution which can be widely applied and can aim at all point clouds in a laser radar view field in a full scene is lacking.
Disclosure of Invention
The embodiment of the application provides a laser radar point cloud compensation method and device, electronic equipment and a storage medium, so as to realize that different compensation strategies can be respectively adopted for dynamic target point cloud and static target point cloud in laser radar point cloud data.
The embodiment of the application adopts the following technical scheme:
in a first aspect, an embodiment of the present application provides a laser radar point cloud compensation method, where the method includes:
acquiring laser point cloud data to be processed, wherein the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar and a millimeter wave radar and an IMU inertial measurement unit also carried on the vehicle are subjected to joint calibration and time synchronization;
performing point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data;
and adopting detection results in the millimeter wave radar or the IMU inertial measurement unit to respectively carry out compensation processing on the static point cloud data and the dynamic point cloud data, and taking the result as a point cloud compensation result of the laser radar.
In some embodiments, the compensation process includes:
And compensating and correcting the position offset of the point cloud caused by the scanning time difference of the laser radar in the motion process of the target object in the perception range of the laser radar during the motion of the vehicle.
In some embodiments, the performing point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data includes:
determining dynamic target motion state parameters in the identification result of the millimeter wave radar;
according to the dynamic target position in the dynamic target motion state parameter as a reference, carrying out space region division on the laser point cloud data to obtain a plurality of sub-regions;
dividing the laser point cloud by a preset clustering algorithm for each sub-area according to the obtained sub-areas, clustering the dynamic target points to obtain dynamic point clouds, and simultaneously obtaining the relative speed between the dynamic target identified by the millimeter wave radar and the vehicle;
and marking the rest point clouds which are not clustered as static point cloud data.
In some embodiments, the compensating the static point cloud data and the dynamic point cloud data respectively, as a result of point cloud compensation of the lidar, includes:
According to the self-vehicle pose change data in the data frame of the IMU inertial measurement unit, which are matched in advance, a rotation matrix and a translation matrix corresponding to laser point cloud data under different time stamps are calculated, and the data marked as the static point cloud are compensated.
In some embodiments, the compensating the static point cloud data and the dynamic point cloud data respectively, as a result of point cloud compensation of the lidar, includes:
based on the dynamic point cloud set, calculating a rotation matrix and a translation matrix corresponding to the laser point cloud under different time stamps according to the speed measurement result of the millimeter wave radar identification target, and dynamically compensating the data marked as the dynamic point cloud.
In some embodiments, the compensating the static point cloud data and the dynamic point cloud data with the detection result in the millimeter wave radar or the IMU inertial measurement unit, as a point cloud compensation result of the laser radar, includes:
the self-vehicle attitude difference value is adopted to carry out compensation processing on the static point cloud data, and the static point cloud data is used as a static point cloud compensation result of the laser radar;
and/or the number of the groups of groups,
and carrying out compensation processing on the dynamic point cloud data by adopting a dynamic speed difference value to serve as a dynamic point cloud compensation result of the laser radar.
In some embodiments, the acquiring laser point cloud data to be processed, where the laser point cloud data is acquired by a laser radar mounted on a vehicle, and the laser radar is jointly calibrated and time-synchronized with a millimeter wave radar and an IMU inertial measurement unit that are also mounted on the vehicle, includes:
the laser radar LiDAR, the millimeter wave radar and the IMU inertial measurement unit are mounted on the vehicle;
acquiring coordinate conversion parameters of the IMU-LiDAR-radar under the same coordinate system, and completing unification of the space coordinates of each sensor;
after the acquisition clocks of various sensors are synchronized to the same time reference, IMU data, liDAR data and radar data are synchronously acquired in real time in the realization scenes of different automatic driving functions;
and carrying out data frame matching on the acquired IMU data, liDAR data and radar data to acquire laser point cloud data to be processed.
In a second aspect, an embodiment of the present application further provides a laser radar point cloud compensation device, where the device includes:
the acquisition module is used for acquiring laser point cloud data to be processed, the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar is jointly calibrated and time-synchronized with a millimeter wave radar and an IMU inertial measurement unit which are also carried on the vehicle;
The point cloud classification module is used for carrying out point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data;
and the point cloud compensation module is used for respectively carrying out compensation processing on the static point cloud data and the dynamic point cloud data by adopting the detection result in the millimeter wave radar or the IMU inertial measurement unit, and taking the static point cloud data and the dynamic point cloud data as the point cloud compensation result of the laser radar.
In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the above method.
In a fourth aspect, embodiments of the present application also provide a computer-readable storage medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the above-described method.
The above at least one technical scheme adopted by the embodiment of the application can achieve the following beneficial effects: by acquiring laser point cloud data to be processed, the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar and a millimeter wave radar and an IMU inertial measurement unit also carried on the vehicle are subjected to joint calibration and time synchronization. In this way, an association relationship can be established among the millimeter wave radar, the IMU inertial measurement unit and the laser radar. And then, carrying out point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data. And finally, adopting detection results in the millimeter wave radar or the IMU inertial measurement unit to respectively compensate the static point cloud data and the dynamic point cloud data. By the method, the aim of panoramic dynamic compensation of the laser radar point cloud can be achieved by adopting the sensor which is frequently used in an actual scene instead of a specific sensor.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
FIG. 1 is a schematic flow chart of a method for compensating a laser radar point cloud in an embodiment of the application;
fig. 2 is a schematic diagram of an implementation principle of a laser radar point cloud compensation method in an embodiment of the present application;
FIG. 3 is a schematic diagram of a laser radar point cloud compensation device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The inventor finds that the point cloud dynamic compensation scheme based on the laser radar mainly comprises the following steps:
The technical path 1 comprises the steps of firstly, synchronizing a time axis and unifying a space coordinate system of a laser radar and an IMU (inertial measurement unit) for realizing the method, and respectively sequencing three-dimensional point cloud data and IMU data of the laser radar according to a time stamp sequence; secondly, providing a laser radar three-dimensional point cloud rotation compensation method based on data block division, carrying out data block division on each frame of laser radar three-dimensional point cloud data according to an IMU output laser radar three-dimensional point cloud data time sequence, obtaining a three-axis rotation transformation matrix R of each data point relative to a frame tail according to the data block division thought, and carrying out three-axis rotation compensation on the laser radar three-dimensional point cloud data; and finally, estimating the quantity T of the inter-frame motion of the point cloud according to the point cloud data frame subjected to the rotation compensation, and carrying out translation compensation on the point cloud data. According to research, when the technical path 1 adopts the vehicle speed and position information to perform motion compensation, the compensation effect is suitable for a static target. In an actual road test environment, the static targets comprise road points, signboards, road static traffic participants, roadside environment information and the like. Therefore, the actual compensation effect of the technical path 1 on the dynamic target, such as the dynamic traffic participants of the running vehicle, is not good, and the appearance size information of the dynamic target cannot be accurately determined.
When the technical path 2 adopts the speed of the detected target point to carry out the point cloud motion compensation, the method is also divided into several methods:
method 1: when the point cloud is acquired, speed information of the point is acquired incidentally, and the speed information is used for carrying out motion compensation correction of the point cloud. The method depends on the specific sensor type, and the application scope is not wide enough, and meanwhile, the performance of the method in various scenes such as turning, jolting and the like is not better than that of motion compensation by using the motion state information of the bicycle.
Method 2: and capturing motion information of the dynamic target by using other speed measuring sensors, and performing motion compensation after fusing the motion information with the point cloud. This approach focuses only on dynamic compensation of specific moving objects, essentially giving up dynamic compensation of static objects including road points, signs, road static traffic participants, roadside environmental information, etc.
Method 3: placing inertial sensors on a target vehicle to record data, so as to dynamically compensate specific target point clouds: the precision is higher, but application scope is smaller, can't use under the condition that only this car is equipped with the sensor.
In summary, the technical problem in the related art is that there is a lack of a motion compensation algorithm that can be widely applied, i.e. does not need a specific sensor, and can be aimed at all point clouds in the laser radar view field in the full scene.
Aiming at the defects in the related art, the laser radar point cloud compensation method in the embodiment of the application utilizes a multi-sensor fusion algorithm to carry out information complementation on the point cloud, thereby achieving the purpose of carrying out dynamic compensation on a dynamic target and a static target point cloud in various driving scene applications, correcting the position offset of the point cloud caused by the laser radar scanning time difference in the motion process of a self-vehicle and the motion process of a target object in a laser radar sensing range so as to improve the fusion effect in the fusion process before sensing, further improve the segmentation accuracy and finally realize the improvement of the target detection accuracy.
The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.
The embodiment of the application provides a laser radar point cloud compensation method, as shown in fig. 1, and provides a schematic flow chart of the laser radar point cloud compensation method in the embodiment of the application, wherein the method at least comprises the following steps S110 to S130:
step S110, laser point cloud data to be processed are acquired, wherein the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar is jointly calibrated and time-synchronized with a millimeter wave radar and an IMU inertial measurement unit which are also carried on the vehicle.
In a specific scenario, a vehicle is often equipped with various sensors to realize a perceived positioning function in an automatic driving system. The various sensors on the vehicle include, but are not limited to, lidar, millimeter wave radar, cameras, and IMU inertial measurement units, among others.
It will be appreciated that in embodiments of the present application, the type or number of specific sensors on the vehicle are not specifically limited, and those skilled in the art may build them according to the actual scenario.
It should be noted that the lidar, the millimeter wave radar and the IMU inertial measurement unit need to be unified into the same world coordinate system, for example, a vehicle body coordinate system.
In addition, the laser radar, the millimeter wave radar and the IMU inertial measurement unit can be calibrated in advance in a combined mode, and meanwhile time stamps of the multiple sensors are unified.
And step S120, carrying out point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data.
And further performing point cloud classification processing according to the laser point cloud data obtained in the steps. And static point cloud data and dynamic point cloud data can be obtained after the point cloud classification processing. It should be noted that when performing laser point cloud classification, it is possible to distinguish whether it belongs to a dynamic point cloud or a static point cloud by means of the moving speed of the point cloud or the target.
By the point cloud classification, different compensation strategies can be respectively adopted for the dynamic target point cloud and the static target point cloud, so that good compensation effects can be achieved for the point clouds corresponding to the dynamic target and the static target.
And step S130, adopting detection results in the millimeter wave radar or the IMU inertial measurement unit to respectively carry out compensation processing on the static point cloud data and the dynamic point cloud data, and taking the result as a point cloud compensation result of the laser radar.
And respectively carrying out compensation processing on the obtained static point cloud data and dynamic point cloud data by adopting a detection result in the millimeter wave radar or a detection result in the IMU inertial measurement unit. The processed result is used as a point cloud compensation result of the laser radar.
In addition, data fusion can be performed after compensation, so that target recognition can be better realized.
And correcting the static point cloud data based on the point cloud position offset caused by the laser radar scanning time difference, and compensating by combining with the vehicle pose information. Based on pose information of the vehicle, the position offset situation of the point cloud which possibly occurs can be corrected because the position offset situation is static point cloud data. In particular, road points, signs, static road traffic participants, and roadside environmental information may be compensated for.
And correcting the dynamic point cloud data based on the point cloud position offset caused by the laser radar scanning time difference, and compensating by combining the dynamic speed. Based on the dynamic speed, the point cloud position offset situation which may occur can be corrected because of the dynamic point cloud data.
By adopting the method, the static point cloud data and the dynamic point cloud data are respectively compensated by adopting the detection results in the millimeter wave radar or the IMU inertial measurement unit. The sensor with a specific model is not needed, and the sensor widely applied in practice can be applied to achieve the purpose of dynamically compensating the laser radar point cloud.
By adopting the method, the laser point cloud data is subjected to point cloud classification according to the identification result in the millimeter wave radar, so that static point cloud data and dynamic point cloud data are obtained. By means of point cloud classification, different compensation strategies are respectively adopted for the dynamic target point cloud and the static target point cloud, and good compensation effects can be achieved for the corresponding point clouds of the dynamic target and the static target.
By adopting the method, the detection results in the millimeter wave radar or the IMU inertial measurement unit are respectively subjected to compensation processing on the static point cloud data and the dynamic point cloud data, and the result is used as a point cloud compensation result of the laser radar. The high-frequency inertial navigation IMU is used for acquiring the accurate pose of the vehicle, and the data is used for carrying out point cloud compensation, so that the vehicle has good effect under certain special scenes such as jolt, turning and the like compared with the prior art.
Unlike the related art, there is a lack of a motion compensation solution that can be widely used, in a full-field scenario, for all point clouds within the lidar view. By adopting the method, the aim of panoramic dynamic compensation of the laser radar point cloud can be realized by adopting the sensor which is frequently used in the actual scene instead of a specific sensor.
In one embodiment of the present application, the compensation process includes: and compensating and correcting the position offset of the point cloud caused by the scanning time difference of the laser radar in the motion process of the target object in the perception range of the laser radar during the motion of the vehicle.
And for compensation processing, the relevant point cloud position offset caused by the scanning time difference of the laser radar is compensated and corrected mainly in the motion process of the target object in the perception range of the laser radar during the motion of the vehicle. Considering that the vehicle moves and the target object moves in the laser radar sensing range, the position change of the point cloud can be generated at the moment, and the position is corrected in a compensation mode.
In one embodiment of the present application, the performing point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data includes: determining dynamic target motion state parameters in the identification result of the millimeter wave radar; according to the dynamic target position in the dynamic target motion state parameter as a reference, carrying out space region division on the laser point cloud data to obtain a plurality of sub-regions; dividing the laser point cloud by a preset clustering algorithm for each sub-area according to the obtained sub-areas, clustering the dynamic target points to obtain dynamic point clouds, and simultaneously obtaining the relative speed between the dynamic target identified by the millimeter wave radar and the vehicle; and marking the rest point clouds which are not clustered as static point cloud data.
It should be noted that, in the processing of the point cloud classification, detection processing is performed on the laser point cloud data of each frame. The following is a detailed description of a certain frame (current frame).
Firstly, acquiring a dynamic target position in a dynamic target motion state parameter according to the millimeter wave radar identification result in a current frame. The millimeter wave radar can measure the speed of the vehicle relative to the point cloud, so as to judge whether the vehicle belongs to a dynamic target. The dynamic object motion state parameters at least comprise the moving speed of the object, the position of the object and the like.
Further, space region division is carried out on all point cloud data in the current frame by taking the dynamic target position as a reference, and the obtained point cloud is divided by adopting a Euclidean clustering algorithm for each sub-region. And clustering results (a plurality of subareas) of the point clouds in the range taking the dynamic target position as a reference (circle center) in the current frame can be obtained according to the corresponding clustering results in each subarea.
It will be appreciated that the euclidean clustering algorithm is merely exemplary, and that one skilled in the art may choose the algorithm based on actual use.
And finally, clustering the dynamic target points to obtain a dynamic point cloud set P. Meanwhile, the relative speed dV between the dynamic target identified by the millimeter wave radar and the vehicle is obtained from the dynamic point cloud set P. The remaining point clouds are marked as static after the dynamic point clouds are obtained.
The clustering process for the dynamic target point can determine whether to cluster according to the position of the dynamic point cloud in each sub-area.
In one embodiment of the present application, the compensating process for the static point cloud data and the dynamic point cloud data respectively includes: according to the self-vehicle pose change data in the data frame of the IMU inertial measurement unit which is matched in advance, a rotation matrix and a translation matrix corresponding to laser point cloud data under different time stamps are calculated, and the data marked as the static point cloud (marked point cloud data) are compensated.
In the implementation, when the static point cloud processing is performed, according to the self-vehicle pose change data in the previous matched IMU inertial navigation data frame (matched with the millimeter wave radar and the laser radar), the corresponding rotation matrix and translation matrix (corresponding to the time difference caused by laser radar scanning) of different time stamp point clouds are calculated, and the point clouds marked as static are compensated.
In one embodiment of the present application, the compensating process for the static point cloud data and the dynamic point cloud data respectively includes: based on the dynamic point cloud set, according to the speed measurement result of the millimeter wave radar identification target, calculating a rotation matrix and a translation matrix corresponding to the laser point cloud under different time stamps, and dynamically compensating the data marked as the dynamic point cloud (marked point cloud data).
In the specific implementation, when the dynamic point cloud processing is performed, the dynamic point cloud collection obtained in the steps and the velocity measurement value dV of the radar millimeter wave radar on the dynamic point cloud are calculated, and the dynamic compensation is performed on the point cloud by corresponding rotation matrixes and translation matrixes (corresponding to time differences caused by laser radar scanning) of the point cloud with different time stamps.
In one embodiment of the present application, the compensating the static point cloud data and the dynamic point cloud data with the detection result in the millimeter wave radar or the IMU inertial measurement unit, as a point cloud compensation result of the laser radar, includes: the self-vehicle attitude difference value is adopted to carry out compensation processing on the static point cloud data, and the static point cloud data is used as a static point cloud compensation result of the laser radar; and/or adopting the dynamic speed difference value to carry out compensation processing on the dynamic point cloud data to serve as a dynamic point cloud compensation result of the laser radar.
As shown in fig. 3, the self-vehicle pose difference value is adopted to compensate the static point cloud data, and the self-vehicle pose interpolation is adopted as a static point cloud compensation result of the laser radar. And carrying out compensation processing on the dynamic point cloud data by adopting a dynamic speed difference value, and taking dynamic speed interpolation of different areas as a dynamic point cloud compensation result of the laser radar.
In one embodiment of the present application, the acquiring laser point cloud data to be processed, where the laser point cloud data is acquired by a laser radar mounted on a vehicle, and the laser radar is jointly calibrated and time-synchronized with a millimeter wave radar and an IMU inertial measurement unit that are also mounted on the vehicle, includes: the laser radar LiDAR, the millimeter wave radar and the IMU inertial measurement unit are mounted on the vehicle; acquiring coordinate conversion parameters of the IMU-LiDAR-radar under the same coordinate system, and completing unification of the space coordinates of each sensor; after the acquisition clocks of various sensors are synchronized to the same time reference, IMU data, liDAR data and radar data are synchronously acquired in real time in the realization scenes of different automatic driving functions; and carrying out data frame matching on the acquired IMU data, liDAR data and radar data to acquire laser point cloud data to be processed.
Before the point cloud classification is performed, test vehicles, calibration, time synchronization, etc. also need to be built for multiple sensors (LiDAR, radar, and IMU). After calibration and time synchronization, data acquisition, data frame matching and other processes can be started, so that the subsequent point cloud classification process is facilitated.
a. Building a test vehicle: the laser radar, the millimeter wave radar, the IMU, the camera, the industrial personal computer server and the like are additionally arranged on the test vehicle, and the detection range (ROI: region of interest) meeting the sensing requirement of the automatic driving function is required to be covered by various sensors so as to ensure that each laser radar point cloud has description information from other sensors.
b. Calibrating: and acquiring IMU-LiDAR-radar coordinate conversion parameters, and completing unification of the space coordinates of each sensor.
c. Time synchronization: and synchronizing the acquisition clocks of the various sensors to the same time reference so as to avoid point cloud parameter errors caused by the difference of the acquisition clock references of the various sensors.
d. And (3) data acquisition: IMU data, liDAR data and radar data are synchronously collected in real time in a typical implementation scene of various automatic driving functions.
e. Data frame matching: and sequencing the laser radar point clouds belonging to the same point cloud frame according to the time stamps respectively to obtain a minimum time stamp Tmin and a maximum time stamp Tmax of each point cloud frame, wherein the time period covered by the laser radar point cloud frame is Tmin-Tmax. Screening all acquired IMU data frames, and if the time stamp IMU_Tn of the IMU data of the nth frame accords with the time stamp IMU_Tn less than or equal to Tmin less than or equal to Tmax, considering that the IMU data frame is matched with the laser radar point cloud frame, wherein the number of the IMU frames matched with each point cloud frame can be one or more.
Further, a dynamic target list perceived by the millimeter wave radar is screened, if the m detection time MO_Tm of the dynamic target accords with Tmin and is smaller than or equal to MO_Tm and smaller than or equal to Tmax, the motion state parameter of the dynamic target at the moment MO_Tm is considered to be matched with the point cloud frame of the laser radar, and the motion parameters of the dynamic target matched with each point cloud frame can be zero, one or a plurality of motion parameters of the dynamic target, and the embodiment of the application is not particularly limited.
The embodiment of the application also provides a laser radar point cloud compensation device 200, as shown in fig. 2, and provides a schematic structural diagram of the laser radar point cloud compensation device in the embodiment of the application, where the laser radar point cloud compensation device 200 at least includes: an acquisition module 210, a point cloud classification module 220, and a point cloud compensation module 230, wherein:
in one embodiment of the present application, the obtaining module 210 is specifically configured to: and acquiring laser point cloud data to be processed, wherein the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar and a millimeter wave radar and an IMU inertial measurement unit also carried on the vehicle are subjected to joint calibration and time synchronization.
In a specific scenario, a vehicle is often equipped with various sensors to realize a perceived positioning function in an automatic driving system. The various sensors on the vehicle include, but are not limited to, lidar, millimeter wave radar, cameras, and IMU inertial measurement units, among others.
It will be appreciated that in embodiments of the present application, the type or number of specific sensors on the vehicle are not specifically limited, and those skilled in the art may build them according to the actual scenario.
It should be noted that the lidar, the millimeter wave radar and the IMU inertial measurement unit need to be unified into the same world coordinate system, for example, a vehicle body coordinate system.
In addition, the laser radar, the millimeter wave radar and the IMU inertial measurement unit can be calibrated in advance in a combined mode, and meanwhile time stamps of the multiple sensors are unified.
In one embodiment of the present application, the point cloud classification module 220 is specifically configured to: and carrying out point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data.
And further performing point cloud classification processing according to the laser point cloud data obtained in the steps. And static point cloud data and dynamic point cloud data can be obtained after the point cloud classification processing. It should be noted that when performing laser point cloud classification, it is possible to distinguish whether it belongs to a dynamic point cloud or a static point cloud by means of the moving speed of the point cloud or the target.
By the point cloud classification, different compensation strategies can be respectively adopted for the dynamic target point cloud and the static target point cloud, so that good compensation effects can be achieved for the point clouds corresponding to the dynamic target and the static target.
In one embodiment of the present application, the point cloud compensation module 230 is specifically configured to: and adopting detection results in the millimeter wave radar or the IMU inertial measurement unit to respectively carry out compensation processing on the static point cloud data and the dynamic point cloud data, and taking the result as a point cloud compensation result of the laser radar.
And respectively carrying out compensation processing on the obtained static point cloud data and dynamic point cloud data by adopting a detection result in the millimeter wave radar or a detection result in the IMU inertial measurement unit. The processed result is used as a point cloud compensation result of the laser radar.
In addition, data fusion can be performed after compensation, so that target recognition can be better realized.
And correcting the static point cloud data based on the point cloud position offset caused by the laser radar scanning time difference, and compensating by combining with the vehicle pose information. Based on pose information of the vehicle, the position offset situation of the point cloud which possibly occurs can be corrected because the position offset situation is static point cloud data.
And correcting the dynamic point cloud data based on the point cloud position offset caused by the laser radar scanning time difference, and compensating by combining the dynamic speed. Based on the dynamic speed, the point cloud position offset situation which may occur can be corrected because of the dynamic point cloud data.
It can be understood that the above-mentioned laser radar point cloud compensation device can implement each step of the laser radar point cloud compensation method provided in the foregoing embodiment, and the relevant explanation about the transaction checking method is applicable to the laser radar point cloud compensation device, which is not described herein again.
Fig. 4 is a schematic structural view of an electronic device according to an embodiment of the present application. Referring to fig. 4, at the hardware level, the electronic device includes a processor, and optionally an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus, or EISA (Extended Industry Standard Architecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 4, but not only one bus or type of bus.
And the memory is used for storing programs. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.
The processor reads the corresponding computer program from the nonvolatile memory to the memory and then runs the computer program to form the laser radar point cloud compensation device on the logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:
acquiring laser point cloud data to be processed, wherein the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar and a millimeter wave radar and an IMU inertial measurement unit also carried on the vehicle are subjected to joint calibration and time synchronization;
performing point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data;
and adopting detection results in the millimeter wave radar or the IMU inertial measurement unit to respectively carry out compensation processing on the static point cloud data and the dynamic point cloud data, and taking the result as a point cloud compensation result of the laser radar.
The method executed by the laser radar point cloud compensation apparatus disclosed in the embodiment of fig. 1 of the present application may be applied to a processor or implemented by the processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.
The electronic device may also execute the method executed by the laser radar point cloud compensation device in fig. 1, and implement the function of the laser radar point cloud compensation device in the embodiment shown in fig. 1, which is not described herein.
The embodiment of the application also proposes a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device comprising a plurality of application programs, enable the electronic device to perform a method performed by the lidar point cloud compensation device in the embodiment shown in fig. 1, and in particular to perform:
acquiring laser point cloud data to be processed, wherein the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar and a millimeter wave radar and an IMU inertial measurement unit also carried on the vehicle are subjected to joint calibration and time synchronization;
performing point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data;
and adopting detection results in the millimeter wave radar or the IMU inertial measurement unit to respectively carry out compensation processing on the static point cloud data and the dynamic point cloud data, and taking the result as a point cloud compensation result of the laser radar.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.
Claims (10)
1. A method of lidar point cloud compensation, wherein the method comprises:
acquiring laser point cloud data to be processed, wherein the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar and a millimeter wave radar and an IMU inertial measurement unit also carried on the vehicle are subjected to joint calibration and time synchronization;
Performing point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data;
and adopting detection results in the millimeter wave radar or the IMU inertial measurement unit to respectively carry out compensation processing on the static point cloud data and the dynamic point cloud data, and taking the result as a point cloud compensation result of the laser radar.
2. The method of claim 1, wherein the compensation process comprises:
and compensating and correcting the position offset of the point cloud caused by the scanning time difference of the laser radar in the motion process of the target object in the perception range of the laser radar during the motion of the vehicle.
3. The method of claim 2, wherein the performing the point cloud classification on the laser point cloud data according to the recognition result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data includes:
determining dynamic target motion state parameters in the identification result of the millimeter wave radar;
according to the dynamic target position in the dynamic target motion state parameter as a reference, carrying out space region division on the laser point cloud data to obtain a plurality of sub-regions;
Dividing the laser point cloud by a preset clustering algorithm for each sub-area according to the obtained sub-areas, clustering the dynamic target points to obtain dynamic point clouds, and simultaneously obtaining the relative speed between the dynamic target identified by the millimeter wave radar and the vehicle;
and marking the rest point clouds which are not clustered as static point cloud data.
4. The method of claim 3, wherein the compensating the static point cloud data and the dynamic point cloud data respectively as a result of point cloud compensation of the lidar comprises:
according to the self-vehicle pose change data in the data frame of the IMU inertial measurement unit, which are matched in advance, a rotation matrix and a translation matrix corresponding to laser point cloud data under different time stamps are calculated, and the data marked as the static point cloud are compensated.
5. The method of claim 3, wherein the compensating the static point cloud data and the dynamic point cloud data respectively as a result of point cloud compensation of the lidar comprises:
based on the dynamic point cloud set, calculating a rotation matrix and a translation matrix corresponding to the laser point cloud under different time stamps according to the speed measurement result of the millimeter wave radar identification target, and dynamically compensating the data marked as the dynamic point cloud.
6. The method of claim 1, wherein the compensating the static point cloud data and the dynamic point cloud data with the detection result in the millimeter wave radar or the IMU inertial measurement unit, respectively, includes:
the self-vehicle attitude difference value is adopted to carry out compensation processing on the static point cloud data, and the static point cloud data is used as a static point cloud compensation result of the laser radar;
and/or the number of the groups of groups,
and carrying out compensation processing on the dynamic point cloud data by adopting a dynamic speed difference value to serve as a dynamic point cloud compensation result of the laser radar.
7. The method of claim 1, wherein the acquiring laser point cloud data to be processed, the laser point cloud data being acquired by a lidar mounted on a vehicle, and the lidar being jointly calibrated and time synchronized with a millimeter wave radar, an IMU inertial measurement unit, also mounted on the vehicle, comprises:
the laser radar LiDAR, the millimeter wave radar and the IMU inertial measurement unit are mounted on the vehicle;
acquiring coordinate conversion parameters of the IMU-LiDAR-radar under the same coordinate system, and completing unification of the space coordinates of each sensor;
After the acquisition clocks of various sensors are synchronized to the same time reference, IMU data, liDAR data and radar data are synchronously acquired in real time in the realization scenes of different automatic driving functions;
and carrying out data frame matching on the acquired IMU data, liDAR data and radar data to acquire laser point cloud data to be processed.
8. A lidar point cloud compensation device, wherein the device comprises:
the acquisition module is used for acquiring laser point cloud data to be processed, the laser point cloud data are acquired through a laser radar carried on a vehicle, and the laser radar is jointly calibrated and time-synchronized with a millimeter wave radar and an IMU inertial measurement unit which are also carried on the vehicle;
the point cloud classification module is used for carrying out point cloud classification on the laser point cloud data according to the identification result in the millimeter wave radar to obtain static point cloud data and dynamic point cloud data;
and the point cloud compensation module is used for respectively carrying out compensation processing on the static point cloud data and the dynamic point cloud data by adopting the detection result in the millimeter wave radar or the IMU inertial measurement unit, and taking the static point cloud data and the dynamic point cloud data as the point cloud compensation result of the laser radar.
9. An electronic device, comprising:
A processor; and
a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the method of any of claims 1 to 7.
10. A computer readable storage medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310993146.XA CN116990776A (en) | 2023-08-08 | 2023-08-08 | Laser radar point cloud compensation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310993146.XA CN116990776A (en) | 2023-08-08 | 2023-08-08 | Laser radar point cloud compensation method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116990776A true CN116990776A (en) | 2023-11-03 |
Family
ID=88529930
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310993146.XA Pending CN116990776A (en) | 2023-08-08 | 2023-08-08 | Laser radar point cloud compensation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116990776A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117612070A (en) * | 2024-01-19 | 2024-02-27 | 福思(杭州)智能科技有限公司 | Static truth value data correction method and device and storage medium |
| CN117612070B (en) * | 2024-01-19 | 2024-05-03 | 福思(杭州)智能科技有限公司 | Static truth value data correction method and device and storage medium |
-
2023
- 2023-08-08 CN CN202310993146.XA patent/CN116990776A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117612070A (en) * | 2024-01-19 | 2024-02-27 | 福思(杭州)智能科技有限公司 | Static truth value data correction method and device and storage medium |
| CN117612070B (en) * | 2024-01-19 | 2024-05-03 | 福思(杭州)智能科技有限公司 | Static truth value data correction method and device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113447923A (en) | Target detection method, device, system, electronic equipment and storage medium | |
| CN115546315A (en) | Sensor on-line calibration method and device for automatic driving vehicle and storage medium | |
| CN112041767A (en) | System and method for synchronizing vehicle sensors and devices | |
| CN114705121A (en) | Vehicle pose measuring method and device, electronic equipment and storage medium | |
| CN114973198A (en) | Course angle prediction method and device of target vehicle, electronic equipment and storage medium | |
| CN117079238A (en) | Road edge detection method, device, equipment and storage medium | |
| CN116990776A (en) | Laser radar point cloud compensation method and device, electronic equipment and storage medium | |
| CN116164763A (en) | Target course angle determining method and device, electronic equipment and storage medium | |
| CN115841519A (en) | Calibration precision detection method, device and equipment of image acquisition equipment | |
| CN115661014A (en) | Point cloud data processing method and device, electronic equipment and storage medium | |
| CN114187328B (en) | Object detection method and device and electronic equipment | |
| CN113253256B (en) | Monitoring method and device based on sensor fusion equipment and sensor fusion equipment | |
| CN115112125A (en) | Positioning method and device for automatic driving vehicle, electronic equipment and storage medium | |
| CN115760636A (en) | Distortion compensation method, device and equipment for laser radar point cloud and storage medium | |
| CN113890668A (en) | Multi-sensor time synchronization method and device, electronic equipment and storage medium | |
| CN114755663A (en) | External reference calibration method and device for vehicle sensor and computer readable storage medium | |
| CN111753901A (en) | Data fusion method, device and system and computer equipment | |
| EP3811349A2 (en) | System and method for automatic calibration of vehicle position determining device in a traffic system | |
| CN116625384B (en) | Data association method and device and electronic equipment | |
| CN111664829A (en) | Correction method, correction device and computer storage medium | |
| WO2020053165A1 (en) | Method for determining at least one position parameter of an object in an environment of a vehicle, computer program product, driver assistance system and vehicle | |
| CN116228820B (en) | Obstacle detection method and device, electronic equipment and storage medium | |
| CN116935370A (en) | Cloud license plate recognition method and device, electronic equipment and storage medium | |
| CN116994179A (en) | Target tracking method and device and electronic equipment | |
| CN116610766A (en) | Map updating method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |