CN116400349A - Calibration method of low-resolution millimeter wave radar and optical camera - Google Patents
Calibration method of low-resolution millimeter wave radar and optical camera Download PDFInfo
- Publication number
- CN116400349A CN116400349A CN202211589130.4A CN202211589130A CN116400349A CN 116400349 A CN116400349 A CN 116400349A CN 202211589130 A CN202211589130 A CN 202211589130A CN 116400349 A CN116400349 A CN 116400349A
- Authority
- CN
- China
- Prior art keywords
- millimeter wave
- wave radar
- dimensional
- point cloud
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000003287 optical effect Effects 0.000 title claims abstract description 26
- 239000011159 matrix material Substances 0.000 claims abstract description 24
- 238000001914 filtration Methods 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 5
- 238000013519 translation Methods 0.000 claims description 7
- 238000005457 optimization Methods 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000005259 measurement Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 4
- 238000013139 quantization Methods 0.000 description 4
- 230000004927 fusion Effects 0.000 description 2
- 239000012535 impurity Substances 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
- G01S13/88—Radar or analogous systems specially adapted for specific applications
- G01S13/89—Radar or analogous systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
- G01S7/40—Means for monitoring or calibrating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10044—Radar image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Electromagnetism (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
The invention discloses a calibration method of a low-resolution millimeter wave radar and an optical camera, which is characterized in that single-frame radar point cloud information acquired by the millimeter wave radar is fused with pose information acquired by a pose sensor, and a point cloud map containing position information of a calibration plate is output; processing the point cloud map by utilizing a point cloud filtering algorithm and a clustering algorithm to extract millimeter wave Lei Dadian clouds of the calibration plate; dividing the position information of the two-dimensional edge characteristic points of the calibration plate in the original image information by using a target dividing algorithm; constructing a three-dimensional model through the structure priori of the calibration plate, registering the three-dimensional model with the millimeter wave radar point cloud block, so as to obtain the fitted three-dimensional edge characteristic point position information of the calibration plate, and matching the fitted three-dimensional edge characteristic point position information with the partitioned two-dimensional edge characteristic point position information to obtain a plurality of groups of two-dimensional and three-dimensional matching characteristic point pair information; and taking a plurality of sets of two-dimensional and three-dimensional matching characteristic point pair information as input of a PnP algorithm, and calculating to obtain an external parameter matrix for calibrating the millimeter wave radar and the camera.
Description
Technical Field
The invention relates to the technical field of external parameter calibration of multi-sensor fusion, in particular to a calibration method of a low-resolution millimeter wave radar and an optical camera.
Background
Three-dimensional environment perception is always the key of unmanned vehicle positioning, path planning navigation, three-dimensional reconstruction and other tasks, and in the task of three-dimensional environment perception, sensors such as a laser radar, a millimeter wave radar, an optical camera and the like play an important role. Compared to radar, optical cameras are receiving much attention because they can access dense scene RGB information, but neither monocular nor binocular cameras can access scene accurate three-dimensional information. Therefore, the method is often fused with other ranging sensors such as a laser radar in practical application, and is used for completing three-dimensional reconstruction, real-time positioning, mapping and other works with dimensions. In recent years, millimeter wave radars are highly focused by students at home and abroad because the millimeter wave radars can be used in special environments such as heavy fog, heavy rain and the like, and a plurality of related research results of the millimeter wave radars are generated. However, there are still many disadvantages of millimeter wave radars on the market compared to lidar, such as: the single frame point cloud data is sparse, low in precision, easy to interfere, multiple in impurity points and the like. This also makes the task of spatial calibration of millimeter wave radars with other sensors more difficult. The spatial calibration of multiple sensors is usually a key ring for completing the multi-source information fusion sensing algorithm.
The calibration work of the millimeter wave radar and the camera generally uses the speed, distance and horizontal azimuth angle information of the target measured by the radar to be matched with the image or image stream, for example, a method for realizing the registration of the millimeter wave radar and the camera of 360 degrees based on a strong reflection calibration object and adding geometric constraint in a calibration equation, a method for calibrating the millimeter wave radar and the camera by using the speed dimension information of the target measured by the radar and searching the corresponding target in an image sequence, and the like. However, the calibration work is mainly used for three-dimensional reconstruction of a large outdoor environment, the resolution of the millimeter wave radar adopted by the calibration work is higher, the calibration scheme does not have universality, in practical engineering, the price is limited, the millimeter wave radar with sparse data is adopted in most application scenes, and the calibration method cannot be used for completing the calibration of the millimeter wave radar and the camera in the space calibration work of the millimeter wave radar and the optical camera.
Disclosure of Invention
The invention aims to provide a calibration method of a low-resolution millimeter wave radar and an optical camera, which has high accuracy of a calibration result, and has effectiveness and robustness in a relatively open environment and an environment with more sundries, and has wide application fields.
In order to achieve the above purpose, the invention adopts the following technical scheme:
a calibration method of a low-resolution millimeter wave radar and an optical camera comprises the following steps:
s1, constructing a calibration plate and calibration acquisition equipment, wherein the calibration acquisition equipment comprises a camera, a millimeter wave radar and a pose sensor;
s2, acquiring original image information of a calibration plate by a camera, acquiring point cloud information of the calibration plate Shan Zhenlei by a millimeter wave radar, and acquiring pose information of a mobile platform by a pose sensor;
s3, fusing Shan Zhenlei point cloud information and pose information, and outputting a point cloud map containing calibration plate position information;
s4, processing the point cloud map by using a point cloud filtering algorithm and a clustering algorithm to extract millimeter wave radar point cloud blocks of the calibration plate;
s5, dividing the position information of the two-dimensional edge characteristic points of the calibration plate in the original image information by using a target dividing algorithm;
s6, constructing a three-dimensional model through the structure priori of the calibration plate, registering the three-dimensional model with the millimeter wave radar point cloud block, so as to obtain the fitted three-dimensional edge characteristic point position information of the calibration plate, and matching the fitted three-dimensional edge characteristic point position information with the two-dimensional edge characteristic point position information in the step S5 to obtain a plurality of groups of two-dimensional and three-dimensional matching characteristic point pair information;
and S7, taking a plurality of sets of two-dimensional and three-dimensional matching characteristic point pair information as input of a PnP algorithm to calculate and obtain an external parameter matrix for calibrating the millimeter wave radar and the camera.
Preferably, step S3 includes the steps of:
s3.1, defining an effective area in radar point cloud information, and filtering out low-precision points of a single frame of the effective area;
s3.2, filtering millimeter wave radar noise points based on a preset CFAR threshold value;
s3.3, according to pose information corresponding to the radar point cloud information of each frame, unifying the radar point cloud information to a global coordinate system;
s3.4, judging the probability of the voxel being occupied when a measured value is given by using a probability model to determine whether the map points are adopted;
and S3.5, completing millimeter wave radar mapping according to the map points adopted in the step S3.4, and outputting a point cloud map containing the position information of the calibration plate.
As a preferred solution, in step S3.3, according to pose information corresponding to radar point cloud information of each frame, the radar point cloud information is unified to a global coordinate system, and the adopted calculation formula is as follows:
wherein,,
representing a single frame Lei Dadian cloud coordinate of millimeter wave radar detection under a global coordinate system at t moment, < >>
Representing a single frame Lei Dadian cloud coordinate of millimeter wave radar detection under a local coordinate system at t moment, < >>
Three-dimensional rotation matrix representing t-moment lower global coordinate system to camera coordinate system>
And representing the three-dimensional translation vector from the global coordinate system to the camera coordinate system at the time t.
Preferably, the probability model used in step S3.4 is:
wherein the method comprises the steps of
For the probability that node n was occupied given the measurement before time t, P (n) is a priori,
for the probability that node n is occupied given the measured value at time t-1, < >>
Representing the probability of node n being occupied given a measured value at time t,/, for>
And representing radar point cloud coordinates of all frames detected by the millimeter wave radar before the time t under a global coordinate system.
Preferably, step S4 includes the steps of:
s4.1, downsampling a point cloud map, and filtering outliers by using a point cloud filtering algorithm;
s4.2, clustering different parts in the point cloud map by using an European clustering algorithm;
and S4.3, extracting millimeter wave radar point cloud blocks of the calibration plate according to the edge characteristics and the surface characteristics of each point cloud class.
As a preferred scheme, step S1 specifically includes:
two reflecting plates with an included angle of 90 degrees are used as calibration plates, and a millimeter wave radar and a camera are fixedly installed on a mobile platform.
Preferably, in step S6, the three-dimensional model is constructed based on the actual dimensions and the included angle of the calibration plate.
In the step S6, an iterative nearest point algorithm is adopted to match the three-dimensional model of the calibration plate with the millimeter wave radar point cloud block.
As a preferable scheme, the pose sensor adopts a wheel type odometer.
Preferably, step S7 includes the steps of:
s7.1, constructing an optimization problem:
s7.2, solving an extrinsic matrix into a nonlinear least square problem:
s7.3, solving a calibration matrix by using an LM optimization method;
wherein z is c The scale factor is M, the internal reference matrix of the camera is M, R is a three-dimensional rotation matrix from the millimeter wave radar coordinate system to the camera coordinate system, T is a three-dimensional translation vector of the millimeter wave radar coordinate system and the camera coordinate system, and p '' P For two-dimensional edge feature point coordinate estimation value, P r For calibrating the coordinates of the point positions of the three-dimensional edge feature points of the plate, p r Is two-dimensional edge feature point coordinates.
The beneficial effects of the invention are as follows:
1. according to the method, single-frame radar point cloud information acquired by the millimeter wave radar and pose information acquired by the pose sensor are fused, and a point cloud map containing position information of the calibration plate is output; processing the point cloud map by utilizing a point cloud filtering algorithm and a clustering algorithm to extract millimeter wave Lei Dadian clouds of the calibration plate; dividing the position information of the two-dimensional edge characteristic points of the calibration plate in the original image information by using a target dividing algorithm; constructing a three-dimensional model through the structure priori of the calibration plate, registering the three-dimensional model with the millimeter wave radar point cloud block, so as to obtain the fitted three-dimensional edge characteristic point position information of the calibration plate, and matching the fitted three-dimensional edge characteristic point position information with the partitioned two-dimensional edge characteristic point position information to obtain a plurality of groups of two-dimensional and three-dimensional matching characteristic point pair information; and taking a plurality of sets of two-dimensional and three-dimensional matching characteristic point pair information as input of a PnP algorithm, and calculating to obtain an external parameter matrix for calibrating the millimeter wave radar and the camera. The external parameter matrix obtained by the algorithm has high accuracy of the obtained calibration result in the calibration tasks of the millimeter wave radar and the optical camera, has effectiveness and robustness in a relatively open environment and an environment with more sundries, and has wide application scene.
2. According to the invention, the calibration tasks of the low-resolution millimeter wave radar and the optical camera are finished by using the calibration plate for the first time, and the built calibration environment and calibration acquisition equipment are simple and easy to operate.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a calibration method of a low-resolution millimeter wave radar and an optical camera according to the present invention.
Fig. 2 is a position diagram of a millimeter wave radar and an optical camera fixedly mounted on a mobile platform.
FIG. 3 is a data comparison graph of the accuracy quantization result of the re-projection effect obtained by manually extracting the feature points of the calibration plate and the algorithm of the invention in the calibration environment and scene one, respectively.
Fig. 4 is a graph of the accuracy quantization result of the re-projection effect obtained by two experiments under the first and second scenes, respectively, using the algorithm of the present invention.
Detailed Description
The following specific examples are presented to illustrate the present invention, and those skilled in the art will readily appreciate the additional advantages and capabilities of the present invention as disclosed herein. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict.
Referring to fig. 1, the embodiment provides a calibration method of a low-resolution millimeter wave radar and an optical camera, which includes the steps of:
s1, constructing a calibration plate and calibration acquisition equipment, wherein the calibration acquisition equipment comprises a camera, a millimeter wave radar and a pose sensor;
s2, acquiring original image information of a calibration plate by a camera, acquiring point cloud information of the calibration plate Shan Zhenlei by a millimeter wave radar, and acquiring pose information of a mobile platform by a pose sensor;
s3, fusing Shan Zhenlei point cloud information and pose information, and outputting a point cloud map containing calibration plate position information;
s4, processing the point cloud map by using a point cloud filtering algorithm and a clustering algorithm to extract millimeter wave radar point cloud blocks of the calibration plate;
s5, dividing the position information of the two-dimensional edge characteristic points of the calibration plate in the original image information by using a target dividing algorithm;
s6, constructing a three-dimensional model through the structure priori of the calibration plate, registering the three-dimensional model with the millimeter wave radar point cloud block, so as to obtain the fitted three-dimensional edge characteristic point position information of the calibration plate, and matching the fitted three-dimensional edge characteristic point position information with the two-dimensional edge characteristic point position information in the step S5 to obtain a plurality of groups of two-dimensional and three-dimensional matching characteristic point pair information;
and S7, taking a plurality of sets of two-dimensional and three-dimensional matching characteristic point pair information as input of a PnP algorithm to calculate and obtain an external parameter matrix for calibrating the millimeter wave radar and the camera.
The external parameter matrix obtained by the algorithm has high accuracy of the obtained calibration result in the calibration task of the millimeter wave radar and the optical camera, has effectiveness and robustness in a relatively open environment and an environment with more sundries, and has wide application scene.
Specific:
the step S3 includes the steps of:
s3.1, defining an effective area in radar point cloud information, and filtering out low-precision points of a single frame of the effective area;
s3.2, filtering millimeter wave radar noise points based on a preset CFAR threshold value;
s3.3, according to pose information corresponding to the radar point cloud information of each frame, unifying the radar point cloud information to a global coordinate system;
s3.4, judging the probability of the voxel being occupied when a measured value is given by using a probability model to determine whether the map points are adopted;
and S3.5, completing millimeter wave radar mapping according to the map points adopted in the step S3.4, and outputting a point cloud map containing the position information of the calibration plate.
In step S3.3, according to pose information corresponding to the radar point cloud information of each frame, the radar point cloud information is unified to the global coordinate system, and the adopted calculation formula is as follows:
wherein,,
representing a single frame Lei Dadian cloud coordinate of millimeter wave radar detection under a global coordinate system at t moment, < >>
Representing a single frame Lei Dadian cloud coordinate of millimeter wave radar detection under a local coordinate system at t moment, < >>
Three-dimensional rotation matrix representing t-moment lower global coordinate system to camera coordinate system>
And representing the three-dimensional translation vector from the global coordinate system to the camera coordinate system at the time t.
The probability model used in step S3.4 is:
wherein the method comprises the steps of
For the probability that node n was occupied given the measurement before time t,p (n) is the prior probability,
for the probability that node n is occupied given the measured value at time t-1, < >>
Representing the probability of node n being occupied given a measured value at time t,/, for>
And representing radar point cloud coordinates of all frames detected by the millimeter wave radar before the time t under a global coordinate system.
The step S4 includes the steps of:
s4.1, downsampling a point cloud map, and filtering outliers by using a point cloud filtering algorithm;
s4.2, clustering different parts in the point cloud map by using an European clustering algorithm;
and S4.3, extracting millimeter wave radar point cloud blocks of the calibration plate according to the edge characteristics and the surface characteristics of each point cloud class.
The step S1 specifically comprises the following steps:
two reflecting plates with an included angle of 90 degrees are used as calibration plates, and a millimeter wave radar and a camera are fixedly installed on a mobile platform. The optical camera of the invention uses the left camera of the zed2 binocular camera as a monocular camera, the resolution is 672×376, the horizontal viewing angle is 110 degrees, the vertical viewing angle is 70 degrees at most, and the camera internal reference uses official data of zed2 by default. The millimeter wave radar uses IWR1443 of TI company, which has a horizontal viewing angle of 120 ° at maximum, a pitch angle of 30 ° at maximum, a distance resolution of 4cm, a horizontal angular resolution of 15 ° and a vertical angular resolution of 60 °. The mobile robotic platform uses a Turtlebot2. The fixed mounting locations of millimeter wave radar and camera are seen in fig. 2.
The pose sensor in the step S2 adopts a wheel type odometer. The pose information of the mobile platform acquired by the pose sensor is the position and the pose of the mobile platform, and the change of the pose information comprises translation and rotation.
And step S6, constructing the three-dimensional model based on the real size and the included angle of the calibration plate.
In step S6, an iterative closest point algorithm is adopted to match the three-dimensional model of the calibration plate with millimeter wave radar point cloud blocks.
In step S7, the method includes the steps of:
s7.1, constructing an optimization problem:
s7.2, solving an extrinsic matrix into a nonlinear least square problem:
s7.3, solving a calibration matrix by using an LM optimization method;
wherein z is c The scale factor is M, the internal reference matrix of the camera is M, R is a three-dimensional rotation matrix from the millimeter wave radar coordinate system to the camera coordinate system, T is a three-dimensional translation vector of the millimeter wave radar coordinate system and the camera coordinate system, and p '' P For two-dimensional edge feature point coordinate estimation value, P r For calibrating the coordinates of the point positions of the three-dimensional edge feature points of the plate, p r Is two-dimensional edge feature point coordinates.
The following describes the advantages of the calibration method according to this embodiment through specific experiments:
the calibration environment in the experiment is that in an open room, only two reflecting plates which are used as calibration plates and form an included angle of 90 degrees; the first scene is that in the open room, other interference objects exist besides the two calibration plates; scene two is a room containing more sundries on the basis of scene one.
Six three-dimensional characteristic points are extracted on the calibration plate point cloud block by manpower and the algorithm of the invention respectively. In the first calibration environment and scene, the three-dimensional characteristic point position information extracted by using the manual and the algorithm is matched with the split two-dimensional characteristic point position information, and six sets of two-dimensional and three-dimensional matching characteristic point pair information obtained by using the two methods are used as the input of a calibration algorithm to obtain a calibration result. In the built calibration environment and in the first scene, the comparison of the precision quantization result data of the re-projection effect obtained by using two different methods is shown in fig. 3.
According to the experimental result shown in fig. 3, the method of manually extracting the feature points is used, so that the re-projection effect is better in the calibration environment, but the false local optimal solution is obtained, so that the point cloud cannot be projected to the pixel area of the target well in the first scene. The algorithm of the invention is used for extracting the characteristic points, the accuracy of the re-projection is very high in the calibration environment and the scene I, and the re-projection effect is far better than that of a method for manually extracting the characteristic points in the scene I.
The external parameter matrix obtained by the algorithm of the invention is subjected to two experiments under the first scene and the second scene respectively, and the accuracy quantization result of the obtained re-projection effect is shown in fig. 4. According to the experimental result shown in fig. 4, the external parameter matrix obtained by using the algorithm has effectiveness and robustness in open environment and environment with more impurities, and the re-projection verification accuracy is about 0.7 on average.
The above examples are merely illustrative of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention, and various modifications and improvements made by those skilled in the art to the technical solution of the present invention should fall within the protection scope of the present invention without departing from the design spirit of the present invention.
Claims (10)
1. The method for calibrating the low-resolution millimeter wave radar and the optical camera is characterized by comprising the following steps of:
s1, constructing a calibration plate and calibration acquisition equipment, wherein the calibration acquisition equipment comprises a camera, a millimeter wave radar and a pose sensor;
s2, acquiring original image information of a calibration plate by a camera, acquiring point cloud information of the calibration plate Shan Zhenlei by a millimeter wave radar, and acquiring pose information of a mobile platform by a pose sensor;
s3, fusing Shan Zhenlei point cloud information and pose information, and outputting a point cloud map containing calibration plate position information;
s4, processing the point cloud map by using a point cloud filtering algorithm and a clustering algorithm to extract millimeter wave radar point cloud blocks of the calibration plate;
s5, dividing the position information of the two-dimensional edge characteristic points of the calibration plate in the original image information by using a target dividing algorithm;
s6, constructing a three-dimensional model through the structure priori of the calibration plate, registering the three-dimensional model with the millimeter wave radar point cloud block, so as to obtain the fitted three-dimensional edge characteristic point position information of the calibration plate, and matching the fitted three-dimensional edge characteristic point position information with the two-dimensional edge characteristic point position information in the step S5 to obtain a plurality of groups of two-dimensional and three-dimensional matching characteristic point pair information;
and S7, taking a plurality of sets of two-dimensional and three-dimensional matching characteristic point pair information as input of a PnP algorithm to calculate and obtain an external parameter matrix for calibrating the millimeter wave radar and the camera.
2. The method for calibrating a low-resolution millimeter wave radar and an optical camera according to claim 1, wherein in step S3, the method comprises the steps of:
s3.1, defining an effective area in radar point cloud information, and filtering out low-precision points of a single frame of the effective area;
s3.2, filtering millimeter wave radar noise points based on a preset CFAR threshold value;
s3.3, according to pose information corresponding to the radar point cloud information of each frame, unifying the radar point cloud information to a global coordinate system;
s3.4, judging the probability of the voxel being occupied when a measured value is given by using a probability model to determine whether the map points are adopted;
and S3.5, completing millimeter wave radar mapping according to the map points adopted in the step S3.4, and outputting a point cloud map containing the position information of the calibration plate.
3. The method for calibrating a low-resolution millimeter wave radar and an optical camera according to claim 2, wherein in step S3.3, according to pose information corresponding to each frame of radar point cloud information, the radar point cloud information is unified to a global coordinate system, and a calculation formula is adopted as follows:
wherein,,
representing a single frame Lei Dadian cloud coordinate under a global coordinate system, which is detected by the millimeter wave radar at the time t,
representing a single frame Lei Dadian cloud coordinate of millimeter wave radar detection under a local coordinate system at t moment, < >>
Three-dimensional rotation matrix representing lower global coordinate system to camera coordinate system at time T, T t O And representing the three-dimensional translation vector from the global coordinate system to the camera coordinate system at the time t.
4. The method for calibrating a low-resolution millimeter wave radar and an optical camera according to claim 3, wherein the probability model used in step S3.4 is:
wherein the method comprises the steps of
For the probability that node n was occupied given the measurement before time t, P (n) is a priori,
for the probability that node n is occupied given the measured value at time t-1, < >>
Representing the probability of node n being occupied given a measured value at time t,/, for>
And representing radar point cloud coordinates of all frames detected by the millimeter wave radar before the time t under a global coordinate system.
5. The method for calibrating a low-resolution millimeter wave radar and an optical camera according to claim 1, wherein in step S4, the method comprises the steps of:
s4.1, downsampling a point cloud map, and filtering outliers by using a point cloud filtering algorithm;
s4.2, clustering different parts in the point cloud map by using an European clustering algorithm;
and S4.3, extracting millimeter wave radar point cloud blocks of the calibration plate according to the edge characteristics and the surface characteristics of each point cloud class.
6. The method for calibrating a low-resolution millimeter wave radar and an optical camera according to claim 1, wherein step S1 specifically comprises:
two reflecting plates with an included angle of 90 degrees are used as calibration plates, and a millimeter wave radar and a camera are fixedly installed on a mobile platform.
7. The method for calibrating a low-resolution millimeter wave radar and an optical camera according to claim 6, wherein in step S6, the three-dimensional model is constructed based on the actual dimensions and the included angles of the calibration plate.
8. The method for calibrating a low-resolution millimeter wave radar and an optical camera according to claim 7, wherein in step S6, an iterative closest point algorithm is adopted to match a three-dimensional model of a calibration plate with a point cloud of the millimeter wave radar.
9. The method for calibrating a low-resolution millimeter wave radar and an optical camera according to claim 1, wherein the pose sensor is a wheel type odometer.
10. The method for calibrating a low-resolution millimeter wave radar and an optical camera according to claim 1, wherein in step S7, the method comprises the steps of:
s7.1, constructing an optimization problem:
s7.2, solving an extrinsic matrix into a nonlinear least square problem:
s7.3, solving a calibration matrix by using an LM optimization method;
wherein z is c The scale factor is M, the internal reference matrix of the camera is M, R is a three-dimensional rotation matrix from the millimeter wave radar coordinate system to the camera coordinate system, T is a three-dimensional translation vector of the millimeter wave radar coordinate system and the camera coordinate system, and p '' P For two-dimensional edge feature point coordinate estimation value, P r For calibrating the coordinates of the point positions of the three-dimensional edge feature points of the plate, p r Is two-dimensional edge feature point coordinates.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211564786 | 2022-12-07 | ||
| CN2022115647860 | 2022-12-07 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116400349A true CN116400349A (en) | 2023-07-07 |
Family
ID=87011140
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211589130.4A Pending CN116400349A (en) | 2022-12-07 | 2022-12-09 | Calibration method of low-resolution millimeter wave radar and optical camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116400349A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117036511A (en) * | 2023-10-08 | 2023-11-10 | 之江实验室 | Calibration method and device for multi-type sensor, computer equipment and storage medium |
-
2022
- 2022-12-09 CN CN202211589130.4A patent/CN116400349A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117036511A (en) * | 2023-10-08 | 2023-11-10 | 之江实验室 | Calibration method and device for multi-type sensor, computer equipment and storage medium |
| CN117036511B (en) * | 2023-10-08 | 2024-03-26 | 之江实验室 | Calibration method and device for multi-type sensor, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110070615B (en) | Multi-camera cooperation-based panoramic vision SLAM method | |
| CN108369743B (en) | Mapping a space using a multi-directional camera | |
| Fruh et al. | Fast 3D model generation in urban environments | |
| CN113989450A (en) | Image processing method, image processing apparatus, electronic device, and medium | |
| US20230222688A1 (en) | Mobile device positioning method and positioning apparatus | |
| Pirker et al. | GPSlam: Marrying Sparse Geometric and Dense Probabilistic Visual Mapping. | |
| CN110910389B (en) | Laser SLAM loop detection system and method based on graph descriptor | |
| Zhang | LILO: A novel LiDAR–IMU SLAM system with loop optimization | |
| CN116400349A (en) | Calibration method of low-resolution millimeter wave radar and optical camera | |
| John et al. | Automatic calibration and registration of lidar and stereo camera without calibration objects | |
| CN113450334B (en) | Overwater target detection method, electronic equipment and storage medium | |
| CN112258631B (en) | Three-dimensional target detection method and system based on deep neural network | |
| CN113838129A (en) | Method, device and system for obtaining pose information | |
| Wang et al. | Improved LeGO-LOAM method based on outlier points elimination | |
| CN113256696A (en) | External parameter calibration method of laser radar and camera based on natural scene | |
| Le Besnerais et al. | Dense height map estimation from oblique aerial image sequences | |
| Song et al. | Registration for 3D LiDAR datasets using pyramid reference object | |
| CN114648639B (en) | Target vehicle detection method, system and device | |
| WO2023283929A1 (en) | Method and apparatus for calibrating external parameters of binocular camera | |
| CN115830116A (en) | Robust visual odometer method | |
| Chen et al. | Amobile system combining laser scanners and cameras for urban spatial objects extraction | |
| Grelsson et al. | Efficient 7D aerial pose estimation | |
| Steinemann et al. | Determining the outline contour of vehicles In 3D-LIDAR-measurements | |
| Madake et al. | Visualization of 3D Point Clouds for Vehicle Detection Based on LiDAR and Camera Fusion | |
| Zhao et al. | The construction method of the digital operation environment for bridge cranes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |