CN112162252B - Data calibration method for millimeter wave radar and visible light sensor - Google Patents
Data calibration method for millimeter wave radar and visible light sensor Download PDFInfo
- Publication number
- CN112162252B CN112162252B CN202011021124.XA CN202011021124A CN112162252B CN 112162252 B CN112162252 B CN 112162252B CN 202011021124 A CN202011021124 A CN 202011021124A CN 112162252 B CN112162252 B CN 112162252B
- Authority
- CN
- China
- Prior art keywords
- visible light
- light sensor
- millimeter wave
- wave radar
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
- G01S7/40—Means for monitoring or calibrating
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
The invention discloses a data calibration method of millimeter wave radar and visible light sensor. The method comprises the following steps: six metal panels are placed in a calibration field; acquiring data of the millimeter wave radar and images of the visible light sensor of the metal panel in the calibration field simultaneously by the millimeter wave radar and the visible light sensor to obtain six groups of alignment data sets of the millimeter wave radar and the visible light sensorAnd (3) withThe method comprises the steps of carrying out a first treatment on the surface of the Aligning the six sets of alignment data setsAnd (3) withCalculating a homography matrix H converted between a scanning plane coordinate system of the millimeter wave radar and a plane coordinate system of the visible light sensor image by adopting a linear least square method; projecting the millimeter wave radar target onto an image of the visible light sensor through homography transformation by utilizing the homography matrix H; and carrying out data association on the millimeter wave radar and the visible light sensor target by adopting a nearest neighbor algorithm, and finally finishing accurate data calibration of the millimeter wave radar and the visible light sensor.
Description
Technical Field
The invention relates to a millimeter wave radar and visible light sensor data calibration method, in particular to a method for calibrating data of a millimeter wave radar and a visible light sensor by combining a single-response transformation and a data correlation algorithm.
Background
Currently, advanced assisted driving systems (ADAS) mainly use millimeter wave radar and visible light sensors to perform environmental sensing. The millimeter wave radar has the capabilities of long-distance detection, all-weather work, vehicle speed measurement and the like, is low in cost, is not influenced by weather and light conditions, and has remarkable advantages in severe environments such as rain, snow, smoke and the like. But its object recognition capability is poor and resolution is low. The visible light sensor has strong target recognition capability, can recognize lane lines and traffic signs, is easily influenced by environment and weather, particularly heavy fog and rainy and snowy weather, and has obviously reduced sensing capability. Therefore, through data fusion of the millimeter wave radar and the visible light sensor and complementary advantages, the surrounding environment of the vehicle can be sensed more accurately, and the accuracy and the robustness of the ADAS system are improved. Because the installation positions of the millimeter wave radar and the visible light sensor are different, in order to make the information acquired by the millimeter wave radar and the visible light sensor consistent in space position, the data of the two sensors must be calibrated so as to ensure that the data acquired by the two sensors have a uniform reference standard and can be mutually converted.
Disclosure of Invention
The invention aims to provide a simple and feasible millimeter wave radar and visible light sensor data calibration method with high accuracy, so that a unified reference standard of the two sensors is obtained, and target position information acquired by the millimeter wave radar can be accurately calibrated on a visible light sensor image.
In order to solve the technical problems, the invention provides a data calibration method for a millimeter wave radar and a visible light sensor combining homography conversion and data association. The technical scheme is as follows:
in a first aspect, a method for calibrating data of a millimeter wave radar and a visible light sensor is provided, the method comprising: selecting an open calibration field, and placing six metal panels in the calibration field;
acquiring data of the millimeter wave radar and images of the visible light sensor of the metal panel in the calibration field simultaneously by the millimeter wave radar and the visible light sensor to obtain six groups of the millimeter wave radar and the visible light sensorVisible light sensor alignment dataset (x r ,y r ) And (u, v);
using the six sets of aligned data sets (x r ,y r ) And (u, v) calculating a homography matrix H converted between a scanning plane coordinate system of the millimeter wave radar and an image plane coordinate system of the visible light sensor by adopting a linear least square method;
projecting the target of the millimeter wave radar onto the image of the visible light sensor through homography transformation by the homography matrix H;
the visible light sensor obtains the target position information of the visible light sensor through a target detection algorithm, and a nearest neighbor algorithm is adopted to match the target of the millimeter wave radar with the target of the visible light sensor and correlate the data of multiple targets;
and determining data calibration of the millimeter wave radar and the visible light sensor according to the correlation matching result.
Further, the size of each metal panel is 30cm by 30cm.
Further, the homography matrix H is a spatial coordinate transformation parameter matrix between two sensor data, where h= [ H ]' 1 H′ 2 H′ 3 ]′。
Further, the similarity measurement method of the millimeter wave radar and the visible light sensor is determined by calculating the Euclidean distance between targets.
In a second aspect, a computer-readable storage medium is provided, in which one or more instructions are stored, which when executed by a processor within an electronic device, implement a data calibration method for a millimeter wave radar and a visible light sensor as described in any one of the above.
In a third aspect, there is provided a terminal device comprising:
a memory and a processor;
at least one program instruction is stored in the memory;
the processor loads and executes the at least one program instruction to realize the data calibration method of the millimeter wave radar and the visible light sensor.
The technical scheme provided by the embodiment of the invention has the beneficial effects that:
the data calibration method of the millimeter wave radar and the visible light sensor is simple and feasible, has high accuracy, can obtain a unified reference standard of the two sensors, and can accurately calibrate the target position information acquired by the millimeter wave radar onto the visible light sensor image. The data calibration method of the millimeter wave radar and the visible light sensor avoids complex calculation of the internal and external parameters of the sensor, has high calibration efficiency, improves the calibration precision, and can obtain accurate and stable calibration results.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data acquisition platform and its mounting location for millimeter wave radar and visible light sensors;
FIG. 2 is a millimeter wave radar coordinate system (x r ,y r ,z r ) Visible light sensor coordinate system (x c ,y c ,z c ) Scanning plane coordinate system (x) r ,y r ) Schematic diagram of the conversion relation between the image plane coordinate systems (u, v) of the visible light sensor, wherein r and alpha represent the radial distance and azimuth angle between the target object and the millimeter wave radar;
FIG. 3 is a schematic illustration of an image acquired by a visible light sensor in a calibration field;
FIG. 4 is a schematic diagram of data collected by a millimeter wave radar in a calibration field;
FIG. 5 is a schematic diagram of a millimeter wave radar target projected onto a visible light sensor image by homography transformation in a test scenario;
FIG. 6 is a schematic diagram of a millimeter wave radar and visible light sensor target association threshold in a test scenario;
fig. 7 is a schematic diagram of calibration results of a millimeter wave radar and a visible light sensor combined with a homography transformation and a data correlation algorithm in a test scene.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
Referring to fig. 1-7, an embodiment of the present invention provides a data calibration method for a millimeter wave radar and a visible light sensor, where the data calibration method for the millimeter wave radar and the visible light sensor includes: selecting an open calibration field, placing six metal panels in the calibration field, wherein the size of each metal panel is 30cm x 30cm, and acquiring data by utilizing a millimeter wave radar and a visible light sensor at the same time, wherein the data acquisition platform of the millimeter wave radar and the visible light sensor and the mounting position of the sensor are shown in figure 1;
acquiring data of the millimeter wave radar and an image of the visible light sensor of the metal panel in the calibration field simultaneously by the millimeter wave radar and the visible light sensor to obtain six groups of alignment data sets (x r ,y r ) And (u, v); specifically, according to the sensor mounting position, a conversion relation of each coordinate system between the two sensors is established as shown in FIG. 2, wherein (x c ,y c ,z c )、(x r ,y r ,z r ) Respectively representing a visible light sensor coordinate system and a millimeter wave radar coordinate system, (x) r ,y r ) (u, v) represent the scanning plane coordinate system of millimeter wave radar and the image plane coordinate system of visible light sensor, respectively, r and α represent the radial distance and azimuth angle between the target object and millimeter wave radar, since the data information collected by millimeter wave radar is established under the polar coordinate system, by triangle functionThe conversion formula is as follows under the conversion of the numerical formula to the Cartesian coordinate system:in the calibration field, the image of the metal panel acquired by the visible light sensor is shown in fig. 3, and the position distance data of the metal panel acquired by the millimeter wave radar is shown in fig. 4.
Using the six sets of aligned data sets (x r ,y r ) And (u, v) calculating a homography matrix H converted between a scanning plane coordinate system of the millimeter wave radar and an image plane coordinate system of the visible light sensor, that is, a spatial coordinate transformation parameter matrix between two sensor data, using a linear least square method, wherein h= [ H ]' 1 H′ 2 H′ 3 ]' the calculation formula is as follows:
let H i ＝[h i1 h i2 h i3 ]′，U＝[u 1 u 2 … u n ]′，V＝[v 1 v 2 … v n ]′，I n×1 ＝[1 1 … 1]′，Wherein n is the number of alignment data of the target point acquired by the millimeter wave radar and the visible light sensor, and n=6;
and projecting the target of the millimeter wave radar onto the image of the visible light sensor through homography transformation by utilizing the homography matrix H, wherein the transformation formula is as follows:
in the test scene, the effect of the target of the millimeter wave radar projected onto the image of the visible light sensor is shown in fig. 4, wherein black dots are projected millimeter wave radar target points;
the visible light sensor obtains the position information of the visible light sensor target in the image thereof through a target detection algorithm, and the nearest neighbor algorithm is adopted to match the target of the millimeter wave radar with the target of the visible light sensor and correlate the data of multiple targets;
specifically, the similarity measurement method of the target of the millimeter wave radar and the target of the visible light sensor is determined by calculating the Euclidean distance between the targets. The position information of the visible light sensor target on the image is obtained by utilizing a visible light sensor target detection algorithm-yolov 4 regression, the millimeter wave radar is matched with the visible light sensor target and the data of multiple targets are associated by adopting a nearest neighbor algorithm, a data association schematic diagram of the nearest neighbor algorithm is shown in fig. 7, wherein C1 and C2 represent the visible light sensor target, R1, R2 and R3 represent the millimeter wave radar target projected on the visible light sensor image through homography transformation, and the association flow is as follows:
(1) Establishing an association gate, and determining an association threshold: an ellipse associated gate.
(2) Threshold filtering: boundary conditions are set and R3 is filtered out.
(3) Determining a similarity measurement method: the Euclidean distance is calculated as follows:
wherein S is ij 、C i 、R j The method respectively represents Euclidean distance of the millimeter wave radar target and the visible light sensor target under the image plane coordinate system of the visible light sensor, coordinate values of the visible light sensor target under the image plane coordinate system of the visible light sensor and coordinate values of the millimeter wave radar target under the image plane coordinate system of the visible light sensor.
(4) Establishing an association matrix:
(5) Determining association decision criteria: the cost value is minimum, and the association pair is formed
(6) Forming an association pair: c (C) 1 →R 1 ,C 2 →R 2
According to the correlation matching result, the accurate calibration of the millimeter wave radar and the visible light sensor is completed, and the calibration result is shown in fig. 7.
An embodiment of the present invention further provides a computer readable storage medium, where one or more instructions are stored, where the one or more instructions implement the data calibration method of the millimeter wave radar and the visible light sensor according to any one of the above when executed by a processor in an electronic device.
The invention further provides a terminal device, which comprises a memory and a processor; at least one program instruction is stored in the memory; the processor loads and executes the at least one program instruction to realize the data calibration method of the millimeter wave radar and the visible light sensor.
The invention provides a data calibration method of millimeter wave radar and visible light sensor, and provides a data calibration method of millimeter wave radar and visible light sensor combining homography transformation and data association, aiming at the problems of accuracy and robustness of data calibration of millimeter wave radar and visible light sensor. Firstly, converting a homography matrix between a scanning plane coordinate system of a millimeter wave radar and an image plane coordinate system of a visible light sensor by applying homography conversion, and projecting a millimeter wave radar target onto an image of the visible light sensor by using the homography matrix; then, the position information of the visible light sensor target in the image is obtained by regression of a visible light sensor target detection algorithm; and finally, matching the visible light sensor target with the millimeter wave radar target and correlating the multi-target data by adopting a nearest neighbor algorithm, and finishing the accurate calibration of the millimeter wave radar and the visible light sensor according to the correlation matching result. The experimental result of the real sensor data shows that the algorithm of the invention not only avoids the complex acquisition of the internal and external parameters of the sensor, but also effectively improves the accuracy of the data calibration of the millimeter wave radar and the visible light sensor in the shielding and multi-target scene. The invention has the advantages that: complicated calculation of internal and external parameters of the sensor is avoided, the calibration efficiency is high, the calibration precision is improved, and an accurate and stable calibration result can be obtained.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention, but rather to enable any modification, equivalent replacement, improvement or the like to be included within the spirit and principles of the invention.
Claims (5)
1. The data calibration method for the millimeter wave radar and the visible light sensor is characterized by comprising the following steps of: selecting an open calibration field, and placing six metal panels in the calibration field;
simultaneously acquiring data of the millimeter wave radar and an image of the visible light sensor of the metal panel in the calibration field by the millimeter wave radar and the visible light sensor to obtain six groups of alignment data sets (x r ，y r ) And (u, v);
the data information collected by the millimeter wave radar is established under a polar coordinate system, and is converted into a Cartesian coordinate system through a triangle function, and the conversion formula is as follows:
using the six sets of aligned data sets (x r ，y r ) And (u, v) calculating a homography matrix H of conversion between a scanning plane coordinate system of the millimeter wave radar and a plane coordinate system of the visible light sensor image, that is, two-sensor data, using a linear least square methodA spatial coordinate transformation parameter matrix between the two, wherein H= [ H '' 1 H′ 2 H′ 3 ]' the calculation formula is as follows:
let H i ＝[h i1 h i2 h i3 ]′，U＝[u 1 u 2 …u n ]′，V＝[v 1 v 2 …v n ]′，I n×1 ＝[11…1]′，Wherein n is the number of alignment data of the target points acquired by the millimeter wave radar and the visible light sensor, and n=6;
projecting a target of the millimeter wave radar onto an image of the visible light sensor through homography transformation by utilizing the homography matrix H;
the image of the visible light sensor obtains the target position information of the visible light sensor through a target detection algorithm, and a nearest neighbor algorithm is adopted to match the target of the millimeter wave radar with the target of the visible light sensor and correlate the data of multiple targets;
and determining data calibration of the millimeter wave radar and the visible light sensor according to the correlation matching result.
2. The method of claim 1, wherein each of the metal panels has a size of 30cm by 30cm.
3. The method of claim 1, wherein the measure of similarity of the millimeter wave radar to the visible light sensor is determined by calculating the euclidean distance between objects.
4. A computer readable storage medium having one or more instructions stored therein, wherein the one or more instructions, when executed by a processor within an electronic device, implement the data calibration method of the millimeter wave radar and visible light sensor of any one of claims 1-3.
5. A terminal device, characterized in that the terminal device comprises:
a memory and a processor;
at least one program instruction is stored in the memory;
the processor is configured to implement the data calibration method of the millimeter wave radar and the visible light sensor according to any one of claims 1 to 3 by loading and executing the at least one program instruction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011021124.XA CN112162252B (en) | 2020-09-25 | 2020-09-25 | Data calibration method for millimeter wave radar and visible light sensor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011021124.XA CN112162252B (en) | 2020-09-25 | 2020-09-25 | Data calibration method for millimeter wave radar and visible light sensor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112162252A CN112162252A (en) | 2021-01-01 |
| CN112162252B true CN112162252B (en) | 2023-07-18 |
Family
ID=73863912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011021124.XA Active CN112162252B (en) | 2020-09-25 | 2020-09-25 | Data calibration method for millimeter wave radar and visible light sensor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112162252B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102798867A (en) * | 2012-09-10 | 2012-11-28 | 电子科技大学 | Correlation method for flight tracks of airborne radar and infrared sensor |
| CN103065323A (en) * | 2013-01-14 | 2013-04-24 | 北京理工大学 | Subsection space aligning method based on homography transformational matrix |
| KR20140078436A (en) * | 2012-12-17 | 2014-06-25 | 현대자동차주식회사 | Sensor fusion system and method thereof |
| WO2014160027A1 (en) * | 2013-03-13 | 2014-10-02 | Image Sensing Systems, Inc. | Roadway sensing systems |
| CN111429530A (en) * | 2020-04-10 | 2020-07-17 | 浙江大华技术股份有限公司 | Coordinate calibration method and related device |
| CN111830470A (en) * | 2019-04-16 | 2020-10-27 | 杭州海康威视数字技术股份有限公司 | Combined calibration method and device, and target object detection method, system and device |
-
2020
- 2020-09-25 CN CN202011021124.XA patent/CN112162252B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102798867A (en) * | 2012-09-10 | 2012-11-28 | 电子科技大学 | Correlation method for flight tracks of airborne radar and infrared sensor |
| KR20140078436A (en) * | 2012-12-17 | 2014-06-25 | 현대자동차주식회사 | Sensor fusion system and method thereof |
| CN103065323A (en) * | 2013-01-14 | 2013-04-24 | 北京理工大学 | Subsection space aligning method based on homography transformational matrix |
| WO2014160027A1 (en) * | 2013-03-13 | 2014-10-02 | Image Sensing Systems, Inc. | Roadway sensing systems |
| CN111830470A (en) * | 2019-04-16 | 2020-10-27 | 杭州海康威视数字技术股份有限公司 | Combined calibration method and device, and target object detection method, system and device |
| CN111429530A (en) * | 2020-04-10 | 2020-07-17 | 浙江大华技术股份有限公司 | Coordinate calibration method and related device |
Non-Patent Citations (1)
| Title |
|---|
| 秦汉.基于视觉和毫米波雷达融合的前方车辆检测算法研究.《中国优秀硕士学位论文全文数据库工程科技Ⅱ辑》.2020,(第07期),第C035-324页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112162252A (en) | 2021-01-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109490890B (en) | Intelligent vehicle-oriented millimeter wave radar and monocular camera information fusion method | |
| US8401240B2 (en) | Passive single camera imaging system for determining motor vehicle speed | |
| CN114972532B (en) | External parameter calibration method, device, equipment and storage medium between laser radars | |
| CN112083441A (en) | Obstacle detection method and system based on deep fusion of laser radar and millimeter wave radar | |
| CN115079143B (en) | Multi-radar external parameter quick calibration method and device for double-bridge steering mine card | |
| CN114413958A (en) | Monocular vision distance and speed measurement method of unmanned logistics vehicle | |
| CN111964680A (en) | Real-time positioning method of inspection robot | |
| CN116643291A (en) | SLAM method for removing dynamic targets by combining vision and laser radar | |
| CN113296120B (en) | Obstacle detection method and terminal | |
| Ye et al. | Extrinsic calibration of a monocular camera and a single line scanning Lidar | |
| CN113971697A (en) | Air-ground cooperative vehicle positioning and orienting method | |
| CN117630911A (en) | Unmanned ship radar photoelectric fusion method based on Singer model | |
| CN111862208B (en) | Vehicle positioning method, device and server based on screen optical communication | |
| CN105403886A (en) | Automatic extraction method for airborne SAR scaler image position | |
| CN112162252B (en) | Data calibration method for millimeter wave radar and visible light sensor | |
| CN117310627A (en) | Combined calibration method applied to vehicle-road collaborative road side sensing system | |
| CN112255604A (en) | Method and device for judging accuracy of radar data and computer equipment | |
| CN116679314A (en) | Three-dimensional laser radar synchronous mapping and positioning method and system for fusion point cloud intensity | |
| Pishehvari et al. | Robust range-doppler registration with hd maps | |
| CN111854678B (en) | Pose measurement method based on semantic segmentation and Kalman filtering under monocular vision | |
| CN113112551B (en) | Camera parameter determining method and device, road side equipment and cloud control platform | |
| CN111964681B (en) | Real-time positioning system of inspection robot | |
| CN113763457B (en) | Method and device for calibrating drop terrain, electronic equipment and storage medium | |
| CN115267756A (en) | Monocular real-time distance measurement method based on deep learning target detection | |
| CN114488037A (en) | Data calibration method for millimeter wave radar and visible light sensor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |