CN114755663A - External reference calibration method and device for vehicle sensor and computer readable storage medium - Google Patents
External reference calibration method and device for vehicle sensor and computer readable storage medium Download PDFInfo
- Publication number
- CN114755663A CN114755663A CN202210347834.4A CN202210347834A CN114755663A CN 114755663 A CN114755663 A CN 114755663A CN 202210347834 A CN202210347834 A CN 202210347834A CN 114755663 A CN114755663 A CN 114755663A
- Authority
- CN
- China
- Prior art keywords
- road identification
- road
- sensor
- identification data
- points
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/497—Means for monitoring or calibrating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
Abstract
The application discloses a method and a device for calibrating external references of a vehicle sensor and a computer-readable storage medium, wherein the method comprises the following steps: acquiring first road identification data of a corresponding high-precision map according to the current position of a vehicle; acquiring and processing first sensor data acquired by a first sensor of a vehicle to obtain second road identification data; acquiring and processing second sensor data acquired by a second sensor of the vehicle to obtain third road sign identification data; constructing a nonlinear optimization model according to the first road identification data, the second road identification data and the third road identification data; and determining the external parameters of the first sensor and the second sensor after calibration according to the nonlinear optimization model. According to the method and the device, the external parameters of the sensor can be calibrated in real time according to the road identification data provided by the high-precision map and the road identification data of different sensors, and the robustness and the precision of the external parameters of the sensor are improved through mutual constraint between different sources of road identification data.
Description
Technical Field
The present application relates to the field of automatic driving technologies, and in particular, to a method and an apparatus for calibrating an external reference of a vehicle sensor, and a computer-readable storage medium.
Background
In recent years, with the development of Advanced Driver Assistance System (ADAS) and Automated Driving (AD) technologies, research based on multi-sensor sensing such as cameras and laser radars has been widely used.
The calibration precision of the sensor relative to the external parameter at the center of the vehicle body influences the measurement of the sensor, and further influences the practical application. For example, road marking information such as lane lines, stop lines, arrows and the like detected from a road image acquired by a camera are registered with high-precision data in a high-precision map, so that the position of the vehicle in the driving process can be judged; the laser radar can sense the surrounding three-dimensional traffic identification, determine the accurate position of the vehicle in the road, and then plan and control according to the driving destination. Since all detection results need to be converted into the body coordinate system through the external parameters for application, the quality of the external parameters of the sensor affects the application of subsequent modules.
The existing calibration method for many sensors usually adopts off-line calibration, and because the vehicle inevitably vibrates in the driving process, the installation position, angle and the like of the sensors are inconsistent with the initial position, so that the initially calibrated external parameters have errors. In view of the above, in the prior art, some real-time calibration methods for sensors are provided, however, these calibration methods usually use only one kind of sensor data for calibration, resulting in poor robustness and accuracy of the calibration result.
Disclosure of Invention
The embodiment of the application provides a method and a device for calibrating external parameters of a vehicle sensor and a computer-readable storage medium, so as to improve the robustness and the precision of the external parameters of the vehicle sensor.
The embodiment of the application adopts the following technical scheme:
in a first aspect, an embodiment of the present application provides an external reference calibration method for a vehicle sensor, where the method includes:
acquiring first road identification data in a corresponding high-precision map according to the current position of the vehicle;
acquiring first sensor data acquired by a first sensor of the vehicle, and processing the first sensor data to obtain second road identification data;
acquiring second sensor data acquired by a second sensor of the vehicle, and processing the second sensor data to obtain third road sign identification data;
constructing a nonlinear optimization model according to the first road identification data, the second road identification data and the third road identification data, wherein the nonlinear optimization model is used for calibrating external parameters of the first sensor and external parameters of the second sensor;
and determining the external parameters of the calibrated first sensor and the external parameters of the calibrated second sensor according to the nonlinear optimization model.
Optionally, the constructing a nonlinear optimization model according to the first road identification data, the second road identification data, and the third road identification data includes:
converting the first road identification data, the second road identification data and the third road identification data into a vehicle body coordinate system;
constructing residual constraint according to the first road identification data, the second road identification data and the third road identification data in the vehicle body coordinate system;
and constructing the nonlinear optimization model according to the residual error constraint.
Optionally, the first road identification data includes 3D road marking points of a high-precision map, the second road identification data includes 2D road marking points of a road image, the third road identification data includes 3D road marking points of a lidar, according to the first road identification data the second road identification data and the third road identification data, constructing the nonlinear optimization model includes:
constructing a first residual constraint according to the 3D road identification points of the high-precision map and the 2D road identification points of the road image;
constructing a second residual constraint according to the 2D road identification points of the road image and the 3D road identification points of the laser radar;
constructing a third residual constraint according to the 3D road identification points of the high-precision map and the 3D road identification points of the laser radar;
and constructing the nonlinear optimization model according to the first residual constraint, the second residual constraint and the third residual constraint.
Optionally, the constructing a first residual constraint according to the 3D road identification points of the high-precision map and the 2D road identification points of the road image includes:
projecting the 3D road identification points of the high-precision map into the road image to obtain 2D road identification points of the high-precision map;
and constructing the first residual constraint according to the 2D road identification points of the high-precision map and the 2D road identification points of the road image.
Optionally, the 2D road identification points of the high-precision map and the 2D road identification points of the road image are 2D road identification points of a linear road identification, and constructing the first residual constraint according to the 2D road identification points of the high-precision map and the 2D road identification points of the road image includes:
fitting the 2D road identification points of the linear road identification in the high-precision map into a straight line;
and constructing residual constraint of the distance from the point to the straight line according to the distance between the 2D road identification point of the linear road identification in the road image and the straight line.
Optionally, the constructing a second residual constraint according to the 2D road identification point of the road image and the 3D road identification point of the lidar includes:
projecting the 3D road identification points of the laser radar to the road image to obtain 2D road identification points of the laser radar;
and constructing the second residual constraint according to the 2D road identification points of the laser radar and the 2D road identification points of the road image.
Optionally, the constructing a third residual constraint according to the 3D road identification point of the high-precision map and the 3D road identification point of the lidar includes:
matching the 3D road identification points of the high-precision map with the 3D road identification points of the laser radar;
and constructing the third residual constraint according to the matching result.
Optionally, after constructing a residual constraint according to the first road identification data, the second road identification data, and the third road identification data in the vehicle body coordinate system, the method further includes:
acquiring historical external parameters of the first sensor and historical external parameters of the second sensor;
constructing an external parameter prior constraint according to the historical external parameters of the first sensor and the historical external parameters of the second sensor;
and constructing the nonlinear optimization model according to the residual constraints and the external reference prior constraints.
In a second aspect, an embodiment of the present application further provides an external reference calibration apparatus for a vehicle sensor, where the apparatus includes:
the first acquisition unit is used for acquiring first road identification data in a corresponding high-precision map according to the current position of the vehicle;
the second acquisition unit is used for acquiring first sensor data acquired by a first sensor of the vehicle and processing the first sensor data to obtain second road identification data;
the third acquisition unit is used for acquiring second sensor data acquired by a second sensor of the vehicle and processing the second sensor data to obtain third road sign identification data;
a first constructing unit, configured to construct a nonlinear optimization model according to the first road identification data, the second road identification data, and the third road identification data, where the nonlinear optimization model is used to calibrate external parameters of the first sensor and external parameters of the second sensor;
and the determining unit is used for determining the external parameters of the calibrated first sensor and the calibrated second sensor according to the nonlinear optimization model.
In a third aspect, an embodiment of the present application further provides an electronic device, including:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to perform any of the methods described above.
In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium storing one or more programs that, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform any of the methods described above.
The embodiment of the application adopts at least one technical scheme which can achieve the following beneficial effects: according to the external reference calibration method of the vehicle sensor, first road identification data in a corresponding high-precision map are obtained according to the current position of a vehicle; then acquiring first sensor data acquired by a first sensor of the vehicle, and processing the first sensor data to obtain second road identification data; then, second sensor data acquired by a second sensor of the vehicle are acquired, and the second sensor data are processed to obtain third road sign identification data; then, according to the first road identification data, the second road identification data and the third road identification data, a nonlinear optimization model is constructed, and the nonlinear optimization model is used for calibrating external parameters of the first sensor and external parameters of the second sensor; and finally, determining the external parameters of the calibrated first sensor and the calibrated second sensor according to the nonlinear optimization model. The external reference calibration method for the vehicle sensor can calibrate the external reference of the sensor in real time according to the road identification data provided by the high-precision map and the road identification data of different sensors, and improves the robustness and the precision of the external reference of the sensor through mutual constraint between different sources of road identification data.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a schematic flow chart illustrating a method for external reference calibration of a vehicle sensor according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram illustrating a comparison between before and after calibration of an external reference of a vehicle sensor according to an embodiment of the present application;
FIG. 3 is a schematic structural diagram of an external reference calibration device of a vehicle sensor according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only a few embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.
The embodiment of the present application provides an external reference calibration method for a vehicle sensor, and as shown in fig. 1, provides a schematic flow chart of the external reference calibration method for a vehicle sensor in the embodiment of the present application, where the method at least includes the following steps S110 to S140:
step S110, acquiring first road identification data in the corresponding high-precision map according to the current position of the vehicle.
When external references of a sensor of a vehicle are calibrated, a current position of the vehicle may be determined first, and the current position of the vehicle may be acquired based on an RTK (Real-Time Kinematic) positioning device installed on the vehicle, or may also be obtained according to a positioning result output by a combined navigation system on the vehicle, such as an RTK + IMU (Inertial Measurement Unit), in a fusion manner, specifically, how to determine the current position of the vehicle.
After the current position of the vehicle is obtained, the first road identification data in the local high-precision map corresponding to the position can be further acquired, and the first road identification data is used as a first data source for subsequent external reference calibration. The first road identification data may include road surface identification data such as lane lines and arrows, and three-dimensional identification data such as speed limit signs and road signs.
Step S120, first sensor data collected by a first sensor of the vehicle is obtained, and the first sensor data is processed to obtain second road identification data.
When the external reference of the sensor of the vehicle is calibrated, the embodiment of the application needs to acquire first sensor data collected by a first sensor of the vehicle, where the first sensor may be, for example, a camera mounted on the vehicle, and the first sensor data may include a road image collected by the camera. The road image is recognized by using an existing road surface recognition model such as Lanenet, so that the second road marking data can be obtained, and the second road marking data specifically can include road marking data such as lane lines, stop lines, arrows, and the like, and serves as a second data source for performing external reference calibration subsequently.
Step S130, second sensor data collected by a second sensor of the vehicle is obtained, and the second sensor data is processed to obtain third road sign identification data.
When the external parameter of the sensor of the vehicle is calibrated, the second sensor data collected by the second sensor of the vehicle are required to be acquired, the second sensor can be a laser radar installed on the vehicle, the second sensor data can comprise three-dimensional point cloud data collected by the laser radar, and the three-dimensional point cloud data are processed, so that the third road sign identification data can be obtained, and the third road sign identification data specifically comprise road sign data such as a lane line, a stop line and an arrow, a speed limit board, three-dimensional identification data such as a road sign board and the like, and serve as a follow-up third data source for external parameter calibration.
It should be noted that the types of the first sensor and the second sensor are only exemplary descriptions, and in practical application, the sensors can be flexibly extended to other types of vehicle sensors, such as millimeter-wave radar, as long as the data collected by the sensors can provide road identification information. In addition, the number of the sensors is not limited to two, and may be three or more, and the entire process flow of the embodiment of the present application may be applied.
Step S140, a nonlinear optimization model is constructed according to the first road identification data, the second road identification data and the third road identification data, and the nonlinear optimization model is used for calibrating the external parameters of the first sensor and the external parameters of the second sensor.
And S150, determining the external parameters of the calibrated first sensor and the calibrated second sensor according to the nonlinear optimization model.
The external reference of the first sensor in the embodiment of the present application may be regarded as a conversion relation of the first sensor to the vehicle body coordinate system, and the external reference of the second sensor may be regarded as a conversion relation of the second sensor to the vehicle body coordinate system. The conversion relationship may specifically include pitch angle, yaw angle, roll angle, sensor height, and the like.
Under the condition that the initially calibrated external parameters are not changed, the same road identification data acquired by different sensors on the vehicle aiming at the current position of the vehicle should be consistent after the initially calibrated external parameters are converted, and under the actual condition, along with the running of the vehicle, the installation positions of the different sensors can deviate, so that certain deviation exists among the road identification data of the multiple sensors obtained based on the initially calibrated external parameters, and the initially calibrated external parameters need to be calibrated.
Based on this, according to the embodiment of the application, a nonlinear optimization model can be constructed according to the road identification data of the multiple different sources, and the nonlinear optimization model is solved to constrain the deviation between the road identification data of the multiple different sources, so as to obtain the external parameters of the calibrated first sensor and the external parameters of the calibrated second sensor.
The nonlinear optimization model may be implemented by using a least square algorithm, and specifically, the calibrated sensor parameter may be solved by using a preset gradient descent algorithm, where the preset gradient descent algorithm may use a GN (Gauss-Newton) algorithm or an LM (Levenberg-Marquardt ) algorithm, for example. Of course, which solving algorithm is specifically adopted can be flexibly selected by those skilled in the art according to actual requirements, and is not specifically limited herein.
The external reference calibration method for the vehicle sensor can calibrate the external reference of the sensor in real time according to the road identification data provided by the high-precision map and the road identification data of different sensors, and improves the robustness and the precision of the external reference of the sensor through mutual constraint between different sources of road identification data.
In an embodiment of the application, the constructing a nonlinear optimization model according to the first road identification data, the second road identification data and the third road identification data includes: converting the first road identification data, the second road identification data and the third road identification data into a vehicle body coordinate system; constructing residual constraint according to the first road identification data, the second road identification data and the third road identification data in the vehicle body coordinate system; and constructing the nonlinear optimization model according to the residual constraint.
When the nonlinear optimization model is constructed, road identification data from a plurality of different sources can be uniformly converted into a vehicle body Coordinate System, for example, for a high-precision map, the first road identification data provided by the high-precision map is in a World geographic System-1984 Coordinate System (WGS-84 Coordinate System), so that when the road identification data provided by the high-precision map is subjected to subsequent processing, the first road identification data can be converted into the vehicle body Coordinate System.
Similarly, the second road identification data provided by the camera is in the camera coordinate system, so that the second road identification data can be converted into the vehicle body coordinate system through the initial external reference of the camera, and the third road identification data provided by the laser radar is in the laser radar coordinate system, and can be converted into the vehicle body coordinate system through the initial external reference of the laser radar.
It should be noted that, unless otherwise specified, the road marking data mentioned in the following embodiments may be regarded as the road marking data converted into the vehicle body coordinate system.
Through the processing, the first road identification data, the second road identification data and the third road identification data in the vehicle body coordinate system can be obtained, residual constraint of the nonlinear optimization model can be further constructed based on the first road identification data, the residual constraint can be used for constraining deviation among the road identification data of a plurality of different sources, and the nonlinear optimization model is gradually converged by constraining the residual of the nonlinear optimization model, so that the optimal solution of the nonlinear optimization model is obtained.
In an embodiment of this application, first road identification data includes the 3D road identification point of high-precision map, second road identification data includes the 2D road identification point of road image, third road identification data includes lidar's 3D road identification point, according to first road identification data second road identification data and third road identification data, it includes to found nonlinear optimization model: constructing a first residual constraint according to the 3D road identification points of the high-precision map and the 2D road identification points of the road image; constructing a second residual constraint according to the 2D road identification points of the road image and the 3D road identification points of the laser radar; constructing a third residual constraint according to the 3D road identification points of the high-precision map and the 3D road identification points of the laser radar; and constructing the nonlinear optimization model according to the first residual constraint, the second residual constraint and the third residual constraint.
Because high-precision map and laser radar can both gather three-dimensional data, and the camera can only gather two-dimensional planar data, consequently the first road sign identification data of this application embodiment specifically can include the 3D road identification point in the high-precision map, and the second road identification data specifically can include the 2D road identification point in the road image, and the third road sign identification data specifically can include laser radar's 3D road identification point.
Because the high-precision map is usually acquired and constructed based on high-precision positioning equipment, the data such as the 3D road identification points provided by the high-precision map can be regarded as data with higher precision and can be used as the basis for calibrating the external parameters of the sensor. Therefore, on one hand, the embodiment of the application can compare the 2D road identification points of the road image with the 3D road identification points provided by the high-precision map, so that the first residual constraint can be constructed according to the deviation degree between the two, and then the real-time calibration of the camera external parameters is realized by utilizing various road surface identification information in the high-precision map. On the other hand, the 3D road identification points of the laser radar can be compared with the 3D road identification points provided by the high-precision map, so that third residual constraint can be constructed according to the deviation degree between the three points, and then the real-time calibration of the laser radar external parameters is realized by utilizing various pavement identifications and three-dimensional identification information in the high-precision map.
Besides the construction of corresponding residual constraints for the external reference calibration of a plurality of sensors based on the road identification data of the high-precision map, the road identification data among a plurality of sensors can also be mutually constrained. Specifically, the 2D road identification points of the road image and the 3D road identification points of the laser radar can be compared, so that second residual constraint can be constructed according to the deviation degree between the two, calibration of self external parameters is constrained through data between different sensors, and robustness and precision of sensor external parameter calibration are further improved.
The above process can be regarded as mutual constraint among the road identification data of the high-precision map, the camera and the sensor, and the data of the high-precision map is accurate, so that the external reference calibration process of the camera and the sensor is essentially constrained. When the sensor is expanded to more sensors, the processing logic can be adopted to realize external reference calibration of more sensors.
In an embodiment of the application, the constructing a first residual constraint according to the 3D road identification points of the high-precision map and the 2D road identification points of the road image includes: projecting the 3D road identification points of the high-precision map into the road image to obtain 2D road identification points of the high-precision map; and constructing the first residual constraint according to the 2D road identification points of the high-precision map and the 2D road identification points of the road image.
When the first residual constraint is constructed, the 3D road identification points of the high-precision map can be projected into the road image through the conversion of the initial external reference and the initial internal reference of the camera. Specifically, assume that the camera imaging model is as follows:
for one point (x) in the vehicle body coordinate systemc，yc，zc) External reference of camera
And camera internal reference
May be projected to the (u, v) of the road image.
And then, comparing the 2D road identification points of the high-precision map in the road image with the original 2D road identification points of the road image, thereby constructing a first residual constraint according to the deviation degree between the two points.
In an embodiment of the application, the 2D road marking points of the high-precision map and the 2D road marking points of the road image are both 2D road marking points of a linear road marking, and constructing the first residual constraint according to the 2D road marking points of the high-precision map and the 2D road marking points of the road image includes: fitting the 2D road identification points of the linear road identification in the high-precision map into a straight line; and constructing residual constraint of the distance from the point to the straight line according to the distance between the 2D road identification point of the linear road identification in the road image and the straight line.
Since the road image can only provide two-dimensional road identification information, and the high-precision map provides three-dimensional road identification information, the corresponding relationship between the road identification points detected in the road image and the road identification points in the high-precision map cannot be directly determined, that is, the specific corresponding relationship between the points cannot be determined. Therefore, for linear road signs such as lane lines, stop lines and the like, the embodiment of the application can measure errors by adopting the distance from points to straight lines, and thus residual constraints of straight line distances are constructed. For example, for the 2D road identification points of the high-precision map and the road image, the 2D road identification points of the high-precision map may be fitted to a straight line, and then the distance from each 2D road identification point in the road image to the straight line is calculated, so as to construct the residual constraint of the distance from the point to the straight line.
In order to clarify the embodiments of the present application, the lane line is taken as an example for further explanation, during the straight-ahead driving process of the vehicle, the vehicle fits the lane line points within a certain radius around the vehicle, and assuming that the lane line conforms to a bivariate linear equation, i.e. y ═ ax + b, the lane line points in the vehicle body coordinate system and the representation form of the parametric equation can be obtained by least square fitting:
assuming lane lines detected by the image
The coordinate of the point in the vehicle body coordinate system is Pcar＝[xc yc zc]TAnd is and
the lane lines in the matched high-precision map are
Can be started from the starting point Pa＝[xa ya za]TAnd end point Pb＝[xbyb zb]TCorresponding vector representation, then point PcarTo a straight line
The error of (d) can be expressed as:
wherein, VabIs a vector (P)b-Pa) The unit vector of (2).
In an embodiment of the application, the constructing a second residual constraint according to the 2D road identification points of the road image and the 3D road identification points of the lidar comprises: projecting the 3D road identification points of the laser radar to the road image to obtain 2D road identification points of the laser radar; and constructing the second residual constraint according to the 2D road identification points of the laser radar and the 2D road identification points of the road image.
When the second residual constraint is constructed, the 3D road identification point of the laser radar can be projected into the road image through the conversion of the initial external reference and the initial internal reference of the camera. And then, comparing the 2D road identification points corresponding to the laser radar in the road image with the original 2D road identification points of the road image, thereby constructing a second residual constraint according to the deviation degree between the two.
Specifically, based on the foregoing embodiment, the road identification points in the laser radar and the road identification points in the image may not correspond to each other one by one, and therefore, the distance from the points to the straight line may also be used to measure the error, that is, the 2D road identification points of the road image may be fitted to a straight line, and then the distance from each 2D road identification point of the laser radar to the straight line is calculated, and thus the residual constraint of the distance from the point to the straight line is constructed. Of course, it should be noted that, specifically, which source of the road identification point is to be fitted to the straight line, those skilled in the art may flexibly adjust the straight line according to actual requirements.
In an embodiment of the present application, the constructing a third residual constraint according to the 3D road identification points of the high-precision map and the 3D road identification points of the lidar includes: matching the 3D road identification points of the high-precision map with the 3D road identification points of the laser radar; and constructing the third residual constraint according to the matching result.
Because the high-precision map and the laser radar can both provide three-dimensional road identification information, the 3D road identification points in the high-precision map and the 3D road identification points obtained by the laser radar can be directly matched with each other point by point, for example, errors can be measured by Euclidean distances between the 3D road identification points in the high-precision map and the 3D road identification points of the laser radar, and third residual constraint is constructed according to the errors.
In an embodiment of the application, after constructing a residual constraint according to the first road identification data, the second road identification data, and the third road identification data in the vehicle body coordinate system, the method further includes: acquiring historical external parameters of the first sensor and historical external parameters of the second sensor; constructing an external parameter prior constraint according to the historical external parameters of the first sensor and the historical external parameters of the second sensor; and constructing the nonlinear optimization model according to the residual constraints and the external reference prior constraints.
In addition to the construction of the residual constraint in the foregoing embodiment, the embodiment of the present application may further obtain the historical external parameters corresponding to each sensor, use the historical external parameters corresponding to each sensor as the prior constraint, and construct the nonlinear optimization model together with the residual constraint. Since the external reference calibration process of the embodiment of the application is performed in real time on line, the historical external reference can be regarded as the external reference of each sensor after calibration at the previous time, and the external reference which is initially calibrated can be used as a fixed prior condition for constraint.
Although the external parameters corresponding to the sensors change in the vehicle motion process, so that the external parameters are not accurate any more, the change is not large, and therefore, the historical external parameters corresponding to the sensors are introduced as prior constraints, so that the convergence speed of the nonlinear optimization model can be greatly improved.
To illustrate the calibration effect of the external reference calibration method of the vehicle sensor of the present application, as shown in fig. 2, a comparison schematic diagram before and after the external reference calibration of the vehicle sensor of the present application is provided, where black lane line points in the left half of fig. 2 represent lane line points in a high-precision map, gray lane line points represent lane line points obtained based on initial external reference detection, black lane line points in the right half of fig. 2 also represent lane line points in the high-precision map, and gray lane line points represent lane line points obtained based on the calibrated external reference detection. Therefore, the external reference calibrated by the external reference calibration method of the vehicle sensor can detect more accurate lane line identification, and provides powerful data support for other automatic driving modules.
The embodiment of the present application further provides an external reference calibration apparatus 300 of a vehicle sensor, as shown in fig. 3, which provides a schematic structural diagram of the external reference calibration apparatus of the vehicle sensor in the embodiment of the present application, where the apparatus 300 includes: a first obtaining unit 310, a second obtaining unit 320, a third obtaining unit 330, a first constructing unit 340, and a determining unit 350, wherein:
a first obtaining unit 310, configured to obtain, according to a current position of a vehicle, first road identification data in a corresponding high-precision map;
the second obtaining unit 320 is configured to obtain first sensor data collected by a first sensor of the vehicle, and process the first sensor data to obtain second road identifier data;
the third obtaining unit 330 is configured to obtain second sensor data collected by a second sensor of the vehicle, and process the second sensor data to obtain third road identification data;
a first constructing unit 340, configured to construct a nonlinear optimization model according to the first road identification data, the second road identification data, and the third road identification data, where the nonlinear optimization model is used to calibrate an external parameter of the first sensor and an external parameter of the second sensor;
a determining unit 350, configured to determine an external parameter of the calibrated first sensor and an external parameter of the calibrated second sensor according to the nonlinear optimization model.
In an embodiment of the present application, the first constructing unit 340 is specifically configured to: converting the first road identification data, the second road identification data and the third road identification data into a vehicle body coordinate system; constructing residual constraint according to the first road identification data, the second road identification data and the third road identification data in the vehicle body coordinate system; and constructing the nonlinear optimization model according to the residual constraint.
In an embodiment of this application, first road sign identification data includes the 3D road identification point of high-precision map, second road identification data includes the 2D road identification point of road image, third road sign identification data includes lidar's 3D road identification point, first construction unit 340 specifically is used for: constructing a first residual constraint according to the 3D road identification points of the high-precision map and the 2D road identification points of the road image; constructing a second residual constraint according to the 2D road identification points of the road image and the 3D road identification points of the laser radar; constructing a third residual constraint according to the 3D road identification points of the high-precision map and the 3D road identification points of the laser radar; and constructing the nonlinear optimization model according to the first residual constraint, the second residual constraint and the third residual constraint.
In an embodiment of the present application, the first constructing unit 340 is specifically configured to: projecting the 3D road identification points of the high-precision map into the road image to obtain 2D road identification points of the high-precision map; and constructing the first residual constraint according to the 2D road identification points of the high-precision map and the 2D road identification points of the road image.
In an embodiment of the application, the 2D road marking points of the high-precision map and the 2D road marking points of the road image are 2D road marking points of a linear road marking, and the first constructing unit 340 is specifically configured to: fitting the 2D road identification points of the linear road identification in the high-precision map into a straight line; and constructing residual constraint of the distance from the point to the straight line according to the distance between the 2D road identification point of the linear road identification in the road image and the straight line.
In an embodiment of the present application, the first constructing unit 340 is specifically configured to: projecting the 3D road identification points of the laser radar to the road image to obtain 2D road identification points of the laser radar; and constructing the second residual constraint according to the 2D road identification points of the laser radar and the 2D road identification points of the road image.
In an embodiment of the present application, the first building unit 340 is specifically configured to: matching the 3D road identification points of the high-precision map with the 3D road identification points of the laser radar; and constructing the third residual constraint according to the matching result.
In one embodiment of the present application, the apparatus further comprises: a fourth acquisition unit, configured to acquire historical parameters of the first sensor and historical parameters of the second sensor; the second construction unit is used for constructing external parameter prior constraints according to the historical external parameters of the first sensor and the historical external parameters of the second sensor; and the third construction unit is used for constructing the nonlinear optimization model according to the residual constraint and the external parameter prior constraint.
It can be understood that the above-mentioned external reference calibration apparatus for a vehicle sensor can implement the steps of the external reference calibration method for a vehicle sensor provided in the foregoing embodiments, and the explanations regarding the external reference calibration method for a vehicle sensor are applicable to the external reference calibration apparatus for a vehicle sensor, and are not repeated herein.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Referring to fig. 4, at a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 4, but that does not indicate only one bus or one type of bus.
And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.
The processor reads a corresponding computer program from the non-volatile memory into the memory and then runs the computer program to form an external reference calibration device of the vehicle sensor on a logic level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:
acquiring first road identification data in a corresponding high-precision map according to the current position of the vehicle;
acquiring first sensor data acquired by a first sensor of the vehicle, and processing the first sensor data to obtain second road identification data;
acquiring second sensor data acquired by a second sensor of the vehicle, and processing the second sensor data to obtain third road sign identification data;
constructing a nonlinear optimization model according to the first road identification data, the second road identification data and the third road identification data, wherein the nonlinear optimization model is used for calibrating external parameters of the first sensor and external parameters of the second sensor;
and determining the external parameters of the calibrated first sensor and the external parameters of the calibrated second sensor according to the nonlinear optimization model.
The method executed by the external reference calibration device of the vehicle sensor disclosed in the embodiment of fig. 1 of the present application can be applied to or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and combines hardware thereof to complete the steps of the method.
The electronic device may further execute the method executed by the external reference calibration apparatus of the vehicle sensor in fig. 1, and implement the functions of the external reference calibration apparatus of the vehicle sensor in the embodiment shown in fig. 1, which are not described herein again.
Embodiments of the present application further provide a computer-readable storage medium storing one or more programs, where the one or more programs include instructions, which when executed by an electronic device including a plurality of application programs, enable the electronic device to perform the method performed by the external reference calibration apparatus for a vehicle sensor in the embodiment shown in fig. 1, and are specifically configured to perform:
acquiring first road identification data in a corresponding high-precision map according to the current position of the vehicle;
acquiring first sensor data acquired by a first sensor of the vehicle, and processing the first sensor data to obtain second road identification data;
acquiring second sensor data acquired by a second sensor of the vehicle, and processing the second sensor data to obtain third road sign identification data;
constructing a nonlinear optimization model according to the first road identification data, the second road identification data and the third road identification data, wherein the nonlinear optimization model is used for calibrating external parameters of the first sensor and external parameters of the second sensor;
and determining the external parameters of the calibrated first sensor and the external parameters of the calibrated second sensor according to the nonlinear optimization model.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A method of external reference calibration of a vehicle sensor, wherein the method comprises:
acquiring first road identification data in a corresponding high-precision map according to the current position of the vehicle;
acquiring first sensor data acquired by a first sensor of the vehicle, and processing the first sensor data to obtain second road identification data;
acquiring second sensor data acquired by a second sensor of the vehicle, and processing the second sensor data to obtain third road sign identification data;
constructing a nonlinear optimization model according to the first road identification data, the second road identification data and the third road identification data, wherein the nonlinear optimization model is used for calibrating external parameters of the first sensor and external parameters of the second sensor;
and determining the external parameters of the calibrated first sensor and the external parameters of the calibrated second sensor according to the nonlinear optimization model.
2. The method of claim 1, wherein said constructing a non-linear optimization model based on said first road identification data, said second road identification data, and said third road identification data comprises:
converting the first road identification data, the second road identification data and the third road identification data into a vehicle body coordinate system;
constructing residual constraint according to the first road identification data, the second road identification data and the third road identification data in the vehicle body coordinate system;
and constructing the nonlinear optimization model according to the residual constraint.
3. The method of claim 1, wherein the first road identification data comprises 3D road identification points of a high-precision map, the second road identification data comprises 2D road identification points of a road image, the third road identification data comprises 3D road identification points of a lidar, and constructing a non-linear optimization model from the first road identification data, the second road identification data, and the third road identification data comprises:
constructing a first residual constraint according to the 3D road identification points of the high-precision map and the 2D road identification points of the road image;
constructing a second residual constraint according to the 2D road identification points of the road image and the 3D road identification points of the laser radar;
constructing a third residual constraint according to the 3D road identification points of the high-precision map and the 3D road identification points of the laser radar;
and constructing the nonlinear optimization model according to the first residual constraint, the second residual constraint and the third residual constraint.
4. The method of claim 3, wherein the constructing a first residual constraint from the 3D road identification points of the high-precision map and the 2D road identification points of the road image comprises:
projecting the 3D road identification points of the high-precision map into the road image to obtain 2D road identification points of the high-precision map;
and constructing the first residual constraint according to the 2D road identification points of the high-precision map and the 2D road identification points of the road image.
5. The method of claim 4, wherein the 2D road identification points of the high-precision map and the 2D road identification points of the road image are 2D road identification points of line type road identification, and the constructing the first residual constraint according to the 2D road identification points of the high-precision map and the 2D road identification points of the road image comprises:
fitting the 2D road identification points of the linear road identification in the high-precision map into a straight line;
and constructing residual constraint of the distance from the point to the straight line according to the distance between the 2D road identification point of the linear road identification in the road image and the straight line.
6. The method of claim 3, wherein constructing a second residual constraint from the 2D road identification points of the road image and the 3D road identification points of the lidar comprises:
projecting the 3D road identification points of the laser radar to the road image to obtain 2D road identification points of the laser radar;
and constructing the second residual constraint according to the 2D road identification points of the laser radar and the 2D road identification points of the road image.
7. The method of claim 3, wherein constructing a third residual constraint from the 3D road marking points of the high-precision map and the 3D road marking points of the lidar comprises:
matching the 3D road identification points of the high-precision map with the 3D road identification points of the laser radar;
and constructing the third residual constraint according to the matching result.
8. The method of claim 2, wherein after constructing the residual constraint based on the first road identification data, the second road identification data, and the third road identification data in the body coordinate system, the method further comprises:
acquiring historical external parameters of the first sensor and historical external parameters of the second sensor;
constructing an external parameter prior constraint according to the historical external parameters of the first sensor and the historical external parameters of the second sensor;
and constructing the nonlinear optimization model according to the residual constraints and the external reference prior constraints.
9. An external reference calibration apparatus for a vehicle sensor, wherein the apparatus comprises:
the first acquisition unit is used for acquiring first road identification data in a corresponding high-precision map according to the current position of the vehicle;
the second acquisition unit is used for acquiring first sensor data acquired by a first sensor of the vehicle and processing the first sensor data to obtain second road identification data;
the third acquisition unit is used for acquiring second sensor data acquired by a second sensor of the vehicle and processing the second sensor data to obtain third road sign identification data;
a first constructing unit, configured to construct a nonlinear optimization model according to the first road identification data, the second road identification data, and the third road identification data, where the nonlinear optimization model is used to calibrate an external parameter of the first sensor and an external parameter of the second sensor;
and the determining unit is used for determining the external parameters of the calibrated first sensor and the calibrated second sensor according to the nonlinear optimization model.
10. A computer readable storage medium storing one or more programs which, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform any of the methods described above.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210347834.4A CN114755663A (en) | 2022-04-01 | 2022-04-01 | External reference calibration method and device for vehicle sensor and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210347834.4A CN114755663A (en) | 2022-04-01 | 2022-04-01 | External reference calibration method and device for vehicle sensor and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114755663A true CN114755663A (en) | 2022-07-15 |
Family
ID=82329468
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210347834.4A Pending CN114755663A (en) | 2022-04-01 | 2022-04-01 | External reference calibration method and device for vehicle sensor and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114755663A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117475399A (en) * | 2023-12-27 | 2024-01-30 | 新石器慧通（北京）科技有限公司 | Lane line fitting method, electronic device and readable medium |
-
2022
- 2022-04-01 CN CN202210347834.4A patent/CN114755663A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117475399A (en) * | 2023-12-27 | 2024-01-30 | 新石器慧通（北京）科技有限公司 | Lane line fitting method, electronic device and readable medium |
| CN117475399B (en) * | 2023-12-27 | 2024-03-29 | 新石器慧通（北京）科技有限公司 | Lane line fitting method, electronic device and readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| RU2727164C1 (en) | Method and apparatus for correcting map data | |
| US20200042803A1 (en) | Information processing method, information processing apparatus, and recording medium | |
| CN114111775B (en) | Multi-sensor fusion positioning method and device, storage medium and electronic equipment | |
| CN114279453B (en) | Automatic driving vehicle positioning method and device based on vehicle-road cooperation and electronic equipment | |
| CN112284416B (en) | Automatic driving positioning information calibration device, method and storage medium | |
| CN113850867A (en) | Camera parameter calibration method, camera parameter calibration device control method, camera parameter calibration device control device, and storage medium | |
| CN115546315A (en) | Sensor on-line calibration method and device for automatic driving vehicle and storage medium | |
| CN109029418A (en) | A method of vehicle is positioned in closed area | |
| CN115376090A (en) | High-precision map construction method and device, electronic equipment and storage medium | |
| CN114966632A (en) | Laser radar calibration method and device, electronic equipment and storage medium | |
| CN114754761A (en) | Optimization method and device for lane line of high-precision map, electronic equipment and storage medium | |
| US11908206B2 (en) | Compensation for vertical road curvature in road geometry estimation | |
| CN114755663A (en) | External reference calibration method and device for vehicle sensor and computer readable storage medium | |
| CN114705121A (en) | Vehicle pose measuring method and device, electronic equipment and storage medium | |
| US20220316909A1 (en) | Method and Communication System for Supporting at Least Partially Automatic Vehicle Control | |
| CN115950441B (en) | Fusion positioning method and device for automatic driving vehicle and electronic equipment | |
| CN115112125A (en) | Positioning method and device for automatic driving vehicle, electronic equipment and storage medium | |
| CN116148821A (en) | Laser radar external parameter correction method and device, electronic equipment and storage medium | |
| JP6819441B2 (en) | Target position estimation method and target position estimation device | |
| CN113253256B (en) | Monitoring method and device based on sensor fusion equipment and sensor fusion equipment | |
| CN116295508A (en) | Road side sensor calibration method, device and system based on high-precision map | |
| CN115546130A (en) | Height measuring method and device for digital twins and electronic equipment | |
| WO2020021596A1 (en) | Vehicle position estimation device and vehicle position estimation method | |
| US20240142590A1 (en) | Online sensor alignment using feature registration | |
| CN115127563A (en) | Positioning method and device for automatic driving vehicle, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |