WO2020062434A1

WO2020062434A1 - Static calibration method for external parameters of camera

Info

Publication number: WO2020062434A1
Application number: PCT/CN2018/113667
Authority: WO
Inventors: 陈亮; 李晓东; 芦超
Original assignee: 初速度（苏州）科技有限公司; 北京初速度科技有限公司
Priority date: 2018-09-30
Filing date: 2018-11-02
Publication date: 2020-04-02
Also published as: CN110969663B; CN110969663A

Abstract

A static calibration method for the external parameters of a camera, comprising: step S1: setting multiple marking points, acquiring an image of the marking points, and identifying the positions of the marking points in the image; step S2: converting the actual GPS positions of the marking points into an IMU coordinate system using an IMU as the origin of coordinates to obtain a point P1; step S3: by means of a rotation translation transformation matrix, converting P1 into a camera coordinate system using a camera as the origin of coordinates to obtain a point P2: step S4: by means of an internal reference matrix of an image capture apparatus, converting P2 into an image coordinate system to obtain projection coordinates of the point on an image; step S5: constructing a re-projection error function, and calculating a re-projection error; step S6: optimizing the rotation translation transformation matrix, and repeating steps S3-S5 until the re-projection error is lower than a specified threshold. In the calibration method, by means of the GPS coordinates of the marking points and pixel coordinates of marking points in the image capture apparatus, a loss function is constructed so as to ultimately obtain a Euclidean transformation relationship, i.e., the rotation translation transformation matrix, between the two. In the calibration method, the calibration of the external parameters of a camera may be completed in a static state without needing to modify an acquisition device and without needing to modify a vehicle.

Description

Static calibration method for camera external parameters

Technical field

The invention relates to the field of intelligent driving, in particular to a static calibration method for external parameters of a camera.

Background technique

At present, the accuracy of autonomous driving maps is measured in the GPS coordinate system, which means that each point on the map needs to be represented by GPS coordinates. In the multi-sensor solution, the camera and GPS may not be installed at the same location of the vehicle at the same time, but may be separated by a distance of 2 to 3 meters. Therefore, the external parameters of the camera need to be calibrated to establish the spatial position relationship between the camera and the GPS module. If the external parameters of the camera are not calibrated, and the image is directly constructed based on the camera image and the position of the car body, an error of two or three meters may eventually occur.

In the traditional calibration method, the camera and the IMU (the GPS module on the IMU) need to be tied closely together, and then shakes violently, exciting the axis of the IMU and the camera, so that the trajectory of the IMU and the camera's The trajectory fits to complete the calibration of the camera. However, the camera and IMU installed on the car cannot shake, so the traditional calibration method is not applicable.

And in the existing technical solutions, there is still a lack of related research on how to establish a connection between the reprojection and the transformation matrix to adjust the transformation matrix during the coordinate transformation process.

Summary of the Invention

In view of the problems existing in the prior art, the present invention adopts the following technical solutions:

A method for calibrating external parameters of a camera, wherein the method includes the following steps:

Step S1: Set one or more marked points, acquire an image including the marked points through a camera device, and identify the position (u0, v0) of the marked points in the image;

Step S2: The actual GPS position of the marked point is converted to the IMU coordinate system with the IMU as the origin of the coordinate to obtain the point P1;

Step S3: P1 is transformed into a coordinate system with the camera device as the coordinate origin by rotating the translation transformation matrix M [R | t] to obtain point P2;

Step S4: P2 is transformed into the coordinate system of the image by the internal parameter matrix K of the camera device itself to obtain the projection coordinates (u1, v1) of the point on the image;

Step S5: construct re-projection error functions of (u0, v0) and (u1, v1), and calculate re-projection errors, which are related to the rotation translation transformation matrix M [R | t]; wherein the re-projection error functions As follows:

[Corrected 16.01.2019 in accordance with Rule 26]

Among them, K: the internal parameter matrix of the camera;

R ^b , t ^b : respectively the attitude and position of the IMU;

[Corrected 16.01.2019 in accordance with Rule 26]

GPS coordinates of the i-th marker point;

[Corrected 16.01.2019 in accordance with Rule 26]

The position of the i-th marker point in the image.

Step S6: Optimize the value of M [R | t], and repeat steps S3-Step S5 until the re-projection error is lower than the specified threshold, at which time the value of M [R | t] is used as the calibration result;

In the rotation and translation transformation matrix M [R | t], R represents rotation, and t represents translation.

Preferably, in step S1, the camera device is disposed on the roof of the vehicle or around the vehicle.

Preferably, the camera includes a plurality of cameras, and there is an overlapping area between the images acquired by each camera.

Preferably, in step S1, the position (u0, v0) of the marked point in the image is obtained by searching by rows or columns manually or by a computer.

Preferably, in step S1, the position (u0, v0) of the marked point in the image is obtained through a trained neural network.

Preferably, in step S5, the re-projection error function is as follows:

[Corrected 16.01.2019 in accordance with Rule 26]

Where K: the internal parameter matrix of the camera;

R ^b , t ^b : respectively the attitude and position of the IMU;

[Corrected 16.01.2019 in accordance with Rule 26]

GPS coordinates of the i-th marker point;

[Corrected 16.01.2019 in accordance with Rule 26]

The position of the i-th marker point in the image.

Preferably, in step S6, the value of M [R | t] is optimized by a least square method.

Preferably, the initial values of the rotation R and the translation t both take 0.

Preferably, in step S1, the number of the marked points is 10-20.

Preferably, in step S2, the actual GPS position of the marked point is measured in advance, or directly measured by using an RTK device during calibration.

The invention of the present invention lies in the following aspects, but is not limited to the following aspects:

The invention is based on the following aspects: (1) a static calibration method for the external parameters of the camera is provided; the calibration of the external parameters of the camera can be completed in a stationary state without the need to modify the acquisition equipment or the vehicle; The traditional punctuation method requires the camera and the IMU (the GPS module on the IMU) be tied together very closely, and then shakes violently. When the vehicle is stationary, the traditional method cannot be used. The present invention provides a method for marking points The GPS coordinates and pixel coordinates of the marker points in the camera device are used to construct a loss function to finally obtain the Euclidean transformation relationship between the two, that is, the rotation and translation transformation matrix. This method does not need to modify the acquisition equipment or the vehicle. , In the static state, you can complete the calibration of the camera external parameters. This is one of the inventive points of the present invention.

(2) A new loss function is designed; the rotation translation transformation matrix can be accurately and effectively obtained through this function. This is mainly reflected in the following aspects: the loss function takes into account the attitude and position of the IMU and the GPS coordinates of the marked points; in the mathematical model, the loss function is associated with the rotation translation transformation matrix, and the specified threshold is set by reprojection error To adjust the value of M [R | t] as the result of calibration. This is one of the inventive points of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a step diagram of a static calibration method for external parameters of a camera according to the present invention.

The present invention is described in further detail below. However, the following examples are merely simple examples of the present invention, and do not represent or limit the scope of protection of the rights of the present invention. The scope of protection of the present invention is subject to the claims.

detailed description

The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

In order to better illustrate the present invention and facilitate understanding of the technical solutions of the present invention, typical but non-limiting examples of the present invention are as follows:

Three-dimensional maps have good performance effects, and have the characteristics of small data volume, fast transmission speed, and low cost. After years of development, three-dimensional maps have been widely used in social management, public services, publicity and display fields.

Generally, 3D map modelers use high-resolution satellite maps or aerial photographs as reference maps based on the data collected by the collectors to complete the 3D modeling work by stitching the images; however, aerial photography or drone photography has many disadvantages, such as The cost is too high and the resolution is not high. This application provides a new data acquisition method for three-dimensional maps, specifically by installing a 360-degree all-around high-resolution camera device on the vehicle, such as four high-resolution cameras on the front and back of the vehicle, Cameras are installed on the front, back, left and right of the vehicle, and the vehicle traverses each street intersection. At the same time, the camera on the vehicle captures high-resolution images as the reference map, and then the stitching of the images completes the 3D modeling work.

Camera external parameters are parameters in the world coordinate system, such as camera position, rotation direction, etc. The camera external parameter calibration method is the mapping of world coordinates to pixel coordinates. Here, world coordinates are considered to be defined, and calibration is a known calibration control. The world coordinates and pixel coordinates of the point, we go to solve this mapping relationship. Once this relationship is solved, we can infer its world coordinates from the pixel coordinates of the point.

The purpose of camera calibration is to determine the values of some camera parameters. The staff can use these parameters to map points in a three-dimensional space to the image space, or vice versa.

The invention is a static calibration method. When the vehicle is stationary during calibration, the position of the marking point does not change. The marking point can be a special marking plate or a marking rod. The GPS position of the marker point is known. This position can be measured in advance and obtained directly during calibration, or it can be measured directly using RTK equipment during calibration.

Generally, there are 6 external parameters of the camera, and the rotation parameters of the three axes are (ω, δ, θ), and then the 3 * 3 rotation matrix of each axis is combined (that is, the matrix is multiplied first) to obtain The three-axis rotation information R is collected, and its size is still 3 * 3; the translation parameters (Tx, Ty, Tz) of the three axes of t.

Example 1

In this embodiment, a plurality of high-resolution cameras or cameras on the top of the vehicle can realize 360-degree high-definition video recording or taking pictures. When the vehicle is stationary, the method of calibrating the external parameters of the camera includes the following steps:

Step S1: Set one or more marked points, and obtain the image including the marked points by the camera on the top of the vehicle, and identify the position (u0, v0) of the marked points in the image. Here the position of the marker point in the image is the observation value. The marker point here can be a special marker card or a marker rod. Because it occupies a small area and the GPS position is known, the accuracy of the acquired signal is higher; the marker point The GPS position is known. The position can be measured in advance and obtained directly during calibration, or it can be measured directly using RTK equipment during calibration.

Since there are 6 external parameters of the camera, in order to calculate the corresponding parameters through the corresponding mathematical equations, at least 6 marked points are required. Generally, in order to reduce the impact of data noise on the solution, about 10 to 20 are selected. Although theoretically more markers are more conducive to solving, it will increase the cost of solving; here the coordinates of the markers can be found in the image captured by the camera, and the corresponding coordinates can be searched line by line by column. Find it on your computer.

Step S2: The actual GPS position of the marked point is converted to the IMU coordinate system with the IMU as the origin of the coordinates to obtain the point P1. The longitude and latitude height system of GPS is an ellipsoidal system, not orthogonal, but the rotation translation transformation matrix M [R | t] is based on an orthogonal coordinate system. Therefore, transforming GPS points to the IMU coordinate system means that the latitude and longitude coordinates can be converted into Cartesian coordinates (that is, orthogonal coordinate systems). This conversion relationship is a standard conversion method in a GIS system. After the transformation, the coordinates of this point are a three-dimensional coordinate (X, Y, Z), which indicates the position of this marked point relative to the IMU. The above-mentioned rotation and translation transformation matrix M [R | t], R represents rotation, and t represents translation.

Step S3: P1 is transformed into the camera coordinate system with the camera as the coordinate origin by rotating the translation transformation matrix M [R | t] to obtain point P2.

Step S4: P2 is transformed into the image coordinate system by the projection matrix of the camera itself, and the projection coordinates (u1, v1) of the point on the image are obtained. Here, the projection coordinates on the image are estimated values. The projection matrix here is the internal parameter matrix of the camera. This step is actually converting the three-dimensional point coordinate P2 into the theoretical two-dimensional coordinate of the point on the camera image through the internal parameter matrix of the camera. (u1, v1).

Step S5: Construct the reprojection error functions of (u0, v0) and (u1, v1), and calculate the reprojection error, which is related to M [R | t]; the reprojection error function is as follows:

[Corrected 16.01.2019 in accordance with Rule 26]

Where K: the internal parameter matrix of the camera;

R ^b , t ^b : respectively the attitude and position of the IMU;

[Corrected 16.01.2019 in accordance with Rule 26]

GPS coordinates of the i-th marker point;

[Corrected 16.01.2019 in accordance with Rule 26]

The position of the i-th marker point in the image. This location is also an observation and can be manually identified.

[Corrected 16.01.2019 in accordance with Rule 26]
among them

Is the coordinate (u1, v1) of the i-th marker point, and is an estimated value.

Step S6: Optimize the value of M [R | t] by the method of least squares, repeat steps S3-step S5, and iterate the value of M [R | t] until the reprojection error is lower than the specified threshold, at this time M [R | t The value of] is used as the calibration result. Here, optimizing the value of M [R | t] by the method of least squares is the prior art. There are various ways to calculate the value of M [R | t]. For example, you can use the reprojection error function to derive the derivative and judge by the size of the derivative. Find the optimized value of M [R | t]; the initial value of M [R | t] can be 0 for simplicity, and its true size is around the negative sixth power of 10.

Example 2

In this embodiment, multiple high-resolution cameras or cameras can be set around the vehicle, as long as 360-degree high-definition photography or photography is achieved. The calibration method of the camera's external parameters includes the following steps:

Step S1: Set one or more marked points, and the cameras or cameras around the vehicle obtain an image including the marked points, and identify the position (u0, v0) of the marked points in the image. Here, the position coordinates of the marked points in the image can be calculated by a trained neural network. The neural network was previously trained through a large amount of existing data (that is, the positions of the known marked points in the image have corresponding coordinates).

Step S2: The actual GPS position of the marked point is converted to the IMU coordinate system with the IMU as the origin of the coordinates to obtain the point P1. Here the GPS position and IMU coordinates are both three-dimensional coordinates, and the resulting point P1 is also a three-dimensional coordinate.

Step S3: P1 is transformed into the camera coordinate system with the camera as the coordinate origin by rotating the translation transformation matrix M [R | t] to obtain point P2. The point P2 here is also a three-dimensional coordinate.

Step S4: P2 is transformed into the image coordinate system by the projection matrix of the camera itself, and the projection coordinates (u1, v1) of the point on the image are obtained. The projection matrix here is the internal parameter matrix of the camera. This step is actually converting the three-dimensional point coordinate P2 into the theoretical two-dimensional coordinate (u1, v1) of the point on the camera image through the internal parameter matrix of the camera.

[Corrected 16.01.2019 in accordance with Rule 26]

Where K: the internal parameter matrix of the camera;

R ^b , t ^b : respectively the attitude and position of the IMU;

[Corrected 16.01.2019 in accordance with Rule 26]

GPS coordinates of the i-th marker point;

[Corrected 16.01.2019 in accordance with Rule 26]

[Corrected 16.01.2019 in accordance with Rule 26]
among them

Is the coordinate (u1, v1) of the i-th marker point, and is an estimated value.

Step S6: Optimize the value of M [R | t] by the method of least squares, and repeat steps S3-S5 until the re-projection error is lower than the specified threshold. At this time, the value of M [R | t] is used as the calibration result.

The applicant states that the present invention illustrates the detailed structural features of the present invention through the foregoing embodiments, but the present invention is not limited to the detailed structural features described above, which does not mean that the present invention must rely on the detailed structural features described above to be implemented. Those skilled in the art should understand that any improvement to the present invention, equivalent replacement of the components used in the present invention, addition of auxiliary components, selection of specific methods, etc., all fall within the scope of protection and disclosure of the present invention.

The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention. These simple modifications All belong to the protection scope of the present invention.

In addition, it should be noted that the specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention provides various possible The combination is not explained separately.

In addition, various combinations of the embodiments of the present invention can also be arbitrarily combined, as long as it does not violate the idea of the present invention, it should also be regarded as the content disclosed by the present invention.

Claims

A method for calibrating external parameters of a camera, wherein the method includes the following steps:

Step S1: Set one or more marked points, acquire an image including the marked points through a camera device, and identify the position (u0, v0) of the marked points in the image;

Step S2: The actual GPS position of the marked point is converted to the IMU coordinate system with the IMU as the origin of the coordinate to obtain the point P1;

Step S3: P1 is transformed into a coordinate system with the camera device as the coordinate origin by rotating the translation transformation matrix M [R | t] to obtain point P2;

Step S4: P2 is transformed into the coordinate system of the image by the internal parameter matrix K of the camera device itself to obtain the projection coordinates (u1, v1) of the point on the image;

Step S5: construct re-projection error functions of (u0, v0) and (u1, v1), and calculate re-projection errors, which are related to the rotation translation transformation matrix M [R | t]; wherein the re-projection error functions As follows:

Among them, K: the internal parameter matrix of the camera;

R b , t b : respectively the attitude and position of the IMU;

GPS coordinates of the i-th marker point;

The position of the i-th marker point in the image;

Step S6: Optimize the value of M [R | t], and repeat steps S3-S5 until the reprojection error is lower than the specified threshold, at which time the value of M [R | t] is used as the calibration result;

In the rotation and translation transformation matrix M [R | t], R represents rotation, and t represents translation.
The method according to claim 1, wherein in step S1, the camera device is disposed on a roof of a vehicle or around a vehicle.
The method according to any one of claims 1-2, wherein the camera device comprises a plurality of cameras, and there is an overlapping area between the images acquired by each camera.
The method according to any one of claims 1-3, characterized in that, in step S1, the position (u0, v0) of the marked point in the image is obtained by human or computer by row and column.
The method according to any one of claims 1-4, wherein in step S1, the position (u0, v0) of the marked point in the image is obtained through a trained neural network.
The method according to any one of claims 1-6, wherein in step S6, the value of M [R | t] is optimized by a least square method.
The method according to any one of claims 1 to 7, wherein the initial values of the rotation R and the translation t are both taken as 0.
The method according to any one of claims 1-8, wherein in step S1, the number of the marked points is 10-20.
The method according to any one of claims 1-9, wherein in step S2, the actual GPS position of the marked point is measured in advance, or directly measured using an RTK device during calibration.