WO2022142759A1

WO2022142759A1 - Lidar and camera joint calibration method

Info

Publication number: WO2022142759A1
Application number: PCT/CN2021/129942
Authority: WO
Inventors: 王军; 李玉莲; 刘欢; 张文琪; 曲颂
Original assignee: 中国矿业大学
Priority date: 2020-12-31
Filing date: 2021-11-11
Publication date: 2022-07-07
Also published as: CN112669393B; CN112669393A

Abstract

A lidar and camera joint calibration method, where a utilized lidar scanning mode is non-repetitive scanning, i.e. a trajectory is different every time the lidar scans, and after static capture for a number of seconds, point cloud coverage within a field of view approaches 100%. According to a non-repetitive scanning characteristic of this type of lidar, a self-made large size checkerboard calibration board is sequentially placed at different positions in an overlapping field of view of a lidar and a camera, and while the camera captures an image, the lidar captures three-dimensional point cloud data for a relatively long amount of time, and obtained three-dimensional point cloud data is converted into a two-dimensional normalized gray scale image according to point cloud intensity, then corner point detection is performed on the normalized gray scale image and the camera image, a corresponding two-dimensional gray scale image and camera image corner point pair is obtained, and subsequently, precise three-dimensional point cloud corner point coordinates are found in a backtracking manner according to a corner point in the two-dimensional gray scale image, and finally a joint calibration result is obtained according to a corresponding three-dimensional point cloud corner point and camera image corner point coordinates. The present method is more precise compared to a traditional method.

Description

A joint calibration method of lidar and camera

technical field

The invention belongs to the field of multi-sensor data fusion, and in particular relates to a joint calibration method of a laser radar and a camera.

Background technique

Lidar and cameras are widely used in driverless, intelligent robots and other fields.

The advantage of lidar is that it can accurately reflect the spatial three-dimensional information of the environment, but it is lacking in detailed description. Although the camera cannot reflect the spatial three-dimensional information of the environment, it has outstanding effects in detail and color description. Therefore, in an unmanned system, it is necessary to integrate the use of lidar and camera to give full play to their respective advantages. However, the premise of fusion is that it needs to be jointly calibrated and unified in space. Most of the existing joint calibration methods of lidar and camera are based on multi-line repetitive scanning lidar. Due to the late product launch, few people have been involved in non-repetitive scanning lidar.

The open source autopilot framework Autoware provides a joint calibration method of lidar and camera, and encapsulates it in the Autoware_Camera_Lidar_Calibrator toolkit. This method needs to manually circle the position of the calibration board in the three-dimensional point cloud to determine the plane and distance where the calibration board is located, so as to calculate the lidar attitude through the angle of the lidar relative to this plane, and then compare it with the camera, and finally get the lidar. Joint calibration results with the camera. However, this method has the problem of inaccurate manual circle points, which makes it impossible to accurately determine the plane and distance of the calibration plate in the 3D point cloud, resulting in inaccurate calibration results.

201910498286.3 discloses "a multi-camera system and lidar combined system and its joint calibration method", the method selects the point cloud on the checkerboard calibration board from the point cloud data through the calibration software, and after selecting the point cloud, the Project the point cloud into the camera coordinate system, observe whether the selected point cloud is located in the center of the calibration plate, adjust the selected point cloud so that the selected point cloud is projected in the center of the checkerboard calibration plate, and click Calibration after all adjustments are made. Output lidar calibration results. This method mainly relies on the manual selection of the point cloud at the center of the calibration plate. There is an error in the judgment of the human eye, and the number of samples is small, so the joint calibration results are difficult to be accurate.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a joint calibration method of laser radar and camera, based on the feature that the longer the scanning integration time of the non-repetitive scanning laser radar is, the higher the point cloud coverage, so as to solve the problem of low precision caused by the traditional joint calibration method.

The technical solution for realizing the purpose of the present invention is: a joint calibration method for a laser radar and a camera. The calibration steps are as follows:

Step 1: Fix the lidar and the camera on the same base, keep the relative position of the lidar and the camera unchanged, and ensure that the overlapping field of view of the lidar and the camera accounts for more than 50% of the camera's field of view. Go to step 2.

Step 2: Calibrate the camera to get the camera internal parameters

f _x , f _y represent the focal length of the camera, c _x , _cy represent the offset of the camera's optical axis on the image coordinate system, go to step 3.

Step 3: Place the checkerboard calibration board at different positions in the overlapping field of view of the lidar and the camera to collect the data of the camera and the lidar. At each position, the camera collects one frame of image data, and the lidar collects it for 20 to 30 seconds. 3D point cloud data, go to step 4.

Step 4: Screen the lidar 3D point cloud data and camera image data collected at each position. If there is data that cannot fully and clearly reflect the checkerboard calibration board, discard the data and fine-tune the checkerboard calibration board. attitude, re-collect data for the position, and go to step five; otherwise, go to step five directly.

Step 5: Normalize the collected three-dimensional point cloud data of the lidar in the x-axis direction to generate a two-dimensional normalized grayscale image of the lidar, wherein, according to the point cloud intensity information in the three-dimensional point cloud data of the lidar Determine the pixel grayscale in the two-dimensional normalized grayscale image, and go to step six.

Step 6: Perform corner detection on the obtained two-dimensional normalized grayscale image of the lidar and the camera image, and obtain the coordinates of the corner points of the checkerboard in the two-dimensional normalized grayscale image of the lidar and the camera image, respectively, and transfer to Step seven.

Step 7: Reverse the corner coordinates of each group of lidar 2D normalized grayscale images to obtain the corresponding lidar 3D point cloud checkerboard corner coordinates, and go to step 8.

Step 8: Return to Step 4, traverse the lidar 3D point cloud and camera image data collected at each position, and obtain a series of lidar 3D point cloud and camera image checkerboard corner coordinate point pairs, camera image corner coordinates and The three-dimensional laser point cloud coordinate transformation relationship is:

[u _c ,v _c ]=K(R*P(x,y,z)+T)

In the formula, [u _c , v _c ] is the camera image checkerboard corner coordinates, K is the camera internal parameter, R is the joint calibration rotation matrix, P(x, y, z) is the three-dimensional lidar point on the checkerboard corner. Cloud coordinates, T is the joint calibration displacement matrix; input each group of lidar 3D point cloud and camera image checkerboard corner coordinates, and finally obtain the rotation matrix R and displacement matrix T, complete a non-repetitive scanning lidar and Joint calibration of cameras.

Compared with the prior art, the present invention has the following significant advantages:

(1) The existing methods mostly use traditional multi-line lidar for joint calibration with cameras. The scanning trajectory of traditional multi-line lidar remains unchanged, and the point cloud coverage in the field of view is low. It is difficult to accurately reflect the calibration board information, which affects the Accuracy of joint calibration. The laser radar scanning method used in the present invention is non-repetitive scanning. The present invention collects data on the checkerboard calibration plate for 20 to 30 seconds while keeping the laser radar in a static state. The corner point information in the checkerboard calibration board can be clearly distinguished, and then the accurate corner point coordinate information can be extracted.

(2) Different from the prior art, the plane and distance of the checkerboard calibration board are estimated by selecting the three-dimensional point cloud data printed on the checkerboard calibration board. The invention proposes a 3D-2D-3D method to accurately obtain the three-dimensional point cloud coordinates played on the corners of the checkerboard calibration board. That is, the 3D point cloud data collected by the lidar is normalized on the x-axis, and a 2D normalized grayscale image of the lidar is established. The point cloud intensity information of the point is provided, and then the corner point detection is performed on the two-dimensional normalized grayscale image of the lidar to obtain the coordinates of the corner points of the calibration plate in the grayscale image, and finally the calibration plate angle in the detected grayscale image is used. The point coordinate backtracking finds the corresponding precise three-dimensional point cloud coordinates of the lidar on the corners of the checkerboard calibration board, which improves the accuracy of joint calibration.

Description of drawings

FIG. 1 is a flow chart of a joint calibration method for a lidar and a camera according to the present invention.

FIG. 2 is a schematic diagram of a black and white checkerboard calibration board used in the present invention.

Figure 3 is a schematic diagram of the definition of the lidar point cloud coordinate system.

FIG. 4 is a schematic structural diagram of the present invention.

Detailed ways

The invention adopts a non-repetitive scanning laser radar camera for joint calibration. The non-repetitive scanning laser radar has the characteristic that each scanning trajectory is not repeated. With the increase of the scanning time, the coverage rate of the output three-dimensional point cloud field increases continuously. After a few seconds of static scanning, the field of view coverage is close to 100%, which can fully reflect the precise environmental details. Combined with the large-scale self-made checkerboard calibration plate, the joint calibration accuracy is greatly improved.

With reference to Fig. 1, a method for joint calibration of a laser radar and a camera according to the present invention, the specific steps are as follows:

Step 1: Fix the lidar and the camera on the same base, keep the relative position of the lidar and the camera unchanged, and ensure that the overlapping field of view of the lidar and the camera accounts for more than 50% of the camera's field of view.

The lidar scanning method is non-repetitive scanning, and the lidar scanning trajectory is not repeated. After a few seconds of static scanning, the coverage rate of the field of view approaches 100%, that is, almost all areas in the field of view are covered.

Step 2: Calibrate the camera to get the camera internal parameters

f _x , f _y represent the focal length of the camera, and c _x , _cy represent the offset of the optical axis of the camera on the image coordinate system.

Step 3: Place the checkerboard calibration board in different positions (9 to 20, depending on the size of the overlapping field of view) in the overlapping field of view of the lidar and the camera to collect data from the camera and the lidar. At each position, The camera collects a frame of image data, and the lidar collects 3D point cloud data for 20 to 30 seconds, which can ensure that the lidar point cloud coverage in the field of view is close to 100%, which can fully reflect the checkerboard corner information.

In order to fully collect the camera image and lidar 3D point cloud data at different positions in the overlapping field of view, on the premise that the lidar and the camera can fully collect all the checkerboard calibration board data, the checkerboard calibration board should be placed in a position that covers the laser. The radar and the camera overlap the near, far, left and right borders and the middle position of the field of view, and the adjacent positions are separated by 3 to 5 meters.

Step 5: As shown in Figure 3, in the lidar point cloud coordinate system, the front of the lidar is the x-axis of the lidar point cloud coordinate. Therefore, in order to fully reflect the information of each corner of the checkerboard calibration board in the field of view, it is necessary to normalize the collected three-dimensional point cloud data of the lidar in the x-axis direction to generate a two-dimensional normalized grayscale image of the lidar. , and according to the point cloud intensity information in the lidar 3D point cloud data, the pixel grayscale in the two-dimensional normalized grayscale image is determined.

In the process of establishing the normalized grayscale image of the lidar, the coordinates of the 3D point cloud are first normalized, and the 3D point cloud data point P ₀ (x ₀ , y ₀ , z ₀ , i ₀ ) of the lidar is set; , i ₀ is the intensity information of the three-dimensional point cloud coordinate point of the lidar, which is directly output by the lidar;

Normalize the x-axis direction to get the coordinates

Second, set the resolution of the two-dimensional normalized grayscale image of the lidar to u ₀ *v ₀ ,

then gain

Pixel coordinates [u,v]=K ₀ *P ₁ (x ₁ ,y ₁ ,z ₁ ) of the lidar 2D normalized grayscale image transformed from the lidar 3D point cloud;

At the same time, the maximum value i _max of the point cloud intensity in all the three-dimensional point cloud data of lidar collected at each position is counted, then the gray value at the pixel coordinates [u, v] of the two-dimensional normalized gray image of the lidar is calculated. for

Step 6: Perform corner detection on the obtained two-dimensional normalized grayscale image of the lidar and the camera image, and obtain the coordinates of the corner points of the checkerboard in the two-dimensional normalized grayscale image of the lidar and the camera image, respectively.

Step 7: Reverse the corner coordinates in the two-dimensional normalized grayscale image of the lidar to obtain the corresponding three-dimensional point cloud coordinates of the lidar.

[u _c ,v _c ]=K(R*P(x,y,z)+T)

Example 1

Combined with Figure 1, a method for joint calibration of lidar and camera, the steps are as follows:

Step 1: Fix the lidar and the camera side by side on the same base facing the same direction, and the overlapping field of view of the lidar and the camera accounts for more than 50% of the camera's field of view.

Step 2: Calibrate the camera to get the camera internal parameters

Step 3: In order to fully collect camera images and lidar 3D point cloud data at different positions in the overlapping field of view, a, b, c, d, e, f, g, h, i are selected in the overlapping field of view this time. A total of 9 different positions (as shown in Figure 4, arranged in the form of concentric circles with different radii). At each location, the camera collects one frame of image data, and the lidar collects 20 seconds of 3D point cloud data. Among them, the checkerboard calibration board used is shown in Figure 2. In order to ensure that the camera and lidar can clearly collect the data of the checkerboard calibration board at a distance, the size of each grid of the checkerboard calibration board is set to 20 cm, 20 The grid is arranged alternately in five rows and four columns.

Step 5: Normalize the three-dimensional point cloud data of the laser radar at each position collected in the x-axis direction to generate a two-dimensional normalized grayscale image of the laser radar. Among them, the image pixel grayscale in the two-dimensional normalized grayscale image is determined according to the point cloud intensity in the lidar three-dimensional point cloud data.

In the process of establishing the normalized grayscale image of the lidar, the coordinates of the 3D point cloud are first normalized, and the 3D point cloud data point of the lidar is set as P ₀ (x ₀ , y ₀ , z ₀ , i ₀ ). Among them, i ₀ is the intensity information of the three-dimensional point cloud coordinate point of the lidar, which is directly output by the lidar.

Normalize the x-axis direction to get the coordinates

Second, set the resolution of the two-dimensional normalized grayscale image of the lidar to u ₀ *v ₀ , the gain

The pixel coordinates of the lidar 2D normalized grayscale image transformed from the lidar 3D point cloud are expressed as [u, v]=K ₀ *P ₁ (x ₁ , y ₁ , z ₁ ).

Step 6: Perform corner detection on the obtained two-dimensional normalized grayscale image of the lidar and the camera image at each position, and obtain the chessboard corner points in the two-dimensional normalized grayscale image of the lidar and the camera image respectively. coordinate.

Step 7: Reverse the corner coordinates in each group of laser radar two-dimensional normalized grayscale images to obtain the corresponding three-dimensional point cloud coordinates of laser radar. The secondary data acquisition time is 20 seconds, the coverage rate of the 3D point cloud is close to 100%, and the amount of 3D point cloud data is relatively large, so the coordinates of the lidar 3D point cloud corresponding to the corner points of the lidar 2D normalized grayscale image are searched in reverse. When there are multiple results, here we average the three-axis coordinate values of several lidar 3D point clouds found to reduce errors.

Step 8: Return to Step 4, traverse the lidar 3D point cloud and camera image data collected at all locations, and obtain a series of lidar 3D point cloud and camera image corner coordinate point pairs. The transformation relationship between camera image corner coordinates and 3D laser point cloud coordinates is:

[u _c ,v _c ]=K(R*P(x,y,z)+T)

In the formula, [u _c , v _c ] is the camera image checkerboard corner coordinates, K is the camera internal parameter, R is the joint calibration rotation matrix, P(x, y, z) is the three-dimensional lidar point on the checkerboard corner. Cloud coordinates, T is the joint calibration displacement matrix. Input the obtained three-dimensional point cloud of the lidar and the coordinates of the corners of the camera image checkerboard, and finally obtain the rotation matrix R and the displacement matrix T, and complete a non-repetitive scanning lidar and camera joint calibration.

Claims

A method for joint calibration of lidar and camera, characterized in that the calibration steps are as follows:

Step 1: Fix the lidar and the camera on the same base, keep the relative position of the lidar and the camera unchanged, and ensure that the overlapping field of view of the lidar and the camera accounts for more than 50% of the camera's field of view, and then go to step 2;

Step 2: Calibrate the camera to get the camera internal parameters
f x , f y represent the focal length of the camera, c x , c y represent the offset of the camera's optical axis on the image coordinate system, go to step 3;

Step 3: Place the checkerboard calibration board at different positions in the overlapping field of view of the lidar and the camera to collect the data of the camera and the lidar. At each position, the camera collects one frame of image data, and the lidar collects it for 20 to 30 seconds. 3D point cloud data, go to step 4;

Step 4: Screen the lidar 3D point cloud data and camera image data collected at each position. If there is data that cannot fully and clearly reflect the checkerboard calibration board, discard the data and fine-tune the checkerboard calibration board. attitude, re-collect data for the position, and go to step five; otherwise, go directly to step five;

Step 5: Normalize the collected three-dimensional point cloud data of the lidar in the x-axis direction to generate a two-dimensional normalized grayscale image of the lidar, wherein, according to the point cloud intensity information in the three-dimensional point cloud data of the lidar Determine the pixel grayscale in the two-dimensional normalized grayscale image, and go to step 6;

Step 6: Perform corner detection on the obtained two-dimensional normalized grayscale image of the lidar and the camera image, and obtain the coordinates of the corner points of the checkerboard in the two-dimensional normalized grayscale image of the lidar and the camera image, respectively, and transfer to Step seven;

Step 7: Reverse the corner coordinates of each group of lidar 2D normalized grayscale images to obtain the corresponding lidar 3D point cloud checkerboard corner coordinates, and go to step 8;

Step 8: Return to Step 4, traverse the lidar 3D point cloud and camera image data collected at each position, and obtain a series of lidar 3D point cloud and camera image checkerboard corner coordinate point pairs, camera image corner coordinates and The three-dimensional laser point cloud coordinate transformation relationship is:

[u c ,v c ]=K(R*P(x,y,z)+T)

In the formula, [u c , v c ] is the camera image checkerboard corner coordinates, K is the camera internal parameter, R is the joint calibration rotation matrix, P(x, y, z) is the three-dimensional lidar point on the checkerboard corner. Cloud coordinates, T is the joint calibration displacement matrix; input each group of lidar 3D point cloud and camera image checkerboard corner coordinates, and finally obtain the rotation matrix R and displacement matrix T, complete a non-repetitive scanning lidar and Joint calibration of cameras.
The joint calibration method for a lidar and a camera according to claim 1, wherein the lidar scanning mode is non-repetitive scanning, the lidar scanning trajectory does not repeat, and after static scanning for several seconds, the coverage of the field of view approaches 100% .
The method for jointly calibrating a lidar and a camera according to claim 1, wherein in step 3, the number of different positions is 9-20, depending on the size of the overlapping field of view.
The joint calibration method for lidar and camera according to claim 1, characterized in that: in step 3, in order to fully collect the camera image and lidar three-dimensional point cloud data at different positions in the overlapping field of view, between lidar and camera On the premise that all the checkerboard calibration board data can be fully collected, the position of the checkerboard calibration board should cover the near, far, left and right borders and middle positions of the overlapping field of view of the lidar and the camera, and the adjacent positions are separated by 3~ 5 meters.
The method for joint calibration of lidar and camera according to claim 1, characterized in that: in step 5, in the process of establishing the normalized grayscale image of lidar, first normalize the coordinates of the three-dimensional point cloud, set the laser Radar 3D point cloud data point P 0 (x 0 , y 0 , z 0 , i 0 ); where i 0 is the intensity information of the lidar 3D point cloud coordinate point, which is directly output by lidar;

Normalize the x-axis direction to get the coordinates

Second, set the resolution of the two-dimensional normalized grayscale image of the lidar to u 0 *v 0 ,

then gain

Pixel coordinates [u,v]=K 0 *P 1 (x 1 ,y 1 ,z 1 ) of the lidar 2D normalized grayscale image transformed from the lidar 3D point cloud;

At the same time, the maximum value i max of the point cloud intensity in all the three-dimensional point cloud data of lidar collected at each position is counted, then the gray value at the pixel coordinates [u, v] of the two-dimensional normalized gray image of the lidar is calculated. for