CN114966578A - Radar external parameter calibration method and device based on shooting equipment and computer equipment - Google Patents

Abstract

The application relates to a radar external reference calibration method and device based on shooting equipment, computer equipment, storage media and computer program products. The method comprises the following steps: acquiring an acquired image of target shooting equipment and point cloud data of a radar to be calibrated, wherein the target shooting equipment and the radar to be calibrated have a view field overlapping region, and the radar to be calibrated comprises a first radar and a second radar; performing target detection on the point cloud data of the first radar to obtain a first target object point cloud, and performing target detection on the point cloud data of the second radar to obtain a second target object point cloud; determining a target object image position based on a collected image of a target shooting device; and carrying out external reference calibration on the first radar and the second radar according to the target object image position, the first target object point cloud and the second target object point cloud to obtain a radar external reference calibration result. By adopting the method, the radar external parameter calibration can be realized.

Description

Radar external parameter calibration method, device and computer equipment based on shooting equipment

technical field

The present application relates to the technical field of intelligent driving, and in particular, to a method, device and computer equipment for calibrating external parameters of radar based on photographing equipment.

Background technique

Radar external parameter calibration refers to the external parameter calibration of the radar installed on the vehicle, which is a very important link in the field of intelligent driving. In the field of intelligent driving, both positioning and perception need to rely on accurate external parameters. The external parameter calibration here refers to Relative positional relationship between radars, including translation and rotation.

In the traditional technology, the method of calibrating the external parameters of the radar is to select one of the radars on the vehicle as the main radar, and then perform the calibration through the overlapping area of the field of view of the main radar and other radars. However, for solid-state radar, its field of view is relatively small, and it may not overlap with other radars. It is difficult to calibrate based on the overlap of the field of view. It is impossible to calibrate the radar without the overlap of the field of view according to the traditional method. .

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide a radar external parameter calibration method, device, computer equipment, computer readable storage medium and computer program product based on a photographing device that can realize the radar external parameter calibration.

In a first aspect, the present application provides a radar external parameter calibration method based on a photographing device. The method includes:

Obtain the collected image of the target shooting device and the point cloud data of the radar to be calibrated. The target shooting device and the radar to be calibrated have a field of view overlapping area, and the radar to be calibrated includes a first radar and a second radar;

Perform target detection on the point cloud data of the first radar to obtain a first target object point cloud, and perform target detection on the point cloud data of the second radar to obtain a second target object point cloud;

Determine the image position of the target object based on the captured image of the target shooting device;

According to the image position of the target object, the point cloud of the first target object and the point cloud of the second target object, the external parameters of the first radar and the second radar are calibrated, and the external parameter calibration results of the radars are obtained.

In one of the embodiments, the target photographing device is a single photographing device, and the single photographing device and the first radar and the second radar both have a field of view overlapping area;

Based on the image position of the target object, the point cloud of the first target object, and the point cloud of the second target object, the external parameters of the first radar and the second radar are calibrated, and the obtained radar external parameter calibration results include:

Based on the image position of the target object and the point cloud of the first target object, the single shooting device and the first radar are calibrated, and the first external parameter calibration result is obtained;

Based on the image position of the target object and the point cloud of the second target object, the single shooting device and the second radar are calibrated, and the second external parameter calibration result is obtained;

Based on the first external parameter calibration result and the second external parameter calibration result, the radar external parameter calibration result is obtained.

In one embodiment, based on the image position of the target object and the point cloud of the first target object, a single shooting device and the first radar are calibrated, and obtaining the first external parameter calibration result includes:

Based on the obtained external parameter transformation relationship, the first target object point cloud is projected to the coordinate system where the captured image is located, and the point cloud image position that matches the first target object point cloud is obtained. The external parameter transformation relationship is a single shooting device and the third The external parameter transformation relationship between radars;

According to the position of the point cloud image and the image position of the target object, a single shooting device and the first radar are calibrated, and the first external parameter calibration result is obtained.

In one of the embodiments, the target photographing device includes a first photographing device having a field of view overlapping area with the first radar, a second photographing device having a field of view overlapping area with the second radar, and the first photographing device and the second photographing device respectively. The photographing device has a third photographing device in the overlapping area of the field of view;

Determining the image position of the target object based on the captured image of the target shooting device includes:

Based on the captured image of each photographing device, the image position of the target object matched with each photographing device is determined respectively.

In one embodiment, based on the image position of the target object, the point cloud of the first target object, and the point cloud of the second target object, the external parameter calibration is performed on the first radar and the second radar, and obtaining the external parameter calibration result of the radar includes:

According to the first target object point cloud and the target object image position matched with the first shooting device, the first radar and the first shooting device are calibrated, and the third external parameter calibration result is obtained;

According to the second target object point cloud and the target object image position matched with the second photographing equipment, the second radar and the second photographing equipment are calibrated, and the fourth external parameter calibration result is obtained;

Based on the image position of the target object matched with each shooting device, the first shooting device, the second shooting device and the third shooting device are calibrated to obtain a calibration result of the shooting devices;

According to the calibration result of the third external parameter, the calibration result of the fourth external parameter and the calibration result of the shooting equipment, the calibration result of the radar external parameter is obtained.

In one embodiment, based on the image position of the target object matched with each shooting device, the first shooting device, the second shooting device and the third shooting device are calibrated, and the obtained shooting device calibration result includes:

Based on the respective matching target object image positions of the first photographing device and the third photographing device, the first photographing device and the third photographing device are calibrated to obtain a first device calibration result;

Based on the target object image positions matched with the second photographing equipment and the third photographing equipment, the second photographing equipment and the third photographing equipment are calibrated to obtain a calibration result of the second equipment;

According to the calibration result of the first device and the calibration result of the second device, the calibration result of the photographing device is obtained.

In a second aspect, the present application also provides a radar external parameter calibration device based on a photographing device. The device includes:

The acquisition module is used to acquire the collected images of the target shooting device and the point cloud data of the radar to be calibrated. The target shooting device and the radar to be calibrated have an overlapping field of view, and the radar to be calibrated includes a first radar and a second radar;

a target detection module, configured to perform target detection on the point cloud data of the first radar to obtain the point cloud of the first target object, and perform target detection on the point cloud data of the second radar to obtain the point cloud of the second target object;

a processing module for determining the image position of the target object based on the captured image of the target shooting device;

The calibration module is used to calibrate the external parameters of the first radar and the second radar according to the image position of the target object, the point cloud of the first target object and the point cloud of the second target object, and obtain the calibration result of the external parameters of the radar.

In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Obtain the collected image of the target shooting device and the point cloud data of the radar to be calibrated. The target shooting device and the radar to be calibrated have a field of view overlapping area, and the radar to be calibrated includes a first radar and a second radar;

Perform target detection on the point cloud data of the first radar to obtain a first target object point cloud, and perform target detection on the point cloud data of the second radar to obtain a second target object point cloud;

Determine the image position of the target object based on the captured image of the target shooting device;

According to the image position of the target object, the point cloud of the first target object and the point cloud of the second target object, the external parameters of the first radar and the second radar are calibrated, and the external parameter calibration results of the radars are obtained.

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by the processor, the following steps are implemented:

Obtain the collected image of the target shooting device and the point cloud data of the radar to be calibrated. The target shooting device and the radar to be calibrated have a field of view overlapping area, and the radar to be calibrated includes a first radar and a second radar;

Perform target detection on the point cloud data of the first radar to obtain a first target object point cloud, and perform target detection on the point cloud data of the second radar to obtain a second target object point cloud;

Determine the image position of the target object based on the captured image of the target shooting device;

According to the image position of the target object, the point cloud of the first target object and the point cloud of the second target object, the external parameters of the first radar and the second radar are calibrated, and the external parameter calibration results of the radars are obtained.

In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

Obtain the collected image of the target shooting device and the point cloud data of the radar to be calibrated. The target shooting device and the radar to be calibrated have a field of view overlapping area, and the radar to be calibrated includes a first radar and a second radar;

Perform target detection on the point cloud data of the first radar to obtain a first target object point cloud, and perform target detection on the point cloud data of the second radar to obtain a second target object point cloud;

Determine the image position of the target object based on the captured image of the target shooting device;

According to the image position of the target object, the point cloud of the first target object and the point cloud of the second target object, the external parameters of the first radar and the second radar are calibrated, and the external parameter calibration results of the radars are obtained.

The above-mentioned method, device, computer equipment, storage medium and computer program product for calibrating external parameters of radar based on photographing equipment, by acquiring the collected image of the target photographing equipment and the point cloud data of the radar to be calibrated, respectively, to the first radar and the second radar. The point cloud data is used for target detection, and the target object can be detected from the point cloud data of the first radar and the second radar, and the first target object point cloud and the second target object point cloud can be obtained, which are determined by the collected images based on the target shooting equipment. The image position of the target object, the target object can be detected from the collected image, and then the target object can be used as a benchmark, according to the target object image position of the target object in the collected image, the first target object point cloud matched by the first radar, and the first target object point cloud matched by the first radar. The second target object point cloud that matches the second radar, the first radar and the second radar are calibrated with external parameters, and the radar external parameter calibration result is obtained. The whole process can use the target shooting equipment that overlaps the field of view of the radar to be calibrated. , and based on the target object captured at the same time, the external parameter calibration of the radar is realized.

Description of drawings

1 is an application environment diagram of a method for calibrating radar external parameters based on a photographing device in one embodiment;

2 is a schematic flowchart of a method for calibrating radar external parameters based on a photographing device in one embodiment;

3 is a schematic diagram of a radar to be calibrated in one embodiment;

4 is a schematic diagram of point cloud data of a radar to be calibrated in one embodiment;

5 is a schematic diagram of a field of view overlapping region in one embodiment;

6 is a schematic diagram of the first target object point cloud projected to the coordinate system where the captured image is located in one embodiment;

7 is a schematic diagram of a field of view overlap region in another embodiment;

8 is a schematic flowchart of a method for calibrating external parameters of radar based on a photographing device in another embodiment;

9 is a schematic flowchart of a method for calibrating radar external parameters based on a photographing device in another embodiment;

10 is a schematic flowchart of a method for calibrating radar external parameters based on a photographing device in yet another embodiment;

11 is an application environment diagram of a radar external parameter calibration method based on a photographing device in another embodiment;

12 is a structural block diagram of an apparatus for calibrating external parameters of radar based on photographing equipment in one embodiment;

Figure 13 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

The radar external parameter calibration method based on the photographing device provided in the embodiment of the present application can be applied to the application environment shown in FIG. 1 . The server 102 obtains the collected image of the target photographing device 104 and the point cloud data of the radar to be calibrated. The target photographing device 104 and the radar to be calibrated have a field of view overlapping area. The radar to be calibrated includes a first radar 106 and a second radar 108. The point cloud data of a radar 106 and a second radar 108 are used for target detection, and a first target object point cloud matching the first radar 106 and a second target object point cloud matching with the second radar 108 are obtained. Based on the target shooting The collected images of the device 104 determine the image position of the target object, and perform external parameter calibration on the first radar 106 and the second radar 108 according to the target object image position, the point cloud of the first target object and the point cloud of the second target object, and obtain the external parameters of the radar. Calibration results. The server 102 may be implemented by an independent server or a server cluster composed of multiple servers.

In one embodiment, as shown in FIG. 2 , a method for calibrating radar external parameters based on a photographing device is provided, and the method is applied to the server 102 in FIG. 1 as an example for description, including the following steps:

Step 202 , acquiring the captured image of the target photographing device and the point cloud data of the radar to be calibrated. The target photographing device and the radar to be calibrated have an overlapping field of view, and the radar to be calibrated includes a first radar and a second radar.

The target shooting device refers to a shooting device that has a field of view overlapping area with the radar to be calibrated. For example, the target photographing device may specifically refer to a camera or a camera that has a field of view overlapping with the field of view of the radar to be calibrated. The captured image refers to the image including the target object captured by the target photographing device. The target object here refers to the object that appears in the field of view of the target shooting device and the radar to be calibrated at the same time, and can be used for external parameter calibration between the camera and the radar. For example, the target object may be a calibration plate that can be used for calibration. The calibration board is used in machine vision, image measurement, photogrammetry, three-dimensional reconstruction and other applications to correct lens distortion, determine the conversion relationship between physical size and pixels, and determine the three-dimensional geometric position of a point on the surface of a space object and its corresponding image in the image. The relationship between the points requires the establishment of a geometric model for camera imaging. For another example, the target object may be a vehicle that can be used for calibration. For another example, the target object can be a building that can be used for calibration.

Among them, the radar to be calibrated refers to the radar on the vehicle that needs to calibrate external parameters. For example, the radar to be calibrated may specifically refer to a solid-state radar on a vehicle that needs to calibrate external parameters. The point cloud data of the radar to be calibrated refers to the point cloud image obtained by the radar to be calibrated. For example, the point cloud data of the radar to be calibrated may specifically refer to a frame of point cloud obtained by calibrating the radar. It should be noted that the radar to be calibrated usually includes at least two radars, namely a first radar and a second radar. There is no overlapping area of the field of view between the first radar and the second radar, or the first radar and the second radar only exist. The small overlapping area of the field of view, the traditional calibration method based on the overlapping area of the field of view is not enough to complete the external parameter calibration. For example, as shown in FIG. 3 , the first radar and the second radar can be forward-facing lidar and rearward-facing lidar that are installed on the vehicle without overlapping fields of view, and the point cloud data corresponding to the radar to be calibrated can be As shown in FIG. 4 , the first area 402 is the point cloud data of the first radar, and the second area 404 is the point cloud data of the second radar. It can be seen from FIG. 4 that the point clouds of the first radar and the second radar are The data does not have any overlapping areas.

Specifically, when the radar external parameter calibration is performed on the radar to be calibrated on any vehicle, the server will obtain the captured image of the target shooting device and the point cloud data of the radar to be calibrated, wherein the target shooting device and the radar to be calibrated have overlapping fields of view area, the radar to be calibrated includes a first radar and a second radar. For example, the target photographing device may specifically be a photographing device that is placed on the same vehicle and has a field of view overlapping area with the radar to be calibrated. As another example, the target shooting device may be a shooting device that is placed on a nearby vehicle and has a field of view overlapping with the radar to be calibrated. The nearby vehicle may be located on the left or right side of the vehicle equipped with the radar to be calibrated, etc. The embodiments are not specifically limited here.

Step 204 , perform target detection on the point cloud data of the first radar to obtain a first target object point cloud, and perform target detection on the point cloud data of the second radar to obtain a second target object point cloud.

Among them, the target detection is to detect the target object in the point cloud data. The first target object point cloud refers to the point cloud including the target object in the first radar. The second target object point cloud refers to the point cloud including the target object in the second radar.

Specifically, after acquiring the point cloud data of the first radar and the second radar, the server will perform target detection on the point cloud data of the first radar and the second radar respectively, so as to obtain the point cloud of the first target object and the second target Object point cloud. In this embodiment, the method of performing target detection on the point cloud data of the first radar and the second radar is not limited, as long as the target detection can be realized. For example, the method of performing target detection on the point cloud data of the first radar and the second radar may be, based on the point information of the point data in the point cloud data, calculate the normal vector corresponding to each point data, according to each point data corresponding The normal vector of the point data is determined, and the target object is determined based on the plane where the point data is located. The point data corresponding to the target object is in a plane, wherein the point information of the point data includes the coordinate information and reflection intensity of the point data. As another example, when the target object is a calibration plate, the corners of the calibration plate are points with high reflectivity, and the method of performing target detection on the point cloud data of the first radar and the second radar can be based on the midpoint of the point cloud data. According to the point information of the data, the reflection intensity corresponding to each point data is determined, the corner points of the calibration plate are located according to the reflection intensity, and the target object is determined based on the corner points of the calibration plate.

Step 206: Determine the image position of the target object based on the captured image of the target photographing device.

The image position of the target object refers to the position of the target object in the captured image. For example, the image position of the target object may specifically refer to a set of pixel coordinates of the target object in the captured image.

Specifically, after acquiring the captured image of the target shooting device, the server will perform target detection based on the captured image of the target shooting device, and determine the image position of the target object. In this embodiment, the method of performing target detection on the collected image is not limited, as long as the target detection can be realized. For example, when the target object is a vehicle or a building, the method of performing target detection on the collected image may be to use a target detection network that can realize target detection to realize the detection. As another example, when the target object is a calibration plate, the method of performing target detection on the captured image may be: performing corner detection on the captured image, determining the corners of the calibration plate, and then determining the target object based on the corners of the calibration plate. In the position in the collected image, the corner points of the standard calibration board are usually the corner points of the two-dimensional code, and the corner point detection can be realized by identifying the two-dimensional code.

Step 208 , perform external parameter calibration on the first radar and the second radar according to the image position of the target object, the point cloud of the first target object and the point cloud of the second target object, and obtain the external parameter calibration result of the radar.

Specifically, if the target shooting device is a single shooting device, and the single shooting device and the first radar and the second radar have overlapping fields of view, the server will The point cloud of the target object is to calibrate the single shooting device and the first radar, the single shooting device and the second radar respectively, and then obtain the external parameter calibration result of the radar according to the calibration results of the two external parameters. Specifically, if the target photographing device at least includes a first photographing device that has a field of view overlapping area with the first radar and a second photographing device that has a field of view overlapping area with the second radar, according to the image position of the target object, the first target The object point cloud and the second target object point cloud, when performing external parameter calibration on the first radar and the second radar, the server will first calibrate the first radar and the first shooting device, and the second radar and the second shooting device. After calibration, based on the two calibration results, the calibration results of the external parameters of the radar are obtained.

Further, when obtaining the external parameter calibration result of the radar based on the two calibration results, if the external parameters between the first shooting device and the second shooting device are known, the server may use the two calibration results and the first shooting device and the second shooting device. The external parameters between the two shooting devices are obtained, and the calibration results of the external parameters of the radar are obtained. The known external parameters between the first shooting device and the second shooting device means that the external parameters have been calibrated in advance on the first shooting device and the second shooting device, and the external parameters between the two are determined.

Further, when the external parameter calibration result of the radar is obtained based on the two calibration results, if the external parameters between the first shooting device and the second shooting device are unknown, the target shooting device will also include a separate camera with the first shooting device. There is a third shooting device in the overlapping area of the field of view with the second shooting device. At this time, the server will also calibrate the first shooting device, the second shooting device and the third shooting device according to the image position of the target object, and then combine the two The calibration results and the calibration results of the shooting equipment are used to obtain the calibration results of the external parameters of the radar.

The above-mentioned method for calibrating external parameters of radar based on photographing equipment, by acquiring the collected image of the target photographing equipment and the point cloud data of the radar to be calibrated, respectively, the target detection is performed on the point cloud data of the first radar and the second radar, and the target detection can be performed from the first radar. The target object is detected from the point cloud data of the second radar and the first target object, and the first target object point cloud and the second target object point cloud are obtained. The target object, and then the target object can be used as a benchmark, according to the target object image position of the target object in the collected image, the first target object point cloud matched by the first radar, and the second target object point cloud matched by the second radar, Perform external parameter calibration on the first radar and the second radar, and obtain the external parameter calibration result of the radar. The whole process can be achieved by using the target shooting equipment that overlaps the field of view of the radar to be calibrated, and based on the target object captured at the same time. Radar external parameter calibration.

In one embodiment, the target photographing device is a single photographing device, and the single photographing device and the first radar and the second radar both have a field of view overlapping area;

Based on the image position of the target object, the point cloud of the first target object, and the point cloud of the second target object, the external parameters of the first radar and the second radar are calibrated, and the obtained radar external parameter calibration results include:

Based on the image position of the target object and the point cloud of the first target object, the single shooting device and the first radar are calibrated, and the first external parameter calibration result is obtained;

Based on the image position of the target object and the point cloud of the second target object, the single shooting device and the second radar are calibrated, and the second external parameter calibration result is obtained;

Based on the first external parameter calibration result and the second external parameter calibration result, the radar external parameter calibration result is obtained.

Wherein, the target photographing device may be a single photographing device, and when the target photographing device is a single photographing device, the single photographing device and the first radar and the second radar both have overlapping fields of view. For example, the schematic diagram of the overlapping area of the field of view can be shown in FIG. 5 , 502 is the field of view area of the first radar, and 504 is the field of view area of the second radar. It can be seen from FIG. 5 that the first radar and the second radar There is no overlapping area of the field of view between them, 506 is the area of the field of view of a single photographing device, and the single photographing device and the first radar and the second radar both have the overlapping area of the field of view.

The first external parameter calibration result refers to the external parameter transformation relationship between a single shooting device and the first radar, including translation and rotation. The second external parameter calibration result refers to the external parameter transformation relationship between a single shooting device and the second radar, including translation and rotation. The radar external parameter calibration result refers to the external parameter transformation relationship between the first radar and the second radar, including translation and rotation.

Specifically, when the target photographing device is a single photographing device, the server will first based on the acquired external parameter transformation relationship between the single photographing device and the first radar, the image position of the target object, and the point cloud of the first target object. The single shooting device and the first radar are calibrated to obtain the first external parameter calibration result, and then based on the obtained external parameter transformation relationship between the single shooting device and the second radar, the image position of the target object and the second target object point Cloud, calibrate a single shooting device and the second radar, and obtain the calibration result of the second external parameter.

Specifically, since the first external parameter calibration result refers to the external parameter transformation relationship between a single shooting device and the first radar, and the second external parameter calibration result refers to the external parameter between a single shooting device and the second radar Transformation relationship, the server can use a single shooting device as an intermediary, based on the first external parameter calibration result and the second external parameter calibration result, determine the external parameter transformation relationship between the first radar and the second radar, that is, obtain the radar external parameter calibration result.

In this embodiment, by first determining the first external parameter calibration result between the single shooting device and the first radar, and then determining the second external parameter calibration result between the single shooting device and the second radar, it is possible to use a single device The shooting device is used as an intermediary, and based on the first external parameter calibration result and the second external parameter calibration result, the external parameter calibration between the first radar and the second radar is realized, and the radar external parameter calibration result is obtained.

In one embodiment, based on the image position of the target object and the point cloud of the first target object, calibrating a single shooting device and the first radar, and obtaining the first external parameter calibration result includes:

Based on the obtained external parameter transformation relationship, the first target object point cloud is projected to the coordinate system where the captured image is located, and the point cloud image position that matches the first target object point cloud is obtained. The external parameter transformation relationship is a single shooting device and the third The external parameter transformation relationship between radars;

According to the position of the point cloud image and the image position of the target object, a single shooting device and the first radar are calibrated, and the first external parameter calibration result is obtained.

The external parameter transformation relationship refers to the initial value of external parameters between the single shooting device and the first radar, which can be obtained based on the positional relationship between the single shooting device and the first radar, and is a rough external parameter transformation relationship. The position of the point cloud image that matches the point cloud of the first target object refers to the position of the point cloud of the first target object in the coordinate system where the captured image is located. For example, the position of the point cloud image that matches the point cloud of the first target object may specifically refer to the set of pixel coordinates of the point cloud of the first target object in the coordinate system where the captured image is located. Match each point data in the object point cloud.

Specifically, when calibrating a single shooting device and the first radar, the server will project the point cloud of the first target object to the collected image based on the acquired external parameter transformation relationship between the single shooting device and the first radar. In the coordinate system, the position of the point cloud image that matches the point cloud of the first target object is obtained. At this time, since the obtained external parameter transformation relationship is rough, as shown in Figure 6, after the first target object point cloud is projected to the coordinate system where the captured image is located, it does not completely overlap with the image position of the target object in the captured image. , the error here is the precision error between the single shooting device and the first radar when the device is installed, and the purpose of calibration is to eliminate the error here.

Specifically, in order to eliminate errors, the server will continuously optimize and iterate the obtained external parameter transformation relationship according to the point cloud image position and the target object image position to correct the external parameter transformation relationship, so that the point cloud image position and the target object image position Nearly overlapping, the first external parameter calibration result is obtained. Further, the obtained external parameter transformation relationship is continuously optimized and iterated to correct the external parameter transformation relationship. The server first determines the position error based on the position of the point cloud image and the image position of the target object, and then performs the external parameter transformation relationship. Optimization, based on the optimized external parameter transformation relationship, project the point cloud of the first target object to the coordinate system where the captured image is located again to obtain a new point cloud image position, and finally determine according to the new point cloud image position and the target object image position. For the new position error, when the optimization iteration does not meet the preset stop iteration condition, continue the optimization iteration until the optimization iteration meets the preset stop iteration condition, stop the optimization iteration, and take the latest optimized external parameter transformation relationship as the first external parameter. Parameter calibration results.

Wherein, the preset stop iteration condition can be set according to needs, which is not specifically limited in this embodiment. For example, the preset stop iteration condition may specifically be that the number of iterations reaches the threshold of the number of iterations, and the threshold of the number of iterations may be set by itself as required. As another example, the preset stop iteration condition may specifically be that the latest position error satisfies the error threshold, and the error threshold may be set by itself as required.

In the following, taking the stop iteration condition as the latest position error meeting the error threshold as an example, based on the image position of the target object and the point cloud of the first target object, a single shooting device and the first radar are calibrated, and the first external parameter calibration result is obtained. Explanation, assuming that the point cloud of the first target object is A, the obtained external parameter transformation relationship is B (the internal parameter transformation of the camera is coupled inside, here it is assumed that the internal parameters of a single shooting device have been accurately calibrated), and the image position of the target object is C , then there should be A*B=C, but because the external parameter transformation relationship B is not accurate enough, the actual obtained is A*B=C`, that is, the position error error=C-C`=C-A*B, through continuous optimization iteration, correction The external parameters in B finally make the error approach the error threshold and reach convergence, that is, the calibration is completed.

In this embodiment, by projecting the point cloud of the first target object to the coordinate system where the captured image is based on the obtained external parameter transformation relationship, the position of the point cloud image that matches the point cloud of the first target object can be obtained. The cloud image position and the target object image position realize the calibration of a single shooting device and the first radar.

In one embodiment, the target photographing device includes a first photographing device having a field of view overlapping area with the first radar, a second photographing device having a field of view overlapping area with the second radar, and the first photographing device and the second photographing device respectively The device has a third shooting device in the overlapping area of the field of view;

Determining the image position of the target object based on the captured image of the target shooting device includes:

Based on the captured image of each photographing device, the image position of the target object matched with each photographing device is determined respectively.

Wherein, the target photographing devices may be at least three photographing devices, and when the target photographing devices are at least three photographing devices, the target photographing devices include a first photographing device that has a field of view overlapping area with the first radar, and a first photographing device that has a visual field with the second radar A second photographing device in the field overlap area and a third photographing device having a field overlapping area with the first photographing device and the second photographing device, respectively. For example, the schematic diagram of the overlapping area of the field of view can be shown in FIG. 7 , 702 is the field of view area of the first radar, and 704 is the field of view area of the second radar. It can be seen from FIG. 7 that the first radar and the second radar There is no overlapping area of the field of view between them, 706 is the field of view area of the first photographing device, the first photographing device and the first radar have the overlapping area of the field of view, 708 is the field of view area of the second photographing device, the second photographing device and the The second radar has a field of view overlapping area, 710 is the field of view area of the third photographing device, and the third photographing device has a field of view overlapping area with the first photographing device and the second photographing device respectively.

Specifically, when the shooting devices are at least three shooting devices, each shooting device will have a corresponding captured image. When determining the image position of the target object, the server will determine the corresponding image for each shooting device based on the captured image of each shooting device. The image position of the target object matched by the device, that is, the position of the target object in the captured image.

In this embodiment, when the target photographing device includes at least three photographing devices, the image position of the target object matching each photographing device can be separately determined based on the collected images of each photographing device.

In one embodiment, based on the image position of the target object, the point cloud of the first target object, and the point cloud of the second target object, performing external parameter calibration on the first radar and the second radar, and obtaining the radar external parameter calibration result includes:

According to the first target object point cloud and the target object image position matched with the first shooting device, the first radar and the first shooting device are calibrated, and the third external parameter calibration result is obtained;

According to the second target object point cloud and the target object image position matched with the second photographing equipment, the second radar and the second photographing equipment are calibrated, and the fourth external parameter calibration result is obtained;

Based on the image position of the target object matched with each shooting device, the first shooting device, the second shooting device and the third shooting device are calibrated to obtain a calibration result of the shooting devices;

According to the calibration result of the third external parameter, the calibration result of the fourth external parameter and the calibration result of the shooting equipment, the calibration result of the radar external parameter is obtained.

The third external parameter calibration result refers to the external parameter transformation relationship between the first radar and the first photographing device, including translation and rotation. The fourth external parameter calibration result refers to the external parameter transformation relationship between the second radar and the second photographing device, including translation and rotation. The shooting device calibration result refers to the extrinsic parameter transformation relationship between the first shooting device and the third shooting device and between the second shooting device and the third shooting device.

Specifically, when the target photographing device includes a first photographing device, a second photographing device, and a third photographing device, since there is an overlapping field of view between the first photographing device and the first radar, and the second photographing device and the second radar There is a field of view overlapping area between them, and the server can calibrate the first radar and the first shooting device according to the first target object point cloud and the target object image position matching the first shooting device, and obtain the third external parameter calibration result , according to the second target object point cloud and the target object image position matching the second shooting device, the second radar and the second shooting device can be calibrated, and the fourth external parameter calibration result can be obtained.

Specifically, since the third photographing device and the first photographing device and the second photographing device respectively have overlapping fields of view, the server can realize the first photographing device, the first photographing device, the second photographing device The second shooting device and the third shooting device are calibrated, and the shooting device calibration result is obtained. After obtaining the third external parameter calibration result, the fourth external parameter calibration result and the shooting device calibration result, the server can use the first shooting device and the second shooting device. The device and the third shooting device are intermediary, and based on the third external parameter calibration result and the shooting device calibration result, the calibration relationship between the first radar and the second shooting device is determined, based on the calibration between the first radar and the second shooting device. The relationship and the fourth external parameter calibration result, through parameter conversion, to determine the external parameter transformation relationship between the first radar and the second radar, that is, to obtain the radar external parameter calibration result.

For example, in this embodiment, it is assumed that the first photographing device is camera A, the second photographing device is camera B, the third photographing device is camera C, the first radar is radar A, and the second radar is radar B. By calibrating the first radar and the first shooting device, the calibration result of the third external parameter can be obtained, that is, the calibration relationship of radar A->camera A. By calibrating the second radar and the second shooting device, the fourth external parameter can be obtained. The calibration result, that is, the calibration relationship of radar B->camera B, by calibrating the first shooting device, the second shooting device and the third shooting device, the calibration result of the shooting device can be obtained, that is, camera A->camera C->camera The calibration relationship of B, according to the calibration result of the third external parameter (the calibration relationship of radar A->camera A) and the calibration result of the shooting equipment (calibration relationship of camera A->camera C->camera B), the first radar and the camera can be obtained. The calibration relationship of the second shooting device, that is, radar A->camera A->camera C->camera B=radar A->camera B, which can be based on the calibration relationship of radar A->camera B and radar B->camera B The calibration relationship of radar A -> radar B is obtained, that is, the calibration result of radar external parameters is obtained, and the calibration of radar A and radar B is completed.

In this embodiment, by first calibrating the first radar and the first photographing device, the third external parameter calibration result is obtained, and then the second radar and the second photographing device are calibrated to obtain the fourth external parameter calibration result, and then the calibration result of the fourth external parameter is obtained. The first photographing equipment, the second photographing equipment and the third photographing equipment are calibrated to obtain the photographing equipment calibration result, which can be mediated by the first photographing equipment, the second photographing equipment and the third photographing equipment. The calibration results of the fourth external parameter and the calibration results of the shooting equipment are used to realize the calibration of the external parameters of the radar.

In one embodiment, the first photographing device, the second photographing device, and the third photographing device are calibrated based on the target object image position matched with each photographing device, and obtaining a photographing device calibration result includes:

Based on the respective matching target object image positions of the first photographing device and the third photographing device, the first photographing device and the third photographing device are calibrated to obtain a first device calibration result;

Based on the target object image positions matched with the second photographing equipment and the third photographing equipment, the second photographing equipment and the third photographing equipment are calibrated to obtain a calibration result of the second equipment;

According to the calibration result of the first device and the calibration result of the second device, the calibration result of the photographing device is obtained.

The first device calibration result refers to the calibration relationship between the first photographing device and the third photographing device. The calibration result of the second device refers to the calibration relationship between the second photographing device and the third photographing device.

Specifically, the server will first calibrate the first and third photographing devices based on the target object image positions that match the respective first and third photographing devices, to obtain the first device calibration result, and then based on the first and third photographing devices. According to the respective matching target image positions of the second photographing device and the third photographing device, the second photographing device and the third photographing device are calibrated to obtain the calibration result of the second device, and finally, according to the calibration result of the first device and the calibration result of the second device , to get the calibration results of the shooting equipment.

For example, in this embodiment, assuming that the first shooting device is camera A, the second shooting device is camera B, and the third shooting device is camera C, the server will first calibrate the first shooting device and the third shooting device, Obtain the calibration result of the first device (the calibration relationship of camera A->camera C), then calibrate the second shooting device and the third shooting device, and obtain the calibration result of the second device (the calibration relationship of camera A->camera C), Finally, according to the calibration result of the first device and the calibration result of the second device, the calibration result of the shooting device (the calibration relationship of camera A->camera C->camera B) is obtained.

Specifically, when calibrating the first photographing device and the third photographing device, the server will base on the acquired initial calibration relationship between the first photographing device and the third photographing device and the target object matching the first photographing device Image position, back-project the target object collected by the first shooting device to the coordinate system where the captured image of the third shooting device is located, and then compare the projected image position of the target object with the image position that matches the third shooting device. Whether the image position of the target object overlaps, by continuously optimizing and iterating the initial calibration relationship between the first shooting device and the third shooting device, until the optimization iteration reaches the preset stop condition, so that the projected image position of the target object is the same as the third one. The image position of the target object matched by the photographing device overlaps or is nearly overlapped, and the calibration is completed. The preset stop condition may be set by yourself as required. For example, the preset stop condition may specifically be that the number of optimization iterations reaches the threshold of the number of optimization iterations, and the threshold of the number of optimization iterations may be set by yourself as required. As another example, the preset stop condition may specifically be that the image position error between the two image positions is smaller than the preset image position error threshold, the preset image position error threshold may be set by yourself as required, and the image position error may specifically be the Euclidean distance, The Mahalanobis distance and the like are not specifically limited in this embodiment.

In this embodiment, by first calibrating the first photographing device and the third photographing device to obtain the calibration result of the first device, and then calibrating the second photographing device and the third photographing device to obtain the calibration result of the second device, the calibration result of the second device can be obtained according to The calibration result of the first device and the calibration result of the second device are used to obtain the calibration result of the photographing device, so as to realize the calibration between the photographing devices.

In one embodiment, the target photographing device includes a first photographing device having a field of view overlapping area with the first radar, a second photographing device having a field of view overlapping area with the second radar, and the first photographing device and the second photographing device When the external parameters of the shooting equipment between them are known, based on the image position of the target object, the point cloud of the first target object and the point cloud of the second target object, the external parameters of the first radar and the second radar are calibrated, and the external parameters of the radar are obtained. Calibration results include:

According to the first target object point cloud and the target object image position matched with the first shooting device, the first radar and the first shooting device are calibrated, and the third external parameter calibration result is obtained;

According to the second target object point cloud and the target object image position matched with the second photographing equipment, the second radar and the second photographing equipment are calibrated, and the fourth external parameter calibration result is obtained;

Based on the external parameters of the shooting equipment, the calibration results of the third external parameters and the calibration results of the fourth external parameters, the calibration results of the external parameters of the radar are obtained.

Wherein, the external parameters of the shooting device refer to external parameters obtained by calibrating the first shooting device and the second shooting device, and are used to represent the conversion relationship between the parameters of the first shooting device and the second shooting device.

Specifically, if the external parameters of the shooting equipment between the first shooting equipment and the second shooting equipment are known, it indicates that the first shooting equipment and the second shooting equipment have been calibrated in advance. At this time, the server only needs to firstly calibrate the first radar Perform calibration with the first shooting device to obtain the third external parameter calibration result, calibrate the second radar and the second shooting device, and obtain the fourth external parameter calibration result, which can be based on the external parameters of the shooting device and the third external parameter calibration results. and the fourth external parameter calibration result to obtain the radar external parameter calibration result. The method for calibrating the first photographing device and the second photographing device is not limited in this embodiment, as long as the calibration can be realized, and the calibration method can be the same as that for the first photographing device and the third photographing device in the above-mentioned embodiment. The calibration method is the same, and details are not described here in this embodiment.

In this embodiment, in the case where the external parameters of the shooting device between the first shooting device and the second shooting device are known, it can be achieved based on the third external parameter calibration result, the fourth external parameter calibration result, and the shooting device external parameters. Radar external parameter calibration.

In one embodiment, when the target photographing device is a single photographing device, and the single photographing device and the first radar and the second radar both have overlapping fields of view, as shown in FIG. 8 , a schematic flowchart is used to illustrate this The proposed method for calibrating radar external parameters based on photographing equipment specifically includes the following steps:

Step 802, acquiring the collected image of a single shooting device and the point cloud data of the radar to be calibrated, the single shooting device and the radar to be calibrated have an overlapping field of view, and the radar to be calibrated includes a first radar and a second radar;

Step 804, performing target detection on the point cloud data of the first radar to obtain a first target object point cloud, and performing target detection on the point cloud data of the second radar to obtain a second target object point cloud;

Step 806, determining the image position of the target object based on the captured image of a single shooting device;

Step 808: Based on the obtained external parameter transformation relationship, project the point cloud of the first target object to the coordinate system where the captured image is located to obtain the point cloud image position that matches the point cloud of the first target object, and the external parameter transformation relationship is a single camera shooting. The external parameter transformation relationship between the device and the first radar;

Step 810, according to the position of the point cloud image and the position of the target object image, calibrate the single shooting device and the first radar, and obtain the first external parameter calibration result;

Step 812, based on the image position of the target object and the point cloud of the second target object, calibrate the single shooting device and the second radar to obtain a second external parameter calibration result;

Step 814, based on the first external parameter calibration result and the second external parameter calibration result, obtain the radar external parameter calibration result.

In one embodiment, when the target photographing device includes a first photographing device having a field of view overlapping area with the first radar, a second photographing device having a field of view overlapping area with the second radar, and the first photographing device and the second photographing device respectively When the photographing equipment has a third photographing equipment in the overlapping area of the field of view, as shown in FIG. 9 , the method for calibrating the external parameters of radar based on the photographing equipment of the present application is illustrated by a schematic flowchart. Include the following steps:

Step 902 , acquiring the captured image of the target photographing device and the point cloud data of the radar to be calibrated, the target photographing device and the radar to be calibrated have a field of view overlapping area, and the radar to be calibrated includes a first radar and a second radar;

Step 904, performing target detection on the point cloud data of the first radar to obtain a first target object point cloud, and performing target detection on the point cloud data of the second radar to obtain a second target object point cloud;

Step 906, based on the captured images of each shooting device, respectively determine the target object image position matching each shooting device;

Step 908, according to the first target object point cloud and the target object image position matched with the first shooting device, calibrate the first radar and the first shooting device, and obtain a third external parameter calibration result;

Step 910, according to the second target object point cloud and the target object image position matched with the second shooting device, calibrate the second radar and the second shooting device, and obtain a fourth external parameter calibration result;

Step 912, based on the respective matching target object image positions of the first photographing device and the third photographing device, calibrate the first photographing device and the third photographing device, and obtain a calibration result of the first device;

Step 914, based on the target object image positions matched with the second photographing equipment and the third photographing equipment respectively, calibrate the second photographing equipment and the third photographing equipment, and obtain the second equipment calibration result;

Step 916, obtaining the calibration result of the photographing device according to the calibration result of the first device and the calibration result of the second device;

Step 918: Obtain a radar external parameter calibration result according to the third external parameter calibration result, the fourth external parameter calibration result and the photographing device calibration result.

In one embodiment, when the target photographing device includes a first photographing device having a field of view overlapping area with the first radar, a second photographing device having a field of view overlapping area with the second radar, and the first photographing device and the second photographing device When the external parameters of the shooting device between the devices are known, as shown in FIG. 10 , a schematic flowchart is used to illustrate the method for calibrating the radar external parameters based on the shooting device of the present application. The method for calibrating the external parameters of the radar based on the shooting device specifically includes the following: step:

Step 1002, acquiring the collected image of the target shooting device and the point cloud data of the radar to be calibrated, the target shooting device and the radar to be calibrated have a field of view overlapping area, and the radar to be calibrated includes a first radar and a second radar;

Step 1004, performing target detection on the point cloud data of the first radar to obtain a first target object point cloud, and performing target detection on the point cloud data of the second radar to obtain a second target object point cloud;

Step 1006, based on the captured images of each shooting device, respectively determine the target object image position that matches each shooting device;

Step 1008, according to the first target object point cloud and the target object image position matched with the first shooting device, calibrate the first radar and the first shooting device, and obtain a third external parameter calibration result;

Step 1010, according to the second target object point cloud and the target object image position matched with the second shooting device, calibrate the second radar and the second shooting device, and obtain a fourth external parameter calibration result;

Step 1012 , based on the external parameters of the photographing device, the calibration results of the third external parameters and the calibration results of the fourth external parameters, obtain the calibration results of the external parameters of the radar. In one embodiment, the radar external parameter calibration method based on the photographing device provided in the embodiment of the present application can be applied to the application environment shown in FIG. 11 .

Suppose there are two cars parked on the road, car A is the target car equipped with lidar A (first radar) and lidar B (second radar), and there is no overlap in the field of view between lidar A and lidar B area, car B is an automatic driving vehicle equipped with a camera (target shooting device), then car A can use the image information of car B to perform online calibration and inspection work, that is, use the shooting device provided by the embodiment of the present application. The external parameter calibration method of Lidar A and Lidar B on vehicle A is realized. At this time, the server will obtain the collected images of the target shooting device and the point cloud data of Lidar A and Lidar B. Perform target detection on the point cloud data of lidar A and lidar B respectively, and obtain the first target object point cloud matching lidar A and the second target object point cloud matching lidar B, based on the camera's acquisition The image determines the image position of the target object. According to the image position of the target object, the point cloud of the first target object and the point cloud of the second target object, the external parameters of the laser radar A and the laser radar B are calibrated, and the external parameter calibration result of the radar is obtained.

Through this operation of using the images collected by the cameras of the surrounding vehicles to update the calibration parameters online, the calibration parameters of the vehicle can always be kept in the most accurate state, even if the sensor is displaced due to some vibration, it can still be Quickly self-correct.

It should be understood that, although the steps in the flowcharts involved in the above embodiments are sequentially displayed according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but may be performed at different times The execution order of these steps or phases is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or phases in the other steps.

Based on the same inventive concept, an embodiment of the present application further provides a camera-based device-based radar external parameter calibration device for implementing the above-mentioned camera-based device-based radar external parameter calibration method. The solution to the problem provided by the device is similar to the solution described in the above method, so the specific limitations in one or more embodiments of the radar external parameter calibration device based on the photographing device provided below can refer to the above for The limitation of the radar external parameter calibration method based on the photographing equipment will not be repeated here.

In one embodiment, as shown in FIG. 12 , a radar external parameter calibration device based on a photographing device is provided, including: an acquisition module 1202 , a target detection module 1204 , a processing module 1206 and a calibration module 1208 , wherein:

The acquisition module 1202 is used for acquiring the collected image of the target shooting device and the point cloud data of the radar to be calibrated, the target shooting device and the radar to be calibrated have a field of view overlapping area, and the radar to be calibrated includes a first radar and a second radar;

The target detection module 1204 is configured to perform target detection on the point cloud data of the first radar to obtain the first target object point cloud, and perform target detection on the point cloud data of the second radar to obtain the second target object point cloud;

a processing module 1206, configured to determine the image position of the target object based on the captured image of the target shooting device;

The calibration module 1208 is configured to perform external parameter calibration on the first radar and the second radar according to the image position of the target object, the point cloud of the first target object and the point cloud of the second target object, and obtain a calibration result of the external parameters of the radar.

The above-mentioned radar external parameter calibration device based on the shooting equipment, by acquiring the collected images of the target shooting equipment and the point cloud data of the radar to be calibrated, respectively performs target detection on the point cloud data of the first radar and the second radar, and can obtain the target detection from the first radar. The target object is detected from the point cloud data of the second radar and the first target object, and the first target object point cloud and the second target object point cloud are obtained. The target object, and then the target object can be used as a benchmark, according to the target object image position of the target object in the collected image, the first target object point cloud matched by the first radar, and the second target object point cloud matched by the second radar, Perform external parameter calibration on the first radar and the second radar, and obtain the external parameter calibration result of the radar. The whole process can be achieved by using the target shooting equipment that overlaps the field of view of the radar to be calibrated, and based on the target object captured at the same time. Radar external parameter calibration.

In one embodiment, the target photographing device is a single photographing device, and the single photographing device and the first radar and the second radar both have a field of view overlapping area, and the calibration module is further configured to be based on the image position of the target object and the point of the first target object Cloud, calibrate a single shooting device and the first radar to obtain the first external parameter calibration result, based on the image position of the target object and the point cloud of the second target object, calibrate the single shooting device and the second radar, and obtain the second The external parameter calibration result, based on the first external parameter calibration result and the second external parameter calibration result, obtains the radar external parameter calibration result.

In one embodiment, the calibration module is further configured to project the point cloud of the first target object to the coordinate system where the captured image is located based on the obtained external parameter transformation relationship, to obtain a point cloud image position that matches the point cloud of the first target object, The external parameter transformation relationship is the external parameter transformation relationship between a single shooting device and the first radar. According to the point cloud image position and the target object image position, the single shooting device and the first radar are calibrated to obtain the first external parameter calibration. result.

In one embodiment, the target photographing device includes a first photographing device having a field of view overlapping area with the first radar, a second photographing device having a field of view overlapping area with the second radar, and the first photographing device and the second photographing device respectively The device has a third photographing device in the overlapping area of the field of view, and the processing module is further configured to determine the image position of the target object matching each photographing device based on the captured images of each photographing device.

In one embodiment, the calibration module is further configured to calibrate the first radar and the first photographing device according to the point cloud of the first target object and the image position of the target object matched with the first photographing device, so as to obtain the third external parameter calibration As a result, according to the second target object point cloud and the target object image position matched with the second shooting device, the second radar and the second shooting device are calibrated, and a fourth external parameter calibration result is obtained. Matching the image position of the target object, calibrate the first shooting device, the second shooting device and the third shooting device, and obtain the shooting device calibration result, according to the third external parameter calibration result, the fourth external parameter calibration result and the shooting device calibration result , and get the calibration results of radar external parameters.

In one embodiment, the calibration module is further configured to calibrate the first photographing device and the third photographing device based on the target object image positions matched with the first photographing device and the third photographing device respectively, to obtain a first device calibration result , based on the target object image positions matched with the second and third shooting devices respectively, the second and third shooting devices are calibrated to obtain the second device calibration result, according to the first device calibration result and the second The equipment calibration result is obtained, and the shooting equipment calibration result is obtained.

In one embodiment, the target photographing device includes a first photographing device having a field of view overlapping area with the first radar, a second photographing device having a field of view overlapping area with the second radar, and the first photographing device and the second photographing device In the case where the external parameters of the photographing equipment are known, the calibration module is further configured to calibrate the first radar and the first photographing equipment according to the first target object point cloud and the target object image position matching the first photographing equipment. , obtain the third external parameter calibration result, and calibrate the second radar and the second shooting device according to the second target object point cloud and the target object image position matched with the second shooting device, and obtain the fourth external parameter calibration result, Based on the external parameters of the shooting equipment, the calibration results of the third external parameters and the calibration results of the fourth external parameters, the calibration results of the external parameters of the radar are obtained.

Each module in the above-mentioned apparatus for calibrating external parameters of radar based on photographing equipment can be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, and the computer device may be a server, and its internal structure diagram may be as shown in FIG. 13 . The computer device includes a processor, a memory, an input/output interface (Input/Output, I/O for short) and a communication interface. Wherein, the processor, the memory and the input/output interface are connected through the system bus, and the communication interface is connected to the system bus through the input/output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The nonvolatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used to store the collected images of the target shooting equipment, the point cloud data of the radar to be calibrated and other data. The input/output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for calibrating external parameters of radar based on a photographing device is realized.

Those skilled in the art can understand that the structure shown in FIG. 13 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is also provided, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the foregoing method embodiments when the processor executes the computer program.

In one embodiment, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, implements the steps in the foregoing method embodiments.

In one embodiment, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the foregoing method embodiments.

It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all data authorized by the user or fully authorized by all parties, and the relevant data Collection, use and processing need to comply with relevant laws, regulations and standards of relevant countries and regions.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to a memory, a database or other media used in the various embodiments provided in this application may include at least one of a non-volatile memory and a volatile memory. Non-volatile memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetic variable memory (Magnetoresistive Random Memory) Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration and not limitation, the RAM may be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM). The database involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the present application should be determined by the appended claims.

Claims (10)

1. A radar external reference calibration method based on shooting equipment is characterized by comprising the following steps:

acquiring an acquired image of target shooting equipment and point cloud data of a radar to be calibrated, wherein the target shooting equipment and the radar to be calibrated have a field-of-view overlapping region, and the radar to be calibrated comprises a first radar and a second radar;

performing target detection on the point cloud data of the first radar to obtain a first target object point cloud, and performing target detection on the point cloud data of the second radar to obtain a second target object point cloud;

determining a target object image position based on the acquired image of the target shooting device;

and carrying out external reference calibration on the first radar and the second radar according to the target object image position, the first target object point cloud and the second target object point cloud to obtain a radar external reference calibration result.

2. The method of claim 1, wherein the target capture device is a single capture device having an overlapping region of fields of view with both the first radar and the second radar;

the external reference calibration of the first radar and the second radar is performed based on the target object image position, the first target object point cloud and the second target object point cloud, and obtaining a radar external reference calibration result comprises:

calibrating the single shooting device and the first radar based on the target object image position and the first target object point cloud to obtain a first external reference calibration result;

calibrating the single shooting device and the second radar based on the target object image position and the second target object point cloud to obtain a second external reference calibration result;

and obtaining a radar external reference calibration result based on the first external reference calibration result and the second external reference calibration result.

3. The method of claim 2, wherein calibrating the single camera and the first radar based on the target object image position and the first target object point cloud to obtain a first external reference calibration result comprises:

based on the obtained external parameter transformation relation, projecting the first target object point cloud to a coordinate system where the collected image is located to obtain a point cloud image position matched with the first target object point cloud, wherein the external parameter transformation relation is the external parameter transformation relation between the single shooting device and the first radar;

and calibrating the single shooting device and the first radar according to the position of the point cloud image and the position of the target object image to obtain a first external reference calibration result.

4. The method according to claim 1, wherein the target photographing apparatus includes a first photographing apparatus having a field of view overlapping region with the first radar, a second photographing apparatus having a field of view overlapping region with the second radar, and a third photographing apparatus having a field of view overlapping region with the first photographing apparatus and the second photographing apparatus, respectively;

the determining a target object image position based on the captured image of the target capturing device comprises:

and respectively determining the position of the target object image matched with each shooting device based on the acquired image of each shooting device.

5. The method of claim 4, wherein the performing external reference calibration on the first radar and the second radar based on the target object image position, the first target object point cloud, and the second target object point cloud, and obtaining a radar external reference calibration result comprises:

calibrating the first radar and the first shooting equipment according to the first target object point cloud and the target object image position matched with the first shooting equipment to obtain a third external reference calibration result;

calibrating the second radar and the second shooting equipment according to the second target object point cloud and the target object image position matched with the second shooting equipment to obtain a fourth external reference calibration result;

calibrating the first shooting device, the second shooting device and the third shooting device based on the target object image position matched with each shooting device to obtain shooting device calibration results;

and obtaining a radar external reference calibration result according to the third external reference calibration result, the fourth external reference calibration result and the shooting equipment calibration result.

6. The method according to claim 5, wherein the calibrating the first shooting device, the second shooting device and the third shooting device based on the target object image position matched with each shooting device comprises:

calibrating the first shooting device and the third shooting device based on the positions of the target object images respectively matched with the first shooting device and the third shooting device to obtain a first device calibration result;

calibrating the second shooting device and the third shooting device based on the target object image positions matched with the second shooting device and the third shooting device respectively to obtain a second device calibration result;

and obtaining a calibration result of the shooting equipment according to the first equipment calibration result and the second equipment calibration result.

7. A radar external reference calibration device based on shooting equipment is characterized by comprising:

the system comprises an acquisition module, a calibration module and a calibration module, wherein the acquisition module is used for acquiring an acquired image of target shooting equipment and point cloud data of a radar to be calibrated, the target shooting equipment and the radar to be calibrated have a field-of-view overlapping region, and the radar to be calibrated comprises a first radar and a second radar;

the target detection module is used for carrying out target detection on the point cloud data of the first radar to obtain a first target object point cloud and carrying out target detection on the point cloud data of the second radar to obtain a second target object point cloud;

the processing module is used for determining the position of a target object image based on the acquired image of the target shooting equipment;

and the calibration module is used for carrying out external parameter calibration on the first radar and the second radar according to the target object image position, the first target object point cloud and the second target object point cloud to obtain a radar external parameter calibration result.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 6.

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 6 when executed by a processor.

Images