CN110555886B

CN110555886B - Vehicle-mounted camera external parameter calibration method and device, electronic equipment and storage medium

Info

Publication number: CN110555886B
Application number: CN201810550979.8A
Authority: CN
Inventors: 谢小燕
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2018-05-31
Filing date: 2018-05-31
Publication date: 2021-09-17
Anticipated expiration: 2038-05-31
Also published as: CN110555886A

Abstract

The embodiment of the invention provides a method and a device for calibrating external parameters of a vehicle-mounted camera, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a first image obtained by shooting three parallel straight lines on the ground by using vehicle-mounted cameras in the left and right directions of a vehicle body, wherein the three straight lines are parallel to a transverse shaft of the vehicle during shooting; respectively calculating external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to a first vehicle body coordinate system based on the first image and the first distances between the three straight lines and the vehicle transverse axis; according to the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system, the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are calculated; therefore, the problem that the accuracy of the vehicle-mounted camera external parameters on the left and right of the vehicle body obtained through calculation in the prior art is not high can be solved.

Description

Vehicle-mounted camera external parameter calibration method and device, electronic equipment and storage medium

Technical Field

The invention relates to the technical field of computer vision, in particular to a vehicle-mounted camera external parameter calibration method and device, electronic equipment and a storage medium.

Background

In order to obtain a panoramic image of an automobile, it is generally necessary to shoot an environment around the automobile body by vehicle-mounted cameras mounted in four directions, namely, front, rear, left and right directions of the automobile body, and to stitch four shot images together to form the panoramic image.

After the vehicle-mounted cameras are installed at the front, rear, left and right directions of the vehicle body, in order to determine the relative position of each vehicle-mounted camera relative to the vehicle body, external parameters of each vehicle-mounted camera need to be calibrated, wherein the external parameters represent the relative position relationship between the coordinate system of the vehicle-mounted camera and the coordinate system of the vehicle body.

The existing vehicle-mounted camera external parameter calibration method is a three-line calibration method, firstly, a longitudinal axis of a vehicle is kept parallel to three parallel lines on a flat ground, then the distance from the three parallel lines to the longitudinal axis of the vehicle is measured, then a vehicle-mounted camera on the vehicle is used for shooting the three parallel lines, according to a perspective projection principle, imaging straight lines of the three straight lines in a shot image have the same vanishing point (namely intersection point) and different slopes, and according to the vanishing point and the different slopes of the three imaging straight lines, the distances from the three parallel lines to the longitudinal axis of the vehicle, which are measured before, are combined, and the external parameter of the vehicle-mounted camera can be calculated.

However, when the method is used for performing external reference calibration on the two vehicle-mounted cameras on the left and right sides of the vehicle body, because the optical axis directions of the two vehicle-mounted cameras on the left and right sides of the vehicle body are perpendicular to the three parallel lines, after the three parallel lines are photographed by the two vehicle-mounted cameras on the left and right sides of the vehicle body, the three straight lines in the photographed image are approximately parallel, that is, the common vanishing point intersects a far place, a large error exists in calculating the slope, and the accuracy of the external reference of the vehicle-mounted cameras on the left and right sides of the vehicle body calculated by the method is not high.

Disclosure of Invention

The embodiment of the invention aims to provide a method and a device for calibrating vehicle-mounted camera external parameters, electronic equipment and a storage medium, which can solve the problem that the accuracy of the vehicle-mounted camera external parameters on the left and right of a vehicle body obtained through calculation in the prior art is not high.

The specific technical scheme is as follows:

the embodiment of the invention provides a method for calibrating external parameters of a vehicle-mounted camera, which comprises the following steps:

obtaining a first image obtained by shooting at least three parallel straight lines on the ground by using vehicle-mounted cameras in the left and right directions of a vehicle body, wherein the at least three straight lines are parallel to a transverse shaft of the vehicle during shooting;

respectively calculating external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to a first vehicle body coordinate system based on the first image and the first distances between the at least three straight lines and the vehicle transverse axis; the first vehicle body coordinate system takes a vehicle transverse axis as an X axis, and a vehicle longitudinal axis as a Y axis;

according to the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system, calculating the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system; the second body coordinate system takes the longitudinal axis of the vehicle as an X axis, and the transverse axis of the vehicle as a Y axis.

Optionally, the method further includes:

obtaining a second image obtained by shooting at least three parallel straight lines on the ground by using vehicle-mounted cameras in the front and rear directions of a vehicle body, wherein the at least three straight lines are parallel to the longitudinal axis of the vehicle during shooting;

and calculating the external parameters of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to the second vehicle body coordinate system respectively based on the second image and the second distances between the at least three straight lines and the longitudinal axis of the vehicle.

Optionally, the method further includes:

and calculating the external parameters of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to the first vehicle body coordinate system according to the external parameters of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to the second vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system.

Optionally, the calculating the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system includes:

respectively determining rotation matrixes of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the first vehicle body coordinate system according to external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system;

respectively calculating rotation matrixes of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the rotation matrixes of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrixes of the second vehicle body coordinate system relative to the first vehicle body coordinate system;

and calculating the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the rotation matrix of the vehicle-mounted camera coordinate system in the left and right directions of the vehicle body relative to the second vehicle body coordinate system.

Optionally, the calculating, according to the rotation matrix of the vehicle-mounted camera coordinate system in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system, the rotation matrix of the vehicle-mounted camera coordinate system in the left and right directions of the vehicle body relative to the second vehicle body coordinate system includes:

according to a rotation matrix R of the vehicle-mounted camera coordinate system in the left direction of the vehicle body relative to the first vehicle body coordinate system_l1And a rotation matrix R of a second body coordinate system with respect to the first body coordinate system₂₁Calculating a rotation matrix R of the vehicle-mounted camera coordinate system in the left direction of the vehicle body relative to the second vehicle body coordinate system according to the following formula_l2：

R_l2＝R₂₁*R_l1；

According to the rotation matrix R of the vehicle-mounted camera coordinate system in the right direction of the vehicle body relative to the first vehicle body coordinate system_r1And a rotation matrix R of a second body coordinate system with respect to the first body coordinate system₂₁Calculating a rotation matrix R of the vehicle-mounted camera coordinate system in the right direction of the vehicle body relative to the second vehicle body coordinate system according to the following formula_r2：

R_r2＝R₂₁*R_r1。

Optionally, the method further includes:

acquiring multiple groups of external parameters of the vehicle-mounted camera, wherein each group of external parameters comprises external parameters of the vehicle-mounted camera in the front, rear, left and right directions of a vehicle body, and at least three parallel straight lines on which different groups of external parameters are calculated are not all the same;

splicing images shot by the vehicle-mounted camera by using each group of external parameters to obtain panoramic images;

and determining the panoramic image with the least splicing gaps in the panoramic image, and taking the external parameter corresponding to the panoramic image as the calibration external parameter of the vehicle-mounted camera.

Optionally, the method further includes:

acquiring multiple groups of external parameters of the vehicle-mounted camera, wherein each group of external parameters comprises external parameters of the vehicle-mounted camera in the front, rear, left and right directions of a vehicle body, at least three parallel straight lines on which different groups of external parameters are calculated are the same, and the distances between the vehicle body and the at least three straight lines are different when the different groups of external parameters are calculated;

The embodiment of the invention also provides a vehicle-mounted camera external parameter calibration device, which comprises:

the first image acquisition module is used for acquiring a first image obtained by shooting at least three parallel straight lines on the ground by using vehicle-mounted cameras in the left and right directions of a vehicle body, wherein the at least three straight lines are parallel to a transverse shaft of the vehicle during shooting;

the first external parameter calculating module is used for calculating external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to a first vehicle body coordinate system respectively based on the first image and first distances between the at least three straight lines and the vehicle transverse axis; the first vehicle body coordinate system takes a vehicle transverse axis as an X axis, and a vehicle longitudinal axis as a Y axis;

the first external parameter conversion module is used for calculating external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and a rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system; the second body coordinate system takes the longitudinal axis of the vehicle as an X axis, and the transverse axis of the vehicle as a Y axis.

Optionally, the apparatus further comprises:

the second image acquisition module is used for acquiring a second image obtained by shooting at least three parallel straight lines on the ground by using vehicle-mounted cameras in the front and rear directions of a vehicle body, wherein the at least three straight lines are parallel to the longitudinal axis of the vehicle during shooting;

and the second external parameter calculating module is used for calculating external parameters of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to the second vehicle body coordinate system respectively based on the second image and second distances between the at least three straight lines and the longitudinal axis of the vehicle.

Optionally, the apparatus further comprises:

and the second external parameter conversion module is used for calculating the external parameters of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to the first vehicle body coordinate system according to the external parameters of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to the second vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system.

Optionally, the first external parameter converting module includes:

the first matrix determining module is used for respectively determining rotation matrixes of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the first vehicle body coordinate system according to external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system;

the second matrix determination module is used for respectively calculating rotation matrixes of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the rotation matrixes of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrixes of the second vehicle body coordinate system relative to the first vehicle body coordinate system;

and the calculation module is used for calculating the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the rotation matrix of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the second vehicle body coordinate system.

Optionally, the second matrix determining module is specifically configured to:

R_l2＝R₂₁*R_l1；

R_r2＝R₂₁*R_r1。

Optionally, the apparatus further comprises:

the first external parameter acquisition module is used for acquiring multiple groups of external parameters of the vehicle-mounted camera, each group of external parameters comprises external parameters of the vehicle-mounted camera in the front, rear, left and right directions of the vehicle body, and at least three parallel straight lines based on which different groups of external parameters are calculated are not all the same;

the first splicing module is used for splicing images shot by the vehicle-mounted camera by using each group of external parameters to obtain a panoramic image;

and the first external parameter determining module is used for determining the panoramic image with the least splicing gaps in the panoramic image and taking the external parameter corresponding to the panoramic image as the calibration external parameter of the vehicle-mounted camera.

Optionally, the apparatus further comprises:

the second external parameter acquisition module is used for acquiring multiple groups of external parameters of the vehicle-mounted camera, each group of external parameters comprises external parameters of the vehicle-mounted camera in the front, rear, left and right directions of the vehicle body, at least three parallel straight lines based on which different groups of external parameters are calculated are the same, and the distances between the vehicle body and the at least three straight lines are different when different groups of external parameters are calculated;

the second splicing module is used for splicing the images shot by the vehicle-mounted camera by using each group of external parameters to obtain panoramic images;

and the second external parameter determining module is used for determining the panoramic image with the least splicing gaps in the panoramic image and taking the external parameter corresponding to the panoramic image as the calibration external parameter of the vehicle-mounted camera.

Embodiments of the present invention also provide an electronic device, including a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to: implementing any of the above method steps.

An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the computer program implements any of the above method steps.

By using the method, the device, the electronic equipment and the storage medium for calibrating the external parameters of the vehicle-mounted camera, the external parameters of the vehicle-mounted camera in the left and right directions of the vehicle body relative to a first vehicle body coordinate system can be calculated, wherein the first vehicle body coordinate system takes a vehicle transverse axis as an X axis and a vehicle longitudinal axis as a Y axis, and the external parameters of the vehicle-mounted camera in the left and right directions of the vehicle body relative to a second vehicle body coordinate system are calculated by combining a rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system. Because the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are calculated according to the rotation matrix, the problem that when the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are solved by shooting images, three straight lines in the images are approximately parallel, and the intersection point is at infinity, so that the calculation result is inaccurate is avoided. Namely, the calculation precision of the external parameters is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a flowchart of a method for calibrating external parameters of a vehicle-mounted camera according to an embodiment of the present invention;

fig. 2 is a schematic diagram of marking external references of the left and right direction vehicle-mounted cameras according to the embodiment of the present invention;

FIG. 3 is a schematic diagram of three straight imaging lines in a captured image according to an embodiment of the present invention;

FIG. 4 is a schematic view of the present invention for adjusting the orientation of a vehicle body;

FIG. 5 is a schematic view of the vehicle body according to the embodiment of the present invention after being adjusted in direction;

FIG. 6 is a schematic diagram of an external reference calibration based on multiple parallel lines according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of vehicle-mounted camera external reference calibration based on different vehicle body positions according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of an external reference calibration apparatus of a vehicle-mounted camera according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In order to solve the problems in the prior art, the embodiment of the invention discloses a method and a device for calibrating external parameters of a vehicle-mounted camera, an electronic device and a storage medium, which can acquire the external parameters of the vehicle-mounted camera in the front direction, the rear direction, the left direction and the right direction of a vehicle body relative to a common vehicle body coordinate system.

Referring to fig. 1, an embodiment of the present invention provides a method for calibrating external reference of a vehicle-mounted camera, including the following steps:

step S101: and acquiring a first image obtained by shooting three parallel straight lines on the ground by using vehicle-mounted cameras in the left and right directions of the vehicle body, wherein the three straight lines are parallel to the transverse axis of the vehicle during shooting. Step S102: respectively calculating external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to a first vehicle body coordinate system based on the first image and the first distances between the three straight lines and the vehicle transverse axis; the first vehicle body coordinate system takes a vehicle transverse axis as an X axis, and a vehicle longitudinal axis as a Y axis.

Referring to fig. 2, fig. 2 is a schematic diagram of marking vehicle exterior references in left and right directions of a vehicle body according to an embodiment of the present invention, and taking three straight lines as an example, as shown in fig. 2, three parallel straight lines l1, l2, and l3 on the ground are kept parallel to a transverse axis X axis of a vehicle carrying vehicle-mounted cameras in front, rear, left, and right directions, where the three straight lines on the ground may be existing, such as zebra stripes, and the like, and may also be drawn. The distances d1, d2, and d3 of the three straight lines from the lateral axis of the vehicle are then measured.

Three straight lines l1, l2 and l3 are photographed by using vehicle-mounted cameras in the left and right directions of a vehicle body to respectively obtain two images, and external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to a first vehicle body coordinate system can be calculated according to the two images and the distances d1, d2 and d3 of the three straight lines l1, l2 and l3 relative to the transverse axis of the vehicle respectively, wherein the external parameters are the roll angle psi, the pitch angle theta and the direction angle theta of the vehicle-mounted cameras relative to the vehicle body

As can be seen from fig. 2, the first vehicle body coordinate system is a coordinate system with the vehicle transverse axis as the X axis and the vehicle longitudinal axis as the Y axis.

How to specifically solve the external reference of the vehicle-mounted camera belongs to the category of the prior art, and is not described herein.

Step S103: according to the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system, calculating the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system; the second body coordinate system takes the longitudinal axis of the vehicle as an X axis, and the transverse axis of the vehicle as a Y axis.

By using the vehicle-mounted camera external parameter calibration method provided by the embodiment of the invention, the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system can be calculated, wherein the first vehicle body coordinate system takes the vehicle transverse axis as the X axis and the vehicle longitudinal axis as the Y axis, and the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are calculated by combining the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system. Because the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are calculated according to the rotation matrix, the problem that when the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are solved by shooting images, three straight lines in the images are approximately parallel, and the intersection point is at infinity, so that the calculation result is inaccurate is avoided. Namely, the calculation precision of the external parameters is improved.

It should be noted that, in the present invention, under the condition that the front and back cameras are normally calibrated, the external references of the left and right cameras are continuously calibrated by rotating the vehicle body or adding parallel lines, and the situation that three straight lines are approximately parallel does not occur when the external references of the left and right cameras are calibrated, so the calibration accuracy is high, but the coordinate system of the external references of the calibrated left and right cameras relative to the coordinate system is changed by rotating the vehicle body or adding parallel lines, so the external references of the left and right cameras are converted by multiplying the rotation matrix, and the obtained external references in the front, back, left and right directions can be relative to the same coordinate system, so the accuracy of external reference calibration is improved.

In the embodiment of the invention, after the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system are obtained through calculation, the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system can be calculated.

The specific calculation process may include the following three steps:

the method comprises the following steps: respectively determining rotation matrixes of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the first vehicle body coordinate system according to external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system;

taking the vehicle-mounted camera in the left direction of the vehicle body as an example, the external reference of the vehicle-mounted camera relative to the first vehicle body coordinate system is assumed as follows: side inclination psi₁Angle of pitch theta₁And angle of direction

Then a rotation matrix R of the vehicle-mounted camera coordinate system in the left direction of the vehicle body relative to the first vehicle body coordinate system can be obtained_l1Comprises the following steps:

it should be noted that any method that can determine the rotation matrix of the vehicle-mounted camera coordinate system relative to the vehicle body coordinate system by using the external reference of the vehicle-mounted camera relative to the vehicle body coordinate system can be applied to the above method flow of the present invention, and the present invention is not limited in particular.

Step two: respectively calculating rotation matrixes of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the rotation matrixes of the vehicle-mounted camera coordinate systems in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrixes of the second vehicle body coordinate system relative to the first vehicle body coordinate system;

in this step, the rotation matrix R with respect to the first vehicle body coordinate system may be based on the vehicle-mounted camera coordinate system in the left direction of the vehicle body_l1And a rotation matrix R of the second body coordinate system relative to the first body coordinate system₂₁A rotation matrix R of the vehicle-mounted camera coordinate system in the left direction of the vehicle body with respect to the second vehicle body coordinate system is calculated as follows_l2：

R_l2＝R₂₁*R_l1；

The rotation matrix R of the vehicle-mounted camera coordinate system relative to the first vehicle body coordinate system can be also determined according to the right direction of the vehicle body_r1And a rotation matrix R of the second body coordinate system relative to the first body coordinate system₂₁A rotation matrix R of the vehicle-mounted camera coordinate system in the left direction of the vehicle body with respect to the second vehicle body coordinate system is calculated as follows_r2：

R_r2＝R₂₁*R_r1；

Step three: and calculating the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the rotation matrix of the vehicle-mounted camera coordinate system in the left and right directions of the vehicle body relative to the second vehicle body coordinate system.

Therefore, in the embodiment of the invention, the rotation matrix of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system can be multiplied, and the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system can be calculated according to the multiplication result. Just because the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are calculated according to the rotation matrix, the problem that when the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are solved by shooting images, three straight lines in the images are approximately parallel, and the intersection point is at infinity, so that the calculation result is inaccurate is avoided. In other words, in the embodiment of the invention, the obtained external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body are accurate external parameters relative to the first vehicle body coordinate system and the second vehicle body coordinate system, so that the accuracy of the external parameters is improved by using the camera external parameter calibration method provided by the embodiment of the invention.

Taking the vehicle-mounted camera in the left direction of the vehicle body as an example, in the second step, a rotation matrix R of the vehicle-mounted camera coordinate system in the left direction of the vehicle body relative to the second vehicle body coordinate system is obtained_l2. Can be according to R_l2And determining external parameters of the vehicle-mounted camera in the left direction of the vehicle body relative to a second vehicle body coordinate system, wherein the step can be regarded as a reverse process of the step I, and is not described any more.

In the embodiment of the invention, the external reference of the vehicle-mounted camera in the right direction of the vehicle body relative to the second vehicle body coordinate system can be determined by using the same method.

Therefore, by using the vehicle-mounted camera external parameter calibration method provided by the embodiment of the invention, the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system can be calculated, wherein the first vehicle body coordinate system takes the transverse axis of the vehicle as the X axis and the longitudinal axis of the vehicle as the Y axis, and the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are calculated by combining the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system. Because the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are calculated according to the rotation matrix, the problem that when the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system are solved by shooting images, three straight lines in the images are approximately parallel, and the intersection point is at infinity, so that the calculation result is inaccurate is avoided. Namely, the calculation precision of the external parameters is improved.

In the embodiment of the invention, the external parameters of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to the second vehicle body coordinate system can be obtained, namely the external parameters of the vehicle-mounted cameras in the front, rear, left and right directions of the vehicle body relative to the common second vehicle body coordinate system can be obtained.

The step of acquiring the external reference of the vehicle-mounted cameras in the front and rear directions of the vehicle body with respect to the second vehicle body coordinate system may be performed before or after the step of acquiring the external reference of the vehicle-mounted cameras in the left and right directions of the vehicle body with respect to the second vehicle body coordinate system, and is not limited thereto.

It is now assumed that the step of acquiring the external reference of the in-vehicle cameras in the front and rear directions of the vehicle body with respect to the second vehicle body coordinate system follows the step of acquiring the external reference of the in-vehicle cameras in the left and right directions of the vehicle body with respect to the second vehicle body coordinate system. After obtaining the external reference of the vehicle-mounted cameras in the left and right directions of the vehicle body with respect to the second vehicle body coordinate system, if the position of the vehicle body is kept unchanged, and the three straight lines are shot by using the vehicle-mounted cameras in the front and rear directions of the vehicle body, the imaging straight lines of the three straight lines in the shot image are approximately parallel, as shown in fig. 3, when the vehicle-mounted camera external parameters in the vehicle body front and rear directions are calibrated by photographing fig. 3, the calculated external parameters are relative to the first vehicle body coordinate system, however, the optical axes of the vehicle-mounted cameras in the vehicle body front and rear directions are perpendicular to the three straight lines, and therefore, the three straight lines are approximately parallel in the captured image, and when the external parameters of the vehicle-mounted camera in the front and rear directions of the vehicle body are calculated using fig. 3, when the slope of the imaging straight line is calculated, a large error exists, so that the accuracy of the external parameter obtained by calculation is not high.

To avoid such calculation errors, embodiments of the present invention may adjust the vehicle body direction in advance before capturing the image.

Referring to fig. 4, fig. 4 is a schematic diagram of adjusting the direction of the vehicle body according to an embodiment of the present invention, in order to obtain an available image after shooting three straight lines l1, l2, and l3 by using the vehicle-mounted cameras in the front and rear directions of the vehicle body, the direction of the vehicle body may be adjusted to be perpendicular to the original direction according to the method shown in fig. 4, and the adjustment of the direction of the vehicle body may be completed by using a rotatable mechanical platform, for example, the entire vehicle body may be rotated by 90 ° counterclockwise by using the mechanical platform. Fig. 5 is a schematic diagram of the vehicle body direction adjusted according to the embodiment of the present invention, as shown in fig. 5, after the vehicle body direction is adjusted, the three straight lines l1, l2, and l3 can be captured by the vehicle-mounted cameras in the front and rear directions of the vehicle body, and the imaging straight lines in the captured image are no longer approximately parallel.

In addition, it should be noted that, in the above method flow, three straight lines are taken as an example for description, and as an alternative, calibration of external parameters of the vehicle-mounted camera may also be performed according to the above method flow based on three or more parallel straight lines.

In the embodiment of the invention, at least three straight lines which are perpendicular to the original three straight lines can be added, and then the three newly added straight lines are shot by using the vehicle-mounted cameras in the front and rear directions of the vehicle body, so that the imaging straight lines in the shot image are not approximately parallel any more.

Based on the two methods provided by the embodiment of the invention, a second image obtained by shooting three parallel straight lines on the ground by using the vehicle-mounted cameras in the front and rear directions of the vehicle body can be obtained, and the three straight lines are parallel to the longitudinal axis of the vehicle during shooting; it can be seen that the external parameters of the onboard camera in the vehicle body front and rear directions calculated from the second image at this time are relative to the second vehicle body coordinate system. The method specifically comprises the following steps: and calculating the external parameters of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to a second vehicle body coordinate system respectively based on the second image and the second distances between the three straight lines and the longitudinal axis of the vehicle. The second body coordinate system is a coordinate system in which the longitudinal axis of the vehicle is the X axis and the lateral axis is the Y axis. The method for calculating the external reference of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to the second vehicle body coordinate system and the external reference of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system are the same, and are not repeated.

And combining the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system to obtain the external parameters of the vehicle-mounted cameras in the front, rear, left and right directions of the vehicle body relative to the common second vehicle body coordinate system.

Therefore, by using the vehicle-mounted camera external parameter calibration method provided by the embodiment of the invention, the external parameters of the vehicle-mounted cameras in the front, rear, left and right directions of the vehicle body relative to the common second vehicle body coordinate system can be calculated.

In the embodiment of the present invention, the external parameters of the vehicle-mounted cameras in the vehicle body front and rear directions relative to the first vehicle body coordinate system may be further calculated according to the external parameters of the vehicle-mounted cameras in the vehicle body front and rear directions relative to the second vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system. The calculation process may refer to the above embodiments, and is not described herein.

Therefore, by using the external parameter calibration method of the vehicle-mounted camera provided by the embodiment of the invention, the external parameters of the vehicle-mounted camera in the front, rear, left and right directions of the vehicle body relative to the common first vehicle body coordinate system can be calculated.

In the embodiment of the invention, a plurality of groups of external parameters of the vehicle-mounted camera can be obtained, each group of external parameters comprises the external parameters of the vehicle-mounted camera in the front, rear, left and right directions of the vehicle body, and three parallel straight lines based on which different groups of external parameters are calculated are not all the same;

The meaning that three parallel straight lines are not completely the same is, for example, 5 parallel straight lines, three of the 5 straight lines are arbitrarily selected to have various combinations, any combination can be used as the three parallel straight lines, the various combinations can have the same straight line or can be completely different, the various combinations are subjected to the external reference calibration, and a better group of external references can be selected from the external reference calibration results.

Referring to fig. 6, fig. 6 is a schematic diagram of external reference calibration based on multiple parallel lines according to an embodiment of the present invention, in the embodiment shown in fig. 6, there are 5 parallel lines l1, l2, l3, l4, and l5 that are parallel to each other, when external reference calibration of a vehicle-mounted camera in the left-right direction of a vehicle body is performed, calculation may be performed based on any three parallel lines, and the external reference calculated based on three parallel lines is taken as a group of external reference, so that an external reference that is obtained by calculation based on three parallel lines may be obtained

Group external ginseng.

In obtaining the above

After the external parameters are combined, the images corresponding to each external parameter can be spliced to obtain a panoramic image. External parameters are needed to be used in the splicing process, the external parameters determine the position of each image when the images are spliced, due to the fact that multiple groups of external parameters are obtained, the splicing effect of the panoramic images obtained by splicing each group of external parameters is different, gaps can exist in the spliced panoramic images at all splicing positions, the panoramic images with the minimum splicing gaps in the spliced panoramic images can be determined, the external parameters corresponding to the panoramic images serve as calibration external parameters of the vehicle-mounted camera, and when the external parameters are used as the calibration external parameters of the vehicle-mounted camera, the panoramic images are spliced according to the determined gapsHowever, in practical applications, the panoramic image with the best stitching effect may also be determined manually, and the external parameter corresponding to the panoramic image is taken as the calibration external parameter, which is not limited herein.

Therefore, the vehicle-mounted camera external parameter calibration method provided by the invention can obtain multiple groups of external parameters of the vehicle-mounted camera, and three parallel straight lines on which different groups of external parameters are calculated can be different, then images shot by the vehicle-mounted camera are spliced by using each group of external parameters to obtain panoramic images, the panoramic images with the minimum splicing gaps in the panoramic images are determined, and the external parameters corresponding to the panoramic images are used as the calibration external parameters of the vehicle-mounted camera. Therefore, the best calibration result can be selected from a plurality of calibration results to serve as the final calibration result, and because a plurality of groups of external parameters obtained by calculation according to different straight lines are considered, the influence of errors on the external parameter calibration result is reduced, and the accuracy of external parameter calibration and the robustness of an algorithm are improved.

In the embodiment of the invention, multiple groups of external parameters of the vehicle-mounted camera can be obtained, each group of external parameters comprises the external parameters of the vehicle-mounted camera in the front, rear, left and right directions of the vehicle body, three parallel straight lines based on which different groups of external parameters are calculated are the same, and the distances between the vehicle body and the three straight lines are different when different groups of external parameters are calculated;

Referring to fig. 7, fig. 7 is a schematic diagram of vehicle-mounted camera external reference calibration based on different vehicle body positions according to an embodiment of the present invention, in the embodiment shown in fig. 7, external reference calibration may be performed once at position 1 to obtain a set of external references and images corresponding to the set of external references, and then the vehicle may be moved to position 2 to perform external reference calibration again to obtain another set of external references and images corresponding to the set of external references. Wherein, the position 1 and the position 2 of the vehicle body are required to satisfy that the axis of the vehicle body is parallel to three straight lines of l1, l2, l3, l4 and l5 on the ground.

After the external parameters and the corresponding images calculated based on the position 1 and the position 2 are obtained, the images corresponding to the two groups of external parameters may be respectively stitched to obtain a panoramic image, and then a panoramic image with the least stitching gaps in the stitched panoramic image is determined, and the external parameters corresponding to the panoramic image are used as the calibration external parameters of the vehicle-mounted camera.

Therefore, the vehicle-mounted camera external parameter calibration method provided by the invention can obtain multiple groups of external parameters of the vehicle-mounted camera, the positions of vehicle bodies based on which different groups of external parameters are calculated can be different, images shot by the vehicle-mounted camera by using each group of external parameters are spliced to obtain a panoramic image, the panoramic image with the least splicing gaps in the panoramic image is determined, and the external parameters corresponding to the panoramic image are used as the calibration external parameters of the vehicle-mounted camera. Therefore, the best calibration result can be selected from a plurality of calibration results to serve as the final calibration result, and because a plurality of groups of external parameters acquired at different positions are considered, the influence of errors on the external parameter calibration result is reduced, and the accuracy of external parameter calibration and the robustness of an algorithm are improved.

Based on the same inventive concept, the embodiments of the present application provide a vehicle-mounted camera external reference calibration apparatus, the contents of which and the contents of the vehicle-mounted camera external reference calibration method in the embodiments described above may be referred to each other, and are not described in detail again.

Referring to fig. 8, fig. 8 is a schematic structural diagram of a vehicle-mounted camera external reference calibration apparatus according to an embodiment of the present invention, where the vehicle-mounted camera external reference calibration apparatus may include:

a first image obtaining module 801, configured to obtain a first image obtained by shooting three parallel straight lines on the ground with vehicle-mounted cameras in left and right directions of a vehicle body, where the three straight lines are parallel to a horizontal axis of the vehicle during shooting;

a first external parameter calculating module 802, configured to calculate external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to a first vehicle body coordinate system, respectively, based on the first image and first distances between the three straight lines and the vehicle transverse axis; the first vehicle body coordinate system takes a vehicle transverse axis as an X axis, and a vehicle longitudinal axis as a Y axis;

a first external parameter conversion module 803, configured to calculate external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and a rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system; the second body coordinate system takes the longitudinal axis of the vehicle as an X axis, and the transverse axis of the vehicle as a Y axis.

On the basis of the vehicle-mounted camera external reference calibration device shown in fig. 8, the method may further include:

the second image acquisition module is used for acquiring a second image obtained by shooting three parallel straight lines on the ground by using the vehicle-mounted cameras in the front and rear directions of the vehicle body, wherein the three straight lines are parallel to the longitudinal axis of the vehicle during shooting;

and the second external parameter calculating module is used for calculating external parameters of the vehicle-mounted cameras in the front and rear directions of the vehicle body relative to the second vehicle body coordinate system respectively based on the second image and second distances between the three straight lines and the longitudinal axis of the vehicle.

In this embodiment of the present invention, the first external reference conversion module 803 may include:

The second matrix determining module may be specifically configured to:

R_l2＝R₂₁*R_l1；

R_r2＝R₂₁*R_r1。

In the embodiment of the invention, the vehicle-mounted camera further comprises a first external parameter acquisition module for acquiring multiple groups of external parameters of the vehicle-mounted camera, wherein each group of external parameters comprises external parameters of the vehicle-mounted camera in the front, rear, left and right directions of the vehicle body, and three parallel straight lines on which different groups of external parameters are calculated are not all the same;

In the embodiment of the present invention, the method may further include: the second external parameter acquisition module is used for acquiring multiple groups of external parameters of the vehicle-mounted camera, each group of external parameters comprises external parameters of the vehicle-mounted camera in the front, rear, left and right directions of the vehicle body, three parallel straight lines based on which different groups of external parameters are calculated are the same, and the distances between the vehicle body and the three straight lines are different when different groups of external parameters are calculated;

By using the method and the device for calibrating the external parameters of the vehicle-mounted camera, provided by the embodiment of the invention, the external parameters of the vehicle-mounted camera in the left and right directions of the vehicle body relative to a first vehicle body coordinate system can be calculated, wherein the first vehicle body coordinate system takes the transverse axis of the vehicle as an X axis, and the longitudinal axis of the vehicle as a Y axis, the calculated external parameters of the vehicle-mounted camera in the left and right directions of the vehicle body relative to the first vehicle body coordinate system are converted to obtain the external parameters of the vehicle-mounted camera in the left and right directions of the vehicle body relative to a second vehicle body coordinate system, wherein the second vehicle body coordinate system takes the longitudinal axis of the vehicle as the X axis, and the transverse axis of the vehicle is the Y axis. Therefore, by using the method and the device for calibrating the external parameters of the vehicle-mounted camera provided by the embodiment of the invention, the external parameters of the vehicle-mounted camera in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the second vehicle body coordinate system can be accurately calculated.

Based on the same inventive concept, embodiments of the present invention further provide an electronic device, referring to fig. 9, which may include a processor 901 and a machine-readable storage medium 902, the machine-readable storage medium storing machine-executable instructions capable of being executed by the processor, the processor being caused by the machine-executable instructions to: implementing any of the method steps in the above embodiments.

An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, the computer program implements any of the above method steps.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A vehicle-mounted camera external parameter calibration method is characterized by comprising the following steps:

according to the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system, calculating the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second vehicle body coordinate system; the second body coordinate system takes the longitudinal axis of the vehicle as an X axis and takes the transverse axis of the vehicle as a Y axis.

2. The method of claim 1, further comprising:

3. The method of claim 2, further comprising:

4. The method of claim 1, wherein the calculating the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the second body coordinate system according to the external parameters of the vehicle-mounted cameras in the left and right directions of the vehicle body relative to the first body coordinate system and the rotation matrix of the second body coordinate system relative to the first body coordinate system comprises:

5. The method of claim 4, wherein the calculating the rotation matrix of the vehicle-mounted camera coordinate system of the left and right directions of the vehicle body relative to the second vehicle body coordinate system according to the rotation matrix of the vehicle-mounted camera coordinate system of the left and right directions of the vehicle body relative to the first vehicle body coordinate system and the rotation matrix of the second vehicle body coordinate system relative to the first vehicle body coordinate system respectively comprises:

according to the rotation matrix of the vehicle-mounted camera coordinate system in the left direction of the vehicle body relative to the first vehicle body coordinate systemR_l1And a rotation matrix R of a second body coordinate system with respect to the first body coordinate system₂₁Calculating a rotation matrix R of the vehicle-mounted camera coordinate system in the left direction of the vehicle body relative to the second vehicle body coordinate system according to the following formula_l2：

R_l2＝R₂₁*R_l1；

R_r2＝R₂₁*R_r1。

6. The method of claim 1, further comprising:

7. The method of claim 1, further comprising:

8. An external reference calibration device of a vehicle-mounted camera is characterized by comprising:

9. The apparatus of claim 8, further comprising:

10. The apparatus of claim 9, further comprising:

11. The apparatus of claim 8, wherein the first external reference conversion module comprises:

12. The apparatus of claim 11, wherein the second matrix determination module is specifically configured to:

according to a rotation matrix R of the vehicle-mounted camera coordinate system in the left direction of the vehicle body relative to the first vehicle body coordinate system_l1And a rotation matrix R of a second body coordinate system with respect to the first body coordinate system₂₁Calculating the left side of the vehicle body according to the following formulaRotation matrix R of the onboard camera coordinate system of directions relative to the second body coordinate system_l2：

R_l2＝R₂₁*R_l1；

R_r2＝R₂₁*R_r1。

13. The apparatus of claim 8, further comprising:

14. The apparatus of claim 8, further comprising:

15. An electronic device comprising a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to: carrying out the method steps of any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that a computer program is stored in the storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 1-7.