CN116664691A - External parameter calibration method, device, equipment, vehicle and medium of vehicle-mounted camera - Google Patents
External parameter calibration method, device, equipment, vehicle and medium of vehicle-mounted camera Download PDFInfo
- Publication number
- CN116664691A CN116664691A CN202310602702.6A CN202310602702A CN116664691A CN 116664691 A CN116664691 A CN 116664691A CN 202310602702 A CN202310602702 A CN 202310602702A CN 116664691 A CN116664691 A CN 116664691A
- Authority
- CN
- China
- Prior art keywords
- vehicle
- key
- key point
- point
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000004590 computer program Methods 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000001514 detection method Methods 0.000 description 19
- 238000004422 calculation algorithm Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 230000009466 transformation Effects 0.000 description 3
- 102100034112 Alkyldihydroxyacetonephosphate synthase, peroxisomal Human genes 0.000 description 2
- 101000799143 Homo sapiens Alkyldihydroxyacetonephosphate synthase, peroxisomal Proteins 0.000 description 2
- 238000000848 angular dependent Auger electron spectroscopy Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000010561 standard procedure Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Studio Devices (AREA)
Abstract
The invention provides an external parameter calibration method, device, equipment, vehicle and medium of a vehicle-mounted camera, comprising the following steps: acquiring the vehicle type information of the target vehicle and key points of the target vehicle from the road surface image, and selecting the key points according to preset key point selection conditions to acquire a first key point corresponding to the first vehicle and a second key point corresponding to the second vehicle; acquiring one of external parameters corresponding to the vehicle-mounted camera, namely a vanishing point, by utilizing cross ratio invariance according to the first key point and the second key point; the preset key point selection condition is that key points which meet the fact that a connecting line between the key points is parallel to the bottom edge or the top edge of the pavement image are selected from a plurality of key points, and the distance between the key points in the first vehicle is equal to the distance between the key points in the second vehicle under a world coordinate system. The external parameter calibration of the invention does not depend on lane line information, and has better generalization capability.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, a vehicle, and a medium for calibrating an external parameter of a vehicle-mounted camera.
Background
In an ADAS system (Advanced Driving Assistant System, advanced driving assistance system), forward-looking related tasks include FCW (Forward Collision Warning ), LDW (Lane Departure Warning, lane departure warning), PCW (Pedestrian Collision Warning ), TSR (Traffic Sign Recognition, traffic sign recognition), and the like. In these tasks, the most central work is to perform distance estimation or speed estimation on a front target or a lane line on a road surface, and in order to achieve accurate distance estimation or speed estimation, external parameter calibration needs to be performed on an on-vehicle camera for achieving a forward looking task.
Traditional external parameter calibration requires manual calibration of the height, vanishing point and the like of a camera by a camera installer, however, errors are easily introduced in the manual calibration process, and various external factors such as jolt, external touch and the like can possibly cause inconsistent actual parameters of the camera and initial calibration along with the time, so that the accuracy of a ranging or speed measuring algorithm is affected.
In the prior art, in order to reduce the manual calibration cost and improve the robustness in the operation process of an algorithm, the self-calibration of external parameters, especially the self-calibration of vanishing points, becomes an important point in an ADAS forward looking task. Currently, the self-calibration for vanishing points in the external parameter calibration algorithm is generally divided into the following two types: (1) According to the perspective principle, the intersection point of two parallel lines is taken as a vanishing point, and the intersection point of a lane line is taken as the vanishing point in general. (2) And directly learning vanishing lines (astronomical lines) through the artificial data, and further taking a point corresponding to the central abscissa of the picture as a vanishing point.
However, the method (1) needs at least two lane lines which are clearly visible on the road surface, and the lane lines need to be kept in a straight line state as far as possible, while the method (2) mainly depends on the generalization performance of the model, and the difference between the artificial data set on which the model is based and the real front view camera scene is large, so that the model is applied to the real scene, and the model effect is not guaranteed.
Since the above method (1) is constrained in road scenes, especially trunk scenes, the method (1) is often adopted as a standard method for camera self-calibration. However, due to the defect of the method (1), in the scene without parallel lane lines, for example, merging lanes, bifurcation lanes, intersections, rural roads and the like, the method (1) often fails the external parameter calibration algorithm because the corresponding parallel lane lines cannot be acquired, and cannot perform distance measurement or speed measurement.
In summary, there is a need for a camera extrinsic calibration method suitable for real and diverse road scenes.
Disclosure of Invention
The invention provides an external parameter calibration method, device, equipment, vehicle and medium for a vehicle-mounted camera, which are used for solving the problems.
The invention provides an external parameter calibration method of a vehicle-mounted camera, which comprises the following steps:
acquiring a road surface image by using a vehicle-mounted camera in the vehicle;
Acquiring model information of a target vehicle and a plurality of key points of the target vehicle from the road surface image, wherein the target vehicle is a vehicle around the own vehicle, and the road surface image comprises a plurality of target vehicles;
acquiring a target vehicle pair from the plurality of target vehicles according to the vehicle type information, wherein the target vehicle pair comprises a first vehicle and a second vehicle;
selecting key points corresponding to the first vehicle and the second vehicle according to preset key point selection conditions, and obtaining at least two first key points corresponding to the first vehicle and at least two second key points corresponding to the second vehicle;
acquiring vanishing points corresponding to the vehicle-mounted camera by utilizing cross ratio invariance according to the first key points and the second key points;
the preset key point selection condition is that key points which meet the fact that connecting lines between the key points are parallel to the bottom edge or the top edge of the pavement image are selected from the plurality of key points, and under a world coordinate system, the distance between the key points in the first vehicle is equal to the distance between the key points in the second vehicle.
According to the external parameter calibration method of the vehicle-mounted camera, when the number of the first key points and the second key points is two, and the first key points are a first key point a and a first key point b, and the second key points are a second key point c and a second key point d, wherein the first key point a corresponds to the second key point c, and the first key point b corresponds to the second key point d; the distance between the first vehicle and the own vehicle is smaller than the distance between the second vehicle and the own vehicle;
The obtaining, according to the first key point and the second key point, the vanishing point corresponding to the vehicle-mounted camera by using cross-ratio invariance includes:
selecting an equidistant point f on a straight line corresponding to a first key point a and a first key point b according to the position relation between the second key point c and the first key point a and the distance between the first key point a and the first key point b;
acquiring an intersection point g between a straight line corresponding to the equidistant point f and the second key point d and a straight line corresponding to the first key point b and the second key point c;
obtaining vanishing lines parallel to the bottom edge or the top edge of the pavement image according to the intersection point g;
and acquiring vanishing points from the vanishing lines according to the size information of the road surface image.
According to the external parameter calibration method of the vehicle-mounted camera provided by the invention, after the vehicle type information of the target vehicle and a plurality of key points of the target vehicle are obtained from the road surface image, the method further comprises the following steps:
under the condition that the vehicle type information of the target vehicles is different, acquiring multi-frame road surface images, wherein the positions of the same target vehicles in the multi-frame road surface images are different;
Combining the multi-frame pavement images to obtain a combined image;
and acquiring the model information of the target vehicle and a plurality of key points of the target vehicle from the combined image.
According to the external parameter calibration method of the vehicle-mounted camera provided by the invention, the target vehicle pairs are a plurality of target vehicle pairs, and the obtaining the vanishing point corresponding to the vehicle-mounted camera by utilizing the cross ratio invariance according to the first key point and the second key point comprises the following steps:
for each target vehicle pair of the plurality of target vehicle pairs, acquiring a vanishing point corresponding to the target vehicle pair by utilizing cross ratio invariance according to a first key point and a second key point corresponding to the target vehicle pair;
and carrying out average value calculation on a plurality of vanishing points corresponding to the plurality of target vehicles to obtain the vanishing points corresponding to the vehicle-mounted camera.
According to the external parameter calibration method of the vehicle-mounted camera provided by the invention, the target vehicle pair is obtained from the plurality of target vehicles according to the vehicle type information, and the method comprises the following steps:
forming a team of target vehicles with the same vehicle type information to obtain a plurality of target vehicle pairs;
the vehicle type information is at least one of a small vehicle, a micro vehicle, a compact vehicle, a medium vehicle, a high-grade vehicle, a luxury vehicle, a three-compartment vehicle, a CDV vehicle, an MPV vehicle or an SUV.
According to the external parameter calibration method of the vehicle-mounted camera, the first key point and the second key point are external wheel key points.
The invention also provides an external parameter calibration device of the vehicle-mounted camera, which comprises:
the image acquisition module is used for acquiring a road surface image by utilizing a vehicle-mounted camera in the vehicle;
the target vehicle information acquisition module is used for acquiring vehicle type information of a target vehicle and a plurality of key points of the target vehicle from the road surface image, wherein the target vehicle is a vehicle around the own vehicle, and the road surface image comprises a plurality of target vehicles;
the vehicle pair determining module is used for acquiring a target vehicle pair from the plurality of target vehicles according to the vehicle type information, wherein the target vehicle pair comprises a first vehicle and a second vehicle;
the key point selection module is used for selecting key points corresponding to the first vehicle and the second vehicle according to preset key point selection conditions, and obtaining at least two first key points corresponding to the first vehicle and at least two second key points corresponding to the second vehicle;
the vanishing point acquisition module is used for acquiring vanishing points corresponding to the vehicle-mounted camera by utilizing cross ratio invariance according to the first key points and the second key points;
The preset key point selection condition is that key points which meet the fact that connecting lines between the key points are parallel to the bottom edge or the top edge of the pavement image are selected from the plurality of key points, and under a world coordinate system, the distance between the key points in the first vehicle is equal to the distance between the key points in the second vehicle.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the external parameter calibration method of the vehicle-mounted camera when executing the program.
The invention further provides a vehicle comprising the electronic equipment.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of calibrating an external parameter of an on-board camera as described in any of the above.
According to the external parameter calibration method, the device, the equipment, the vehicle and the medium of the vehicle-mounted camera, the first key point and the second key point are obtained from the target vehicle of the vehicle type, the distance between the first key point and the second key point is equal in a world coordinate system, the straight line passing through the first key point and the straight line passing through the second key point are parallel to the bottom edge or the top edge of a road surface image, and based on the first key point and the second key point, one of external parameters of the vehicle-mounted camera, namely the vanishing point, is obtained by utilizing the invariance of the cross ratio in projective transformation.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart of an external parameter calibration method of an on-board camera according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of obtaining vanishing points based on cross-ratio invariance according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of determining vanishing points based on key points according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a combined image provided by an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an external parameter calibration device of an on-vehicle camera according to an embodiment of the present invention;
fig. 6 illustrates a physical structure diagram of an electronic device.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a flow chart of an external parameter calibration method of an on-board camera according to an embodiment of the present invention; as shown in fig. 1, the external parameter calibration method of the vehicle-mounted camera comprises the following steps:
s101, a road surface image is acquired by using an in-vehicle camera in the own vehicle.
In this step, a road surface image around the own vehicle is acquired by an in-vehicle camera provided in the own vehicle. The road surface image includes a plurality of target vehicles, and the target vehicles are vehicles around the own vehicle, and it should be noted that the vehicle-mounted camera may be a front-view vehicle-mounted camera or a look-around camera or a look-back camera. In the looking-around camera, the obtained initial road surface image needs to be subjected to distortion correction, and the external parameter calibration of the vehicle-mounted camera is carried out based on the corrected road surface image. The front-view vehicle-mounted camera and the rear-view camera can directly perform external parameter calibration based on the obtained road surface image.
S102, acquiring the vehicle type information of the target vehicle and a plurality of key points of the target vehicle from the road surface image.
In this step, the road surface image is subjected to target detection by the existing target detection algorithm, so that the model information and the prediction frame of the target vehicle are predicted, and meanwhile, a plurality of key points of the target vehicle, such as a wheel key point, a lamp key point, a vehicle rearview mirror and the like, are output.
The existing target detection algorithm may be a conventional target detection algorithm, such as a Viola Jones detector, a HOG detector, and the like. The method can also be a target detection algorithm based on deep learning, such as RCNN series, YOLO series, SSD and the like, and the invention is not limited to a specific target detection algorithm. In addition, in the process of performing target detection by using RCNN series and YOLO series, the key points are obtained by direct regression, or obtained by Stacked Hourglass Networks alone, or obtained directly by using end-to-end target detection algorithm centrnet, and the invention is not limited to the obtaining of the key points.
S103, acquiring target vehicle pairs from the plurality of target vehicles according to the vehicle type information.
The target vehicle pair comprises a first vehicle and a second vehicle. The model information is at least one of a small-sized vehicle, a mini-sized vehicle, a compact vehicle, a medium vehicle, a high-grade vehicle, a luxury vehicle, a three-compartment vehicle, a CDV vehicle, an MPV vehicle or an SUV.
More specifically, the acquiring the target vehicle pair from the plurality of target vehicles according to the vehicle type information includes:
And grouping the target vehicles with the same vehicle type information to obtain a plurality of target vehicle pairs. That is, the target vehicles of the same vehicle type are paired to form a target vehicle pair. For example, if three target vehicles are detected, i.e., a small vehicle, and an SUV, then the two small vehicles serve as a pair of target vehicles. If all three detected target vehicles are small vehicles, the small vehicle C and the small vehicle D are taken as a pair of target vehicles, the small vehicle C and the small vehicle E are taken as a pair of target vehicles, and the small vehicle D and the small vehicle E are taken as a pair of target vehicles, so that 3 pairs of target vehicles are obtained. In order to distinguish between two vehicles in a target vehicle pair, they are designated as a first vehicle and a second vehicle, respectively.
S104, selecting key points corresponding to the first vehicle and the second vehicle according to preset key point selection conditions, and obtaining at least two first key points corresponding to the first vehicle and at least two second key points corresponding to the second vehicle.
The preset key point selection condition is that key points which meet the fact that connecting lines between the key points are parallel to the bottom edge or the top edge of the pavement image are selected from the plurality of key points, and the distance between the first key points is equal to the distance between the second key points under a world coordinate system. Here, the distance between the first key points and the distance between the second key points are calculated not based on the pixel coordinate values of the road surface image, but based on the coordinate values of the respective key points in the world coordinate system. And parallelism here allows some error (within 2 degrees) and does not require absolute parallelism.
For example, assuming that straight lines corresponding to two front wheel keypoints detected in a first vehicle are parallel to a bottom edge or a top edge of a road surface image, straight lines corresponding to two rear wheel keypoints detected in a second vehicle are parallel to a bottom edge or a top edge of a road surface image, and a distance between the two front wheel keypoints and a distance between the two rear wheel keypoints in a world coordinate system are equal (the wheel track of the same vehicle type is substantially the same, and an error is negligible in the present invention), the two front wheel keypoints are determined as first keypoints of the first vehicle, and the two rear wheel keypoints are determined as second keypoints of the second vehicle.
Or, assuming that two rear wheel key points are detected in the first vehicle and the corresponding straight lines are parallel to the bottom edge or the top edge of the road surface image, and the two rear wheel key points are detected in the second vehicle and the corresponding straight lines are parallel to the bottom edge or the top edge of the road surface image, based on the fact that the distance between the two rear wheel key points in the first vehicle and the distance between the two rear wheel key points in the second vehicle are equal, the first key point and the second key point are obtained.
In addition, at least 2 first key points and at least 2 second key points are selected, and then a straight line corresponding to the first key points and a straight line corresponding to the second key points are obtained. If only one straight line corresponding to the first key points or the second key points meets the condition of being parallel to the bottom edge or the top edge of the pavement image, the vanishing points are acquired based on the two straight lines and the first key points and the second key points selected from the straight lines. If the first key point or the second key point corresponds to a plurality of straight lines, vanishing point acquisition is performed based on the paired straight lines, and a final vanishing point is determined from the plurality of vanishing points, and a specific determination process is described below.
It should be noted that the above-mentioned distances are equal to each other and are determined based on the vehicle type information, and are the same as certain parameters in the vehicle of the vehicle type (there may be a small error, which is negligible in the present invention), such as the tread between the wheels, the distance between the outer rear view mirrors, and the like. In the present invention, generally, the position of a first keypoint in a first vehicle corresponds to the position of a second keypoint in a second vehicle, such as the keypoint of the tail of two vehicles. In an exceptional case, the position of the first key point in the first vehicle and the position of the second key point in the second vehicle may not correspond, for example, when the first vehicle and the second vehicle are in the same direction, the rear-wheel key point of the vehicle corresponds, the left-right rearview mirror of the first vehicle corresponds to the left-right rearview mirror of the second vehicle, and so on. When the first vehicle is reversed from the second vehicle, the front wheel keypoints of the first vehicle (assuming the first vehicle rear wheel keypoints are blocked, not detected) correspond to the rear wheel keypoints of the second vehicle, the left rear view mirror of the first vehicle corresponds to the right rear view mirror of the second vehicle, the right rear view mirror of the first vehicle corresponds to the left rear view mirror of the second vehicle, and so on.
S105, obtaining the vanishing point corresponding to the vehicle-mounted camera by utilizing the cross ratio invariance according to the first key point and the second key point. After the vanishing point is obtained, other external parameters such as a pitch angle of the vehicle-mounted camera can be further determined according to the vanishing point, and the determination process of the other external parameters is realized by using a common related vehicle-mounted camera external parameter calibration algorithm. In addition, after all the external parameter calibration data of the vehicle-mounted camera are obtained, the external parameter calibration data can be used for measuring the speed or distance of the vehicle so as to finish the forward looking task in automatic driving.
In this step, vanishing points are obtained from the determined key points and through cross-ratio invariance.
The following describes the application of the cross-ratio invariance in vanishing point acquisition, taking points a ", b", c ", a ', b ' and c ' (6 points are not identical to the first and second key points mentioned in the present invention here, but only for illustrating the principle of cross-ratio invariance) in fig. 2 as an example, the vanishing points are acquired according to the length ratio, and the specific steps are:
1) Three collinear points a ', B ' and c ' are determined in the image, and the spacing ratio between the 3 points is A to B in the world coordinate system.
2) Drawing a straight line l passing through the point a ', the straight line l not coinciding with the straight line n passing through the points a ', B ' and c ', taking the points a ", B" and c "on the straight line l, and noting the point a" =a ', and under the world coordinate system, the distance between the points a "and B" and the distance between the points B "and c" satisfy a: B.
3) The straight line passing through the point b 'and the point b' is obtained, the straight line passing through the point c 'and the point c' is obtained, and then the intersection point o of the two straight lines is obtained.
4) A straight line m passing through the intersection o and parallel to the straight line l is obtained, and the intersection v' of the straight line m and the straight line n is determined as a vanishing point.
And obtaining vanishing points in the road surface image according to the invention by using the cross ratio invariance. It should be noted that, a pair of target vehicles corresponds to one vanishing point, and when a plurality of pairs of target vehicles appear, a plurality of vanishing points are obtained. In this case, the final vanishing point may be obtained by an operation of taking the mean value of the coordinate values of the plurality of vanishing points; the final vanishing point can also be determined from a plurality of vanishing points through preset screening conditions (such as a target vehicle closest to the own vehicle and the like); the vanishing points obtained by the lane line or the vanishing points obtained by other conventional vanishing point obtaining methods may be integrated, and the final vanishing point may be determined from the plurality of vanishing points obtained by the present invention, which is not limited in this regard.
According to the external parameter calibration method of the vehicle-mounted camera, the first key points and the second key points are obtained from the target vehicle of the vehicle type, the distances between the two first key points and the distances between the two second key points are equal in a world coordinate system, the straight line passing through the first key points and the straight line passing through the second key points are parallel to the bottom edge or the top edge of the road surface image, the vanishing points are obtained by utilizing the invariance of the cross ratio in the projective transformation based on the first key points and the second key points, and the external parameter calibration of the vehicle-mounted camera is completed based on the vanishing points.
Further, on the basis of the above embodiment, when the first key point and the second key point are two, and the first key point is a first key point a and a first key point b, and the second key point is a second key point c and a second key point d, where the first key point a corresponds to the second key point c, and the first key point b corresponds to the second key point d; the distance between the first vehicle and the own vehicle is smaller than that between the second vehicle and the own vehicle, and the specific position relationship is shown in fig. 3.
In this embodiment, the description is made with 2 first key points and 2 second key points, where the first key point a corresponds to the second key point c, the first key point b corresponds to the second key point d, specifically, the sequential positions of the point a and the point b in the straight line ab correspond to the sequential positions of the point c and the point d in the straight line cd, as shown in fig. 3, the point a on the left corresponds to the point c on the left, the point b on the right corresponds to the point d on the right, and it is to be noted that, in the positions of the two key points, the left and the right are positions where the non-key points are located in the image.
Correspondingly, the obtaining, according to the first key point and the second key point, the vanishing point corresponding to the vehicle-mounted camera by using cross invariance includes:
and selecting an equidistant point f on a straight line corresponding to the first key point a and the first key point b according to the position relation between the second key point c and the first key point a and the distance between the first key point a and the first key point b.
Taking the position of the key point in fig. 3 as an example, according to the positional relationship between the second key point c and the first key point a, that is, the second key point c is on the right side of the first key point a, an equidistant point f is taken on the right side of the straight line corresponding to the points a and b, where the equidistant point f satisfies that the distance between the point f and the point b is equal to the distance between the point a and the point b, and it should be noted that, the distance herein refers to the distance in the image (i.e., the distance between the pixel points) instead of the distance in the world coordinate system.
In other embodiments, if the point c is on the left side of the point a, the equidistant point f is taken on the left side of the line ab, and will not be described herein.
And acquiring an intersection point g between a straight line corresponding to the equidistant point f and the second key point d and a straight line corresponding to the first key point b and the second key point c.
And acquiring vanishing lines parallel to the bottom edge or the top edge of the pavement image according to the intersection point g.
And acquiring vanishing points from the vanishing lines according to the size information of the road surface image. Specifically, after the vanishing line is obtained, a midpoint is selected on the vanishing line according to the width of the road surface image, for example, the width of the road surface image is 1280 pixels, and then the midpoint is obtained on the vanishing line parallel to the bottom edge of the image according to the 1280 pixels as the vanishing point.
In addition, in fig. 3, the intersection point of the straight line ad and the straight line bc is e, and the world coordinate corresponding to the intersection point e is the intermediate point between the first vehicle and the second vehicle.
From the cross-ratio invariance, ae: ed=ab: bf in fig. 3, and ae: ed is actually equal to ab: cd, and when ab: cd=1:1, bf=ab is taken, based on which the equidistant point f is determined.
In this embodiment, the number of the first key points and the second key points is limited to two, so that the rate of determining vanishing points can be increased.
According to the external parameter calibration method for the vehicle-mounted camera, the vanishing points are obtained through the group of equal-length line segments parallel to the bottom edge of the image, wherein the equal-length line segments are obtained based on some key points fixed in distance with the vehicle of the same vehicle type, and further the external parameter calibration of the camera is finished based on the vanishing points, so that the external parameter calibration of the vehicle-mounted camera does not depend on lane line information excessively in an actual scene, and the method has better generalization.
Further, on the basis of the above embodiment, after the obtaining the vehicle type information of the target vehicle and the plurality of key points of the target vehicle from the road surface image, the method further includes:
and under the condition that the vehicle type information of the target vehicles is different, acquiring multiple frames of road surface images, wherein the positions of the same target vehicles in the multiple frames of road surface images are different.
And combining the multi-frame pavement images to obtain a combined image. Schematically, the combined image is shown in fig. 4.
And acquiring the model information of the target vehicle and a plurality of key points of the target vehicle from the combined image.
In this embodiment, if the detected target vehicles are not the same vehicle type and cannot form the target vehicle pair, the target detection and the key point detection can be performed again on the combined images by combining the front frame image and the rear frame image.
It should be noted that the above-mentioned combined image has a certain requirement for the front-rear position of the target vehicle, that is, the longitudinal position of the vehicle in the two-frame image changes. And the road surface difference between the multi-frame road surface images is small. In addition, in order to save calculation time, only two frames of road surface images meeting the requirements can be selected for merging; if two or more frames of road surface images can be selected for merging in order to pursue accuracy of vanishing point acquisition, the present invention is not limited thereto.
If the detected target vehicles have the same vehicle type, the subsequent target vehicle alignment step is directly performed, and it is noted that under the condition, the external parameter calibration can be completed only by one frame of road surface image.
In addition, in the case of acquiring vanishing points based on the combined image, the first vehicle and the second vehicle in the target vehicle pair are actually the same vehicle, and the acquired key points in the same vehicle more easily conform to preset key point selection conditions, but have a limitation on the multi-frame road surface image in terms of road surface conditions. In the case of acquiring vanishing points based on a single frame road surface image, although there is no limitation on the road surface image, since the first vehicle and the second vehicle are two vehicles with the same vehicle type, there may be a certain difference in distance between some key points of the two vehicles, and thus the acquisition of the first key point and the second key point is not accurate by the method described above. The two methods can be used singly or simultaneously.
According to the external parameter calibration method of the vehicle-mounted camera, aiming at the situation that vehicles of the same vehicle type are not in the road surface images, vehicle team formation is naturally completed through combination of multiple frames of road surface images, and generalization of the external parameter calibration method is further improved.
Further, on the basis of the foregoing embodiment, the target vehicle pair is a plurality of target vehicle pairs, and the obtaining, according to the first key point and the second key point, the vanishing point corresponding to the vehicle-mounted camera using cross invariance includes:
for each target vehicle pair of the plurality of target vehicle pairs, acquiring a vanishing point corresponding to the target vehicle pair by utilizing cross ratio invariance according to a first key point and a second key point corresponding to the target vehicle pair;
and carrying out average value calculation on a plurality of vanishing points corresponding to the plurality of target vehicles to obtain the vanishing points corresponding to the vehicle-mounted camera.
In this embodiment, by the method for acquiring vanishing points, the corresponding vanishing points can be obtained after the key points of each pair of target vehicles are processed, and the pixel coordinates of the final vanishing points are obtained by means of average calculation for the obtained plurality of vanishing points. And then the external parameter calibration is completed based on the final vanishing point.
In addition, in order to increase the calculation rate, only one target vehicle pair (for example, two vehicles of the same vehicle type closest to the own vehicle) may be selected from among the plurality of target vehicle pairs to acquire the vanishing point. In order to improve accuracy of vanishing point acquisition, the above-described average processing may be performed on a plurality of vanishing points.
According to the external parameter calibration method of the vehicle-mounted camera, the average value of the vanishing points is calculated, so that the final vanishing point is obtained, the accuracy of obtaining the vanishing point is improved, and the accuracy of external parameter calibration is further improved.
Further, on the basis of the above embodiment, the first key point and the second key point are points outside the wheel. In this embodiment, the first key point is determined as the two points outside the wheel in the first vehicle, the second key point is determined as the two points outside the wheel in the second vehicle, and in general, 1 or 2 or 3 points outside the wheel in the surrounding vehicle can be detected in the road surface image obtained from the vehicle-mounted camera in the vehicle, and on the basis of meeting the preset key point selection condition, the two points outside the wheel in each vehicle are finally taken as the first key point or the second key point.
In addition, the wheel external connection point of the vehicle and the road surface is used as the first key point or the second key point, so that the wheel external connection point is more clearly visible, and the vanishing point refers to the vanishing point of the lane line, the lane line is road surface information, and the wheel external connection point can also be calculated as the road surface information. If other key points in the vehicle are used as the first key point or the second key point, it is necessary to ensure that the connecting line between the key points is parallel to the road surface.
In fig. 3, the first vehicle and the second vehicle are two vehicles traveling in the same direction, and among the detected key points, two rear wheel circumscribed points in the first vehicle are used as a first key point, and two rear wheel circumscribed points in the second vehicle are used as a second key point.
In another scenario, if the traveling directions of the first vehicle and the second vehicle are different, taking fig. 3 as an example, and assuming that the second vehicle is located on a reverse lane of the bulkhead, two rear wheel circumscribed points in the first vehicle may be taken as a first key point and two front wheel circumscribed points in the second vehicle may be taken as a second key point among the detected key points.
According to the external parameter calibration method of the vehicle-mounted camera, the calculation efficiency is improved by limiting the first key point and the second key point as the wheel external points, wherein the wheel external points are more clear in the key point detection process of the road surface image, and the shielding probability is smaller.
In addition, in order to further improve the accuracy of the external parameter calibration, the external parameter calibration method provided by the invention can be used together with the existing external parameter calibration method, for example, the vanishing point is obtained through a lane line, the vanishing point is obtained through the key point of the target vehicle pair, and the final external parameter calibration is completed by combining the two methods to obtain the vanishing point.
The external parameter calibration device of the vehicle-mounted camera provided by the invention is described below, and the external parameter calibration device of the vehicle-mounted camera described below and the external parameter calibration method of the vehicle-mounted camera described above can be correspondingly referred to each other.
Fig. 5 is a schematic structural diagram of an external parameter calibration device of an on-vehicle camera according to an embodiment of the present invention; as shown in fig. 5, the external parameter calibration device of the vehicle-mounted camera includes an image acquisition module 501, a target vehicle information acquisition module 502, a vehicle pair determination module 503, a key point selection module 504, and a vanishing point acquisition module 505.
An image acquisition module 501 is used for acquiring a road surface image by using an onboard camera in the own vehicle.
In the present module, a road surface image around the host vehicle is obtained by an in-vehicle camera provided in the host vehicle. The road surface image includes a plurality of target vehicles, and the target vehicles are vehicles around the own vehicle, and it should be noted that the vehicle-mounted camera may be a front-view vehicle-mounted camera or a look-around camera or a look-back camera. In the looking-around camera, the obtained initial road surface image needs to be subjected to distortion correction, and the external parameter calibration of the vehicle-mounted camera is carried out based on the corrected road surface image. The front-view vehicle-mounted camera and the rear-view camera can directly perform external parameter calibration based on the obtained road surface image.
The target vehicle information obtaining module 502 is configured to obtain, from the road surface image, vehicle type information of a target vehicle and a plurality of key points of the target vehicle.
In the module, the road surface image is subjected to target detection through the existing target detection algorithm, so that the model information and the prediction frame of the target vehicle are predicted, and meanwhile, a plurality of key points of the target vehicle, such as a wheel key point, a vehicle lamp key point, a vehicle rearview mirror and the like, are output.
The existing target detection algorithm may be a conventional target detection algorithm, such as a Viola Jones detector, a HOG detector, and the like. The method can also be a target detection algorithm based on deep learning, such as RCNN series, YOLO series, SSD and the like, and the invention is not limited to a specific target detection algorithm. In addition, in the process of performing target detection by using RCNN series and YOLO series, the key points are obtained by direct regression, or obtained by Stacked Hourglass Networks alone, or obtained directly by using end-to-end target detection algorithm centrnet, and the invention is not limited to the obtaining of the key points.
A vehicle pair determining module 503, configured to obtain a target vehicle pair from the plurality of target vehicles according to the vehicle type information.
The target vehicle pair comprises a first vehicle and a second vehicle. The model information is at least one of a small-sized vehicle, a mini-sized vehicle, a compact vehicle, a medium vehicle, a high-grade vehicle, a luxury vehicle, a three-compartment vehicle, a CDV vehicle, an MPV vehicle or an SUV.
More specifically, the acquiring the target vehicle pair from the plurality of target vehicles according to the vehicle type information includes:
and grouping the target vehicles with the same vehicle type information to obtain a plurality of target vehicle pairs. That is, the target vehicles of the same vehicle type are paired to form a target vehicle pair. For example, if three target vehicles are detected, i.e., a small vehicle, and an SUV, then the two small vehicles serve as a pair of target vehicles. If all three detected target vehicles are small vehicles, the small vehicle C and the small vehicle D are taken as a pair of target vehicles, the small vehicle C and the small vehicle E are taken as a pair of target vehicles, and the small vehicle D and the small vehicle E are taken as a pair of target vehicles, so that 3 pairs of target vehicles are obtained. In order to distinguish between two vehicles in a target vehicle pair, they are designated as a first vehicle and a second vehicle, respectively.
The key point selection module 504 is configured to select key points corresponding to the first vehicle and the second vehicle according to a preset key point selection condition, so as to obtain at least two first key points corresponding to the first vehicle and at least two second key points corresponding to the second vehicle.
The preset key point selection condition is that key points which meet the fact that connecting lines between the key points are parallel to the bottom edge or the top edge of the pavement image are selected from the plurality of key points, and the distance between the first key points is equal to the distance between the second key points under a world coordinate system. Here, the distance between the first key points and the distance between the second key points are calculated not based on the pixel coordinate values of the road surface image, but based on the coordinate values of the respective key points in the world coordinate system. And parallelism here allows some error (within 2 degrees) and does not require absolute parallelism.
For example, assuming that straight lines corresponding to two front wheel keypoints detected in a first vehicle are parallel to a bottom edge or a top edge of a road surface image, straight lines corresponding to two rear wheel keypoints detected in a second vehicle are parallel to a bottom edge or a top edge of a road surface image, and a distance between the two front wheel keypoints and a distance between the two rear wheel keypoints in a world coordinate system are equal (the wheel track of the same vehicle type is substantially the same, and an error is negligible in the present invention), the two front wheel keypoints are determined as first keypoints of the first vehicle, and the two rear wheel keypoints are determined as second keypoints of the second vehicle.
Or, assuming that two rear wheel key points are detected in the first vehicle and the corresponding straight lines are parallel to the bottom edge or the top edge of the road surface image, and the two rear wheel key points are detected in the second vehicle and the corresponding straight lines are parallel to the bottom edge or the top edge of the road surface image, based on the fact that the distance between the two rear wheel key points in the first vehicle and the distance between the two rear wheel key points in the second vehicle are equal, the first key point and the second key point are obtained.
In addition, at least 2 first key points and at least 2 second key points are selected, and then a straight line corresponding to the first key points and a straight line corresponding to the second key points are obtained. If only one straight line corresponding to the first key points or the second key points meets the condition of being parallel to the bottom edge or the top edge of the pavement image, the vanishing points are acquired based on the two straight lines and the first key points and the second key points selected from the straight lines. If the first key point or the second key point corresponds to a plurality of straight lines, vanishing point acquisition is performed based on the paired straight lines, and a final vanishing point is determined from the plurality of vanishing points, and a specific determination process is described below.
It should be noted that the above-mentioned distances are equal to each other and are determined based on the vehicle type information, and are the same as certain parameters in the vehicle of the vehicle type (there may be a small error, which is negligible in the present invention), such as the tread between the wheels, the distance between the outer rear view mirrors, and the like. In the present invention, generally, the position of a first keypoint in a first vehicle corresponds to the position of a second keypoint in a second vehicle, such as the keypoint of the tail of two vehicles. In an exceptional case, the position of the first key point in the first vehicle and the position of the second key point in the second vehicle may not correspond, for example, when the first vehicle and the second vehicle are in the same direction, the rear-wheel key point of the vehicle corresponds, the left-right rearview mirror of the first vehicle corresponds to the left-right rearview mirror of the second vehicle, and so on. When the first vehicle is reversed from the second vehicle, the front wheel keypoints of the first vehicle (assuming the first vehicle rear wheel keypoints are blocked, not detected) correspond to the rear wheel keypoints of the second vehicle, the left rear view mirror of the first vehicle corresponds to the right rear view mirror of the second vehicle, the right rear view mirror of the first vehicle corresponds to the left rear view mirror of the second vehicle, and so on.
And the vanishing point obtaining module 505 is configured to obtain, according to the first key point and the second key point, a vanishing point corresponding to the vehicle-mounted camera by using cross invariance. After the vanishing point is obtained, other external parameters such as a pitch angle of the vehicle-mounted camera can be further determined according to the vanishing point, and the determination process of the other external parameters is realized by using a common related vehicle-mounted camera external parameter calibration algorithm. In addition, after all the external parameter calibration data of the vehicle-mounted camera are obtained, the external parameter calibration data can be used for measuring the speed or distance of the vehicle so as to finish the forward looking task in automatic driving.
In the module, vanishing points are obtained according to the determined key points and through cross-ratio invariance.
The following describes the application of the cross-ratio invariance in vanishing point acquisition, taking points a ", b", c ", a ', b ' and c ' (6 points are not identical to the first and second key points mentioned in the present invention here, but only for illustrating the principle of cross-ratio invariance) in fig. 2 as an example, the vanishing points are acquired according to the length ratio, and the specific steps are:
1) Three collinear points a ', B ' and c ' are determined in the image, and the spacing ratio between the 3 points is A to B in the world coordinate system.
2) Drawing a straight line l passing through the point a ', which straight line l does not coincide with the straight line n passing through the points a ', B ' and c ', taking the points a ", B" and c "on the straight line l, and note that point a" =a ', and that in the world coordinate system, the distance between three points satisfies point a "point B" and the distance between point B "and point c" satisfies a: and B, a step of performing the process.
3) The straight line passing through the point b 'and the point b' is obtained, the straight line passing through the point c 'and the point c' is obtained, and then the intersection point o of the two straight lines is obtained.
4) A straight line m passing through the intersection o and parallel to the straight line l is obtained, and the intersection v' of the straight line m and the straight line n is determined as a vanishing point.
And obtaining vanishing points in the road surface image according to the invention by using the cross ratio invariance. It should be noted that, a pair of target vehicles corresponds to one vanishing point, and when a plurality of pairs of target vehicles appear, a plurality of vanishing points are obtained. In this case, the final vanishing point may be obtained by an operation of taking the mean value of the coordinate values of the plurality of vanishing points; the final vanishing point can also be determined from a plurality of vanishing points through preset screening conditions (such as a target vehicle closest to the own vehicle and the like); the vanishing points obtained by the lane line or the vanishing points obtained by other conventional vanishing point obtaining methods may be integrated, and the final vanishing point may be determined from the plurality of vanishing points obtained by the present invention, which is not limited in this regard.
According to the external parameter calibration device of the vehicle-mounted camera, the first key points and the second key points are obtained from the target vehicle of the vehicle type, the distances between the two first key points and the distances between the two second key points are equal in a world coordinate system, the straight line passing through the first key points and the straight line passing through the second key points are parallel to the bottom edge or the top edge of the road surface image, the vanishing points are obtained by utilizing the invariance of the cross ratio in the projective transformation based on the first key points and the second key points, and the external parameter calibration of the vehicle-mounted camera is completed based on the vanishing points.
Fig. 6 illustrates a physical schematic diagram of an electronic device, as shown in fig. 6, which may include: processor 610 (processor), communication interface 620 (Communications Interface), memory 630 (memory) and communication bus 640, wherein processor 610, communication interface 620, memory 630 communicate with each other via communication bus 640. The processor 610 may invoke logic instructions in the memory 630 to perform the above-provided method of calibrating external parameters of an onboard camera, the method comprising: acquiring a road surface image by using a vehicle-mounted camera in the vehicle; acquiring model information of a target vehicle and a plurality of key points of the target vehicle from the road surface image, wherein the target vehicle is a vehicle around the own vehicle, and the road surface image comprises a plurality of target vehicles; acquiring a target vehicle pair from the plurality of target vehicles according to the vehicle type information, wherein the target vehicle pair comprises a first vehicle and a second vehicle; selecting key points corresponding to the first vehicle and the second vehicle according to preset key point selection conditions, and obtaining at least two first key points corresponding to the first vehicle and at least two second key points corresponding to the second vehicle; acquiring vanishing points corresponding to the vehicle-mounted camera by utilizing cross ratio invariance according to the first key points and the second key points; the preset key point selection condition is that key points which meet the fact that connecting lines between the key points are parallel to the bottom edge or the top edge of the pavement image are selected from the plurality of key points, and under a world coordinate system, the distance between the key points in the first vehicle is equal to the distance between the key points in the second vehicle.
Further, the logic instructions in the memory 630 may be implemented in the form of software functional units and stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention further provides a non-transitory computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the above-provided method for calibrating an external parameter of a vehicle-mounted camera, the method comprising: acquiring a road surface image by using a vehicle-mounted camera in the vehicle; acquiring model information of a target vehicle and a plurality of key points of the target vehicle from the road surface image, wherein the target vehicle is a vehicle around the own vehicle, and the road surface image comprises a plurality of target vehicles; acquiring a target vehicle pair from the plurality of target vehicles according to the vehicle type information, wherein the target vehicle pair comprises a first vehicle and a second vehicle; selecting key points corresponding to the first vehicle and the second vehicle according to preset key point selection conditions, and obtaining at least two first key points corresponding to the first vehicle and at least two second key points corresponding to the second vehicle; acquiring vanishing points corresponding to the vehicle-mounted camera by utilizing cross ratio invariance according to the first key points and the second key points; the preset key point selection condition is that key points which meet the fact that connecting lines between the key points are parallel to the bottom edge or the top edge of the pavement image are selected from the plurality of key points, and under a world coordinate system, the distance between the key points in the first vehicle is equal to the distance between the key points in the second vehicle.
In another aspect, an embodiment of the present invention further provides a vehicle, including: the electronic device provided in the foregoing embodiment. The implementation principle and the generated technical effects of the vehicle provided by the embodiment of the invention are the same as those of the foregoing method embodiment, and are not described herein again.
In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of executing the above provided method for calibrating a parameter of an onboard camera, the method comprising: acquiring a road surface image by using a vehicle-mounted camera in the vehicle; acquiring model information of a target vehicle and a plurality of key points of the target vehicle from the road surface image, wherein the target vehicle is a vehicle around the own vehicle, and the road surface image comprises a plurality of target vehicles; acquiring a target vehicle pair from the plurality of target vehicles according to the vehicle type information, wherein the target vehicle pair comprises a first vehicle and a second vehicle; selecting key points corresponding to the first vehicle and the second vehicle according to preset key point selection conditions, and obtaining at least two first key points corresponding to the first vehicle and at least two second key points corresponding to the second vehicle; acquiring vanishing points corresponding to the vehicle-mounted camera by utilizing cross ratio invariance according to the first key points and the second key points; the preset key point selection condition is that key points which meet the fact that connecting lines between the key points are parallel to the bottom edge or the top edge of the road surface image are selected from the plurality of key points, and the distance between the key points in the first vehicle is equal to the distance between the key points in the second vehicle under a world coordinate system.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. The external parameter calibration method of the vehicle-mounted camera is characterized by comprising the following steps of:
acquiring a road surface image by using a vehicle-mounted camera in the vehicle;
acquiring model information of a target vehicle and a plurality of key points of the target vehicle from the road surface image, wherein the target vehicle is a vehicle around the own vehicle, and the road surface image comprises a plurality of target vehicles;
acquiring a target vehicle pair from the plurality of target vehicles according to the vehicle type information, wherein the target vehicle pair comprises a first vehicle and a second vehicle;
selecting key points corresponding to the first vehicle and the second vehicle according to preset key point selection conditions, and obtaining at least two first key points corresponding to the first vehicle and at least two second key points corresponding to the second vehicle;
Acquiring vanishing points corresponding to the vehicle-mounted camera by utilizing cross ratio invariance according to the first key points and the second key points;
the preset key point selection condition is that key points which meet the fact that connecting lines between the key points are parallel to the bottom edge or the top edge of the pavement image are selected from the plurality of key points, and under a world coordinate system, the distance between the key points in the first vehicle is equal to the distance between the key points in the second vehicle.
2. The method for calibrating an external parameter of a vehicle-mounted camera according to claim 1, wherein when the first key point and the second key point are two, and the first key point is a first key point a and a first key point b, and the second key point is a second key point c and a second key point d, the first key point a corresponds to the second key point c, and the first key point b corresponds to the second key point d; the distance between the first vehicle and the own vehicle is smaller than the distance between the second vehicle and the own vehicle;
the obtaining, according to the first key point and the second key point, the vanishing point corresponding to the vehicle-mounted camera by using cross-ratio invariance includes:
Selecting an equidistant point f on a straight line corresponding to a first key point a and a first key point b according to the position relation between the second key point c and the first key point a and the distance between the first key point a and the first key point b;
acquiring an intersection point g between a straight line corresponding to the equidistant point f and the second key point d and a straight line corresponding to the first key point b and the second key point c;
obtaining vanishing lines parallel to the bottom edge or the top edge of the pavement image according to the intersection point g;
and acquiring vanishing points from the vanishing lines according to the size information of the road surface image.
3. The method for calibrating an external parameter of an onboard camera according to claim 1, wherein after the obtaining of the vehicle type information of the target vehicle and the plurality of key points of the target vehicle from the road surface image, the method further comprises:
under the condition that the vehicle type information of the target vehicles is different, acquiring multi-frame road surface images, wherein the positions of the same target vehicles in the multi-frame road surface images are different;
combining the multi-frame pavement images to obtain a combined image;
and acquiring the model information of the target vehicle and a plurality of key points of the target vehicle from the combined image.
4. The method for calibrating an external parameter of an on-vehicle camera according to claim 1, wherein the target vehicle pairs are a plurality of target vehicle pairs, and the obtaining the vanishing point corresponding to the on-vehicle camera by using the cross invariance according to the first key point and the second key point comprises:
for each target vehicle pair of the plurality of target vehicle pairs, acquiring a vanishing point corresponding to the target vehicle pair by utilizing cross ratio invariance according to a first key point and a second key point corresponding to the target vehicle pair;
and carrying out average value calculation on a plurality of vanishing points corresponding to the plurality of target vehicles to obtain the vanishing points corresponding to the vehicle-mounted camera.
5. The external parameter calibration method of the vehicle-mounted camera according to claim 1, wherein the obtaining the target vehicle pair from the plurality of target vehicles according to the vehicle type information includes:
forming a team of target vehicles with the same vehicle type information to obtain a plurality of target vehicle pairs;
the vehicle type information is at least one of a small vehicle, a micro vehicle, a compact vehicle, a medium vehicle, a high-grade vehicle, a luxury vehicle, a three-compartment vehicle, a CDV vehicle, an MPV vehicle or an SUV.
6. The method for calibrating an external parameter of an on-vehicle camera according to any one of claims 1 to 5, wherein the first key point and the second key point are points external to a wheel.
7. An external parameter calibration device of a vehicle-mounted camera is characterized by comprising:
the image acquisition module is used for acquiring a road surface image by utilizing a vehicle-mounted camera in the vehicle;
the target vehicle information acquisition module is used for acquiring vehicle type information of a target vehicle and a plurality of key points of the target vehicle from the road surface image, wherein the target vehicle is a vehicle around the own vehicle, and the road surface image comprises a plurality of target vehicles;
the vehicle pair determining module is used for acquiring a target vehicle pair from the plurality of target vehicles according to the vehicle type information, wherein the target vehicle pair comprises a first vehicle and a second vehicle;
the key point selection module is used for selecting key points corresponding to the first vehicle and the second vehicle according to preset key point selection conditions, and obtaining at least two first key points corresponding to the first vehicle and at least two second key points corresponding to the second vehicle;
the vanishing point acquisition module is used for acquiring vanishing points corresponding to the vehicle-mounted camera by utilizing cross ratio invariance according to the first key points and the second key points;
The preset key point selection condition is that key points which meet the fact that connecting lines between the key points are parallel to the bottom edge or the top edge of the pavement image are selected from the plurality of key points, and under a world coordinate system, the distance between the key points in the first vehicle is equal to the distance between the key points in the second vehicle.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method for calibrating external parameters of an onboard camera according to any of claims 1-6 when executing the program.
9. A vehicle comprising the electronic device of claim 8.
10. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the method of calibrating an external parameter of an in-vehicle camera according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310602702.6A CN116664691A (en) | 2023-05-25 | 2023-05-25 | External parameter calibration method, device, equipment, vehicle and medium of vehicle-mounted camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310602702.6A CN116664691A (en) | 2023-05-25 | 2023-05-25 | External parameter calibration method, device, equipment, vehicle and medium of vehicle-mounted camera |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116664691A true CN116664691A (en) | 2023-08-29 |
Family
ID=87721774
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310602702.6A Pending CN116664691A (en) | 2023-05-25 | 2023-05-25 | External parameter calibration method, device, equipment, vehicle and medium of vehicle-mounted camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116664691A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315048A (en) * | 2023-11-22 | 2023-12-29 | 深圳元戎启行科技有限公司 | External parameter self-calibration method of vehicle-mounted camera, electronic equipment and storage medium |
-
2023
- 2023-05-25 CN CN202310602702.6A patent/CN116664691A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315048A (en) * | 2023-11-22 | 2023-12-29 | 深圳元戎启行科技有限公司 | External parameter self-calibration method of vehicle-mounted camera, electronic equipment and storage medium |
| CN117315048B (en) * | 2023-11-22 | 2024-04-12 | 深圳元戎启行科技有限公司 | External parameter self-calibration method of vehicle-mounted camera, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3699048A1 (en) | Travelling track prediction method and device for vehicle | |
| CN109859278B (en) | Calibration method and calibration system for camera external parameters of vehicle-mounted camera system | |
| US11688183B2 (en) | System and method of determining a curve | |
| US8384781B2 (en) | Stereo camera device | |
| CN111442776A (en) | Method and equipment for sequential ground scene image projection synthesis and complex scene reconstruction | |
| US8538205B2 (en) | Multi-function summing machine | |
| CN113312983B (en) | Semantic segmentation method, system, device and medium based on multi-mode data fusion | |
| CN105243655A (en) | System and method for estimating vehicle dynamics using feature points in images from multiple cameras | |
| WO2021260589A1 (en) | Monocular depth supervision from 3d bounding boxes | |
| CN116664691A (en) | External parameter calibration method, device, equipment, vehicle and medium of vehicle-mounted camera | |
| CN114842430B (en) | Vehicle information identification method and system for road monitoring | |
| JP7259309B2 (en) | Image processing device and image processing method | |
| CN114119749A (en) | Monocular 3D vehicle detection method based on dense association | |
| US11869253B2 (en) | Vehicle environment modeling with a camera | |
| CN114821531B (en) | Lane line recognition image display system based on electronic exterior rearview mirror ADAS | |
| US11477371B2 (en) | Partial image generating device, storage medium storing computer program for partial image generation and partial image generating method | |
| CN116740680A (en) | Vehicle positioning method and device and electronic equipment | |
| CN114120260A (en) | Method and system for identifying travelable area, computer device, and storage medium | |
| CN116206483B (en) | Parking position determining method, electronic device and computer readable storage medium | |
| WO2023126142A1 (en) | Method and apparatus for generating ground truth for other road participant | |
| WO2023126097A1 (en) | Method and apparatus for generating ground truth for driving boundaries | |
| CN117079240A (en) | Method and device for detecting driving line based on aerial view | |
| CN117911558A (en) | Method and device for building map by memorizing parking of motor vehicle and readable storage medium | |
| CN117601850A (en) | Method and device for memory parking positioning and deviation rectifying of motor vehicle and readable storage medium | |
| JP2022155102A (en) | Mobile body control device, control method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |