CN117036505B - On-line calibration method and system for vehicle-mounted camera - Google Patents
On-line calibration method and system for vehicle-mounted camera Download PDFInfo
- Publication number
- CN117036505B CN117036505B CN202311069501.0A CN202311069501A CN117036505B CN 117036505 B CN117036505 B CN 117036505B CN 202311069501 A CN202311069501 A CN 202311069501A CN 117036505 B CN117036505 B CN 117036505B
- Authority
- CN
- China
- Prior art keywords
- vehicle
- image
- coordinate system
- automobile
- mounted camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 239000011159 matrix material Substances 0.000 claims description 71
- 230000009466 transformation Effects 0.000 claims description 53
- 238000001514 detection method Methods 0.000 claims description 28
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 22
- 238000012937 correction Methods 0.000 claims description 14
- 238000013519 translation Methods 0.000 claims description 14
- 239000013598 vector Substances 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 238000006073 displacement reaction Methods 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 5
- 230000011218 segmentation Effects 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 5
- 230000009286 beneficial effect Effects 0.000 description 11
- 238000004364 calculation method Methods 0.000 description 5
- 230000000994 depressogenic effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000007373 indentation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 241001292396 Cirrhitidae Species 0.000 description 1
- 240000007594 Oryza sativa Species 0.000 description 1
- 235000007164 Oryza sativa Nutrition 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 235000009566 rice Nutrition 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T7/85—Stereo camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30256—Lane; Road marking
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention provides an on-line calibration method and system for a vehicle-mounted camera, wherein the method comprises the following steps: constructing four coordinate systems; controlling the vehicle-mounted camera to perform translational movement; controlling a vehicle-mounted camera to acquire a first image of a target object at a first moment; controlling the vehicle-mounted camera to acquire a second image of the calibration object at a second moment after translational movement; a third image of the calibration object is acquired at a third moment after the vehicle-mounted camera is controlled to rotate for a certain angle; calibrating internal parameters of the vehicle-mounted camera; and calibrating external parameters of the vehicle-mounted camera according to the first image, the second image and the third image. According to the camera calibration method based on the active vision, the camera calibration is realized, the calibration is completed in the running process of the automobile by fully utilizing the vehicle-mounted camera, the adjustment of the vehicle-mounted camera is realized, the operation is simple, the calculated amount of the vehicle-mounted camera calibration is reduced, and the calibration robustness is improved.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an on-line calibration method and system for a vehicle-mounted camera.
Background
The intelligent trolley is used for collecting data through the vehicle-mounted camera and completing data interaction with the control system. Therefore, calibration of the vehicle-mounted camera is a precondition for completing automatic driving. In reality, when the intelligent automobile is well provided with the vehicle-mounted camera for calibration, certain deviation is generated on the calibration result due to vibration of the automobile. In reality, a camera self-calibration method is often adopted for calibrating the vehicle-mounted camera, and the calibration is carried out through a Kruppa equation, but the camera self-calibration method has the defects of poor robustness, large calculated amount and the like, and the steps are complex.
The prior art CN202210928855 provides a method and a system for calibrating a vehicle-mounted camera online, which judge the type of a target vehicle according to the position of a vehicle identification frame to obtain a road vanishing point detection sample, and detect the road vanishing point through the upper and lower end points of the left/right boundary of the vehicle identification frame of the road vanishing point, so that the calculation step of the camera self-calibration method is not simplified, and the calibration robustness is not improved. Therefore, a simple on-line calibration method for the vehicle-mounted camera is needed to improve the calibration robustness.
Disclosure of Invention
The embodiment of the invention provides an on-line calibration method for a vehicle-mounted camera, which comprises the following steps:
1. the on-line calibration method for the vehicle-mounted camera is characterized by comprising the following steps of:
constructing a pixel coordinate system, an image coordinate system, a camera coordinate system and a world coordinate system;
based on road conditions, controlling the vehicle-mounted camera to perform translational movement;
controlling a vehicle-mounted camera to acquire a first image of a target object at a first moment;
controlling the vehicle-mounted camera to acquire a second image of the calibration object at a second moment after translational movement;
a third image of the calibration object is acquired at a third moment after the vehicle-mounted camera is controlled to rotate for a certain angle;
calibrating internal parameters of the vehicle-mounted camera based on affine transformation between a pixel coordinate system and an image coordinate system and projection transformation between the image coordinate system and a camera coordinate system;
Based on rigid body transformation between the world coordinate system and the camera coordinate system, the external parameters of the vehicle-mounted camera are calibrated through the first image, the second image and the third image.
Preferably, based on the road condition, controlling the vehicle-mounted camera to perform translational movement includes:
acquiring a road image of the future running of the automobile through a vehicle-mounted camera;
based on the road image of the future running of the automobile, predicting whether the automobile runs straight and stably,
wherein, based on the road image that the car is going in the future, judge whether the car is straight line steady and travel, include:
extracting two road edge lines based on a road image of the future running of the automobile;
when the distances from the automobile to the two road edge lines are equal, virtually extending the two road edge lines and intersecting the two road edge lines at one point to determine a road vanishing point;
judging whether the road vanishing point is positioned at the front central line of the automobile, if so, judging that the automobile runs straight, otherwise, judging that the automobile runs non-straight and adjusting the direction;
when the automobile is judged to run straight, the pothole detection is carried out on the road, and whether the automobile runs straight stably is judged;
collecting a first motion state of an automobile;
If the automobile runs linearly and stably, controlling the vehicle-mounted camera to perform translational motion in a world coordinate system based on a first motion state of the automobile, otherwise, not processing.
Preferably, determining that the vehicle is traveling non-linearly and performing the direction adjustment includes:
obtaining horizontal displacement from a road vanishing point to a central line in front of an automobile;
calculating a direction adjustment angle based on horizontal displacement from a road vanishing point to a central line in front of an automobile;
based on the direction adjustment angle, the automobile is controlled to carry out driving direction adjustment, so that the road vanishing point is positioned on the front central line of the automobile.
Preferably, when it is determined that the automobile is traveling straight, pothole detection is performed on the road and it is predicted whether the automobile is traveling straight smoothly, including:
constructing a pothole detection model based on historical data and training to obtain a trained pavement pothole detection model;
carrying out pothole detection on a road image of the future running of the automobile by using the trained pothole detection model;
if no pits are detected, judging that the automobile runs linearly and stably;
if the pothole is detected, carrying out pixel threshold segmentation on a road image of the automobile running in the future, and determining a pothole area;
judging whether wheels of the automobile pass through the hollow area or not based on the first motion state of the automobile;
If the wheels of the automobile do not pass through the hollow area, judging that the automobile runs linearly and stably;
if the wheels of the automobile pass through the hollow area, judging that the automobile runs unstably.
Preferably, based on the first motion state of the automobile, controlling the vehicle-mounted camera to perform translational motion in a world coordinate system includes:
acquiring a first motion state of an automobile, and predicting the direction and the speed of the automobile;
determining the direction of the vehicle-mounted camera on a camera coordinate system based on the predicted direction of the vehicle;
calculating a speed difference between a preset speed of the vehicle-mounted camera and the speed of the predicted vehicle based on the speed of the predicted vehicle, and determining the speed of the vehicle-mounted camera on a camera coordinate system;
determining the direction and the speed of the vehicle-mounted camera on a world coordinate system based on the direction and the speed of the vehicle-mounted camera on the camera coordinate system;
based on the direction and the speed of the vehicle-mounted camera on the world coordinate system, the vehicle-mounted camera is controlled to perform translational motion on the world coordinate system.
Preferably, calibrating the external parameters of the vehicle camera by the first image, the second image and the third image based on the rigid body transformation between the world coordinate system and the camera coordinate system comprises:
Respectively extracting features of the first image, the second image and the third image to obtain a first feature point, a second feature point and a third feature point;
determining a translation vector based on the first feature point and the second feature point;
determining a rotation matrix based on the second feature point and the third feature point;
determining an external parameter matrix based on the rotation matrix and the translation vector;
and calibrating external parameters of the vehicle-mounted camera based on matrix elements in the external parameter matrix.
Preferably, the on-line calibration method of the vehicle-mounted camera further comprises the following steps:
when the random disturbance of the automobile causes deviation, dynamically correcting the rotating torque array;
when the random disturbance of the automobile causes deviation, the method dynamically corrects the external parameters, and comprises the following steps:
acquiring the position of a road vanishing point at a first moment in an image coordinate system;
acquiring the position of a road vanishing point at the second moment in an image coordinate system;
calculating the position change difference of the road vanishing point after any time;
the rotational torque array is corrected based on the difference in position change.
Preferably, the on-line calibration method of the vehicle-mounted camera further comprises the following steps:
based on the internal parameters and the external parameters, carrying out pixel correction on a fourth image acquired by the vehicle-mounted camera;
The method for correcting the pixels of the fourth image acquired by the vehicle-mounted camera based on the internal parameters and the external parameters comprises the following steps:
collecting a fourth image of the calibration object and extracting features to obtain fourth feature points;
acquiring coordinates of a fourth feature point on a pixel coordinate system;
calculating coordinates of the fourth feature point on the world coordinate system based on the position of the fourth feature point on the calibration object, the size of the calibration object and the origin of the world coordinate system;
calculating an actual conversion matrix of the fourth feature point between the world coordinate system and the pixel coordinate system;
comparing the internal parameter matrix with the external parameter matrix with the actual conversion matrix to determine correction parameters;
and correcting any pixel in the fourth image according to the correction parameters to obtain a corrected fourth image.
Preferably, the on-line calibration method of the vehicle-mounted camera further comprises the following steps:
collecting images of adjacent vehicles, determining the distance between the images and the adjacent vehicles based on internal parameters, judging that the distance between the images and the adjacent vehicles is larger than a safety distance, and sending out a warning signal if the distance between the images is smaller than the set safety distance, otherwise, not processing the images;
wherein capturing an image of an adjacent vehicle and determining a distance to the adjacent vehicle based on the calibrated internal parameters comprises:
Obtaining a calibrated internal parameter;
adjusting the origin of the image coordinate system so that the origin of the image coordinate system coincides with the origin of the camera coordinate system;
acquiring images of adjacent vehicles;
based on historical data, constructing a vehicle height recognition model and training to obtain a trained vehicle height recognition model;
identifying adjacent vehicles on the image by using the trained vehicle height identification model to obtain the heights of the adjacent vehicles on a camera coordinate system;
acquiring the height of an adjacent vehicle on a pixel coordinate system;
affine transformation and projective transformation are carried out on the height of the adjacent vehicle on the pixel coordinate system based on affine transformation between the pixel coordinate system and the image coordinate system and projective transformation between the image coordinate system and the camera coordinate system, and the distance between the adjacent vehicle and the camera coordinate system is determined;
and judging that the distance between the vehicle and the adjacent vehicle on the camera coordinate system is greater than the set safety distance, if the distance is less than the set safety distance, sending out a warning signal, otherwise, not processing.
The invention also provides an on-line calibration system of the vehicle-mounted camera, which comprises the following steps:
the coordinate system construction module is used for constructing a pixel coordinate system, an image coordinate system, a camera coordinate system and a world coordinate system;
The motion control module is used for controlling the vehicle-mounted camera to perform translational motion based on road conditions,
the first image acquisition module is used for controlling the vehicle-mounted camera to acquire a first image of the calibration object at a first moment;
the second image acquisition module is used for controlling the vehicle-mounted camera to acquire a second image of the calibration object at a second moment after translational movement;
the third image acquisition module is used for controlling the vehicle-mounted camera to rotate by a certain angle and acquiring a third image of the calibration object at a third moment;
the internal parameter calibration module is used for calibrating the internal parameters of the vehicle-mounted camera based on affine transformation between the pixel coordinate system and the image coordinate system and projection transformation between the image coordinate system and the camera coordinate system;
and the external parameter calibration module is used for calibrating external parameters of the vehicle-mounted camera according to the first image, the second image and the third image based on rigid body transformation between the world coordinate system and the camera coordinate system.
The invention has the beneficial effects that:
according to the camera calibration method based on the active vision, the camera calibration is realized, the calibration is completed in the running process of the automobile by fully utilizing the vehicle-mounted camera, the adjustment of the vehicle-mounted camera is realized, the operation is simple, the calculated amount of the vehicle-mounted camera calibration is reduced, and the calibration robustness is improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, serve to explain the invention. In the drawings:
FIG. 1 is a schematic diagram of an on-line calibration method for a vehicle-mounted camera in an embodiment of the invention;
fig. 2 is a schematic diagram of an on-line calibration system for a vehicle-mounted camera according to an embodiment of the invention.
Detailed Description
The preferred embodiments of the present invention will be described below with reference to the accompanying drawings, it being understood that the preferred embodiments described herein are for illustration and explanation of the present invention only, and are not intended to limit the present invention.
The embodiment of the invention provides an on-line calibration method of a vehicle-mounted camera, which is shown in fig. 1 and comprises the following steps:
Step 1: constructing a pixel coordinate system, an image coordinate system, a camera coordinate system and a world coordinate system;
step 2: based on the road condition, the vehicle-mounted camera is controlled to perform translational movement,
step 3: controlling a vehicle-mounted camera to acquire a first image of a target object at a first moment;
step 4: controlling the vehicle-mounted camera to acquire a second image of the calibration object at a second moment after translational movement;
step 5: a third image of the calibration object is acquired at a third moment after the vehicle-mounted camera is controlled to rotate for a certain angle;
step 6: calibrating internal parameters of the vehicle-mounted camera based on affine transformation between a pixel coordinate system and an image coordinate system and projection transformation between the image coordinate system and a camera coordinate system;
step 7: based on rigid body transformation between the world coordinate system and the camera coordinate system, external parameters of the vehicle-mounted camera are calibrated according to the first image, the second image and the third image.
The working principle and the beneficial effects of the technical scheme are as follows:
the four coordinate systems of the camera are a pixel coordinate system, a camera coordinate system, an image coordinate system and a world coordinate system respectively. The world coordinate system is a coordinate system where an object to be imaged is located, the camera coordinate system is a coordinate system taking a pinhole of a vehicle-mounted camera as an origin, the image coordinate system is a coordinate system taking an intersection point between an optical axis and a projection plane as an origin, the pixel coordinate system is in a direction from the pinhole of the camera to the projection plane, the upper left corner of the projection plane is the origin, two coordinate axes are overlapped with two sides of the projection plane, and the coordinate system and the image coordinate system are in the same plane. Therefore, point coordinates on the world coordinate system and the camera coordinate system are calculated by rigid body change, while the camera coordinate system and the image coordinate system are calculated by perspective projection, and the image coordinate system and the pixel coordinate system are calculated by affine transformation. Finally, the coordinates of a point in the pixel coordinate system are calculated corresponding to the coordinates of the point in the world coordinate system and an internal parameter matrix consisting of internal parameters and an external parameter matrix consisting of external parameters, wherein the internal parameters include the focal length, the physical size of the pixel, and the number of pixels whose center of the image is different from the origin. The external parameter is a rotation matrix, i.e. a matrix of yaw and pitch angles. The present embodiment uses a checkerboard as a calibration object, and other calibration objects such as circular grid, halcon calibration plate, etc. can be implemented. Determining a checkerboard as a calibration object and determining calibration points of the calibration object. The embodiment determines the internal parameters and the external parameters by an active vision based calibration method. However, the calibration method based on active vision needs to ensure that the camera keeps special movement, such as uniform movement or uniform rotation movement. And 1, constructing a pixel coordinate system and a world coordinate system. Step 2, based on road conditions, controlling the vehicle-mounted camera to perform uniform motion, wherein the specific motion, including translational motion and rotational motion, of the vehicle-mounted camera is required to be kept. Step 3 and step 4 are leveled in a time interval set by the vehicle-mounted camera A first image and a second image are acquired separately for the calibration object. And 5, controlling the vehicle-mounted camera to rotate by a certain angle, and acquiring a third image of the calibration object at a third moment. And 6, calibrating internal parameters of the vehicle-mounted camera based on affine transformation between the pixel coordinate system and the image coordinate system and projection transformation between the image coordinate system and the camera coordinate system. For example, the coordinates of a point on the image coordinate system are (x, y), and the coordinates of the point on the corresponding pixel coordinate system are (u, v) calculated by affine transformation between the pixel coordinate system and the image coordinate system. (u, v) is equal to (x, y) times the affine transformation matrix a. The matrix elements of the affine transformation matrix a are represented by coordinates (u) of the origin of the image coordinate system on the pixel coordinate system 0 ,v 0 ) And the same is true for the construction of the variable scale of each pixel, the coordinates of a point on the image coordinate system are (x, y) and the coordinates of the corresponding camera coordinate system (x r ,y r ) And projective transformation between. Based on pinhole imaging principle, (x, y) is equal to (x r ,y r ) Multiplied by a projective transformation matrix B, the matrix elements of which are constructed by the focal length f of the vehicle-mounted camera. Therefore, based on affine transformation between the pixel coordinate system and the image coordinate system and projective transformation between the image coordinate system and the camera coordinate system, an internal parameter matrix M can be determined, which is equal to the cross-multiplication of the affine transformation matrix a and the projective transformation matrix B, and the internal parameters of the vehicle-mounted camera are calibrated according to the internal parameter matrix M. Wherein the internal parameters of the vehicle-mounted camera comprise coordinates (u) of the origin of the image coordinate system on the pixel coordinate system 0 ,v 0 ) The varying scale of each pixel and the focal length f of the onboard camera. And step 7, calibrating external parameters of the vehicle-mounted camera according to the first image, the second image and the third image based on rigid body transformation between the world coordinate system and the camera coordinate system. For example, a point coordinate in the world coordinate system is (x 0 ,y 0 ,z 0 ) The coordinates of the corresponding pixel coordinate system are (u, v). And this point. Thus, a certain point coordinate is converted from (x) in the world coordinate system by a conversion matrix S 0 ,y 0 ,z 0 ) Is converted into a corresponding pixel coordinate system (u, v). WhileThe transformation matrix S can be split into an internal parameter matrix M and an external parameter matrix N according to the nature of the parameters. The conversion matrix S is equal to the inner parameter matrix M multiplied by the outer parameter matrix N. Whereas the internal parameter matrix M has been determined so that an external parameter matrix N can be calculated, which comprises a rotation matrix C and a translation vector D. From the translational movement of the first image and the second image, a translation vector D can be determined. And from the rotational movements of the second image and the third image, a rotation matrix C can be determined. Thus, the calibration of external parameters is realized.
According to the embodiment of the invention, the camera calibration is realized based on the camera calibration method of active vision, the calibration is completed by fully utilizing the vehicle-mounted camera in the running process of the automobile, the adjustment of the vehicle-mounted camera is realized, the operation is simple, the calculated amount of external parameters of the vehicle-mounted camera calibration is reduced, and the calibration robustness is improved.
In one embodiment, step 2 comprises:
step 2.1: acquiring a road image of the future running of the automobile through a vehicle-mounted camera;
step 2.2: based on the road image of the future running of the automobile, predicting whether the automobile runs straight and stably,
wherein, step 2.2 includes:
step 2.2.1: extracting two road edge lines based on a road image of the future running of the automobile;
step 2.2.2: when the distances from the automobile to the two road edge lines are equal, virtually extending the two road edge lines and intersecting the two road edge lines at one point to determine a road vanishing point;
step 2.2.3: judging whether the road vanishing point is positioned at the front central line of the automobile, if so, judging that the automobile runs straight, otherwise, judging that the automobile runs non-straight and adjusting the direction;
step 2.2.4: when the automobile is judged to run straight, the pothole detection is carried out on the road, and whether the automobile runs straight stably is judged;
step 2.3: collecting a first motion state of an automobile;
step 2.4: if the automobile runs linearly and stably, the vehicle-mounted camera is controlled to perform uniform motion in a world coordinate system based on the first motion state of the automobile, otherwise, the vehicle-mounted camera is not processed.
The working principle and the beneficial effects of the technical scheme are as follows:
The vehicle-mounted camera is controlled to keep constant motion in a world coordinate system, the motion of a carrier vehicle of the camera is controlled firstly, and step 2.1 is to collect road images of future running of the vehicle through the vehicle-mounted camera. And 2.2, predicting whether the automobile runs linearly and stably according to the acquired road image. Step 2.3, acquiring a first motion state of the automobile, wherein the first motion state comprises data of automobile driving, the amplitude of up-and-down jolt of the automobile and the like. And 2.4, when the automobile is judged to be in straight stable running, controlling the vehicle-mounted camera to perform uniform motion in a world coordinate system according to the first motion state of the automobile. The vehicle-mounted camera is adjusted according to the motion of the carrier vehicle of the camera, if the vehicle moves forward at a constant speed of 3m/s, the control camera moves forward at a constant speed of-1 m/s, so that the vehicle-mounted camera is enabled. If the automobile does not run straight and stably, the condition of the calibration method based on the active vision is not met, so that a prompt message is sent to prompt a driver to control the automobile to run straight and stably.
The embodiment of the invention predicts the road condition by collecting the road image of the future running of the automobile,
and 2.2.1, extracting two road edge lines based on the road image of the future running of the automobile. The image of an automobile running on a road has two lane lines. Two road edge lines are determined from the two lane lines. Step 2.2.2 when the distance from the car to the two road edge lines is equal, respectively, i.e. the on-board camera on the car is on the centre line in the road. Therefore, two road edge lines are virtually extended and intersected at one point, and the virtual point is the road vanishing point. And 2.2.3, judging whether the road vanishing point is positioned on the front central line of the automobile, if so, judging that the automobile is in straight running, otherwise, judging that the automobile is in non-straight running, and carrying out direction adjustment. And 2.2.4, when the automobile is judged to run straight, detecting the pits on the road, and judging whether the automobile runs straight and stably.
According to the embodiment, the road image of the future running of the automobile is extracted through the automobile, the two road edge lines are extracted to carry out straight running judgment, the straight running of the automobile is conveniently and rapidly judged by artificial intelligence, the calibration time is saved, the translation control of the vehicle-mounted camera is simultaneously realized, the straight stable running of the automobile is met, and the calibration method of active vision is conveniently utilized for calibration.
In one embodiment, determining that the vehicle is traveling non-straight and making a direction adjustment includes:
the first step: obtaining horizontal displacement from a road vanishing point to a central line in front of an automobile;
and a second step of: calculating a direction adjustment angle based on horizontal displacement from a road vanishing point to a central line in front of an automobile;
and a third step of: based on the direction adjustment angle, the automobile is controlled to carry out driving direction adjustment, so that the road vanishing point is positioned on the front central line of the automobile.
The working principle and the beneficial effects of the technical scheme are as follows:
the precondition for the direction adjustment of the vehicle is that the vehicle is traveling on a straight road. According to the first step of the road, horizontal displacement from a road vanishing point to the central line in front of the automobile is obtained from an image generated by the vehicle-mounted camera, and the position of the vehicle-mounted camera is located on the central line in front of the automobile. And a second step of: and calculating the direction adjustment angle based on the horizontal displacement from the road vanishing point to the central line in front of the automobile. And a third step of: based on the calculated direction adjustment angle, the automobile is controlled to carry out driving direction adjustment, so that the road vanishing point is positioned on the front central line of the automobile, namely, the driving direction of the automobile is adjusted, and the vehicle-mounted camera and the road vanishing point are positioned on the front central line of the automobile.
The invention adjusts the running direction through the horizontal displacement of the road vanishing point and the front central line of the automobile, and can conveniently adjust the direction according to the real-time image of the vehicle-mounted camera.
In one embodiment, step 2.2.4 comprises:
step 2.2.4.1: constructing a pothole detection model based on historical data and training to obtain a trained pavement pothole detection model;
step 2.2.4.2: carrying out pothole detection on a road image of the future running of the automobile by using the trained pothole detection model;
step 2.2.4.3: if no pits are detected, judging that the automobile runs linearly and stably;
step 2.2.4.4: if the pothole is detected, image segmentation is carried out on a road image of the future running of the automobile, and a pothole area is determined;
step 2.2.4.5: judging whether wheels of the automobile pass through the hollow area or not based on the first motion state of the automobile;
step 2.2.4.6: if the wheels of the automobile do not pass through the hollow area, judging that the automobile runs linearly and stably;
step 2.2.4.7: if the wheels of the automobile pass through the hollow area, judging that the automobile runs unstably.
The working principle and the beneficial effects of the technical scheme are as follows:
step 2.2.4.1 is to construct a pothole detection model based on the historical data and train the pothole detection model to obtain a trained pavement pothole detection model. In the embodiment, a pavement pothole detection model is built through a convolutional neural network model, and the built pavement pothole detection model is trained by utilizing historical data. Step 2.2.4.2, performing pothole detection on the road image of the automobile running in the future by using the trained pothole detection model to obtain a detection result, and judging whether a pothole exists or not. Step 2.2.4.3: if no pits are detected, the straight-line stable running of the automobile can be judged by combining the straight-line running result of the step 2.2.3. If a depression is detected in step 2.2.4.4, the road image of the vehicle traveling in the future is subjected to pixel threshold segmentation to determine a depression area. The present embodiment detects by computer vision. Since the indentations are essentially depressed and the texture of the indentations is coarser than the surrounding pavement, the pixel intensity is also lower than the surrounding pavement. The detection is thus based on the image characteristics of the depressed features of the depressed areas. If the pothole is detected, semantic segmentation is carried out on the road image of the future running of the automobile, and a pothole image area and a non-pothole image area A are segmented. Step 2.2.4.5 obtains a first motion state of the vehicle, such as a wheel position B and a running speed of the vehicle, calculates whether the wheel position B of the vehicle passes through the pothole area a, that is, whether the area of the wheel of the vehicle coincides with the pothole area a, if so, determines that the wheel of the vehicle passes through the pothole area, and if not, determines that the wheel of the vehicle does not pass through the pothole area. If the step 2.2.4.6 determines that the wheels of the automobile do not pass through the hollow area, it determines that the wheels of the automobile have bypassed the hollow area, and the straight stable running of the automobile is not affected. In step 2.2.4.7, if the wheels of the vehicle pass through the hollow area, it is determined that the wheels of the vehicle will be affected by the hollow area, and the vehicle will not be able to run stably.
According to the embodiment of the invention, whether the road is provided with the pits is detected, if so, the wheel driving area is overlapped with the pit area, so that whether the automobile is in the future or not is judged, and a precondition is provided for controlling the automobile to stably drive and reasonably selecting the calibration time.
In one embodiment, step 2.2.4.5 includes:
step 2.2.4.5.1: acquiring a first motion state of an automobile, and predicting the direction and the speed of the automobile;
step 2.2.4.5.2: determining the direction of the vehicle-mounted camera on a camera coordinate system based on the predicted direction of the vehicle;
step 2.2.4.5.3: calculating a speed difference between a preset speed of the vehicle-mounted camera and the speed of the predicted vehicle based on the speed of the predicted vehicle, and determining the speed of the vehicle-mounted camera on a camera coordinate system;
step 2.2.4.5.4: determining the direction and the speed of the vehicle-mounted camera on a world coordinate system based on the direction and the speed of the vehicle-mounted camera on the camera coordinate system;
step 2.2.4.5.5: based on the direction and the speed of the vehicle-mounted camera on the world coordinate system, the vehicle-mounted camera is controlled to perform translational motion on the world coordinate system.
The working principle and the beneficial effects of the technical scheme are as follows:
step 2.2.4.5.1 obtains a first motion state of the vehicle, predicts a direction of the vehicle And velocity v q . Step 2.2.4.5.2 determines that the direction of the onboard camera in the camera coordinate system is also +.>Step 2.2.4.5.3: speed v of a motor vehicle based on predictions q Calculating the preset speed v of the vehicle-mounted camera c With predicted speed v of car q And determining the speed Deltav of the vehicle-mounted camera on a camera coordinate system according to the speed Deltav. Step 2.2.4.5.4 is based on the direction of the vehicle camera in the camera coordinate system +.>And the speed Deltav, determining the direction and the speed v of the vehicle-mounted camera on a world coordinate system c . Step 2.2.4.5.5 controls the vehicle-mounted camera to perform uniform motion in the world coordinate system based on the direction and the speed of the vehicle-mounted camera in the world coordinate system.
According to the embodiment of the invention, the vehicle-mounted camera is controlled to perform uniform motion in the world coordinate system based on the running state of the vehicle, so that the relative motion of the vehicle-mounted camera on the vehicle is realized, and the vehicle-mounted camera is convenient to control.
In one embodiment, step 7 comprises:
step 7.1: respectively extracting features of the first image, the second image and the third image to obtain a first feature point, a second feature point and a third feature point;
step 7.2: determining a translation vector based on the first feature point and the second feature point;
Step 7.3: determining a rotation matrix based on the second feature point and the third feature point;
step 7.4: determining an external parameter matrix based on the rotation matrix and the translation vector;
step 7.5: and calibrating external parameters of the vehicle-mounted camera based on matrix elements in the external parameter matrix.
The working principle and the beneficial effects of the technical scheme are as follows:
and 7.1, respectively extracting the characteristics of the first image, the second image and the third image to obtain a first characteristic point, a second characteristic point and a third characteristic point. In this embodiment, the checkerboard is used as the calibration object, and therefore, the corner points of the checkerboard are used as the feature points. And 7.2, matching the first characteristic points and the second characteristic points, and calculating coordinate changes of the two first characteristic points and the second characteristic points on a world coordinate system to determine translation vectors. And 7.3, matching the second characteristic points with the third characteristic points, and calculating coordinate changes of the two first characteristic points and the second characteristic points on a world coordinate system to determine a rotation matrix. Since the external parameter matrix includes a rotation matrix and a translation vector. Therefore, the external parameter matrix can be obtained from the rotation matrix and the translation vector, and the external parameters of the vehicle-mounted camera are obtained according to the meaning of matrix elements of the external parameter matrix N, so that the external parameter calibration is realized.
According to the embodiment of the invention, the external parameters of the vehicle-mounted camera are calculated by calibrating the calibration object, so that the on-line calibration of the vehicle-mounted camera is conveniently realized.
In one embodiment, the on-line calibration method of the vehicle-mounted camera further comprises the following steps:
step 8: when the random disturbance of the automobile causes deviation, dynamically correcting the rotating torque array;
wherein, step 8 includes:
step 8.1: acquiring the position of a road vanishing point at a first moment in an image coordinate system;
step 8.2: acquiring the position of a road vanishing point at the second moment in an image coordinate system;
step 8.3: calculating the position change difference of the road vanishing point after any time;
step 8.4: the rotational torque array is corrected based on the difference in position change.
The working principle and the beneficial effects of the technical scheme are as follows:
since unexpected random disturbances may be generated on the car, such as random disturbances generated by the bearings of the car while rotating. Random disturbances are classified into transverse disturbances and longitudinal disturbances. The influence of the longitudinal disturbance on the calibration is small, and the influence of the transverse disturbance on the calibration is large, so that the result of the rotation matrix needs to be corrected, and the calculation result of the external parameters is more accurate. The embodiment passes the positions of the road vanishing points at the first time and the second time in the image coordinate system. And 8.3, calculating the position change difference of the road vanishing point after any time. And 8.4, correcting the rotation matrix based on the position change difference, and deducting the influence of the random disturbance on the rotation matrix to realize the correction of the rotation matrix.
According to the embodiment of the invention, errors generated by random disturbance are eliminated for the rotating torque array, so that the calculation result of external parameters is more accurate.
In one embodiment, the on-line calibration method of the vehicle-mounted camera further comprises the following steps:
step 9: based on the internal parameters and the external parameters, carrying out pixel correction on a fourth image acquired by the vehicle-mounted camera;
wherein, step 9 includes:
step 9.1: collecting a fourth image of the calibration object and extracting features to obtain fourth feature points;
step 9.2: acquiring coordinates of a fourth feature point on a pixel coordinate system;
step 9.3: calculating coordinates of the fourth feature point on the world coordinate system based on the position of the fourth feature point on the calibration object, the size of the calibration object and the origin of the world coordinate system;
step 9.4: calculating an actual conversion matrix of the fourth feature point between the world coordinate system and the pixel coordinate system;
step 9.5: comparing the internal parameter matrix with the external parameter matrix with the actual conversion matrix to determine correction parameters;
step 9.6: and correcting any pixel in the fourth image according to the correction parameters to obtain a corrected fourth image.
The working principle and the beneficial effects of the technical scheme are as follows:
Step 8.1 acquiring a fourth image of the calibration object and characterizingAnd extracting to obtain a fourth characteristic point. And 8.2, acquiring coordinates of the fourth feature point on a pixel coordinate system. And calculating the coordinates of the fourth feature point on the world coordinate system based on the position of the fourth feature point on the calibration object, the size of the calibration object and the origin of the world coordinate system. For example, the fourth feature point is at the fourth corner point, and the size of the calibration object is a square of 1 meter, wherein the square calibration object is divided into 10 x 10 black-white small squares. And calculating the coordinates of the fourth feature point on the world coordinate system according to the origin of the world coordinate system. Step 8.4 calculating an actual transformation matrix of the fourth feature points between the world coordinate system and the pixel coordinate systemStep 8.5: the internal parameter matrix M and the external parameter matrix N are combined with the actual conversion matrix>The correction parameter mu is determined by comparison. And 8.6, correcting any pixel in the fourth image according to the correction parameter mu to obtain a corrected fourth image.
According to the invention, the correction parameters are introduced to carry out pixel correction on the acquired image, so that the image deformation formed in the image acquisition process is overcome, and the calculated result of the image acquired by the calibration object is more accurate.
In one embodiment, the on-line calibration method of the vehicle-mounted camera further comprises the following steps:
step 10: collecting images of adjacent vehicles, determining the distance between the images and the adjacent vehicles based on internal parameters, judging that the distance between the images and the adjacent vehicles is larger than a safety distance, and sending out a warning signal if the distance between the images is smaller than the set safety distance, otherwise, not processing the images;
wherein, step 10 includes:
step 10.1: obtaining a calibrated internal parameter;
step 10.2: adjusting the origin of the image coordinate system so that the origin of the image coordinate system coincides with the origin of the camera coordinate system;
step 10.3: acquiring images of adjacent vehicles;
step 10.4: based on historical data, constructing a vehicle height recognition model and training to obtain a trained vehicle height recognition model;
step 10.5: identifying adjacent vehicles on the image by using the trained vehicle height identification model to obtain the heights of the adjacent vehicles on a camera coordinate system;
step 10.6: acquiring the height of an adjacent vehicle on a pixel coordinate system;
step 10.7: affine transformation and projective transformation are carried out on the height of the adjacent vehicle on the pixel coordinate system based on affine transformation between the pixel coordinate system and the image coordinate system and projective transformation between the image coordinate system and the camera coordinate system, and the distance between the adjacent vehicle and the camera coordinate system is determined;
And 10.8, judging that the distance between the vehicle and the adjacent vehicle on the camera coordinate system is larger than the set safety distance, if the distance is smaller than the set safety distance, sending out a warning signal, otherwise, not processing.
The working principle and the beneficial effects of the technical scheme are as follows:
obtaining calibrated internal parameters such as the origin of an image coordinate system on a pixel coordinate system
(u 0 ,v 0 ) The focal length of the vehicle-mounted camera is 0.064m, and the physical size of the pixel, that is, the size of one pixel (du, dv), du=dv=0.08 mm in the present embodiment. In order to facilitate the measurement of the distance, step 10.2 of this embodiment adjusts the origin of the image coordinate system such that the origin of the image coordinate system coincides with the origin of the camera coordinate system, i.e. (u) 0 ,v 0 ) = (0, 0). Step 10.3 acquires images of adjacent vehicles. And 10.4, constructing a vehicle height identification model by using the neural network model. The vehicle height identification model is trained using historical data. The vehicle height recognition model recognizes the height of the vehicle, such as a truck, and the vehicle height recognition model recognizes the height of the truck, which can be recognized as 3 meters. Step 10.5 identifying neighboring vehicles, such as neighboring vehicles, using the trained vehicle height identification model The vehicle was a truck and the identified height was 3 meters. Step 10.6 acquires the height of the card on the pixel coordinate system, for example 1000 pixels. Step 10.7 distance from the neighboring vehicle in the camera coordinate System according to the calculation formulas of affine transformation and projective transformationAnd (5) rice. Step 10.8, setting the safety distance between the driver and the truck to be 3 meters, and sending out a warning signal to prompt the driver to pull away the distance between the driver and the truck if the distance L is smaller than 3 meters. When the vehicle and the adjacent truck exceed 3 meters, the warning signal of the vehicle stops sending.
According to the invention, the distance between the vehicle and the adjacent vehicle is calculated through affine transformation and projective transformation, and whether the distance between the vehicle and the adjacent vehicle is within the safe distance is judged according to the distance between the vehicle and the adjacent vehicle, so that the safe running of the vehicle is improved.
The invention also provides an on-line calibration system of the vehicle-mounted camera, which comprises the following steps:
the coordinate system construction module 1 is used for constructing a pixel coordinate system, an image coordinate system, a camera coordinate system and a world coordinate system;
a motion control module 2 for controlling the vehicle-mounted camera to perform translational motion based on road conditions,
the first image acquisition module 3 is used for controlling the vehicle-mounted camera to acquire a first image of the calibration object at a first moment;
the second image acquisition module 4 is used for controlling the vehicle-mounted camera to acquire a second image of the calibration object at a second moment after translational movement;
The third image acquisition module 5 is used for controlling the vehicle-mounted camera to rotate by a certain angle and acquiring a third image of the calibration object at a third moment;
the internal parameter calibration module 6 is used for calibrating the internal parameters of the vehicle-mounted camera based on affine transformation between the pixel coordinate system and the image coordinate system and projection transformation between the image coordinate system and the camera coordinate system;
and the external parameter calibration module 7 is used for calibrating external parameters of the vehicle-mounted camera according to the first image, the second image and the third image based on rigid body transformation between the world coordinate system and the camera coordinate system.
The working principle and the beneficial effects of the technical scheme are as follows:
the vehicle-mounted camera online calibration system constructs four coordinate systems, namely an image coordinate system, a camera coordinate system and a world coordinate system through a coordinate system construction module 1. And the motion control module 2 is used for controlling the vehicle-mounted camera to perform translational motion based on road conditions. The first image acquisition module 3, the second image acquisition module 4 and the third image acquisition module 5 acquire a first image, a second image and a third image related to the calibration object respectively at any time through the vehicle-mounted camera. And the internal parameter calibration module 6 is used for calibrating the internal parameters of the vehicle-mounted camera based on affine transformation between the pixel coordinate system and the image coordinate system and projection transformation between the image coordinate system and the camera coordinate system. The external parameter calculation module 7 calibrates external parameters of the vehicle-mounted camera by using the first image, the second image and the third image.
According to the embodiment of the invention, the camera calibration is realized based on the camera calibration method of active vision, the calibration is completed by fully utilizing the vehicle-mounted camera in the running process of the automobile, the adjustment of the vehicle-mounted camera is realized, the operation is simple, the calculated amount of the vehicle-mounted camera calibration is reduced, and the calibration robustness is improved.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (8)
1. The on-line calibration method for the vehicle-mounted camera is characterized by comprising the following steps of:
constructing a pixel coordinate system, an image coordinate system, a camera coordinate system and a world coordinate system;
based on road conditions, controlling the vehicle-mounted camera to perform translational movement;
controlling a vehicle-mounted camera to acquire a first image of a target object at a first moment;
controlling the vehicle-mounted camera to acquire a second image of the calibration object at a second moment after translational movement;
a third image of the calibration object is acquired at a third moment after the vehicle-mounted camera is controlled to rotate for a certain angle;
calibrating internal parameters of the vehicle-mounted camera based on affine transformation between a pixel coordinate system and an image coordinate system and projection transformation between the image coordinate system and a camera coordinate system;
Calibrating external parameters of the vehicle-mounted camera by the first image, the second image and the third image based on rigid body transformation between the world coordinate system and the camera coordinate system;
based on road conditions, control on-vehicle camera and carry out translational motion, include:
acquiring a road image of the future running of the automobile through a vehicle-mounted camera;
based on the road image of the future running of the automobile, predicting whether the automobile runs straight and stably,
wherein, based on the road image that the car is going in the future, judge whether the car is straight line steady and travel, include:
extracting two road edge lines based on a road image of the future running of the automobile;
when the distances from the automobile to the two road edge lines are equal, virtually extending the two road edge lines and intersecting the two road edge lines at one point to determine a road vanishing point;
judging whether the road vanishing point is positioned at the front central line of the automobile, if so, judging that the automobile runs straight, otherwise, judging that the automobile runs non-straight and adjusting the direction;
when the automobile is judged to run straight, the pothole detection is carried out on the road, and whether the automobile runs straight stably is judged;
collecting a first motion state of an automobile;
if the automobile runs linearly and stably, controlling the vehicle-mounted camera to perform translational motion in a world coordinate system based on a first motion state of the automobile, otherwise, not processing;
Calibrating external parameters of the vehicle-mounted camera by the first image, the second image and the third image based on rigid body transformation between the world coordinate system and the camera coordinate system, wherein the method comprises the following steps:
respectively extracting features of the first image, the second image and the third image to obtain a first feature point, a second feature point and a third feature point;
determining a translation vector based on the first feature point and the second feature point;
determining a rotation matrix based on the second feature point and the third feature point;
determining an external parameter matrix based on the rotation matrix and the translation vector;
and calibrating external parameters of the vehicle-mounted camera based on matrix elements in the external parameter matrix.
2. The on-line calibration method of an on-vehicle camera according to claim 1, wherein determining that the vehicle is traveling non-linearly and performing the direction adjustment comprises:
obtaining horizontal displacement from a road vanishing point to a central line in front of an automobile;
calculating a direction adjustment angle based on horizontal displacement from a road vanishing point to a central line in front of an automobile;
based on the direction adjustment angle, the automobile is controlled to carry out driving direction adjustment, so that the road vanishing point is positioned on the front central line of the automobile.
3. The on-vehicle camera online calibration method according to claim 1, wherein when it is determined that the vehicle is traveling straight, pothole detection is performed on a road, and whether the vehicle is traveling straight smoothly is predicted, comprising:
Constructing a pothole detection model based on historical data and training to obtain a trained pavement pothole detection model;
carrying out pothole detection on a road image of the future running of the automobile by using the trained pothole detection model;
if no pits are detected, judging that the automobile runs linearly and stably;
if the pothole is detected, carrying out pixel threshold segmentation on a road image of the automobile running in the future, and determining a pothole area;
judging whether wheels of the automobile pass through the hollow area or not based on the first motion state of the automobile;
if the wheels of the automobile do not pass through the hollow area, judging that the automobile runs linearly and stably;
if the wheels of the automobile pass through the hollow area, judging that the automobile runs unstably.
4. The on-line calibration method of the vehicle-mounted camera according to claim 3, wherein controlling the vehicle-mounted camera to perform translational motion in a world coordinate system based on a first motion state of the vehicle comprises:
acquiring a first motion state of an automobile, and predicting the direction and the speed of the automobile;
determining the direction of the vehicle-mounted camera on a camera coordinate system based on the predicted direction of the vehicle;
calculating a speed difference between a preset speed of the vehicle-mounted camera and the speed of the predicted vehicle based on the speed of the predicted vehicle, and determining the speed of the vehicle-mounted camera on a camera coordinate system;
Determining the direction and the speed of the vehicle-mounted camera on a world coordinate system based on the direction and the speed of the vehicle-mounted camera on the camera coordinate system;
based on the direction and the speed of the vehicle-mounted camera on the world coordinate system, the vehicle-mounted camera is controlled to perform translational motion on the world coordinate system.
5. The on-line calibration method of an on-vehicle camera according to claim 1, further comprising:
when the random disturbance of the automobile causes deviation, dynamically correcting the rotating torque array;
when the random disturbance of the automobile causes deviation, the method dynamically corrects the external parameters, and comprises the following steps:
acquiring the position of a road vanishing point at a first moment in an image coordinate system;
acquiring the position of a road vanishing point at the second moment in an image coordinate system;
calculating the position change difference of the road vanishing point after any time;
the rotational torque array is corrected based on the difference in position change.
6. The on-line calibration method of an on-vehicle camera according to claim 1, further comprising:
based on the internal parameters and the external parameters, carrying out pixel correction on a fourth image acquired by the vehicle-mounted camera;
the method for correcting the pixels of the fourth image acquired by the vehicle-mounted camera based on the internal parameters and the external parameters comprises the following steps:
Collecting a fourth image of the calibration object and extracting features to obtain fourth feature points;
acquiring coordinates of a fourth feature point on a pixel coordinate system;
calculating coordinates of the fourth feature point on the world coordinate system based on the position of the fourth feature point on the calibration object, the size of the calibration object and the origin of the world coordinate system;
calculating an actual conversion matrix of the fourth feature point between the world coordinate system and the pixel coordinate system;
comparing the internal parameter matrix with the external parameter matrix with the actual conversion matrix to determine correction parameters;
and correcting any pixel in the fourth image according to the correction parameters to obtain a corrected fourth image.
7. The on-line calibration method of an on-vehicle camera according to claim 1, further comprising:
collecting images of adjacent vehicles, determining the distance between the images and the adjacent vehicles based on internal parameters, judging that the distance between the images and the adjacent vehicles is larger than a safety distance, and sending out a warning signal if the distance between the images is smaller than the set safety distance, otherwise, not processing the images;
wherein capturing an image of an adjacent vehicle and determining a distance to the adjacent vehicle based on the calibrated internal parameters comprises:
obtaining a calibrated internal parameter;
Adjusting the origin of the image coordinate system so that the origin of the image coordinate system coincides with the origin of the camera coordinate system;
acquiring images of adjacent vehicles;
based on historical data, constructing a vehicle height recognition model and training to obtain a trained vehicle height recognition model;
identifying adjacent vehicles on the image by using the trained vehicle height identification model to obtain the heights of the adjacent vehicles on a camera coordinate system;
acquiring the height of an adjacent vehicle on a pixel coordinate system;
affine transformation and projective transformation are carried out on the height of the adjacent vehicle on the pixel coordinate system based on affine transformation between the pixel coordinate system and the image coordinate system and projective transformation between the image coordinate system and the camera coordinate system, and the distance between the adjacent vehicle and the camera coordinate system is determined;
and judging that the distance between the vehicle and the adjacent vehicle on the camera coordinate system is greater than the set safety distance, if the distance is less than the set safety distance, sending out a warning signal, otherwise, not processing.
8. An on-line calibration system for a vehicle-mounted camera, comprising:
the coordinate system construction module is used for constructing a pixel coordinate system, an image coordinate system, a camera coordinate system and a world coordinate system;
The motion control module is used for controlling the vehicle-mounted camera to perform translational motion based on road conditions,
the first image acquisition module is used for controlling the vehicle-mounted camera to acquire a first image of the calibration object at a first moment;
the second image acquisition module is used for controlling the vehicle-mounted camera to acquire a second image of the calibration object at a second moment after translational movement;
the third image acquisition module is used for controlling the vehicle-mounted camera to rotate by a certain angle and acquiring a third image of the calibration object at a third moment;
the internal parameter calibration module is used for calibrating the internal parameters of the vehicle-mounted camera based on affine transformation between the pixel coordinate system and the image coordinate system and projection transformation between the image coordinate system and the camera coordinate system;
the external parameter calibration module is used for calibrating external parameters of the vehicle-mounted camera according to the first image, the second image and the third image based on rigid body transformation between the world coordinate system and the camera coordinate system;
based on road conditions, control on-vehicle camera and carry out translational motion, include:
acquiring a road image of the future running of the automobile through a vehicle-mounted camera;
based on the road image of the future running of the automobile, predicting whether the automobile runs straight and stably,
Wherein, based on the road image that the car is going in the future, judge whether the car is straight line steady and travel, include:
extracting two road edge lines based on a road image of the future running of the automobile;
when the distances from the automobile to the two road edge lines are equal, virtually extending the two road edge lines and intersecting the two road edge lines at one point to determine a road vanishing point;
judging whether the road vanishing point is positioned at the front central line of the automobile, if so, judging that the automobile runs straight, otherwise, judging that the automobile runs non-straight and adjusting the direction;
when the automobile is judged to run straight, the pothole detection is carried out on the road, and whether the automobile runs straight stably is judged;
collecting a first motion state of an automobile;
if the automobile runs linearly and stably, controlling the vehicle-mounted camera to perform translational motion in a world coordinate system based on a first motion state of the automobile, otherwise, not processing;
calibrating external parameters of the vehicle-mounted camera by the first image, the second image and the third image based on rigid body transformation between the world coordinate system and the camera coordinate system, wherein the method comprises the following steps:
respectively extracting features of the first image, the second image and the third image to obtain a first feature point, a second feature point and a third feature point;
Determining a translation vector based on the first feature point and the second feature point;
determining a rotation matrix based on the second feature point and the third feature point;
determining an external parameter matrix based on the rotation matrix and the translation vector;
and calibrating external parameters of the vehicle-mounted camera based on matrix elements in the external parameter matrix.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311069501.0A CN117036505B (en) | 2023-08-23 | 2023-08-23 | On-line calibration method and system for vehicle-mounted camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311069501.0A CN117036505B (en) | 2023-08-23 | 2023-08-23 | On-line calibration method and system for vehicle-mounted camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117036505A CN117036505A (en) | 2023-11-10 |
| CN117036505B true CN117036505B (en) | 2024-03-29 |
Family
ID=88624396
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311069501.0A Active CN117036505B (en) | 2023-08-23 | 2023-08-23 | On-line calibration method and system for vehicle-mounted camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117036505B (en) |
Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104019751A (en) * | 2014-06-25 | 2014-09-03 | 吉林大学 | Three-dimensional calibrating target for laser scanning plane of active vision measurement of automobile morphology |
| CN107577996A (en) * | 2017-08-16 | 2018-01-12 | 中国地质大学（武汉） | A kind of recognition methods of vehicle drive path offset and system |
| CN207458130U (en) * | 2017-11-03 | 2018-06-05 | 扬州大学 | The vehicle-mounted camera external parameter fast calibration device of machine vision |
| CN108898638A (en) * | 2018-06-27 | 2018-11-27 | 江苏大学 | A kind of on-line automatic scaling method of vehicle-mounted camera |
| CN110222658A (en) * | 2019-06-11 | 2019-09-10 | 腾讯科技（深圳）有限公司 | The acquisition methods and device of road vanishing point position |
| CN110517202A (en) * | 2019-08-30 | 2019-11-29 | 的卢技术有限公司 | A kind of vehicle body camera calibration method and its caliberating device |
| CN110766763A (en) * | 2019-11-14 | 2020-02-07 | 朱宇扬 | Camera calibration method based on vanishing point operation |
| CN111325800A (en) * | 2018-12-17 | 2020-06-23 | 北京华航无线电测量研究所 | Monocular vision system pitch angle calibration method |
| WO2020151212A1 (en) * | 2019-01-24 | 2020-07-30 | 惠州市德赛西威汽车电子股份有限公司 | Calibration method for extrinsic camera parameter of on-board camera system, and calibration system |
| CN111524192A (en) * | 2020-04-20 | 2020-08-11 | 北京百度网讯科技有限公司 | Calibration method, device and system for external parameters of vehicle-mounted camera and storage medium |
| JP2021027370A (en) * | 2019-07-31 | 2021-02-22 | クラリオン株式会社 | Calibration method of on-vehicle camera and calibration device of the on-vehicle camera |
| CN113066133A (en) * | 2021-04-23 | 2021-07-02 | 同致电子科技（厦门）有限公司 | Vehicle-mounted camera online self-calibration method based on pavement marking geometrical characteristics |
| CN114842093A (en) * | 2022-05-18 | 2022-08-02 | 重庆长安汽车股份有限公司 | Automatic calibration system and calibration method for external parameters of vehicle-mounted monocular camera based on key points |
| CN114998452A (en) * | 2022-08-03 | 2022-09-02 | 深圳安智杰科技有限公司 | On-line calibration method and system for vehicle-mounted camera |
| CN115096346A (en) * | 2022-06-17 | 2022-09-23 | 安徽蔚来智驾科技有限公司 | Sensor parameter calibration method, device, medium and vehicle based on automatic driving |
| CN115278220A (en) * | 2022-07-29 | 2022-11-01 | 重庆长安汽车股份有限公司 | Vehicle-mounted camera simulation test method and system, electronic equipment and storage medium |
| WO2022261825A1 (en) * | 2021-06-15 | 2022-12-22 | 华为技术有限公司 | Calibration method and device for automatic driving vehicle |
| CN115511974A (en) * | 2022-09-29 | 2022-12-23 | 浙江工业大学 | Rapid external reference calibration method for vehicle-mounted binocular camera |
| CN115619875A (en) * | 2022-10-21 | 2023-01-17 | 广州汽车集团股份有限公司 | Vehicle-mounted camera calibration method and device, electronic equipment and storage medium |
| CN116523995A (en) * | 2023-04-25 | 2023-08-01 | 哈尔滨工业大学 | Image-based estimation method for automobile centroid slip angle |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8855848B2 (en) * | 2007-06-05 | 2014-10-07 | GM Global Technology Operations LLC | Radar, lidar and camera enhanced methods for vehicle dynamics estimation |
| US20130141520A1 (en) * | 2011-12-02 | 2013-06-06 | GM Global Technology Operations LLC | Lane tracking system |
| JP6843712B2 (en) * | 2017-07-27 | 2021-03-17 | フォルシアクラリオン・エレクトロニクス株式会社 | In-vehicle processing device |
| KR101969030B1 (en) * | 2017-09-13 | 2019-04-16 | (주)베이다스 | Method and apparatus for providing camera calibration for vehicles |
| CN110570475A (en) * | 2018-06-05 | 2019-12-13 | 上海商汤智能科技有限公司 | vehicle-mounted camera self-calibration method and device and vehicle driving method and device |
-
2023
- 2023-08-23 CN CN202311069501.0A patent/CN117036505B/en active Active
Patent Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104019751A (en) * | 2014-06-25 | 2014-09-03 | 吉林大学 | Three-dimensional calibrating target for laser scanning plane of active vision measurement of automobile morphology |
| CN107577996A (en) * | 2017-08-16 | 2018-01-12 | 中国地质大学（武汉） | A kind of recognition methods of vehicle drive path offset and system |
| CN207458130U (en) * | 2017-11-03 | 2018-06-05 | 扬州大学 | The vehicle-mounted camera external parameter fast calibration device of machine vision |
| CN108898638A (en) * | 2018-06-27 | 2018-11-27 | 江苏大学 | A kind of on-line automatic scaling method of vehicle-mounted camera |
| CN111325800A (en) * | 2018-12-17 | 2020-06-23 | 北京华航无线电测量研究所 | Monocular vision system pitch angle calibration method |
| WO2020151212A1 (en) * | 2019-01-24 | 2020-07-30 | 惠州市德赛西威汽车电子股份有限公司 | Calibration method for extrinsic camera parameter of on-board camera system, and calibration system |
| CN110222658A (en) * | 2019-06-11 | 2019-09-10 | 腾讯科技（深圳）有限公司 | The acquisition methods and device of road vanishing point position |
| JP2021027370A (en) * | 2019-07-31 | 2021-02-22 | クラリオン株式会社 | Calibration method of on-vehicle camera and calibration device of the on-vehicle camera |
| CN110517202A (en) * | 2019-08-30 | 2019-11-29 | 的卢技术有限公司 | A kind of vehicle body camera calibration method and its caliberating device |
| CN110766763A (en) * | 2019-11-14 | 2020-02-07 | 朱宇扬 | Camera calibration method based on vanishing point operation |
| CN111524192A (en) * | 2020-04-20 | 2020-08-11 | 北京百度网讯科技有限公司 | Calibration method, device and system for external parameters of vehicle-mounted camera and storage medium |
| CN113066133A (en) * | 2021-04-23 | 2021-07-02 | 同致电子科技（厦门）有限公司 | Vehicle-mounted camera online self-calibration method based on pavement marking geometrical characteristics |
| WO2022261825A1 (en) * | 2021-06-15 | 2022-12-22 | 华为技术有限公司 | Calibration method and device for automatic driving vehicle |
| CN114842093A (en) * | 2022-05-18 | 2022-08-02 | 重庆长安汽车股份有限公司 | Automatic calibration system and calibration method for external parameters of vehicle-mounted monocular camera based on key points |
| CN115096346A (en) * | 2022-06-17 | 2022-09-23 | 安徽蔚来智驾科技有限公司 | Sensor parameter calibration method, device, medium and vehicle based on automatic driving |
| CN115278220A (en) * | 2022-07-29 | 2022-11-01 | 重庆长安汽车股份有限公司 | Vehicle-mounted camera simulation test method and system, electronic equipment and storage medium |
| CN114998452A (en) * | 2022-08-03 | 2022-09-02 | 深圳安智杰科技有限公司 | On-line calibration method and system for vehicle-mounted camera |
| CN115511974A (en) * | 2022-09-29 | 2022-12-23 | 浙江工业大学 | Rapid external reference calibration method for vehicle-mounted binocular camera |
| CN115619875A (en) * | 2022-10-21 | 2023-01-17 | 广州汽车集团股份有限公司 | Vehicle-mounted camera calibration method and device, electronic equipment and storage medium |
| CN116523995A (en) * | 2023-04-25 | 2023-08-01 | 哈尔滨工业大学 | Image-based estimation method for automobile centroid slip angle |
Non-Patent Citations (2)
| Title |
|---|
| 一种开放道路车载双目相机的外参标定方法;刘福明;计算机与多媒体技术;20211215;108-112 * |
| 车载相机自动标定和快速标定技术研究;祁淼等;机械设计与制造;20201208;221-223+227 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117036505A (en) | 2023-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109074069B (en) | Autonomous vehicle with improved visual detection capability | |
| CN102201056B (en) | Vehicle environment recognizing apparatus and method | |
| CN106462968B (en) | Method and device for calibrating a camera system of a motor vehicle | |
| CN110745140B (en) | Vehicle lane change early warning method based on continuous image constraint pose estimation | |
| CN107424116A (en) | Position detecting method of parking based on side ring depending on camera | |
| CN106503636A (en) | A kind of road sighting distance detection method of view-based access control model image and device | |
| US20200250984A1 (en) | Pothole detection system | |
| CN104129389A (en) | Method for effectively judging and recognizing vehicle travelling conditions and device thereof | |
| CN111381248B (en) | Obstacle detection method and system considering vehicle bump | |
| CN108470142B (en) | Lane positioning method based on inverse perspective projection and lane distance constraint | |
| US10363940B2 (en) | Vehicle position attitude-angle estimation device and vehicle position attitude-angle estimation method | |
| CN103196418A (en) | Measuring method of vehicle distance at curves | |
| JP4670528B2 (en) | Imaging device deviation detection method, imaging device deviation correction method, and imaging device | |
| CN107229908A (en) | A kind of method for detecting lane lines | |
| US20040061712A1 (en) | Stereoscopic image processing apparatus and the method of processing stereoscopic images | |
| CN109753841B (en) | Lane line identification method and device | |
| CN112606649B (en) | Vehicle and vehicle body balance control method and control system thereof | |
| CN107796373A (en) | A kind of distance-finding method of the front vehicles monocular vision based on track plane geometry model-driven | |
| CN107284455A (en) | A kind of ADAS systems based on image procossing | |
| CN117036505B (en) | On-line calibration method and system for vehicle-mounted camera | |
| CN106885523B (en) | A kind of vehicle route tracking error vision measurement optimization method | |
| CN116311136A (en) | Lane line parameter calculation method for driving assistance | |
| US11477371B2 (en) | Partial image generating device, storage medium storing computer program for partial image generation and partial image generating method | |
| CN115144849A (en) | Sensor fusion for object avoidance detection | |
| JP2023022415A (en) | Object recognition apparatus, computer program and recording medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |