CN111862236A - Fixed-focus binocular camera self-calibration method and system - Google Patents
Fixed-focus binocular camera self-calibration method and system Download PDFInfo
- Publication number
- CN111862236A CN111862236A CN202010713282.5A CN202010713282A CN111862236A CN 111862236 A CN111862236 A CN 111862236A CN 202010713282 A CN202010713282 A CN 202010713282A CN 111862236 A CN111862236 A CN 111862236A
- Authority
- CN
- China
- Prior art keywords
- binocular camera
- self
- deviation
- focus
- fixed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000012937 correction Methods 0.000 claims abstract description 56
- 230000003068 static effect Effects 0.000 claims abstract description 56
- 238000012545 processing Methods 0.000 claims description 10
- 238000003384 imaging method Methods 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 3
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 3
- 238000005096 rolling process Methods 0.000 claims description 2
- 238000003702 image correction Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000013519 translation Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/005—Tree description, e.g. octree, quadtree
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Processing (AREA)
Abstract
The invention provides a self-calibration method and a self-calibration system for a fixed-focus binocular camera, which comprise the following steps: 1) acquiring left and right original images; 2) correcting the left and right original images; 3) extracting characteristic points from the left and right correction images and matching; 4) counting the average value of the vertical coordinate deviation of the left image and the right image, if the average value is larger than a first threshold value, correcting and estimating a first parameter group, comparing again after calibration, and repeatedly performing iterative correction until the average value is smaller than the first threshold value; 5) finding a static object; 6) tracking the parallax and wheel motion information of the static object when the vehicle is in a moving state; 7) and obtaining the distance deviation between the wheel movement distance and the three-dimensional distance change value of the static object, if the distance deviation is greater than a second threshold value, correcting and estimating a second parameter group, recalculating, repeatedly performing iterative correction until the distance deviation is less than the second threshold value, and finishing self-calibration. The invention utilizes real-time image tracking and vehicle body motion information to carry out optimized calibration on external parameters, completes image correction work and provides accurate three-dimensional identification data for the vehicle body.
Description
Technical Field
The invention relates to the field of image processing, in particular to a self-calibration method and a self-calibration system for a fixed-focus binocular camera.
Background
The binocular sensor can be used as a stereo camera applicable to outdoor scenes and can provide a stereo obstacle detection function for the robot, the binocular camera can calculate three-dimensional position information of obstacles in a visual field through image analysis and identification of the advancing direction of the robot, and guarantee is provided for safe driving of the robot. However, the binocular camera needs to be subjected to a strict calibration process before being used as a stereo camera.
The SLAM-Based Self-Calibration of a Binocular Stereo Vision Rig in real-Time article provides a method for solving Binocular external parameters Based on nonlinear optimization of Slam, but excessive variables may cause the nonlinear optimization to fail to converge.
The CN 109313813 a patent proposes a method for correcting binocular external parameters by using vehicle body motion information, but only considering the deviation of yaw angle, the actual situation is often accompanied by various errors when the position of the binocular camera changes due to the external environment.
Therefore, how to provide a method for performing optimization calibration on external parameters based on real-time data for a fixed-focus binocular camera so that the fixed-focus binocular camera can provide accurate three-dimensional identification data has become one of the problems to be solved by the technical staff in the field.
Disclosure of Invention
In view of the above disadvantages of the prior art, an object of the present invention is to provide a self-calibration method and system for a fixed-focus binocular camera, which are used to solve the problems in the prior art that external parameters change due to environmental influences during use, and non-linear optimization cannot converge.
In order to achieve the above and other related objects, the present invention provides a self-calibration method for a fixed-focus binocular camera, the self-calibration method at least comprising:
1) acquiring left and right original images from two image acquisition units of a binocular camera respectively;
2) constructing a binocular extrinsic parameter matrix according to the current first parameter set and the second parameter set, and correcting the left and right original images to obtain left and right corrected images; wherein the first parameter group affects the parallax in the vertical direction, and the second parameter group affects the parallax in the horizontal direction;
3) respectively extracting characteristic points from the left and right correction images, matching the characteristic points in the left and right correction images to obtain matched characteristic point pairs, and filtering the characteristic points which are mismatched;
4) counting the vertical coordinate deviation of the left image and the right image of each feature point pair, if the average value of the vertical coordinate deviation is larger than a first threshold value, performing correction estimation on at least one parameter in the first parameter group, comparing the corrected value with the first threshold value again after recalibration, repeating iterative correction until the average value of the vertical coordinate deviation is smaller than the first threshold value, and updating the first parameter group;
5) classifying objects in the scene to find static objects in the scene;
6) when the binocular camera is in a moving state, calculating the parallax of a static object based on the characteristic point pairs in the range of the static object, and tracking the parallax and wheel motion information of the static object;
7) obtaining a wheel motion distance based on the wheel motion information, obtaining a three-dimensional distance change value of the static object based on the parallax of the static object, comparing the wheel motion distance with the three-dimensional distance change value of the static object to obtain a distance deviation, if the distance deviation is greater than a second threshold value, carrying out correction estimation on the second parameter group, recalculating the three-dimensional distance of the static object based on the calibrated image, repeatedly carrying out iterative correction until the distance deviation is less than the second threshold value, updating the second parameter group, and completing self-calibration of parameters of the binocular camera.
Optionally, the parameters in the first parameter group include pitch angle deviation, roll angle deviation, altitude deviation, and front-back deviation of the binocular camera.
More optionally, the parameters in the second parameter set include a yaw angle deviation and a baseline length of the binocular camera.
More optionally, the average value of the ordinate deviations in step 4) satisfies the following relation:
wherein, VErr is the average value of the vertical coordinate deviation value, wkAs a weight of the kth pair of characteristic point pairs, VLkFor the ordinate, VR, of the k-th pair of feature point pairs in the left imagekIs the ordinate of the k-th pair of feature point pairs in the right image, N is the number of feature point pairs, UCk,VCk,UDk,VDkAs an intermediate variable, ULkFor the k-th pair of feature point pair, the abscissa, UR, in the left imagekAnd f, the transverse coordinates of the kth pair of characteristic points in the right image are shown, delta p is the pitch angle deviation of the binocular camera, delta r is the rolling angle deviation of the binocular camera, delta H is the height deviation of the binocular camera, delta D is the front-back deviation of the binocular camera, b is the base line length of the binocular camera, and f is the lens focal length of the binocular camera.
More optionally, the method for solving each parameter in the first parameter group is replaced by a matrix operation or a nonlinear optimization mode.
More optionally, the weight w of the kth pair of feature point pairskAs a default value or weight w of the kth pair of characteristic point pairskThe following relationship is satisfied:
where ResU is the number of pixels per row of the left and right images and ResV is the number of pixels per column of the left and right images.
More optionally, when the parameters in the first parameter group are corrected in step 3), the corrected parameters are obtained by multiplying the correction amount by a first coefficient, wherein the first coefficient is greater than 0 and equal to or less than 1.
Optionally, the feature points are extracted in step 3) based on a quadtree extraction strategy, so that the feature points are uniformly distributed in the corrected left and right images.
Optionally, semantic recognition is used in step 5) to find the static object.
Optionally, in step 7), the three-dimensional distance change value of the static object satisfies the following relation:
wherein Δ Z is a three-dimensional distance change value set of the static object, b is a baseline length of the binocular camera, f is a lens focal length of the binocular camera, D1 and D2 are parallax sets of the static object at the time t1 and the time t2, respectively, Δ D is a parallax error, and Δ y is a yaw angle error of the binocular camera.
More optionally, the distance deviation satisfies the following relation:
wherein, Δ b is correction amount of base line length of binocular camera, Δ M is the wheel movement distance, D1i、D2iThe parallax values of the ith static object at the time t1 and the time t2, respectively.
More optionally, when the parameters in the second parameter group are corrected in step 7), the corrected parameters are obtained by multiplying the correction amount by a second coefficient, wherein the second coefficient is greater than 0 and equal to or less than 1.
Optionally, the binocular cameras are oriented in the same direction at the starting time tracked in step 7) and the final time tracked.
Optionally, the self-calibration method of the fixed-focus binocular camera further includes 8) calculating three-dimensional information in the view environment according to the corrected second parameter set.
In order to achieve the above and other related objects, the present invention provides a fixed-focus binocular camera self-calibration system, which at least comprises:
the system comprises a mobile platform, a binocular camera, an image processing unit and a mobile platform control unit;
the binocular camera is arranged on the mobile platform and used for acquiring left and right images;
the image processing unit is arranged on the mobile platform, is connected with the binocular camera and the mobile platform control unit, and executes the self-calibration method of the fixed-focus binocular camera to realize self-calibration;
the mobile platform control unit is arranged on the mobile platform, is connected with the wheels of the mobile platform, and controls the wheels to rotate and obtain the movement distance of the wheels.
Optionally, the two image acquisition units of the binocular camera have fixed-focus lenses with synchronous triggering relationship, same resolution and same focal length, and imaging planes are on the same plane.
Optionally, the wheel of the mobile platform includes a wheel, a motor and an encoder, the motor drives the wheel to rotate, and the encoder is used for recording the rotation angle and the number of turns of the wheel.
As described above, the self-calibration method and system of the fixed-focus binocular camera of the invention have the following beneficial effects:
the self-calibration method and the self-calibration system of the fixed-focus binocular camera do not depend on a calibration tool prepared in advance, parameters including rotation and translation deviation of external parameters are optimized and calibrated by utilizing real-time image tracking and vehicle body motion information, and related image correction work is completed, so that the fixed-focus binocular camera can provide accurate three-dimensional identification data for a vehicle body.
Drawings
Fig. 1 is a schematic flow chart of the self-calibration method of the fixed-focus binocular camera according to the present invention.
FIG. 2 is a schematic diagram of the center coordinate system of the corrected image according to the present invention.
FIG. 3 is a schematic diagram of the moving direction and coordinate system at time t1 and time t2 according to the present invention.
Fig. 4 is a schematic structural diagram of the fixed-focus binocular camera self-calibration system of the present invention.
Description of the element reference numerals
101-102 first-second image acquisition units
110 image processing unit
201 to 204 first to fourth wheels
210 mobile platform control unit
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention.
Please refer to fig. 1 to 4. It should be noted that the drawings provided in the present embodiment are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
Example one
As shown in fig. 1, the present embodiment provides a self-calibration method for a fixed-focus binocular camera, where the self-calibration method for a fixed-focus binocular camera includes:
1) and respectively acquiring left and right original images from two image acquisition units of the binocular camera.
Specifically, the binocular camera comprises two image acquisition units (lenses), the two image acquisition units have a hardware triggering synchronization relationship and are formed by imaging elements with the same resolution and fixed-focus lenses with the same focal length, imaging planes of the two image acquisition units are located on the same plane, the centers of images are located on the same horizontal line, and the distance between the centers of the images is the length of a base line of the binocular camera. And acquiring a left original image and a right original image based on the binocular camera.
2) Constructing a binocular extrinsic parameter matrix according to the current first parameter set and the second parameter set, and correcting the left and right original images to obtain left and right corrected images; the first parameter set mainly affects the parallax in the vertical direction, and the second parameter set mainly affects the parallax in the horizontal direction.
Specifically, assuming that the binocular cameras are calibrated in the factory and the left and right imaging elements are horizontally arranged on the same plane in the structural design, the deformation caused by structural or environmental influences does not cause a large physical error. The internal references of the binocular camera include a lens focal length f (known amount), left and right imaging element center positions (cx1, cy1), and (cx2, cy 2). The external parameters of the binocular camera include a pitch angle deviation Δ p (Δ pitch), a yaw angle deviation Δ y (Δ yaw), a roll angle deviation Δ r (Δ roll), and a base line length b (baseline), a height deviation Δ H, and a front-rear deviation Δ D.
In this embodiment, the parameters are grouped according to whether they affect horizontal parallax matching operation (mainly affect vertical parallax), and the parameters affecting the horizontal parallax matching operation include the vertical coordinate cy of the center positions of the left and right imaging elements, the pitch angle deviation Δ p of the binocular camera, the roll angle deviation Δ r of the binocular camera, the height deviation Δ H of the binocular camera, and the front-back deviation Δ D of the binocular camera, which are denoted as a first parameter group a; parameters that do not affect the horizontal parallax matching operation include the lens focal length f of the binocular camera, the abscissa cx of the center position of the left and right imaging elements, the yaw angle deviation Δ y of the binocular camera (normally, the vertical parallax caused by the trapezoidal distortion can be ignored when the angle of the yaw angle deviation Δ y of the binocular camera is less than 5 degrees), and the base length B of the binocular camera, which are recorded as the second parameter group B. When the yaw angle deviation delta y and the pitch angle deviation delta p of the binocular camera are not more than 5 degrees, the influence of the deviation on the image is similar to the horizontal and vertical coordinates cx and cy of the central position of the element, in the embodiment, the horizontal and vertical coordinates cx and cy of the central position of the element are defaulted to the central position of the image, and the central point deviation of the binocular camera is corrected through the yaw angle deviation delta y and the pitch angle deviation delta p of the binocular camera; in addition, the focal length f of the lens of the binocular camera is a known quantity, and correction is not needed; the finally obtained parameters of the first parameter group a and the second parameter group B are as follows:
the first parameter set a ═ { Δ p, Δ r, Δ H, Δ D }
Second parameter set B ═ { Δ y, B }
The deviation of the parameters in the first parameter group A causes the deviation of binocular images in the vertical direction, the depth matching operation process is influenced, the parameters of the first parameter group A are corrected, and the matched pixels of the left image and the right image can be on the same horizontal line, so that the calculation amount of the parallax calculation step is greatly reduced; the deviation of the parameters in the second parameter group B can cause the binocular image to generate horizontal direction deviation, namely parallax deviation or depth deviation, and the parameters of the second parameter group B can be corrected to obtain an accurate three-dimensional data result through parallax.
In practical use, relevant internal parameters or external parameters may be added to the first parameter group a or the second parameter group B according to needs, and the present embodiment is not limited thereto.
Specifically, under the condition that the inside and outside parameters of the binocular camera are known, the left and right original images are corrected by using a standard correction algorithm, the standard correction algorithm includes, but is not limited to, a stereoRectify function in an opencv algorithm library, and matlab also has such a function, which is not described herein again. After correction, values of a pitch angle deviation delta p, a roll angle deviation delta r, a height deviation delta H and a front-back deviation delta D (all deviation parameters) of the binocular camera in the first parameter group A are all 0 (or within a threshold range close to 0); the value of the yaw angle deviation Δ y (deviation parameter) of the binocular camera in the second parameter group B is 0 (or within a threshold range close to 0); in practical application, the base length b of the binocular camera is updated to b + Δ b, Δ b is a correction amount of the base length of the binocular camera, the correction amount can be obtained through nonlinear optimization calculation, and the base length b of the binocular camera after theoretical update meets the following requirements: sqrt (b + Δ H '. DELTA.H' + Δ D '. DELTA.D'), where Δ H 'is a correction amount of the height deviation Δ H and Δ D' is a correction amount of the front-rear deviation Δ D.
3) And respectively extracting characteristic points from the left and right correction images, matching the characteristic points in the left and right correction images to obtain matched characteristic point pairs, and filtering the characteristic points which are mismatched.
Specifically, as an example, in the present embodiment, the feature points are extracted based on a quadtree extraction strategy so that the respective feature points are uniformly distributed in the corrected left and right images; any method of extracting feature points in actual use is applicable to the present invention, and is not limited to the present embodiment. After extracting the characteristic points, matching the characteristic points in the left and right correction images based on a matching algorithm to obtain characteristic point pairs; and filter the mismatched feature points.
4) And counting the vertical coordinate deviation of the left image and the right image of each characteristic point pair, if the average value of the vertical coordinate deviation is greater than a first threshold value, performing correction estimation on at least one parameter in the first parameter group, comparing the corrected value with the first threshold value again after recalibration, repeating iterative correction until the average value of the vertical coordinate deviation is less than the first threshold value, and updating the first parameter group.
Specifically, the image resolution is noted as ResU x ResV, where ResU is the number of pixels per row and ResV is the number of pixels per column. The pixel coordinates of the feature point pairs in the left and right diagrams are (UL, VL) and (UR, VR), the coordinate system is as shown in fig. 2, the origin is the image center position, the U axis is rightward, the V axis is downward, and the number of feature pairs is N. The average value of the ordinate deviation satisfies the following relational expression:
wherein, the VErr is the average value of the vertical coordinate deviation value; w is akThe weight of the k-th pair of feature point pairs is 1 by default in this example, and the weight w can be set as required in practical usekValue of (1), weight w of each feature point pairkMay be the same value or different values; VLkThe ordinate of the k-th pair of feature point pairs in the left image; VR (virtual reality)kThe ordinate of the kth pair of characteristic points in the right image; n is the number of pairs of characteristic points.
As an implementation manner of the present invention, in the present embodiment, before correcting the first parameter group a, the following intermediate variables are set:
from the geometric relationship, the solving equation of each parameter in the first parameter group a can be derived as follows:
wherein, UCk,VCk,UDk,VDkAs an intermediate variable, ULkFor the k-th pair of feature point pair, the abscissa, UR, in the left imagekIs the abscissa of the k-th pair of feature points in the right image. The base length b of the binocular camera and the lens focal length f of the binocular camera can adopt default values or factory set values, the vertical deviation can be corrected through repeated iterative correction, and the correction effect is not influenced due to inaccurate values. And (3) carrying out translation and/or rotation operation on the whole image based on the correction quantity estimated by the pitch angle deviation delta p, the roll angle deviation delta r, the height deviation delta H and the front-back deviation delta D of the binocular camera, so that the vertical coordinates of all the characteristic points in the left image and the right image are basically consistent (are positioned in the same row), namely the average value VErr of the vertical coordinate deviation values approaches to 0 (is smaller than a first threshold).
It should be noted that, in the actual correction, only one or more parameters having a serious influence may be corrected according to the actual situation, including but not limited to correcting only the pitch angle deviation Δ p and the roll angle deviation Δ r of the binocular camera. The first threshold may be set to a specific value based on actual needs, and is not limited to this embodiment.
As another implementation manner of the present invention, since the above formula has an approximate relationship, and in an actual situation, a plurality of errors are mixed together by the feature point, the first parameter group a may also be solved by a matrix operation or a nonlinear optimization, which is not repeated herein.
As another implementation manner of the present invention, in actual operation, multiple deviations are mixed together, the result of calculating the deviation by the formula is often not accurate enough, and if the deviation is corrected directly according to the formula, the result may cause an excessive correction (overshoot), this example adopts a method similar to PID control, and each correction amount of the first parameter group a is multiplied by a first coefficient and then added to the corresponding current external reference result to obtain a corrected parameter, and by means of repeated iterative correction, a large error is corrected to a small error, and then the small error is gradually reduced, and finally the correction process is completed, so as to obtain an accurate parameter of the first parameter group a, where the first coefficient is greater than 0 and less than or equal to 1.
As another implementation mode of the invention, the weight of each characteristic point pairWeight wkSet according to the distance of the feature point position from the center of the image, as an example, the following relation is satisfied:
5) and classifying the objects in the scene to find out the static objects in the scene.
Specifically, in the present embodiment, objects in a scene are classified by using a semantic recognition algorithm, and objects in a static state in the scene, including but not limited to buildings, roads, and road signs, are found, which is not listed here.
6) And when the binocular camera is in a moving state, calculating the parallax of the static object based on the characteristic point pairs in the range of the static object, and tracking the parallax and wheel motion information of the static object.
Specifically, when the binocular camera is in a moving state, feature point pairs in a range of the static object are obtained through screening, and the parallax of the static object is calculated based on the feature point pairs. When the yaw angle deviation Δ y of the binocular camera is small (the general angle is less than 5 degrees), the same amount of parallax deviation Δ d is caused to the whole parallax of the whole image, and the relationship between Δ y and Δ d is as follows:
specifically, the three-dimensional information of the static object is tracked, and the wheel motion information in the tracking process is recorded to calculate a wheel odometer.
7) Obtaining a wheel motion distance based on the wheel motion information, obtaining a three-dimensional distance change value of the static object based on the parallax of the static object, comparing the wheel motion distance with the three-dimensional distance change value of the static object to obtain a distance deviation, if the distance deviation is greater than a second threshold value, carrying out correction estimation on the second parameter group, recalculating the three-dimensional distance of the static object based on the calibrated image, repeatedly carrying out iterative correction until the distance deviation is less than the second threshold value, updating the second parameter group, and completing self-calibration of parameters of the binocular camera.
As shown in fig. 3, it is assumed that the vehicle body drives the binocular camera to drive in the time period from t1 to t2, and the directions calculated by the wheel odometer at the time t1 and the time t2 are substantially consistent, that is, the driving path between the time t1 and the time t2 may not keep an absolute straight line, and the directions at the time t1 and the time t2 are consistent, for example, in the vehicle lane change process, the directions before and after lane change are consistent, and the wheel movement distance of the vehicle body in the forward direction is Δ M obtained by the wheel odometer. In this embodiment, a coordinate system is established with the optical center of the left image capturing unit as the origin, the X axis is directed to the right side of the vehicle body, and the Z axis is directed to the front of the vehicle body, so that after the vehicle body travels forward by Δ M distance, the moving distance of the static object in the Z direction should be close to the wheel movement distance Δ M. The feature point set of the static object is tracked in this time period, the pixel coordinates of the left image (or the right image) corresponding to the time t1 and the time t2 are (U1, V1) and (U2, V2), the parallax sets are D1 and D2, and the solving equation of the three-dimensional distance change value set Δ Z of the static object at the time t1 and the time t2 in the static feature set is as follows:
since each change value in the three-dimensional distance change value collection Δ Z of the static object is close to the wheel movement distance Δ M, the following nonlinear equation can be derived to solve the second parameter group B:
solving the delta b and the delta D when the right equation has the minimum sub value based on the formula, wherein the delta b is the correction quantity of the base line length of the binocular camera, D1i、D2iThe parallax values of the ith static object at the time t1 and the time t2, respectively. Iteratively correcting the second parameter group B repeatedly through multiple groups of data, finishing parameter correction when the error of the nonlinear equation is smaller than a second threshold value, and updating with new parametersAnd newly updating the second parameter group B to finish the self-calibration of the binocular camera. The second threshold may be set to a specific value based on actual needs, and is not limited to this embodiment.
In the present embodiment, the wheel movement distance Δ M of the vehicle body in the forward direction and the set Δ Z of three-dimensional distance changes of the static object are compared to correct the second parameter set. In practical use, the comparison can be performed based on any moving direction of the vehicle body, and the present embodiment is not limited thereto.
In another embodiment of the present invention, in order to prevent an excessive correction (overshoot), when the parameters in the second parameter group B are corrected, each correction amount in the second parameter group B is multiplied by a second coefficient, which is greater than 0 and equal to or less than 1, and then added to the corresponding current external parameter result to obtain a corrected parameter, and the correction process is finally completed by repeating iterative correction.
As another implementation manner of the present invention, the self-calibration method of the fixed-focus binocular camera further includes: 8) and calculating and outputting three-dimensional information in the visual field environment according to the corrected first parameter group and the second parameter group.
Example two
As shown in fig. 4, the present embodiment provides a fixed-focus binocular camera self-calibration system, which includes:
a mobile platform, a binocular camera, an image processing unit 110, and a mobile platform control unit 210.
As shown in fig. 4, the mobile platform includes, but is not limited to, a mobile robot platform and a vehicle body, which are not listed here.
Specifically, the moving platform includes a bearing platform and wheels, and in this embodiment, includes four wheels, which are a first wheel 201, a second wheel 202, a third wheel 203, and a fourth wheel 204. Each wheel comprises a wheel, a motor and an encoder, and the motor drives the wheel to rotate; the encoders are used for recording the rotation angle and the number of turns of the wheels, reading back the values of the encoders in real time, and calculating the displacement change of the mobile platform according to the values of the encoders.
As shown in fig. 4, the binocular camera is disposed on the mobile platform for acquiring left and right images.
Specifically, the binocular camera includes two image acquisition units, a first image acquisition unit 101 and a second image acquisition unit 102, the first image acquisition unit 101 and the second image acquisition unit 102 are disposed and face the front end of the mobile platform, the first image acquisition unit 101 and the second image acquisition unit 102 have a fixed focus lens with a synchronous triggering relationship, the same resolution and the same focal length, and imaging planes are located on the same plane.
As shown in fig. 4, the image processing unit 110 is disposed on the mobile platform, and is connected to the binocular camera and the mobile platform control unit 210, so as to implement the self-calibration method of the fixed-focus binocular camera according to the first embodiment, thereby implementing self-calibration.
Specifically, the image processing unit 110 obtains real-time image data from the first image capturing unit 101 and the second image capturing unit 102, and performs data processing; simultaneously obtaining wheel mileage from the mobile platform control unit 210; and then, internal and external parameters are optimized and calibrated through real-time image tracking and vehicle body motion information, and relevant image correction work is completed, so that the binocular camera can provide accurate three-dimensional identification data for the mobile platform.
It should be noted that, the self-calibration method is described in the first embodiment, and is not described herein again.
As shown in fig. 4, the moving platform control unit 210 is disposed on the moving platform, and is connected to the wheels of the moving platform to control the wheels to rotate and obtain the movement distance of the wheels.
In summary, the present invention provides a self-calibration method and system for a fixed-focus binocular camera, including: 1) acquiring left and right original images from two image acquisition units of a binocular camera respectively; 2) constructing a binocular extrinsic parameter matrix according to the current first parameter set and the second parameter set, and correcting the left and right original images to obtain left and right corrected images; wherein the first parameter group affects the parallax in the vertical direction, and the second parameter group affects the parallax in the horizontal direction; 3) respectively extracting characteristic points from the left and right correction images, matching the characteristic points in the left and right correction images to obtain matched characteristic point pairs, and filtering the characteristic points which are mismatched; 4) counting the vertical coordinate deviation of the left image and the right image of each feature point pair, if the average value of the vertical coordinate deviation is larger than a first threshold value, performing correction estimation on at least one parameter in the first parameter group, comparing the corrected value with the first threshold value again after recalibration, repeating iterative correction until the average value of the vertical coordinate deviation is smaller than the first threshold value, and updating the first parameter group; 5) classifying objects in the scene to find static objects in the scene; 6) when the binocular camera is in a moving state, calculating the parallax of a static object based on the characteristic point pairs in the range of the static object, and tracking the parallax and wheel motion information of the static object; 7) obtaining a wheel motion distance based on the wheel motion information, obtaining a three-dimensional distance change value of the static object based on the parallax of the static object, comparing the wheel motion distance with the three-dimensional distance change value of the static object to obtain a distance deviation, if the distance deviation is greater than a second threshold value, carrying out correction estimation on the second parameter group, recalculating the three-dimensional distance of the static object based on the calibrated image, repeatedly carrying out iterative correction until the distance deviation is less than the second threshold value, updating the second parameter group, and completing self-calibration of parameters of the binocular camera. The self-calibration method and the self-calibration system of the fixed-focus binocular camera do not depend on a calibration tool prepared in advance, utilize real-time image tracking and vehicle body motion information, perform an optimized calibration method on parameters including rotation and translation deviation of external parameters, and complete related image correction work, so that the binocular camera can provide accurate three-dimensional identification data for a vehicle body. Therefore, the invention effectively overcomes various defects in the prior art and has high industrial utilization value.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (17)
1. A self-calibration method for a fixed-focus binocular camera is characterized by at least comprising the following steps:
1) acquiring left and right original images from two image acquisition units of a binocular camera respectively;
2) constructing a binocular extrinsic parameter matrix according to the current first parameter set and the second parameter set, and correcting the left and right original images to obtain left and right corrected images; wherein the first parameter group affects the parallax in the vertical direction, and the second parameter group affects the parallax in the horizontal direction;
3) respectively extracting characteristic points from the left and right correction images, matching the characteristic points in the left and right correction images to obtain matched characteristic point pairs, and filtering the characteristic points which are mismatched;
4) counting the vertical coordinate deviation of the left image and the right image of each feature point pair, if the average value of the vertical coordinate deviation is larger than a first threshold value, performing correction estimation on at least one parameter in the first parameter group, comparing the corrected value with the first threshold value again after recalibration, repeating iterative correction until the average value of the vertical coordinate deviation is smaller than the first threshold value, and updating the first parameter group;
5) classifying objects in the scene to find static objects in the scene;
6) when the binocular camera is in a moving state, calculating the parallax of a static object based on the characteristic point pairs in the range of the static object, and tracking the parallax and wheel motion information of the static object;
7) obtaining a wheel motion distance based on the wheel motion information, obtaining a three-dimensional distance change value of the static object based on the parallax of the static object, comparing the wheel motion distance with the three-dimensional distance change value of the static object to obtain a distance deviation, if the distance deviation is greater than a second threshold value, carrying out correction estimation on the second parameter group, recalculating the three-dimensional distance of the static object based on the calibrated image, repeatedly carrying out iterative correction until the distance deviation is less than the second threshold value, updating the second parameter group, and completing self-calibration of parameters of the binocular camera.
2. The self-calibration method of the fixed-focus binocular camera according to claim 1, wherein: the parameters in the first parameter group comprise pitch angle deviation, roll angle deviation, height deviation and front-back deviation of the binocular camera.
3. The self-calibration method of the fixed-focus binocular camera according to claim 2, wherein: the parameters in the second parameter group comprise the yaw angle deviation and the base line length of the binocular camera.
4. The self-calibration method of the fixed-focus binocular camera according to claim 3, wherein: the average value of the vertical coordinate deviation in the step 4) satisfies the following relational expression:
wherein, VErr is the average value of the vertical coordinate deviation value, wkAs a weight of the kth pair of characteristic point pairs, VLkFor the ordinate, VR, of the k-th pair of feature point pairs in the left imagekIs the ordinate of the k-th pair of feature point pairs in the right image, N is the number of feature point pairs, UCk,VCk,UDk,VDkAs an intermediate variable, ULkFor the k-th pair of feature point pair, the abscissa, UR, in the left imagekAnd f, the transverse coordinates of the kth pair of characteristic points in the right image are shown, delta p is the pitch angle deviation of the binocular camera, delta r is the rolling angle deviation of the binocular camera, delta H is the height deviation of the binocular camera, delta D is the front-back deviation of the binocular camera, b is the base line length of the binocular camera, and f is the lens focal length of the binocular camera.
5. The self-calibration method of the fixed-focus binocular camera according to claim 4, wherein: and replacing the method for solving each parameter in the first parameter group by a matrix operation or nonlinear optimization mode.
6. The self-calibration method of the fixed-focus binocular camera according to claim 4, wherein: weight w of k-th pair of feature point pairskAs a default value or weight w of the kth pair of characteristic point pairskThe following relationship is satisfied:
where ResU is the number of pixels per row of the left and right images and ResV is the number of pixels per column of the left and right images.
7. The self-calibration method of the fixed-focus binocular camera according to any one of claims 4 to 6, wherein: and 3) when the parameters in the first parameter group are corrected, multiplying the correction quantity by a first coefficient and then correcting to obtain the corrected parameters, wherein the first coefficient is more than 0 and less than or equal to 1.
8. The self-calibration method of the fixed-focus binocular camera according to claim 1, wherein: and 3) extracting the feature points based on a quadtree extraction strategy in the step 3), so that the feature points are uniformly distributed in the corrected left and right images.
9. The self-calibration method of the fixed-focus binocular camera according to claim 1, wherein: and 5) finding the static object by adopting semantic recognition.
10. The self-calibration method of the fixed-focus binocular camera according to claim 1, wherein: in step 7), the three-dimensional distance change value of the static object satisfies the following relation:
wherein Δ Z is a three-dimensional distance change value set of the static object, b is a baseline length of the binocular camera, f is a lens focal length of the binocular camera, D1 and D2 are parallax sets of the static object at the time t1 and the time t2, respectively, Δ D is a parallax error, and Δ y is a yaw angle error of the binocular camera.
11. The self-calibration method of the fixed-focus binocular camera according to claim 10, wherein: the distance deviation satisfies the following relation:
wherein, Δ b is correction amount of base line length of binocular camera, Δ M is the wheel movement distance, D1i、D2iThe parallax values of the ith static object at the time t1 and the time t2, respectively.
12. The self-calibration method of the fixed-focus binocular camera according to any one of claims 10 to 11, wherein: and 7) when the parameters in the second parameter group are corrected, multiplying the correction quantity by a second coefficient and then correcting to obtain the corrected parameters, wherein the second coefficient is more than 0 and less than or equal to 1.
13. The self-calibration method of the fixed-focus binocular camera according to claim 1, wherein: the starting time tracked in the step 7) and the final time tracked are consistent in orientation of the binocular cameras.
14. The self-calibration method of the fixed-focus binocular camera according to claim 1, wherein: the self-calibration method of the fixed-focus binocular camera further comprises 8) calculating three-dimensional information in a visual field environment according to the corrected first parameter group and the second parameter group.
15. The utility model provides a focus binocular camera is from calibration system which characterized in that, focus binocular camera includes at least from calibration system:
the system comprises a mobile platform, a binocular camera, an image processing unit and a mobile platform control unit;
the binocular camera is arranged on the mobile platform and used for acquiring left and right images;
the image processing unit is arranged on the mobile platform, is connected with the binocular camera and the mobile platform control unit, and executes the self-calibration method of the fixed-focus binocular camera according to any one of claims 1-14 to realize self-calibration;
the mobile platform control unit is arranged on the mobile platform, is connected with the wheels of the mobile platform, and controls the wheels to rotate and obtain the movement distance of the wheels.
16. The fixed-focus binocular camera self-calibration system of claim 15, wherein: the two image acquisition units of the binocular camera are provided with fixed-focus lenses with synchronous triggering relation, same resolution and same focal length, and imaging planes are located on the same plane.
17. The fixed-focus binocular camera self-calibration system of claim 15, wherein: the wheels of the mobile platform comprise wheels, motors and encoders, the motors drive the wheels to rotate, and the encoders are used for recording the rotation angles and the number of turns of the wheels.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010713282.5A CN111862236B (en) | 2020-07-22 | 2020-07-22 | Self-calibration method and system for fixed-focus binocular camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010713282.5A CN111862236B (en) | 2020-07-22 | 2020-07-22 | Self-calibration method and system for fixed-focus binocular camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111862236A true CN111862236A (en) | 2020-10-30 |
| CN111862236B CN111862236B (en) | 2023-10-20 |
Family
ID=72950645
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010713282.5A Active CN111862236B (en) | 2020-07-22 | 2020-07-22 | Self-calibration method and system for fixed-focus binocular camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111862236B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112907487A (en) * | 2021-03-23 | 2021-06-04 | 东软睿驰汽车技术（沈阳）有限公司 | Binocular correction result determination method and device and electronic equipment |
| CN113610932A (en) * | 2021-08-20 | 2021-11-05 | 苏州智加科技有限公司 | Binocular camera external parameter calibration method and device |
| CN115988193A (en) * | 2022-12-09 | 2023-04-18 | 广纳四维(广东)光电科技有限公司 | Binocular image combination method based on single lens, optical module and head display equipment |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105469386A (en) * | 2015-11-11 | 2016-04-06 | 浙江大华技术股份有限公司 | Method and device for determining height and pitch angle of stereo camera |
| CN105931261A (en) * | 2016-07-08 | 2016-09-07 | 北京格灵深瞳信息技术有限公司 | Method and device for modifying extrinsic parameters of binocular stereo camera |
| CN107133983A (en) * | 2017-05-09 | 2017-09-05 | 河北科技大学 | Bundled round steel end face binocular vision system and space orientation and method of counting |
| US20180322658A1 (en) * | 2017-02-27 | 2018-11-08 | Anhui Huami Information Technology Co.,Ltd. | Camera Calibration |
| CN108830905A (en) * | 2018-05-22 | 2018-11-16 | 苏州敏行医学信息技术有限公司 | The binocular calibration localization method and virtual emulation of simulating medical instrument cure teaching system |
| CN109313813A (en) * | 2016-06-01 | 2019-02-05 | 奥托立夫开发公司 | Vision system and method for motor vehicles |
| CN109598762A (en) * | 2018-11-26 | 2019-04-09 | 江苏科技大学 | A kind of high-precision binocular camera scaling method |
| CN109859278A (en) * | 2019-01-24 | 2019-06-07 | 惠州市德赛西威汽车电子股份有限公司 | The scaling method and calibration system joined outside in-vehicle camera system camera |
| CN110197510A (en) * | 2019-06-05 | 2019-09-03 | 广州极飞科技有限公司 | Scaling method, device, unmanned plane and the storage medium of binocular camera |
| CN110260801A (en) * | 2019-05-13 | 2019-09-20 | 平安科技（深圳）有限公司 | Method and apparatus for measuring volume of material |
| DE102018108154A1 (en) * | 2018-04-06 | 2019-10-10 | pmdtechnologies ag | Calibration device and calibration method for a camera system |
| CN110796710A (en) * | 2019-10-29 | 2020-02-14 | 轻客智能科技（江苏）有限公司 | Method and module for rapidly calibrating external parameters of camera on line based on manifold space |
| CN111062990A (en) * | 2019-12-13 | 2020-04-24 | 哈尔滨工程大学 | Binocular vision positioning method for underwater robot target grabbing |
| CN111243033A (en) * | 2020-01-10 | 2020-06-05 | 大连理工大学 | Method for optimizing external parameters of binocular camera |
-
2020
- 2020-07-22 CN CN202010713282.5A patent/CN111862236B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105469386A (en) * | 2015-11-11 | 2016-04-06 | 浙江大华技术股份有限公司 | Method and device for determining height and pitch angle of stereo camera |
| CN109313813A (en) * | 2016-06-01 | 2019-02-05 | 奥托立夫开发公司 | Vision system and method for motor vehicles |
| CN105931261A (en) * | 2016-07-08 | 2016-09-07 | 北京格灵深瞳信息技术有限公司 | Method and device for modifying extrinsic parameters of binocular stereo camera |
| US20180322658A1 (en) * | 2017-02-27 | 2018-11-08 | Anhui Huami Information Technology Co.,Ltd. | Camera Calibration |
| CN107133983A (en) * | 2017-05-09 | 2017-09-05 | 河北科技大学 | Bundled round steel end face binocular vision system and space orientation and method of counting |
| DE102018108154A1 (en) * | 2018-04-06 | 2019-10-10 | pmdtechnologies ag | Calibration device and calibration method for a camera system |
| CN108830905A (en) * | 2018-05-22 | 2018-11-16 | 苏州敏行医学信息技术有限公司 | The binocular calibration localization method and virtual emulation of simulating medical instrument cure teaching system |
| CN109598762A (en) * | 2018-11-26 | 2019-04-09 | 江苏科技大学 | A kind of high-precision binocular camera scaling method |
| CN109859278A (en) * | 2019-01-24 | 2019-06-07 | 惠州市德赛西威汽车电子股份有限公司 | The scaling method and calibration system joined outside in-vehicle camera system camera |
| CN110260801A (en) * | 2019-05-13 | 2019-09-20 | 平安科技（深圳）有限公司 | Method and apparatus for measuring volume of material |
| CN110197510A (en) * | 2019-06-05 | 2019-09-03 | 广州极飞科技有限公司 | Scaling method, device, unmanned plane and the storage medium of binocular camera |
| CN110796710A (en) * | 2019-10-29 | 2020-02-14 | 轻客智能科技（江苏）有限公司 | Method and module for rapidly calibrating external parameters of camera on line based on manifold space |
| CN111062990A (en) * | 2019-12-13 | 2020-04-24 | 哈尔滨工程大学 | Binocular vision positioning method for underwater robot target grabbing |
| CN111243033A (en) * | 2020-01-10 | 2020-06-05 | 大连理工大学 | Method for optimizing external parameters of binocular camera |
Non-Patent Citations (3)
| Title |
|---|
| GUODONG ZHAI等: ""Coal Mine Rescue Robots Based on Binocular Vision: A Review of the State of the Art"", 《IEEE》 * |
| 荆旭;邱世广;尹旭悦;范秀敏;何其昌;: "光轴外扩式布置的双目相机图像立体校正", 上海交通大学学报, no. 02 * |
| 高齐竹慧, 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112907487A (en) * | 2021-03-23 | 2021-06-04 | 东软睿驰汽车技术（沈阳）有限公司 | Binocular correction result determination method and device and electronic equipment |
| CN113610932A (en) * | 2021-08-20 | 2021-11-05 | 苏州智加科技有限公司 | Binocular camera external parameter calibration method and device |
| CN113610932B (en) * | 2021-08-20 | 2024-06-04 | 苏州智加科技有限公司 | Binocular camera external parameter calibration method and device |
| CN115988193A (en) * | 2022-12-09 | 2023-04-18 | 广纳四维(广东)光电科技有限公司 | Binocular image combination method based on single lens, optical module and head display equipment |
| CN115988193B (en) * | 2022-12-09 | 2024-02-09 | 广纳四维(广东)光电科技有限公司 | Binocular imaging method based on single lens, optical module and head display equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111862236B (en) | 2023-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111862234B (en) | Binocular camera self-calibration method and system | |
| CN111862235B (en) | Binocular camera self-calibration method and system | |
| CN110569704B (en) | Multi-strategy self-adaptive lane line detection method based on stereoscopic vision | |
| CN109308693B (en) | Single-binocular vision system for target detection and pose measurement constructed by one PTZ camera | |
| CN110136208B (en) | Joint automatic calibration method and device for robot vision servo system | |
| CN109344882B (en) | Convolutional neural network-based robot control target pose identification method | |
| CN112396664B (en) | Monocular camera and three-dimensional laser radar combined calibration and online optimization method | |
| CN111862236B (en) | Self-calibration method and system for fixed-focus binocular camera | |
| CN110146099B (en) | Synchronous positioning and map construction method based on deep learning | |
| CN110842940A (en) | Building surveying robot multi-sensor fusion three-dimensional modeling method and system | |
| CN114399554B (en) | Calibration method and system of multi-camera system | |
| CN105758426A (en) | Combined calibration method for multiple sensors of mobile robot | |
| CN111897349A (en) | Underwater robot autonomous obstacle avoidance method based on binocular vision | |
| CN110163963B (en) | Mapping device and mapping method based on SLAM | |
| CN114022560A (en) | Calibration method and related device and equipment | |
| CN111998862A (en) | Dense binocular SLAM method based on BNN | |
| CN111429571B (en) | Rapid stereo matching method based on spatio-temporal image information joint correlation | |
| CN113313659A (en) | High-precision image splicing method under multi-machine cooperative constraint | |
| CN113345032B (en) | Initialization map building method and system based on wide-angle camera large distortion map | |
| CN111198563B (en) | Terrain identification method and system for dynamic motion of foot type robot | |
| CN112419411A (en) | Method for realizing visual odometer based on convolutional neural network and optical flow characteristics | |
| CN116664622A (en) | Visual movement control method and device | |
| CN112258582B (en) | Camera attitude calibration method and device based on road scene recognition | |
| Tan et al. | A distortion image correction method based on machine vision | |
| CN113781581A (en) | Depth of field distortion model calibration method based on target loose attitude constraint |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |