CN117557657A - Binocular fisheye camera calibration method and system based on Churco calibration plate - Google Patents

Abstract

The invention discloses a binocular fisheye camera calibration method based on a Churco calibration plate, which comprises the following steps of: s110, setting parameters of a ChArUco calibration plate; s120, obtaining an internal reference calibration image of the monocular fisheye camera by moving the position of the ChArUco calibration plate, detecting corner points of the square with the ArUco marks and alternating black and white according to the set parameters of the ChArUco calibration plate, obtaining corner point information and id information, and calibrating an internal reference matrix and a distortion coefficient by using the corner point information and the id information; s130, synchronously acquiring external parameter calibration images of the binocular fisheye camera, detecting corner points of the square with ArUco marks and black and white alternation according to set parameters of the ChArUco calibration plate, and calibrating an external parameter matrix by using complete corner point information and id information. The invention can effectively avoid the problem that the external parameter calibration fails due to the fact that different cameras identify angular points for calibrating pictures at the same moment when the external parameter of the binocular fisheye camera is calibrated.

Description

Binocular fisheye camera calibration method and system based on Churco calibration plate

Technical Field

The invention relates to the field of computer vision, in particular to a binocular fisheye camera calibration method and system based on a Churco calibration plate.

Background

In recent years, as the fisheye camera can shoot a larger view field angle, which can reach 180 degrees or higher, more scenes can be brought into pictures during shooting, and the fisheye camera plays an increasingly important role in the fields of vision measurement, intelligent transportation, virtual reality, panoramic stitching and the like. However, due to the unique lens design structure of the fisheye camera, image distortion is brought while a large visual angle is shot, the most serious is radial distortion, the reason for the generation of the distortion is that light rays are more seriously refracted at a position far away from the center of the lens, and the distortion degree is more serious when the light rays are closer to the edge position of the image, so before the fisheye camera is used, an internal reference matrix, a distortion coefficient and an external reference matrix of the camera are required to be calibrated, and the precision and the accuracy of image measurement can be ensured, and therefore, the calibration of the fisheye camera is particularly important for the use of the fisheye camera.

As shown in fig. 1, in the current method for calibrating a fisheye camera based on a checkerboard, a circular ring grid calibration plate or an AruCo calibration plate, when the internal parameters and distortion coefficients of a binocular fisheye camera are calibrated, the calibration plate needs to be placed in the shooting range of a camera, and the position pictures of the calibration plate under each view angle are shot by changing the position of the calibration plate or moving the camera, so that the calibration method has the following problems for the fisheye camera:

(1) All angular point positions in the calibration plate need to be detected in the calibration process of the checkerboard calibration plate, but when the fisheye camera shoots that the calibration plate is positioned at the edge position of the image, partial angular point detection is often failed due to serious distortion;

(2) In the calibration process of the circular grid calibration plate, a circle is imaged as an ellipse, and the projection of the center of the circle in space is not equal to the center of the projected ellipse, so that the circular grid calibration plate has eccentric error;

(3) The AruCo calibration plate has quick detection and multifunction, but the accuracy of the angular point positions of the calibration plate is not very high even though the calibration plate is thinned by sub-pixels; because of the imaging characteristics of the fisheye camera, the distortion is serious, errors are easily caused by using a common checkerboard calibration plate, and partial corner points can fail to be detected.

When external parameter calibration of the binocular fisheye camera is carried out, a series of pictures with different angles, which simultaneously comprise calibration plates, are shot through the binocular fisheye camera, the detected angular point positions are utilized for carrying out relative external parameter calibration, and the calibration method has the following problems for the fisheye camera: the partial corner points of the calibration pictures shot by the left and right fisheye cameras are failed to detect or the corner points detected by the two groups of pictures are inconsistent due to lens distortion, so that the calibration difficulty is high.

Disclosure of Invention

The invention mainly aims to provide a binocular fisheye camera calibration method and a binocular fisheye camera calibration system based on a Churco calibration plate, which can improve the internal parameter calibration precision of a camera and solve the problem that external parameter calibration fails due to different angular points of different cameras for calibrating pictures at the same moment during external parameter calibration of the binocular fisheye camera.

The technical scheme adopted by the invention is as follows:

the binocular fisheye camera calibration method based on the Churco calibration plate comprises the following steps:

s110, setting parameters of the Chuarco calibration plate, including the number of rows and columns of the Chuarco calibration plate, black and white square side length, arUco code side length and dictionary type;

s120, obtaining an internal reference calibration image of the monocular fisheye camera by moving the position of the ChArUco calibration plate, detecting corner points of the square with the ArUco marks and alternating black and white according to the set parameters of the ChArUco calibration plate, obtaining corner point information and id information, and calibrating an internal reference matrix and a distortion coefficient by using the corner point information and the id information;

s130, synchronously acquiring external reference calibration images of the binocular fisheye camera, detecting corner points of square with ArUco marks and black and white alternation according to set parameters of the ChArUco calibration plate to obtain corner point information and id information, removing the external reference calibration images with the number of corner points lower than 80%, sequentially selecting unidentified corner point positions of the external reference calibration images with the number of corner points between 80% and 100%, searching the corner point positions by adopting a gradient descent algorithm, inserting the corner point information and the id information into an automatically detected corner point and id array, and calibrating an external reference matrix by utilizing the complete corner point information and the complete id information.

In connection with the above technical solution, step S120 specifically includes the following steps:

s121, enabling the Chuarco calibration plate to appear in each position of a fisheye camera picture by moving the position of the Chuarco calibration plate, and collecting calibration images of a single fisheye camera from different visual angles;

s122, carrying out graying treatment on the obtained calibration image;

s123, acquiring angular point information of AruCo codes under a coordinate system of a calibration plate, wherein the angular point information comprises the number of detected AruCo codes, four angular point coordinates and ids of each AruCo code, and the ids of the AruCo codes are sequentially increased from left to right from top to bottom; screening each image according to the number of corner points and projection errors;

s124, calculating an internal reference matrix, distortion coefficients and overall reprojection error value by using a Churco library on the rest of the images.

In step S123, the number of corner points of the chuarco calibration board is detected by adopting Subpixel Corner Detection algorithm, and the number of corner points detected in each image is determined to be 80% lower than the number of real corner points in the chuarco calibration board, and then the images are removed; and simultaneously, independently calculating the reprojection error value of each image, and eliminating if the reprojection error value of the single distorted image is higher than a threshold k.

By adopting the technical scheme, the distortion coefficients are divided into radial distortion and tangential distortion, which are obtained by calculating by minimizing the re-projection error; specifically, for each calibration corner point on the calibration plate, calculating the distance between the two-dimensional pixel coordinates at the ideal position and the two-dimensional pixel coordinates in the actual image, and then using a least square method to fit the radial distortion and tangential distortion coefficients, and minimizing the fitting error.

In connection with the above technical solution, step S130 includes the following steps:

s131, synchronously acquiring a plurality of images of the binocular fisheye camera and the ChArUco calibration plate from different visual angles, so that the ChArUco calibration plate appears in each position of the fisheye camera, and a plurality of distorted images for calibrating external parameters are obtained;

s132, carrying out gray processing on a plurality of acquired distorted images;

s133, acquiring angular point information of AruCo codes under a coordinate system of a calibration plate, wherein the angular point information comprises the number of detected AruCo codes, four angular point coordinates and ids of each AruCo code, and the ids of the AruCo codes are sequentially increased from left to right from top to bottom; screening each image according to the number of corner points and projection errors to obtain candidate calibration images;

s134, comparing the number of the corner points detected by the candidate calibration images with the number of the actual corner points in the ChArUco calibration plate, and if the number of the detected corner points is not more than 80% of the number of the actual corner points, simultaneously eliminating the photos shot by the two cameras at the same time; if the number of the detected angular points exceeds 80% but is less than 100%, entering manual angular point adding, and preventing the detection failure caused by the difference of the number of the angular points detected by two calibration pictures shot by the binocular fisheye camera at the same time;

S135, calculating relative external parameters of the camera according to the angular point coordinates, the internal parameter matrix and the distortion coefficient, wherein the relative external parameters comprise a relative rotation matrix R and a relative translation matrix T.

By adopting the technical scheme, the images for internal reference calibration or external reference calibration are 40 to 60.

In step S134, after all unidentified corner ids in a certain image are detected, a rectangle with the corner as the center is selected by a manual frame from the first unidentified corner position, the rectangle is searched from the center to the periphery, gradient values g_x and g_y in the x direction and the y direction are calculated respectively, the gradient amplitude at the current position is calculated, if the gradient amplitude at the current position is greater than the historical maximum gradient amplitude, the maximum gradient amplitude and the coordinate value of the corresponding position are updated, the searching is stopped until the updated position reaches the image edge or the iteration reaches the maximum number, the searching is stopped, the coordinate values (x, y) are output, the position and id of the point are drawn in a calibration picture, all the corner information is sequentially added, and each of the searched corner coordinates and ids is added to the positions of the corresponding corner array and the id array.

By adopting the technical scheme, the ChArUco calibration plate is set to be 5 rows and 7 columns, wherein the side length of a black-and-white square is 0.04m, the side length of ArUco code is 0.031m, and the dictionary type is DICT_6*6.

The invention also provides a binocular fisheye camera calibration system based on the Churco calibration plate, which is characterized by comprising the following steps:

the calibration photo acquisition module is used for acquiring an internal reference calibration image and an external reference calibration image of the monocular fisheye camera by moving the position of the Churco calibration plate, and presetting parameters of the Churco calibration plate, including the number of rows and columns of the Churco calibration plate, the side length of black and white squares, the side length of Arrco codes and dictionary types;

the camera internal reference calibration module is used for detecting corner points of ArUco marks and black-white alternating squares in an internal reference calibration image according to set parameters of the ChArUco calibration plate to obtain corner point information and id information, and performing internal reference matrix and distortion coefficient calibration by using the corner point information and the id information;

the camera external parameter calibration module is used for detecting angular points of ArUco marks and black and white alternating squares in external parameter calibration images according to set parameters of the ChArUco calibration plate to obtain angular point information and id information, eliminating the external parameter calibration images with the number of angular points lower than 80%, sequentially framing the external parameter calibration images with the number of angular points between 80% and 100%, searching out angular point positions by adopting a gradient descent algorithm after sequentially framing unidentified angular point positions, inserting the angular point information and the id information into an automatically detected angular point and id array, and calibrating an external parameter matrix by utilizing the complete angular point information and the complete id information.

The invention also provides a computer storage medium in which a computer program executable by a processor is stored, and the computer program executes the binocular fisheye camera calibration method based on the Churuco calibration board.

The invention has the beneficial effects that: according to the invention, by adopting the Churco calibration plate and combining the advantages of AruCo codes and checkerboard, the accuracy of a calibration result is improved, and the problems of low accuracy and even calibration failure of a calibration system caused by incomplete calibration images or incomplete corner detection are avoided. When the camera is calibrated by external parameters, eliminating external parameters with the number of angular points lower than 80%, selecting unidentified angular point positions from external parameters calibrated images with the number of angular points between 80% and 100%, and searching out the angular point positions by adopting a gradient descent algorithm, thereby solving the problem that the external parameters are not calibrated due to different angular points of different cameras for calibrating photos at the same moment when the binocular fisheye camera is calibrated by external parameters.

Further, when the internal reference of the camera is calibrated, firstly, calibration photos with the number of detected angle points lower than 80% are screened out, then the rest photos are subjected to re-projection error detection, and the calibration photos with the re-projection error value higher than a threshold value are filtered and removed, so that the internal reference calibration precision is greatly improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic view of a different calibration plate of the prior art;

FIG. 2 is a flow chart of a binocular fisheye camera calibration method based on a Churco calibration plate in an embodiment of the invention;

FIG. 3 is a schematic diagram of a ChArUco calibration plate used in the present invention;

FIG. 4 is a flow chart of a camera internal parameter calibration method according to another embodiment of the invention;

FIG. 5 is a flow chart of a camera exogenous calibration method in accordance with another embodiment of the present invention;

FIG. 6 is a schematic diagram of a binocular fisheye camera calibration system based on a Churco calibration plate in an embodiment of the invention;

FIG. 7 is a schematic diagram of a calibration result detection device according to an embodiment of the present invention;

fig. 8 is a schematic diagram of the embodiment of the present invention, in which each frame of image after the calibration result is processed is spliced again into a video output.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

According to the invention, the fisheye camera is calibrated by mainly adopting the Churco calibration plate, so that the internal parameters of the camera can be estimated more accurately, the distortion of the fisheye camera is serious due to the imaging characteristic of the fisheye camera, and errors are easy to be caused by using a common checkerboard calibration plate. The ChArUco calibration board is added with black and white squares on the basis of a checkerboard, and the squares can provide more information and estimate the internal parameters of the camera more accurately. In addition, when internal reference calculation is carried out, the ChArUco calibration board does not need to detect all angular points, so that the problem that the calibration fails due to incomplete angular point detection caused by distortion of a fisheye camera is greatly avoided; secondly, the Chuarco calibration plate can provide more comprehensive distortion correction, besides radial distortion and tangential distortion, other distortions such as vertical distortion can occur in the fisheye camera, and all the distortions can be corrected simultaneously by using the Chuarco calibration plate; finally, because the distortion of the fisheye camera is serious, the common checkerboard calibration plate can cause larger reprojection error, and the Churco calibration plate can provide better reprojection error, so that the calibration precision is improved.

The reference matrix and the distortion coefficient of the cameras with the same type and the same focal length are always the same, so that when the reference matrix and the distortion coefficient of the binocular fisheye camera are calibrated, the reference matrix and the distortion coefficient of the whole camera system can be calibrated only by calibrating the reference matrix and the distortion coefficient of one camera. The following implementation of binocular fisheye camera calibration based on the chuarco calibration plate is performed by a number of specific embodiments.

Example 1

As shown in fig. 2, the binocular fisheye camera calibration method based on the chuarco calibration plate of this embodiment includes the following steps:

s110, setting parameters of the Chuarco calibration plate, including the number of rows and columns of the Chuarco calibration plate, black and white square side length, arUco code side length and dictionary type;

s120, obtaining an internal reference calibration image of the monocular fisheye camera by moving the position of the ChArUco calibration plate, detecting corner points of the square with the ArUco marks and alternating black and white according to the set parameters of the ChArUco calibration plate, obtaining corner point information and id information, and calibrating an internal reference matrix and a distortion coefficient by using the corner point information and the id information;

s130, synchronously acquiring external reference calibration images of the binocular fisheye camera, detecting corner points of square with ArUco marks and black and white alternation according to set parameters of the ChArUco calibration plate to obtain corner point information and id information, removing the external reference calibration images with the number of corner points lower than 80%, sequentially selecting unidentified corner point positions of the external reference calibration images with the number of corner points between 80% and 100%, searching the corner point positions by adopting a gradient descent algorithm, inserting the corner point information and the id information into an automatically detected corner point and id array, and calibrating an external reference matrix by utilizing the complete corner point information and the complete id information.

When the binocular fisheye camera is subjected to external parameter calibration, the distortion of the fisheye camera can lead to calibration pictures at the same moment, and calibration failure is caused by different numbers of corner points detected by the binocular fisheye camera, so that the embodiment firstly filters out pictures with fewer corner points, then manually adds corner point information to pictures with incomplete corner point detection, automatically searches out the intersection point position of black and white squares, namely the coordinate information of the corner points by adopting a gradient descent algorithm after rectangular frames near undetected corner points are selected through frames, and adds the intersection point position, namely the coordinate information of the corner points into the original corner points and an id array to perform external parameter calibration calculation. The embodiment can perfectly solve the problem that the external parameter calibration fails due to the fact that different cameras are different in angular points recognized by calibration pictures at the same moment when the external parameter of the binocular fisheye camera is calibrated.

Further, the step S120 specifically includes the following steps:

s121, enabling the Chuarco calibration plate to appear in each position of a fisheye camera picture by moving the position of the Chuarco calibration plate, and collecting calibration images of a single fisheye camera from different visual angles;

s122, carrying out graying treatment on the obtained calibration image;

s123, acquiring angular point information of AruCo codes under a coordinate system of a calibration plate, wherein the angular point information comprises the number of detected AruCo codes, four angular point coordinates and ids of each AruCo code, and the ids of the AruCo codes are sequentially increased from left to right from top to bottom; screening each image according to the number of corner points and projection errors;

S124, calculating an internal reference matrix, distortion coefficients and overall reprojection error value by using a Churco library on the rest of the images.

In the same way, when the internal reference of the camera is calibrated, the calibration photos with the number of detected angular points lower than a certain proportion can be firstly calibrated, then the rest photos are subjected to the detection of the reprojection errors, and the calibration photos with the reprojection errors higher than the threshold value are filtered and removed, so that the internal reference calibration precision is greatly improved.

Specifically, step S123 specifically adopts Subpixel Corner Detection algorithm to detect the number of corner points of the chuarco calibration board, determines that the number of corner points detected in each image is lower than 80% of the number of real corner points in the chuarco calibration board, and eliminates; and simultaneously, independently calculating the reprojection error value of each image, and eliminating if the reprojection error value of the single distorted image is higher than a threshold k.

Further, the distortion coefficients are divided into radial distortion and tangential distortion, which are obtained by minimizing the re-projection error; specifically, for each calibration corner point on the calibration plate, calculating the distance between the two-dimensional pixel coordinates at the ideal position and the two-dimensional pixel coordinates in the actual image, and then using a least square method to fit the radial distortion and tangential distortion coefficients, and minimizing the fitting error.

Step S130 includes the steps of:

s131, synchronously acquiring a plurality of images of the binocular fisheye camera and the ChArUco calibration plate from different visual angles, so that the ChArUco calibration plate appears in each position of the fisheye camera, and a plurality of distorted images for calibrating external parameters are obtained;

s132, carrying out gray processing on a plurality of acquired distorted images;

s133, acquiring angular point information of AruCo codes under a coordinate system of a calibration plate, wherein the angular point information comprises the number of detected AruCo codes, four angular point coordinates and ids of each AruCo code, and the ids of the AruCo codes are sequentially increased from left to right from top to bottom; screening each image according to the number of corner points and projection errors to obtain candidate calibration images;

s134, comparing the number of the corner points detected by the candidate calibration images with the number of the actual corner points in the ChArUco calibration plate, and if the number of the detected corner points is not more than 80% of the number of the actual corner points, simultaneously eliminating the photos shot by the two cameras at the same time; if the number of the detected angular points exceeds 80% but is less than 100%, entering manual angular point adding, and preventing the detection failure caused by the difference of the number of the angular points detected by two calibration pictures shot by the binocular fisheye camera at the same time;

S135, calculating relative external parameters of the camera according to the angular point coordinates, the internal parameter matrix and the distortion coefficient, wherein the relative external parameters comprise a relative rotation matrix R and a relative translation matrix T.

In this embodiment, as shown in fig. 3, a chuarco calibration board may be preset to have 5 rows and 7 columns, where the black and white square has a side length of 0.04m, the ararco code has a side length of 0.031m, and the dictionary type is dic_ 6*6. The images for the internal or external reference calibration are 40 to 60.

In step S134, after all unidentified corner ids in a certain image are detected, a rectangle with the corner as the center is selected from the first unidentified corner position by a manual frame, searching is performed from the center of the rectangle to the periphery, gradient values g_x and g_y in the x direction and the y direction are calculated respectively, gradient amplitude values at the current position are calculated, if the gradient amplitude value at the current position is greater than the historical maximum gradient amplitude value, the maximum gradient amplitude value and coordinate values of the corresponding position are updated until the updated position reaches the image edge or the maximum number of iterations, searching is stopped, coordinate values (x, y) are output, the position and id of the point are drawn in a calibration picture, all corner information is sequentially added, and each of the searched corner coordinates and id is added into the positions of the corresponding corner array and id array.

Example 2

This embodiment is based on embodiment 1, except that the preferred embodiment is given for each step.

The binocular fisheye camera calibration method based on the Churco calibration plate does not need special calibration equipment, is suitable for most monocular and binocular fisheye cameras, improves the accuracy of a calibration result, and simultaneously avoids the problems of low accuracy and even calibration failure of a calibration system caused by incomplete calibration images or incomplete corner detection and unequal corner detection.

In a first aspect, an embodiment of the present invention provides a method for calibrating a binocular fisheye camera, including

S110, setting parameters of a ChArUco calibration plate;

step S120, obtaining an internal reference calibration image of the monocular fisheye camera, detecting corner points of an ArUco mark and black-white alternating square, obtaining corner point information and id information, and calibrating an internal reference matrix and a calibration coefficient by using the corner point information and the id information;

step S130, synchronously acquiring external reference calibration images of the binocular fisheye camera, eliminating the angular point information id information of the square with ArUco marks and black and white alternation, eliminating the calibration photos with the detected angular point number lower than 80%, sequentially selecting unidentified angular point positions of external reference calibration photos with the angular point number between 80% and 100%, searching out the angular point positions by adopting a gradient descent algorithm, inserting the angular point information and the id information into an automatically detected angular point and id array, and calibrating an external reference matrix by utilizing the complete angular point information and the complete id information.

The fisheye camera is calibrated by the ChArUco calibration plate, so that the internal parameters of the camera can be estimated more accurately, the distortion of the fisheye camera is serious due to the imaging characteristic of the fisheye camera, errors are easily caused by using a common checkerboard calibration plate, black and white squares are added to the ChArUco calibration plate on the basis of the checkerboard, more information can be provided for the squares, the internal parameters of the camera can be estimated more accurately, and when the internal parameters are calculated, the ChArUco calibration plate does not need to detect all angular points, so that the failure of calibration caused by insufficient angular point detection due to the distortion of the fisheye camera is greatly avoided; secondly, the Chuarco calibration plate can provide more comprehensive distortion correction, besides radial distortion and tangential distortion, other distortions such as vertical distortion can occur in the fisheye camera, and all the distortions can be corrected simultaneously by using the Chuarco calibration plate; finally, because the distortion of the fisheye camera is serious, the common checkerboard calibration plate can cause larger reprojection error, and the Churco calibration plate can provide better reprojection error, so that the calibration precision is improved. The reference matrix and the distortion coefficient of the cameras with the same type and the same focal length are always the same, so that when the reference matrix and the distortion coefficient of the binocular fisheye camera are calibrated, the reference matrix and the distortion coefficient of the whole camera system can be calibrated only by calibrating the reference matrix and the distortion coefficient of one camera. When the external reference is carried out on the binocular fisheye camera, the distortion of the fisheye camera can lead to calibration photos at the same moment, and the number of corner points detected by the binocular fisheye camera is different, so that calibration failure is caused.

In the invention, firstly, the photos with fewer angular points are filtered out, then the angular point information is manually added to the photos with incomplete angular point detection, the rectangular frames near the undetected angular points are selected through the frames, then the intersection point positions of black and white squares, namely the coordinate information of the angular points, are automatically searched out by adopting a gradient descent algorithm, and the coordinate information is added to the original angular points and id arrays to perform external parameter calibration calculation.

The above steps S110 to S130 are exemplarily described below:

in step S110, parameters of the chuarco calibration board are set, including the number of rows and columns of the chuarco calibration board, black and white square side length, arucc code side length and dictionary type, wherein the checkerboard side length and arucc code side length in the chuarco calibration board are both in units of meters. Referring to fig. 3, the chuarco calibration plate is composed of black-and-white squares and arco codes, wherein arco codes are embedded in the white squares in the black-and-white squares, reference marks of each arco code are different, each arco code comprises a black frame, a plurality of white squares and black squares are arranged in the black frame, and the different color squares are arranged, so that each arco code has uniqueness. For example, a ChArUco calibration plate was set to 5 rows and 7 columns, where black and white squares had a side length of 0.04m and ArUco code had a side length of 0.031m, and the dictionary type was DICT_6*6.

In step S120, an internal reference calibration image of the monocular fisheye camera is obtained, angular points of square with arco marks and black and white alternating are detected, angular point information and id information are obtained, and an internal reference matrix and a calibration coefficient are calibrated by using the angular point information and the id information, and specifically as shown in fig. 4, an internal reference calibration result of the camera can be obtained through the following steps.

Step S121, obtaining distortion images for calibrating an internal reference matrix and distortion coefficients of the fisheye camera, and moving the positions of the calibration plates to enable the calibration plates to appear in each position of a picture of the fisheye camera, and collecting calibration images of a single fisheye camera from different visual angles, wherein the number of the calibration images is about 40 to 60.

Step S122, sequentially reading distortion images for calibrating the internal reference matrix and the distortion coefficient, carrying out gray-scale processing, removing color information in the images, enabling the images to only contain brightness information, simplifying the complexity of calculation, and improving the calibration accuracy and stability.

And step S123, acquiring angular point information of AruCo codes under a calibration plate coordinate system, wherein the angular point information comprises the number of detected AruCo codes, four angular point coordinates and ids of each AruCo code, and the ids of the AruCo codes are sequentially increased from left to right from top to bottom.

Preferably: the detected AruCo marks are refined based on a checkerboard model, angular points around the AruCo marks are searched first, the angular points are used for calculating accurate position and posture information of the marks, and then the checkerboard model is utilized for further optimizing the estimation results, so that more accurate and stable calibration position and posture information is obtained.

And acquiring angular point information of the checkerboard formed by black and white squares under a calibration plate coordinate system, wherein the angular point information comprises the number of detected angular points, angular point coordinates and angular point ids, and the angular point ids are sequentially increased from left to right and from bottom to top.

Preferably: screening the number of corner points detected by each picture, removing pictures with the number of corner points of the detected calibration plate being lower than 80% so as to improve the calibration precision, specifically, adopting Subpixel Corner Detection algorithm to detect the corner points of the Churco calibration plate, improving the accuracy and precision of the corner point detection by solving more accurate image gradient and diagonal linear interpolation, determining whether the number of the detected corner points reaches 80% of the number of the real corner points in the Churco calibration plate after the corner points are detected, and if the number of the detected corner points is lower than 80%, indicating that the available information of the picture for calibration is less, thus filtering and removing the calibration picture; meanwhile, the re-projection error value of each distorted image is calculated independently, if the re-projection error value of the single distorted image is higher than a threshold k, the error of the calibration photo is larger, the influence on the final calibration result is larger, and therefore the calibration photo is filtered and removed.

And S124, calculating an internal reference matrix, distortion coefficients and overall reprojection error value by using the Churco library. Specifically, it is first necessary to establish a correspondence relationship between the world coordinate system and the image coordinate system. For each corner point it can be represented as (x, y, z) in the object coordinate system, where z defaults to 0, because the corner points lie on one plane, while in the pixel coordinate system the pixel coordinate representation of each corner point can be assumed as (u, v). Then, calculating the real coordinates of each corner point under the world coordinate system through the known size of the Churco mode and the position of the internal corner point, defining an origin, and then calculating the real coordinates of all the corner points under the world coordinate system according to the distance and the angle of two adjacent corner points in the pattern, wherein the Churco calibration board is exemplified by 5 rows and 7 columns, the side length of a black-and-white square is 0.04m, the side length of an Arrco code is 0.031m, the dictionary type is DICT_6*6, and the real coordinates under the world coordinate system are respectively:

and after the corresponding relation between the world coordinate system and the image coordinate system is obtained, carrying out camera calibration by adopting a Zhang calibration method. Concrete embodimentsIn other words, there are points (X _i ,Y _i ,Z _i ) Their coordinates in the pixel coordinate system are (u) _i ,v _i ) According to the vacuum camera model, three-dimensional points (X _i ,Y _i ,Z _i ) Projected onto the image plane to obtain its corresponding pixel coordinates (u _i ,v _i ) The projection process is as follows:

wherein f _x And f _y Focal length of camera in horizontal and vertical directions, c _x And c _y The coordinates of the principal point of the image plane (i.e. the image center) in the horizontal and vertical directions are respectively obtained by rewriting the above formula into a matrix form to obtain the following overdetermined equation set:

wherein m is _ij The method is characterized in that the method is an element in a projection matrix, a least square method is used for solving an overdetermined equation set to obtain the projection matrix, namely an internal and external parameter matrix of a camera, and an extraction internal parameter matrix K is shown as follows:

for the distortion coefficient, it is calculated by correcting the two-dimensional pixel coordinates, specifically, the distortion coefficient can be divided into two parts: radial distortion and tangential distortion. Radial distortion is a phenomenon in which the image point is not in the ideal position because the light rays are no longer perfectly parallel to the optical axis as they pass through the lens, and are more refracted farther from the center of the lens. Let r be the distance from the center point of the image at a certain position on the image, whereThen radial distortion can beDescribed by the following formula:

wherein x is _p And y _p Representing uncorrected pixel coordinates, x _{_rcorr} And y _{_rcorr} Represents the corrected pixel coordinates, r represents the distance of the pixel coordinates from the optical axis, also called radial distortion radius, k ₁ 、k ₂ 、k ₃ Is the radial distortion coefficient.

Tangential distortion is due to the fact that the lens is not perfectly parallel to the imaging plane, which can be described by the following equation:

wherein p is ₁ And p ₂ Is the tangential distortion coefficient, x _{_tcorr} And y _{_tcorr} Representing the tangentially corrected pixel coordinates.

The radial distortion coefficient and tangential distortion coefficient are calculated by minimizing the re-projection error. Specifically, for each calibration point (corner point on the calibration plate), the distance between its two-dimensional pixel coordinates at the ideal position and the two-dimensional pixel coordinates actually observed is calculated, and then the least square method is used to fit the appropriate radial distortion and tangential distortion coefficients, so that the fitting error is minimized. The Levenberg-Marquardt algorithm is an optimization algorithm for solving the nonlinear least square problem, can minimize the reprojection error, and continuously adjusts the values of coordinates to minimize the sum of the reprojection errors of all the characteristic points, specifically, calculates the reprojection error according to the current coordinate estimation, updates the estimation according to the error, and if the error is larger, needs a larger updating step; if the error is smaller, smaller updating step length is needed, and the optimal solution is gradually approached in the iterative process, so that the aim of minimizing the reprojection error is fulfilled, and finally, the proper radial distortion and tangential distortion coefficient are fitted through minimizing the reprojection error;

In step S130, the external reference calibration image of the binocular fisheye camera is synchronously acquired, the angular point information id information of the square with ArUco marks and black and white alternation is removed, the calibration photo with the detected angular point number lower than 80% is removed, the external reference calibration photo with the angular point number between 80% and 100% is sequentially framed to select the unrecognized angular point position, then the angular point position is searched by adopting a gradient descent algorithm, the angular point information and the id information are inserted into the automatically detected angular point and id array, the external reference matrix calibration is performed by utilizing the complete angular point information and the id information, and the external reference calibration result of the camera can be obtained through the following steps as shown in fig. 5.

In step S131, a distorted image for calibrating the external reference matrix is obtained, and multiple images of the binocular fisheye camera including the chuarco calibration board are synchronously acquired from different viewing angles, so that the chuarco calibration board appears in each position of the fisheye camera, and the number of the calibrated images is about 60 to 80.

In step S132, the distorted images acquired by the binocular fisheye camera and used for calibrating the external parameters are respectively read, gray processing is carried out, color information in the images is removed, so that the images only contain brightness information, the calculation complexity is simplified, and the calibration accuracy and stability are improved;

In step S133, acquiring angular point information of AruCo codes under a calibration plate coordinate system, wherein the angular point information comprises the number of detected AruCo codes, four angular point coordinates and ids of each AruCo code, and the ids of the AruCo codes are sequentially increased from left to right from top to bottom; the method comprises the steps of obtaining corner information of a checkerboard formed by squares of black-white alternating squares under a calibration plate coordinate system, wherein the corner information comprises the number of detected corner points, corner coordinates and corner points id, and the id of each corner point is sequentially increased from left to right and from bottom to top.

Preferably: the detected AruCo marks are refined based on a checkerboard model, angular points around the AruCo marks are searched first, the angular points are used for calculating accurate position and posture information of the marks, and then the checkerboard model is utilized for further optimizing the estimation results, so that more accurate and stable calibration position and posture information is obtained.

In step S134, comparing the number of corner points detected by the candidate calibration photos with the number of actual corner points in the churuco calibration board, and if the number of detected corner points is less than 80% of the number of actual corner points, simultaneously eliminating the calibration photos shot by the two cameras at the same time; if the number of detected corner points exceeds 80% but is less than 100%, entering manual adding of corner points, preventing the failure of detection caused by the difference of the number of corner points detected by two calibration photographs taken by the binocular fisheye camera at the same time, as described in step S135.

In step S135, after all the unidentified corner points id are detected, a rectangle centered on the corner point is selected from the first unidentified corner point on the calibration photo by a manual frame, a gradient descent algorithm is adopted, specifically, a learning rate and a maximum iteration number are set first, searching is performed from the center position of the rectangle to the periphery, gradient values g_x and g_y in the x direction and the y direction are calculated respectively, gradient amplitude values at the current position are calculated, if the gradient amplitude value at the current position is greater than the historical maximum gradient amplitude value, the maximum gradient amplitude value and coordinate values of the corresponding position are updated until the updated position reaches the image edge or the maximum iteration number, searching is stopped, the coordinate values (x, y) are output, and the position and id of the point are drawn in the calibration picture. And sequentially adding all the corner information, and adding each searched corner coordinate and id into the positions corresponding to the corner array and the id array.

In step S136, the relative external parameters of the camera, including the relative rotation matrix R and the relative translation matrix T, are calculated using the coordinates of the corner points obtained in step S135, the coordinates in the world coordinate system obtained in step S124, the camera internal parameter matrix, and the distortion coefficient. Specifically, it is assumed that the coordinate systems of the left and right cameras are O _l And O _r And there is a rotation matrix R and a translation vector T between them, then for any three-dimensional space point P, its projected pixel coordinates at left and right cameras are P respectively _l And p _r . Then there is the following relationship:

wherein K is _l And K _r Respectively an internal reference matrix of a left camera and a right camera, [ I|O ]]Representing a 3 x 4 identity matrix, O is a zero vector of 3*1. Transforming the above equation yields the following equation:

wherein A is _l And A _r An internal reference matrix of the left and right cameras respectively, b is a zero vector of length 2*N, N represents the number of two-dimensional-three-dimensional point pairs, and the i-th point pair comprises a three-dimensional point P _i ＝[X _i ,Y _i ,Z _i ] ^T And its pixel coordinates p in both left and right cameras _li ＝[u _li ,v _li ] ^T And p _ri ＝[u _ri ,v _ri ] ^T . These point pairs can be converted into an overdetermined system of equations:

/>

for each three-dimensional point P _i The corresponding pixel coordinate p can be obtained from the left and right cameras _li And p _ri The rotation matrix R and the translation vector T are estimated by the least square method so that the error of the above equation set is minimized, specifically, for each point P _i Subtracting the estimated values from the pixel coordinates of the left and right cameras to obtain an error vector e with the length of 4*1 _i Minimizing Σ by gradient descent method _i ||e _i || ² Finally, by solving the overdetermined equation set, the function can obtain a rotation matrix R and a translation vector T between the left camera and the right camera.

Example 3

The embodiment is mainly used for realizing the method embodiment, and the binocular fisheye camera calibration system based on the Churco calibration plate mainly comprises the following steps:

the calibration photo acquisition module is used for acquiring an internal reference calibration image and an external reference calibration image of the monocular fisheye camera by moving the position of the Churco calibration plate, and presetting parameters of the Churco calibration plate, including the number of rows and columns of the Churco calibration plate, the side length of black and white squares, the side length of Arrco codes and dictionary types;

the camera internal reference calibration module is used for detecting corner points of ArUco marks and black-white alternating squares in an internal reference calibration image according to set parameters of the ChArUco calibration plate to obtain corner point information and id information, and performing internal reference matrix and distortion coefficient calibration by using the corner point information and the id information;

the camera external parameter calibration module is used for detecting angular points of ArUco marks and black and white alternating squares in external parameter calibration images according to set parameters of the ChArUco calibration plate to obtain angular point information and id information, eliminating the external parameter calibration images with the number of angular points lower than 80%, sequentially framing the external parameter calibration images with the number of angular points between 80% and 100%, searching out angular point positions by adopting a gradient descent algorithm after sequentially framing unidentified angular point positions, inserting the angular point information and the id information into an automatically detected angular point and id array, and calibrating an external parameter matrix by utilizing the complete angular point information and the complete id information.

Example 4

Referring to fig. 6, this embodiment is also a binocular camera calibration system for implementing the above-described method embodiment, and mainly includes:

the calibration photo acquisition module 201 is used for acquiring a calibration image of the Churuco calibration board shot by the fisheye camera;

the angular point information acquisition module 202 of ArUCo marks and black and white squares is used for acquiring angular point information of ArUco marks and black and white squares under a coordinate system of a calibration plate, the reference marks of each ArUco code are different, each ArUco code comprises a black frame, a plurality of white squares and black squares are arranged in the black frame, and the squares with different colors are arranged, so that each ArUco code has uniqueness;

the camera internal reference calibration module 203 is configured to calibrate an internal reference matrix and a distortion coefficient of the fisheye camera by using corner information and id information of a black-and-white square;

the camera external parameter calibration module 204 is configured to calibrate the relative external parameters of the fisheye camera, including the relative rotation matrix R and the relative translation matrix T, by using the corner information and id information of the black-white square, the coordinates of the corner under the world coordinate system, the camera internal parameter matrix and the distortion coefficient.

Example 5

Referring to fig. 7, this embodiment provides a calibration result detection apparatus, which mainly includes:

The video reading module 301 is configured to read a video stream or a video file of the camera;

the frame image correction module 302 is configured to perform frame processing on the read video stream, and correct each frame image by using the calibrated internal reference matrix, distortion coefficient and external reference matrix;

the video display module 303 is shown in fig. 8, and is configured to re-stitch each processed frame image into a video output.

Example 6

The present application also provides a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., having stored thereon a computer program that when executed by a processor performs a corresponding function. The computer readable storage medium of the present embodiment, when executed by a processor, implements the binocular fisheye camera calibration method based on the chuarco calibration board of the method embodiment.

It should be noted that each step/component described in the present application may be split into more steps/components, or two or more steps/components or part of the operations of the steps/components may be combined into new steps/components, as needed for implementation, to achieve the object of the present invention.

The sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of the processes should be determined by the functions and the internal logic, and should not be construed as limiting the implementation process of the embodiments of the present application.

It will be understood that modifications and variations will be apparent to those skilled in the art from the foregoing description, and it is intended that all such modifications and variations be included within the scope of the following claims.

Claims (10)

1. The binocular fisheye camera calibration method based on the Churco calibration plate is characterized by comprising the following steps of:

s110, setting parameters of the Chuarco calibration plate, including the number of rows and columns of the Chuarco calibration plate, black and white square side length, arUco code side length and dictionary type;

s120, obtaining an internal reference calibration image of the monocular fisheye camera by moving the position of the ChArUco calibration plate, detecting corner points of the square with the ArUco marks and alternating black and white according to the set parameters of the ChArUco calibration plate, obtaining corner point information and id information, and calibrating an internal reference matrix and a distortion coefficient by using the corner point information and the id information;

s130, synchronously acquiring external reference calibration images of the binocular fisheye camera, detecting corner points of square with ArUco marks and black and white alternation according to set parameters of the ChArUco calibration plate to obtain corner point information and id information, removing the external reference calibration images with the number of corner points lower than 80%, sequentially selecting unidentified corner point positions of the external reference calibration images with the number of corner points between 80% and 100%, searching the corner point positions by adopting a gradient descent algorithm, inserting the corner point information and the id information into an automatically detected corner point and id array, and calibrating an external reference matrix by utilizing the complete corner point information and the complete id information.

2. The binocular fisheye camera calibration method based on the chuarco calibration plate of claim 1, wherein step S120 specifically comprises the following steps:

s121, enabling the Chuarco calibration plate to appear in each position of a fisheye camera picture by moving the position of the Chuarco calibration plate, and collecting calibration images of a single fisheye camera from different visual angles;

s122, carrying out graying treatment on the obtained calibration image;

s123, acquiring angular point information of AruCo codes under a coordinate system of a calibration plate, wherein the angular point information comprises the number of detected AruCo codes, four angular point coordinates and ids of each AruCo code, and the ids of the AruCo codes are sequentially increased from left to right from top to bottom; screening each image according to the number of corner points and projection errors;

s124, calculating an internal reference matrix, distortion coefficients and overall reprojection error value by using a Churco library on the rest of the images.

3. The binocular fisheye camera calibration method based on the chuarco calibration plate according to claim 1, wherein step S123 specifically adopts Subpixel Corner Detection algorithm to detect the number of corner points of the chuarco calibration plate, and eliminates if the number of detected corner points in each image is less than 80% of the number of real corner points in the chuarco calibration plate; and simultaneously, independently calculating the reprojection error value of each image, and eliminating if the reprojection error value of the single distorted image is higher than a threshold k.

4. The binocular fisheye camera calibration method based on the chuarco calibration plate according to claim 1, wherein the distortion coefficients are divided into radial distortion and tangential distortion, which are calculated by minimizing the re-projection error; specifically, for each calibration corner point on the calibration plate, calculating the distance between the two-dimensional pixel coordinates at the ideal position and the two-dimensional pixel coordinates in the actual image, and then using a least square method to fit the radial distortion and tangential distortion coefficients, and minimizing the fitting error.

5. The binocular fisheye camera calibration method based on the chuarco calibration plate of claim 1, wherein step S130 comprises the steps of:

s131, synchronously acquiring a plurality of images of the binocular fisheye camera and the ChArUco calibration plate from different visual angles, so that the ChArUco calibration plate appears in each position of the fisheye camera, and a plurality of distorted images for calibrating external parameters are obtained;

s132, carrying out gray processing on a plurality of acquired distorted images;

s133, acquiring angular point information of AruCo codes under a coordinate system of a calibration plate, wherein the angular point information comprises the number of detected AruCo codes, four angular point coordinates and ids of each AruCo code, and the ids of the AruCo codes are sequentially increased from left to right from top to bottom; screening each image according to the number of corner points and projection errors to obtain candidate calibration images;

S134, comparing the number of the corner points detected by the candidate calibration images with the number of the actual corner points in the ChArUco calibration plate, and if the number of the detected corner points is not more than 80% of the number of the actual corner points, simultaneously eliminating the photos shot by the two cameras at the same time; if the number of the detected angular points exceeds 80% but is less than 100%, entering manual angular point adding, and preventing the detection failure caused by the difference of the number of the angular points detected by two calibration pictures shot by the binocular fisheye camera at the same time;

s135, calculating relative external parameters of the camera according to the angular point coordinates, the internal parameter matrix and the distortion coefficient, wherein the relative external parameters comprise a relative rotation matrix R and a relative translation matrix T.

6. The binocular fisheye camera calibration method based on the chuarco calibration plate of claim 1, wherein the images for the internal or external reference calibration are 40 to 60.

7. The binocular fisheye camera calibration method based on the chuarco calibration plate according to claim 1, wherein in step S134, when all unidentified corner points id in a certain image are detected, starting from the first unidentified corner point position, a rectangle centered on the corner point is selected by a manual frame, searching is performed from the rectangular center position to the periphery, gradient values g_x and g_y in x-direction and y-direction are calculated respectively, and gradient amplitude values at the current position are calculated, if the gradient amplitude value at the current position is greater than the historical maximum gradient amplitude value, the maximum gradient amplitude value and coordinate values of the corresponding position are updated until the updated position reaches the image edge or the maximum number of iterations, searching is stopped, coordinate values (x, y) are output, the position and id of the point are drawn in the calibration picture, all corner point information is sequentially added, and each corner point coordinate and id searched out are added to the positions of the corresponding corner point array and id array.

8. The method for calibrating a binocular fisheye camera based on a chuarco calibration plate according to any of claims 1-7, wherein the chuarco calibration plate is set to 5 rows and 7 columns, wherein the black and white squares have a side length of 0.04m, the chuco code has a side length of 0.031m, and the dictionary type is dic_ 6*6.

9. Binocular fisheye camera calibration system based on chuarco calibration board, characterized by comprising:

the calibration photo acquisition module is used for acquiring an internal reference calibration image and an external reference calibration image of the monocular fisheye camera by moving the position of the Churco calibration plate, and presetting parameters of the Churco calibration plate, including the number of rows and columns of the Churco calibration plate, the side length of black and white squares, the side length of Arrco codes and dictionary types;

the camera internal reference calibration module is used for detecting corner points of ArUco marks and black-white alternating squares in an internal reference calibration image according to set parameters of the ChArUco calibration plate to obtain corner point information and id information, and performing internal reference matrix and distortion coefficient calibration by using the corner point information and the id information;

the camera external parameter calibration module is used for detecting angular points of ArUco marks and black and white alternating squares in external parameter calibration images according to set parameters of the ChArUco calibration plate to obtain angular point information and id information, eliminating the external parameter calibration images with the number of angular points lower than 80%, sequentially framing the external parameter calibration images with the number of angular points between 80% and 100%, searching out angular point positions by adopting a gradient descent algorithm after sequentially framing unidentified angular point positions, inserting the angular point information and the id information into an automatically detected angular point and id array, and calibrating an external parameter matrix by utilizing the complete angular point information and the complete id information.

10. A computer storage medium, wherein a computer program executable by a processor is stored, the computer program executing the binocular fisheye camera calibration method based on the chuarco calibration plate of any one of claims 1-7.