CN107578450B - Method and system for calibrating assembly error of panoramic camera - Google Patents
Method and system for calibrating assembly error of panoramic camera Download PDFInfo
- Publication number
- CN107578450B CN107578450B CN201710828874.XA CN201710828874A CN107578450B CN 107578450 B CN107578450 B CN 107578450B CN 201710828874 A CN201710828874 A CN 201710828874A CN 107578450 B CN107578450 B CN 107578450B
- Authority
- CN
- China
- Prior art keywords
- camera
- calibration
- coordinate system
- binocular stereo
- coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
A method and a system for assembly error calibration of a panoramic camera are provided, the method comprises the following steps: establishing a monocular calibration model; establishing a binocular three-dimensional calibration model; and calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model. The method calculates the angle between the coordinate axes of the adjacent cameras and the rotation and translation errors of all coordinate systems on the basis of a standard camera calibration model, and the two parameter indexes can accurately and concisely describe the assembly errors of the panoramic camera. Meanwhile, the invention provides a feasible solution method for the assembly error, which can better compensate the splicing accuracy problem caused by the assembly error of the camera in the production application of the panoramic camera, thereby improving the splicing effect.
Description
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a method and a system for calibrating assembly errors of a panoramic camera.
Background
The panoramic camera is used for shooting a spatial scene by using a plurality of cameras simultaneously and outputting a 360-degree panoramic video in real time. The panoramic camera at least has two or more cameras, and the most core technology is real-time splicing of multiple paths of videos.
Because the panoramic camera adopts a plurality of cameras to shoot 360-degree space scenes, partial overlapping areas collected by adjacent cameras are utilized to carry out image fusion, and finally, the partially overlapped images collected by the cameras are spliced into a large-scale seamless high-resolution image. The image stitching technology of the panoramic camera greatly depends on the size of an overlapping area between cameras, and the size of the overlapping area collected between adjacent cameras is related to the view field angle of a lens on one hand, and the other hand is limited by the assembly error of the panoramic camera.
At present, how to calibrate the assembly error of the panoramic camera quickly, simply and accurately has no specific calibration method, so a quantifiable calibration method of the assembly error of the panoramic camera is urgently needed, and the allowable error range of the panoramic camera is determined according to a splicing algorithm, so that the industrial production is guided.
Disclosure of Invention
In the prior art, no specific calibration method exists for how to calibrate the assembly error of the panoramic camera quickly, simply and accurately, and in order to solve the problem, the invention provides a method and a system for calibrating the assembly error of the panoramic camera, and the specific scheme is as follows:
a method for panoramic camera assembly error calibration comprises the following steps:
s1, establishing a monocular calibration model;
s2, establishing a binocular stereo calibration model;
and S3, calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model.
In the method, the specific steps of establishing the monocular calibration model are as follows:
the coordinate of the point P in the world coordinate system is set as Pw＝(Xw，Yw，Zw)TThe coordinate in the camera coordinate system is Pc＝(Xc，Yc，Zc)TAnd they satisfy the following relationship:
Pc＝R×Pw+T (1)
projecting a point in a camera coordinate system into an image coordinate system, the point P having a coordinate P in the camera coordinate systemc＝(Xc，Yc，Zc)TWith the coordinate P ═ x, y in the image coordinate systemTThe following relationship is satisfied:
in formula (2), s is a scale factor of any scale, M is an internal parameter matrix of the camera, and fxRepresenting the equivalent focal length, f, of the camera in the x-direction of the imaging planeyRepresents the equivalent focal length of the camera in the y direction of the imaging plane (C)x，Cy) Representing pixel coordinates of the principal point in x, y directions of the imaging plane;
the homogeneous coordinate form of the camera calibration model is expressed as follows:
in the formula (3), the first and second groups,
representing the coordinates of the feature points in the image plane coordinate system,
representing spatial feature points under a calibration plate coordinate system;
where W denotes a physical transformation for locating the observed object plane and includes the sum of a partial rotation R and a partial translation T associated with the observed image plane, and W ═ R | T.
In step S1, the entire calibration board image is collected by each camera of the panoramic camera, and the feature point coordinates of all the calibration boards in the plane are extracted according to the collected calibration board images of different fields of view.
In the method, 10 to 15 calibration images are acquired by each camera, and the position and the angle of each acquired calibration plate image are different.
In the method, the specific steps of establishing the binocular stereo calibration model are as follows:
establishing a binocular stereo calibration plate consisting of a left sub calibration plate 1 and a right sub calibration plate 2, and respectively establishing a world first coordinate system O1-XY and a second coordinate system O2-XY for the two sub calibration plates 1 and 2, wherein the spatial relationship of the two world coordinate systems is represented by a translation vector T0;
two adjacent cameras are adopted to collect images of the binocular stereo calibration plate, the two adjacent cameras are respectively a left camera and a right camera, a point in space is set as P, and coordinates of the two sub calibration plates 1 and 2 in a world coordinate system are respectively X1、X2Then X1And X2The following relationship is satisfied:
X2＝X1-T0 (4)
the coordinate of the space point P under the coordinate system of the left camera is XlThe coordinate under the coordinate system of the right camera is XrThe following conversion relations exist between the two groups:
Xl＝Rl×X1+Tl，Xr＝Rr×Xx+Tr(5)
in the formula (5), RlAnd TlDenotes the external parameter, R, of the left camerarAnd TrRepresenting the external parameters of the right camera;
by eliminating X in formula (4) and formula (5)1，X2The following can be obtained:
Xr＝Rr×Rl -1×Xl-Rr×T0+Tr-Rr×Rl -1×Tl(6)
further, it can be derived that:
R＝Rr×Rl -1，T＝-Rr×T0+Tr-Rr×Rl -1×Tl(7)
in formula (7), R represents a rotation matrix between the left camera coordinate system and the right camera coordinate system, T represents a translation vector between the left camera coordinate system and the right camera coordinate system, and R and T are results of binocular stereo calibration performed between adjacent cameras.
In step S2, images of binocular stereo calibration plates of different viewing fields of the panoramic camera are collected, coordinates of feature points of all the binocular stereo calibration plates in a plane are extracted, and a rotation matrix R and a translational vector T between adjacent cameras in the panoramic camera are calculated respectively by using the binocular stereo calibration model.
In the method, two sub-calibration plates 1 and 2 in the binocular stereo calibration plate are in a checkerboard form, and the sizes of the checkerboards are the same.
In the method, when the images of the binocular stereo calibration board are collected, the images of the binocular stereo calibration board are shot simultaneously by using the adjacent left and right cameras, the sub-calibration boards 1 and 2 are respectively only shown in the visual field of one camera when the images are collected, and the number of the collected images of the binocular stereo calibration board is 8-12.
In the method, the specific steps of calculating the assembly error of the panoramic camera are as follows:
in step S2, a matrix in which a rotation matrix R between adjacent camera coordinate systems of the panoramic camera is 3 × 3 is obtained, the first column of the rotation matrix R represents a rotation vector of the X axis between the adjacent camera coordinate systems, and the rotation vector is represented as Vx(ii) a The second column represents the rotation vector of the Y axis between the adjacent camera coordinate systems, the rotation vector being set to Vy(ii) a The third column indicates the rotation vector of the Z axis between the adjacent camera coordinate systems, which is set to Vz；
Let a standard vector be (0, 0, 1)TThe included angles between the X-axis, the Y-axis and the Z-axis of the coordinate systems of the adjacent cameras are respectively Amglex，Amgley，AmglezDenotes, then Amglex，Amgley，AnglezThe calculation formula of (a) is as follows:
wherein dot (t)) represents a dot product operation between two vectors, norm (norm)) represents a mode of the vectors, and an included angle between the X axis, the Y axis and the Z axis is an assembly error of the panoramic camera.
The method for calibrating the assembly error of the panoramic camera comprises the steps of firstly, establishing a monocular calibration model, and secondly, establishing a binocular stereo calibration model; and then, calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model. The method calculates the angle between the coordinate axes of the adjacent cameras and the rotation and translation errors of all coordinate systems on the basis of a standard camera calibration model, and the two parameter indexes can accurately and concisely describe the assembly errors of the panoramic camera. Meanwhile, the invention provides a feasible solution method for the assembly error, which can better compensate the splicing accuracy problem caused by the assembly error of the camera in the production application of the panoramic camera, thereby improving the splicing effect.
According to another aspect of the present invention, there is also provided a system for panoramic camera assembly error calibration, comprising:
the monocular calibration model establishing module is used for establishing a monocular calibration model;
the binocular stereo calibration model establishing module is used for establishing a binocular stereo calibration model;
and the assembly error calculation module is used for calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model.
The system for calibrating the assembly error of the panoramic camera comprises the following steps of firstly, establishing a monocular calibration model, and secondly, establishing a binocular stereo calibration model; and then, calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model. The method calculates the angle between the coordinate axes of the adjacent cameras and the rotation and translation errors of all coordinate systems on the basis of a standard camera calibration model, and the two parameter indexes can accurately and concisely describe the assembly errors of the panoramic camera. Meanwhile, the invention provides a feasible solution method for the assembly error, which can better compensate the splicing accuracy problem caused by the assembly error of the camera in the production application of the panoramic camera, thereby improving the splicing effect.
Drawings
FIG. 1 is a flow chart of a method of one implementation provided by the method for panoramic camera assembly error calibration of the present invention;
FIG. 2 is a schematic view of a single target plate of the present invention;
FIG. 3 is a schematic view of a binocular stereo calibration plate of the present invention;
FIG. 4 is a schematic view of an image capture device of the present invention;
fig. 5 is a block diagram of an implementation provided by the system for panoramic camera assembly error calibration of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings in conjunction with the following detailed description. It should be understood that the description is intended to be exemplary only, and is not intended to limit the scope of the present invention. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present invention.
The panoramic camera is used for shooting a spatial scene by using a plurality of cameras simultaneously and outputting a 360-degree panoramic video in real time. The panoramic camera at least has two or more cameras, and the most core technology is real-time splicing of multiple paths of videos.
Because the panoramic camera adopts a plurality of cameras to shoot 360-degree space scenes, partial overlapping areas collected by adjacent cameras are utilized to carry out image fusion, and finally, the partially overlapped images collected by the cameras are spliced into a large-scale seamless high-resolution image. The image stitching technology of the panoramic camera greatly depends on the size of an overlapping area between cameras, and the size of the overlapping area collected between adjacent cameras is related to the view field angle of a lens on one hand, and the other hand is limited by the assembly error of the panoramic camera.
At present, how to calibrate the assembly error of the panoramic camera quickly, simply and accurately has no specific calibration method, so a quantifiable calibration method of the assembly error of the panoramic camera is urgently needed, and the allowable error range of the panoramic camera is determined according to a splicing algorithm, so that the industrial production is guided.
The method for calibrating the assembly error of the panoramic camera provided by the invention can better compensate the splicing accuracy problem caused by the assembly error of the camera in the production application of the panoramic camera, thereby improving the splicing effect, and as shown in figure 1, the method specifically comprises the following steps:
step S1, establishing a monocular calibration model;
in step S1, collecting complete calibration board images by using each camera of the panoramic camera, and extracting feature point coordinates of all calibration boards in a plane according to the collected calibration board images of different fields of view;
step S2, establishing a binocular stereo calibration model;
in step S2, acquiring images of binocular stereo calibration plates of different view fields of the panoramic camera, extracting feature point coordinates of all the binocular stereo calibration plates in a plane, and calculating a rotation matrix R and a translational vector T between adjacent cameras in the panoramic camera respectively by using the binocular stereo calibration model;
and step S3, calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model.
It should be noted that, in the monocular calibration model and the binocular stereo calibration model in the present invention, the coordinate mapping relationship between the world coordinate system W-XYZ and the camera coordinate system C-XYZ is represented by the rotation matrix R and the translation vector T, where R and T are external parameters of the camera, and the world coordinate system W-XYZ is obtained after the camera coordinate system C-XYZ is transformed by the translation vector T and the rotation matrix R.
Specifically, in the above method, the specific steps of establishing the monocular calibration model are as follows:
the coordinate of the point P in the world coordinate system is set as Pw＝(Xw，Yw，Zw)TThe coordinate in the camera coordinate system is Pc＝(Xc，Yc，Zc)TAnd they satisfy the following relationship:
Pc＝R×Pw+T (1)
since the camera coordinate system C-XYZ is three-dimensional and the image coordinate system I-XY is two-dimensional, projecting a point in the camera coordinate system into the image coordinate system, the point P having its coordinate P in the camera coordinate systemc＝(Xc，Yc，Zc)TWith the coordinate P ═ x, y in the image coordinate systemTThe following relationship is satisfied:
in formula (2), s is a scale factor of any scale, M is an internal parameter matrix of the camera, and fxIndicating that the camera is at the imaging planeEquivalent focal length in the x-direction of the plane, fyRepresents the equivalent focal length of the camera in the imaging plane y direction, (c)x，cy) Representing the pixel coordinates of the principal point in the x, y direction of the imaging plane.
As described above, the homogeneous coordinate form of the camera calibration model is represented as:
in the formula (3), the first and second groups,
representing the coordinates of the feature points in the image plane coordinate system,
representing spatial feature points under a calibration plate coordinate system;
where W denotes a physical transformation for locating the observed object plane and includes the sum of a partial rotation R and a partial translation T associated with the observed image plane, W ═ R | T.
Fig. 2 is a schematic diagram of a single target calibration board of the present invention, as shown in fig. 2, in step S1, 10 to 15 calibration images are acquired by each camera, and the position and angle of the calibration board image acquired each time are different, so that the calibration result has high accuracy and moderate calculation amount. And extracting the coordinates of the characteristic points of all calibration plates in a plane according to the collected calibration plate images of different view fields, and respectively calculating the internal parameters and the external parameters of each camera in the panoramic camera by using the camera calibration model.
Specifically, in the above method, the specific steps of establishing the binocular stereo calibration model are as follows:
fig. 3 is a schematic view of the binocular stereo calibration board of the present invention, as shown in fig. 3, a binocular stereo calibration board composed of a left sub-calibration board 1 and a right sub-calibration board 2 is established, a world first coordinate system O1-XY and a second coordinate system O2-XY are respectively established for the two sub-calibration boards 1 and 2, wherein the spatial relationship of the two world coordinate systems is represented by a translation vector T0;
two adjacent cameras are adopted to collect images of the binocular stereo calibration plate, the two adjacent cameras are respectively a left camera and a right camera, a point in space is set as P, and coordinates of the two sub calibration plates 1 and 2 in a world coordinate system are respectively X1、X2Then X1And X2The following relationship is satisfied:
X2＝X1-T0 (4)
the coordinate of the space point P under the coordinate system of the left camera is XlThe coordinate under the coordinate system of the right camera is XrThe following conversion relations exist between the two groups:
Xl＝Rl×X1+Tl，Xr＝Rr×X2+Tr(5)
in the formula (5), RlAnd TlDenotes the external parameter, R, of the left camerarAnd TrRepresenting the extrinsic parameters of the right camera.
By eliminating X in formula (4) and formula (5)1，X2The following can be obtained:
Xr＝Rr×Rl -1×Xl-Rr×T0+Tr-Rr×Rl -1×Tl(6)
further, it can be derived that:
R＝Rr×Rl -1，T＝-Rr×T0+Tr-Rr×Rl -1×Tl(7)
in formula (7), R represents a rotation matrix between the left camera coordinate system and the right camera coordinate system, T represents a translation vector between the left camera coordinate system and the right camera coordinate system, and R and T are results of binocular stereo calibration performed between adjacent cameras.
In the method, two sub-calibration plates 1 and 2 in the binocular stereo calibration plate are in a checkerboard form, and the sizes of the checkerboards are the same.
In the method, when the images of the binocular stereo calibration board are collected, the images of the binocular stereo calibration board are shot simultaneously by using the adjacent left and right cameras, the sub-calibration boards 1 and 2 are respectively only shown in the visual field of one camera when the images are collected, and the number of the collected images of the binocular stereo calibration board is 8-12.
Preferably, the left camera is used for collecting the calibration board picture of the sub-calibration board 1, and the right camera is used for collecting the calibration board picture of the sub-calibration board 2.
Fig. 4 is a schematic diagram of the image capturing apparatus of the present invention, and in step S3, a panoramic camera model is used as shown in fig. 4, where O1-xyz and O2-xyz represent the camera coordinate systems of the first camera O1 and the second camera O2, respectively.
In an ideal case, i.e., without assembly error, the angle between the X-axis and the Y-axis and the Z-axis of the coordinate system of the first camera O1 and the second camera O2 is zero degrees, and the angles between the Y-axis and the Z-axis are ninety degrees. In the actual assembly process, the angle between the coordinate systems of the adjacent cameras of the panoramic camera is difficult to ensure to be in an ideal state.
In order to describe the assembly error, on the basis of a binocular stereo calibration model of the panoramic camera, an included angle between coordinate axes of adjacent camera coordinate systems of the panoramic camera is calculated.
Specifically, in step S3, the specific step of calculating the assembly error of the panoramic camera is as follows:
in step S2, a matrix in which a rotation matrix R between adjacent camera coordinate systems of the panoramic camera is 3 × 3 is obtained, the first column of the rotation matrix R represents a rotation vector of the X axis between the adjacent camera coordinate systems, and the rotation vector is represented as Vx(ii) a The second column represents the rotation vector of the Y axis between the adjacent camera coordinate systems, the rotation vector being set to Vy(ii) a The third column indicates the rotation vector of the Z axis between the adjacent camera coordinate systems, which is set to Vz，
Let a standard vector be (0, 0, 1)TThe included angles between the X-axis, the Y-axis and the Z-axis of the coordinate systems of the adjacent cameras are respectively formed by Anglex，Angley，AnglezIndicates, then Anglex，Angley，AnglezThe calculation formula of (a) is as follows:
where dot () represents the dot product operation between two vectors and norm () represents the norm of the vector. And the included angle among the X axis, the Y axis and the Z axis is the assembly error of the panoramic camera.
Preferably, as in the panoramic camera model shown in fig. 4, there are 4 cameras in total, and it is assumed that the rotation matrix between the first camera O1 and the second camera O2 is R12Translation vector is T12The rotation matrix between the second camera O2 and the third camera (not shown in the figure) is R23Translation vector is T23And the rotation matrix between the third camera and the fourth camera (not shown in the figure) is R34Translation vector is T34And the rotation matrix between the fourth camera and the first camera O1 is R41Translation vector is T41。
01 has a point P in the coordinate system1The coordinate of which is P in the 02 coordinate system2And the coordinate in the 03 coordinate system (not shown in the figure) is P3And the coordinate in the 04 coordinate system (not shown in the figure) is P4Then, the following calculation formula is given:
P2＝R12×P1+T12(8)
P3＝R23×P2+T23(9)
P4＝R34×P3+T34(10)
as can be seen from equations (8), (9) and (10):
in the embodiment of the invention, the rotation error of the panoramic camera is ReTranslation error of TeThen, there are:
Re＝R41×R34×R23×R12
Te＝R41×R34×R23×R12+R41×R34+R41+R41×T34+T41
ideally, ReIs a 3 × 3 identity matrix, translation error Te0, due to errors in the assembly of the cameras of the panoramic camera, in practice, the rotation error ReAnd translation error TeNot equal to the ideal value, calculated rotation error ReAnd translation error TeThe magnitude of the deviation from the ideal value may be used to describe the assembly error of the panoramic camera.
The method for calibrating the assembly error of the panoramic camera comprises the steps of firstly, establishing a monocular calibration model, collecting complete calibration plate images by utilizing all cameras of the panoramic camera, and extracting characteristic point coordinates of all calibration plates in a plane according to the collected calibration plate images of different view fields; secondly, establishing a binocular stereo calibration model, acquiring images of binocular stereo calibration plates of different view fields of the panoramic camera, extracting characteristic point coordinates of all the binocular stereo calibration plates in a plane, and respectively calculating a rotation matrix R and a translational vector T between adjacent cameras in the panoramic camera by using the binocular stereo calibration model; and then, calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model. The method calculates the angle between the coordinate axes of the adjacent cameras and the rotation and translation errors of all coordinate systems on the basis of a standard camera calibration model, and the two parameter indexes can accurately and concisely describe the assembly errors of the panoramic camera. Meanwhile, the invention provides a feasible solution method for the assembly error, which can better compensate the splicing accuracy problem caused by the assembly error of the camera in the production application of the panoramic camera, thereby improving the splicing effect.
As another aspect of the present invention, the present invention further provides a system for calibrating assembly errors of a panoramic camera, as shown in fig. 5, the system comprising:
a monocular calibration model establishing module 51, configured to establish a monocular calibration model;
in the monocular calibration model establishing module 51, each camera of the panoramic camera is used to collect a complete calibration plate image, and the feature point coordinates of all the calibration plates in the plane are extracted according to the collected calibration plate images of different fields of view.
A binocular stereo calibration model establishing module 52, configured to establish a binocular stereo calibration model;
in the binocular stereo calibration model establishing module 52, images of binocular stereo calibration plates of different view fields of the panoramic camera are collected, characteristic point coordinates of all the binocular stereo calibration plates in a plane are extracted, and a rotation matrix R and a translational vector T between adjacent cameras in the panoramic camera are respectively calculated by using the binocular stereo calibration model.
And the assembly error calculation module 53 is used for calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model.
The system for calibrating the assembly error of the panoramic camera comprises the following steps of firstly, establishing a monocular calibration model, collecting complete calibration plate images by utilizing all cameras of the panoramic camera, and extracting characteristic point coordinates of all calibration plates in a plane according to the collected calibration plate images with different view fields; secondly, establishing a binocular stereo calibration model, acquiring images of binocular stereo calibration plates of different view fields of the panoramic camera, extracting characteristic point coordinates of all the binocular stereo calibration plates in a plane, and respectively calculating a rotation matrix R and a translational vector T between adjacent cameras in the panoramic camera by using the binocular stereo calibration model; and then, calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model. The method calculates the angle between the coordinate axes of the adjacent cameras and the rotation and translation errors of all coordinate systems on the basis of a standard camera calibration model, and the two parameter indexes can accurately and concisely describe the assembly errors of the panoramic camera. Meanwhile, the invention provides a feasible solution method for the assembly error, which can better compensate the splicing accuracy problem caused by the assembly error of the camera in the production application of the panoramic camera, thereby improving the splicing effect.
It should be noted that, in the monocular calibration model and the binocular stereo calibration model in the present invention, the coordinate mapping relationship between the world coordinate system W-XYZ and the camera coordinate system C-XYZ is represented by the rotation matrix R and the translation vector T, where R and T are external parameters of the camera, and the world coordinate system W-XYZ is obtained after the camera coordinate system C-XYZ is transformed by the translation vector T and the rotation matrix R.
Specifically, the specific steps of establishing the monocular calibration model are as follows:
the coordinate of the point P in the world coordinate system is set as Pw＝(Xw，Yw，Zw)TThe coordinate in the camera coordinate system is Pc＝(Xc，Yc，Zc)TAnd they satisfy the following relationship:
Pc＝R×Pw+T (1)
since the camera coordinate system C-XYZ is three-dimensional and the image coordinate system I-XY is two-dimensional, projecting a point in the camera coordinate system into the image coordinate system, the point P having its coordinate P in the camera coordinate systemc＝(Xc，Yc，Zc)TWith the coordinate P ═ x, y in the image coordinate systemTThe following relationship is satisfied:
in formula (2), s is a scale factor of any scale, M is an internal parameter matrix of the camera, and fxRepresenting the equivalent focal length, f, of the camera in the x-direction of the imaging planeyRepresents the equivalent focal length of the camera in the imaging plane y direction, (c)x，cy) Representing the pixel coordinates of the principal point in the x, y direction of the imaging plane.
As described above, the homogeneous coordinate form of the camera calibration model is represented as:
in the formula (3), the first and second groups,
representing the coordinates of the feature points in the image plane coordinate system,
representing spatial feature points under a calibration plate coordinate system;
where W denotes a physical transformation for locating the observed object plane and includes the sum of a partial rotation R and a partial translation T associated with the observed image plane, W ═ R | T.
Fig. 2 is a schematic diagram of a single-target calibration board of the present invention, as shown in fig. 2, in the above-mentioned single-target calibration model building module 51, 10 to 15 calibration images are acquired by each camera, and the position and angle of the calibration board image acquired each time are different, so that the calibration result has high accuracy and moderate calculation amount. And extracting the coordinates of the characteristic points of all calibration plates in a plane according to the collected calibration plate images of different view fields, and respectively calculating the internal parameters and the external parameters of each camera in the panoramic camera by using the camera calibration model.
Specifically, the specific steps of establishing the binocular stereo calibration model are as follows:
fig. 3 is a schematic view of the binocular stereo calibration board of the present invention, as shown in fig. 3, a binocular stereo calibration board composed of a left sub-calibration board 1 and a right sub-calibration board 2 is established, a world first coordinate system O1-XY and a second coordinate system O2-XY are respectively established for the two sub-calibration boards 1 and 2, wherein the spatial relationship of the two world coordinate systems is represented by a translation vector T0;
two adjacent cameras are adopted to collect images of the binocular stereo calibration plate, the two adjacent cameras are respectively a left camera and a right camera, a point in space is set as P, and coordinates of the two sub calibration plates 1 and 2 in a world coordinate system are respectively X1、X2Then X1And X2Satisfies the following conditionsThe relationship is as follows:
X2＝X1-T0 (4)
the coordinate of the space point P under the coordinate system of the left camera is XlThe coordinate under the coordinate system of the right camera is XrThe following conversion relations exist between the two groups:
Xl＝Rl×X1+Tl，Xr＝Rr×X2+Tr(5)
in the formula (5), RlAnd TlDenotes the external parameter, R, of the left camerarAnd TrRepresenting the extrinsic parameters of the right camera.
By eliminating X in formula (4) and formula (5)1，X2The following can be obtained:
Xr＝Rr×Rl -1×Xl-Rr×T0+Tr-Rr×Rl -1×Tl(6)
further, it can be derived that:
R＝Rr×Rl -1，T＝-Rr×T0+Tr-Rr×Rl -1×Tl(7)
in formula (7), R represents a rotation matrix between the left camera coordinate system and the right camera coordinate system, T represents a translation vector between the left camera coordinate system and the right camera coordinate system, and R and T are results of binocular stereo calibration performed between adjacent cameras.
Two sub-calibration plates 1 and 2 in the binocular stereo calibration plate are in a checkerboard form, and the sizes of the checkerboards are consistent.
When the images of the binocular stereo calibration plates are collected, the images of the binocular stereo calibration plates are shot simultaneously by using the left camera and the right camera which are adjacent, the sub calibration plates 1 and 2 are respectively only arranged in the visual field of one camera when the images are collected, and the number of the collected images of the binocular stereo calibration plates is 8-12.
Preferably, the left camera is used for collecting the calibration board picture of the sub-calibration board 1, and the right camera is used for collecting the calibration board picture of the sub-calibration board 2.
FIG. 4 is a schematic diagram of the image capturing device of the present invention, wherein in the above-mentioned assembly error calculating module 53, a panoramic camera model is adopted as shown in FIG. 4, wherein O1-xyz and O2-xyz represent the camera coordinate systems of the first camera O1 and the second camera O2, respectively;
in an ideal case, i.e., without assembly error, the angle between the X-axis and the Y-axis and the Z-axis of the coordinate system of the first camera O1 and the second camera O2 is zero degrees, and the angles between the Y-axis and the Z-axis are ninety degrees. In the actual assembly process, the angle between the coordinate systems of the adjacent cameras of the panoramic camera is difficult to ensure to be in an ideal state.
In order to describe the assembly error, on the basis of a binocular stereo calibration model of the panoramic camera, an included angle between coordinate axes of adjacent camera coordinate systems of the panoramic camera is calculated.
Specifically, the specific steps of calculating the assembly error of the panoramic camera are as follows:
a matrix with a rotation matrix R of 3X 3 between the coordinate systems of the adjacent cameras of the panoramic camera is obtained through the binocular stereo calibration model establishing module 52, the first column of the rotation matrix R represents a rotation vector of an X axis between the coordinate systems of the adjacent cameras, and the rotation vector is set as Vx(ii) a The second column represents the rotation vector of the Y axis between the adjacent camera coordinate systems, the rotation vector being set to Vy(ii) a The third column indicates the rotation vector of the Z axis between the adjacent camera coordinate systems, which is set to Vz，
Let a standard vector be (0, 0, 1)TThe included angles between the X-axis, the Y-axis and the Z-axis of the coordinate systems of the adjacent cameras are respectively formed by Anglex，Angley，AnglezIndicates, then Anglex，Angley，AnglezThe calculation formula of (a) is as follows:
where dot () represents the dot product operation between two vectors and norm () represents the norm of the vector. And the included angle among the X axis, the Y axis and the Z axis is the assembly error of the panoramic camera.
Preferably, as in the panoramic camera model shown in fig. 4, there are 4 cameras in total, and it is assumed that the rotation matrix between the first camera O1 and the second camera O2 is R12Translation vector is T12The rotation matrix between the second camera O2 and the third camera (not shown in the figure) is R23Translation vector is T23And the rotation matrix between the third camera and the fourth camera (not shown in the figure) is R34Translation vector is T34And the rotation matrix between the fourth camera and the first camera O1 is R41Translation vector is T41。
01 has a point P in the coordinate system1The coordinate of which is P in the 02 coordinate system2And the coordinate in the 03 coordinate system (not shown in the figure) is P3And the coordinate in the 04 coordinate system (not shown in the figure) is P4Then, the following calculation formula is given:
P2＝R12×P1+T12(8)
P3＝R23×P2+T23(9)
P4＝R34×P3+T34(10)
as can be seen from equations (8), (9) and (10):
in the embodiment of the invention, the rotation error of the panoramic camera is ReTranslation error of TeThen, there are:
Re＝R41×R34×R23×R12
Re＝R41×R34×R23×R12+R41×R34×T23+R41×T34+T41
ideally, TeIs a 3 × 3 identity matrix, translation error Te0, due to errors in the assembly of the cameras of the panoramic camera, in practice, the rotation error ReAnd translation error TeNot equal to the ideal value, calculated rotation error ReAnd translation error TeThe magnitude of the deviation from the ideal value may be used to describe the assembly error of the panoramic camera.
It is to be understood that the above-described embodiments of the present invention are merely illustrative of or explaining the principles of the invention and are not to be construed as limiting the invention. Therefore, any modification, equivalent replacement, improvement and the like made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. Further, it is intended that the appended claims cover all such variations and modifications as fall within the scope and boundaries of the appended claims or the equivalents of such scope and boundaries.
Claims (6)
1. A method for calibrating assembly errors of a panoramic camera is characterized by comprising the following steps:
s1, establishing a monocular calibration model; the specific steps for establishing the monocular calibration model are as follows:
the coordinate of the point P in the world coordinate system is set as Pw＝(Xw，Yw，Zw)TThe coordinate in the camera coordinate system is Pc＝(Xc，Yc，Zc)TAnd they satisfy the following relationship:
Pc＝R×Pw+T (1)
projecting a point in a camera coordinate system into an image coordinate system, the point P having a coordinate P in the camera coordinate systemc＝(Xc，Yc，Zc)TWith image coordinatesCoordinate in the system P ═ (x, y)TThe following relationship is satisfied:
in formula (2), s is a scale factor of any scale, M is an internal parameter matrix of the camera, and fxRepresenting the equivalent focal length, f, of the camera in the x-direction of the imaging planeyRepresents the equivalent focal length of the camera in the imaging plane y direction, (c)x，cy) Representing pixel coordinates of the principal point in x, y directions of the imaging plane;
the homogeneous coordinate form of the camera calibration model is expressed as follows:
in the formula (3), the first and second groups,
representing the coordinates of the feature points in the image plane coordinate system,
representing spatial feature points under a calibration plate coordinate system;
where W represents the physical transformation used to locate the observed object plane and includes the sum of the partial rotation R and the partial translation T associated with the observed image plane, and W ═ R | T;
s2, establishing a binocular stereo calibration model; the specific steps of establishing the binocular stereo calibration model are as follows:
establishing a binocular stereo calibration plate consisting of a left sub calibration plate 1 and a right sub calibration plate 2, and respectively establishing a world first coordinate system O1-XY and a second coordinate system O2-XY for the two sub calibration plates 1 and 2, wherein the spatial relationship of the two world coordinate systems is represented by a translation vector T0;
two adjacent cameras are adopted for collecting images of a binocular stereo calibration board, the two adjacent cameras are respectively a left camera and a right camera, and one camera in a space is arrangedThe point is P, and the coordinates of the two sub-calibration plates 1 and 2 in the world coordinate system are X respectively1、X2Then X1And X2The following relationship is satisfied:
X2＝X1-T0 (4)
the coordinate of the space point P under the coordinate system of the left camera is XlThe coordinate under the coordinate system of the right camera is XrThe following conversion relations exist between the two groups:
Xl＝Rl×X1+Tl，Xr＝Rr×X2+Tr(5)
in the formula (5), RlAnd TlDenotes the external parameter, R, of the left camerarAnd TrRepresenting the external parameters of the right camera;
by eliminating X in formula (4) and formula (5)1，X2The following can be obtained:
Xr＝Rr×Rl -1×Xl-Rr×T0+Tr-Rr×Rl -1×Tl(6)
further, it can be derived that:
R＝Rr×Rl -1，T＝-Rr×T0+Tr-Rr×Rl -1×Tl(7)
in the formula (7), R represents a rotation matrix between a left camera coordinate system and a right camera coordinate system, T represents a translation vector between the left camera coordinate system and the right camera coordinate system, and R and T are results of binocular stereo calibration between adjacent cameras;
s3, calculating the assembly error of the panoramic camera according to the monocular calibration model and the binocular stereo calibration model; the specific steps for calculating the assembly error of the panoramic camera are as follows:
in step S2, a matrix in which a rotation matrix R between adjacent camera coordinate systems of the panoramic camera is 3 × 3 is obtained, the first column of the rotation matrix R represents a rotation vector of the X axis between the adjacent camera coordinate systems, and the rotation vector is represented as Vx(ii) a The second column represents the phaseA rotation vector of Y axis between adjacent camera coordinate systems, the rotation vector is set as Vy(ii) a The third column indicates the rotation vector of the Z axis between the adjacent camera coordinate systems, which is set to Vz；
Let a standard vector be (0, 0, 1)TThe included angles between the X-axis, the Y-axis and the Z-axis of the coordinate systems of the adjacent cameras are respectively formed by Anglex，Angley，AnglezIndicates, then Anglex，Angley，AnglezThe calculation formula of (a) is as follows:
wherein dot (t)) represents a dot product operation between two vectors, norm (norm)) represents a mode of the vectors, and an included angle between the X axis, the Y axis and the Z axis is an assembly error of the panoramic camera.
2. The method as claimed in claim 1, wherein in step S1, the complete calibration board image is captured by each camera of the panoramic camera, and the feature point coordinates of all calibration boards in the plane are extracted according to the captured calibration board images of different fields of view.
3. The method of claim 2, wherein 10 to 15 calibration images are acquired by each camera, and the position and angle of the calibration plate image acquired each time are different.
4. The method according to claim 1, wherein in step S2, images of binocular stereo calibration plates of different view fields of the panoramic camera are acquired, coordinates of feature points of all the binocular stereo calibration plates in a plane are extracted, and the rotation matrix R and the translation vector T between adjacent cameras in the panoramic camera are calculated respectively by using the binocular stereo calibration model.
5. The method according to claim 4, wherein the two sub-calibration plates 1, 2 of the binocular stereo calibration plate are in a checkerboard form, and the size of each checkerboard is consistent.
6. The method according to claim 5, wherein when the images of the binocular stereo calibration plates are collected, the images of the binocular stereo calibration plates are simultaneously photographed by using the adjacent left and right cameras, the sub-calibration plates 1 and 2 are respectively only present in the visual field of one camera when the images are collected, and the number of the collected images of the binocular stereo calibration plates is 8 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710828874.XA CN107578450B (en) | 2017-09-14 | 2017-09-14 | Method and system for calibrating assembly error of panoramic camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710828874.XA CN107578450B (en) | 2017-09-14 | 2017-09-14 | Method and system for calibrating assembly error of panoramic camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107578450A CN107578450A (en) | 2018-01-12 |
| CN107578450B true CN107578450B (en) | 2020-04-14 |
Family
ID=61033440
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710828874.XA Active CN107578450B (en) | 2017-09-14 | 2017-09-14 | Method and system for calibrating assembly error of panoramic camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107578450B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109215085B (en) * | 2018-08-23 | 2021-09-17 | 上海小萌科技有限公司 | Article statistical method using computer vision and image recognition |
| CN109410283B (en) * | 2018-09-14 | 2021-09-24 | 高新兴科技集团股份有限公司 | Space calibration device of indoor panoramic camera and positioning device with space calibration device |
| CN111383277B (en) * | 2018-12-29 | 2023-05-19 | 余姚舜宇智能光学技术有限公司 | Wide-interval double-camera module AA method and system |
| CN110779933A (en) * | 2019-11-12 | 2020-02-11 | 广东省智能机器人研究院 | Surface point cloud data acquisition method and system based on 3D visual sensing array |
| CN112612002A (en) * | 2020-12-01 | 2021-04-06 | 北京天地玛珂电液控制系统有限公司 | Digital construction system and method for scene space of full working face under coal mine |
| CN115065782B (en) * | 2022-04-29 | 2023-09-01 | 珠海视熙科技有限公司 | Scene acquisition method, acquisition device, image pickup equipment and storage medium |
| CN116030145A (en) * | 2023-03-23 | 2023-04-28 | 北京中科慧眼科技有限公司 | Stereo matching method and system for binocular lenses with different focal lengths |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106875339A (en) * | 2017-02-22 | 2017-06-20 | 长沙全度影像科技有限公司 | A kind of fish eye images joining method based on strip scaling board |
| CN107052086A (en) * | 2017-06-01 | 2017-08-18 | 扬州苏星机器人科技有限公司 | Stamping parts surface defect detection apparatus and detection method based on 3D vision |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101577795A (en) * | 2009-06-17 | 2009-11-11 | 深圳华为通信技术有限公司 | Method and device for realizing real-time viewing of panoramic picture |
-
2017
- 2017-09-14 CN CN201710828874.XA patent/CN107578450B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106875339A (en) * | 2017-02-22 | 2017-06-20 | 长沙全度影像科技有限公司 | A kind of fish eye images joining method based on strip scaling board |
| CN107052086A (en) * | 2017-06-01 | 2017-08-18 | 扬州苏星机器人科技有限公司 | Stamping parts surface defect detection apparatus and detection method based on 3D vision |
Non-Patent Citations (1)
| Title |
|---|
| 基于FPGA的航空CCD相机图像畸变校正技术研究;徐芳;《中国优秀硕士学位论文全文数据库信息科技辑》;20140815(第8期);第18-20页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107578450A (en) | 2018-01-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107578450B (en) | Method and system for calibrating assembly error of panoramic camera | |
| CN104835117B (en) | Spherical panorama generation method based on overlapping mode | |
| JP4825980B2 (en) | Calibration method for fisheye camera. | |
| CN106709899B (en) | Method, device and equipment for calculating relative positions of two cameras | |
| CN110809786B (en) | Calibration device, calibration chart, chart pattern generation device, and calibration method | |
| CN110782394A (en) | Panoramic video rapid splicing method and system | |
| CN108122191B (en) | Method and device for splicing fisheye images into panoramic image and panoramic video | |
| CN108648237B (en) | Space positioning method based on vision | |
| CN105809701B (en) | Panoramic video posture scaling method | |
| CN103337094A (en) | Method for realizing three-dimensional reconstruction of movement by using binocular camera | |
| CN108734738B (en) | Camera calibration method and device | |
| JP7218435B2 (en) | CALIBRATION DEVICE, CALIBRATION CHART AND CALIBRATION METHOD | |
| CN109345587B (en) | Hybrid vision positioning method based on panoramic vision and monocular vision | |
| CN103971375A (en) | Panoramic gaze camera space calibration method based on image splicing | |
| CN108234989B (en) | convergent integrated imaging shooting method based on checkerboard calibration plate | |
| CN107038714B (en) | Multi-type visual sensing cooperative target tracking method | |
| CN113329179B (en) | Shooting alignment method, device, equipment and storage medium | |
| CN102692806A (en) | Methods for acquiring and forming free viewpoint four-dimensional space video sequence | |
| CN113450416B (en) | TCSC method applied to three-dimensional calibration of three-dimensional camera | |
| TWI501193B (en) | Computer graphics using AR technology. Image processing systems and methods | |
| WO2018000892A1 (en) | Imaging method, apparatus and system for panoramic stereo image | |
| CN103546680B (en) | A kind of deformation-free omni-directional fisheye photographic device and a method for implementing the same | |
| JP7033294B2 (en) | Imaging system, imaging method | |
| JP2005275789A (en) | Three-dimensional structure extraction method | |
| CN111091595B (en) | Strabismus three-dimensional mapping method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |