CN112629565B - Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit - Google Patents

Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit Download PDF

Info

Publication number
CN112629565B
CN112629565B CN202110249005.8A CN202110249005A CN112629565B CN 112629565 B CN112629565 B CN 112629565B CN 202110249005 A CN202110249005 A CN 202110249005A CN 112629565 B CN112629565 B CN 112629565B
Authority
CN
China
Prior art keywords
image
matrix
relation
equation set
polynomial equation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110249005.8A
Other languages
Chinese (zh)
Other versions
CN112629565A (en
Inventor
关棒磊
余英建
孙祥一
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
National University of Defense Technology
Original Assignee
National University of Defense Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National University of Defense Technology filed Critical National University of Defense Technology
Priority to CN202110249005.8A priority Critical patent/CN112629565B/en
Publication of CN112629565A publication Critical patent/CN112629565A/en
Application granted granted Critical
Publication of CN112629565B publication Critical patent/CN112629565B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C25/00—Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
    • G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
    • G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
    • G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/20—Instruments for performing navigational calculations
    • G06T3/02—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image

Abstract

The application relates to a method, a device and equipment for calibrating the rotation relation between a camera and an inertial measurement unit. The method comprises the following steps: acquiring a first image and a second image shot by a camera, and extracting affine matching point pairs of the first image and the second image; establishing a second relation between the homography matrix and the first homogeneous coordinate and the second homogeneous coordinate according to the first relation between the homography matrix of the first image and the second image and the rotation matrix and the translation vector; establishing a third relation between the affine transformation matrix and the rotation matrix according to the affine transformation matrix; obtaining a polynomial equation set according to the second relation and the third relation; acquiring initial installation parameters, and performing first-order approximation on a polynomial equation set to obtain a simplified polynomial equation set with the order of 2; and solving the simplified polynomial equation set by adopting a resolver to obtain a rotation matrix to be calibrated and a translation vector. By adopting the method, the rapid correction of the rotation relation can be realized.

Description

Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit
Technical Field
The application relates to the technical field of non-contact measurement, in particular to a method, a device and equipment for calibrating the rotation relation between a camera and an inertial measurement unit.
Background
Multi-sensor fusion, particularly the fusion of a camera and an Inertial Measurement Unit (IMU), has a wide range of applications in the fields of computer vision and optical measurement, such as autonomous navigation and automatic landing and simultaneous localization and mapping (SLAM). In these applications, successful fusion of the camera and IMU data requires precise alignment of the spatiotemporal relationship between the camera and IMU, and alignment errors can reduce the fusion accuracy of the sensors, affecting the final measurement. Therefore, precise alignment of the spatiotemporal relationship between the camera and the IMU is a fundamental task that plays an important role in sensor fusion and measurement.
The conventional calibration method treats the alignment of the camera IMU system as a hand-eye calibration problem because the IMU directly outputs rotation information relative to the IMU reference frame, but this method requires the camera pose to be recovered in advance. In addition, methods which do not need known calibration targets and camera attitude prior information are also provided, the relative position is directly calculated from feature matching, and the accuracy of the methods is greatly influenced by feature mismatch because the methods do not have a mechanism for processing outliers. A more recent approach describes a method for solving the minimum solution of rotational alignment of an IMU camera system using homography constraint, which directly calculates the rotational alignment of the IMU system of the camera, avoiding calculating the camera pose or camera relative pose, and uses RANSAC (RANdomS multiple Consensions, Random sampling consistency algorithm) framework (marking A Fishler and Robert C balls. Random sampling consistency sensing: A partial for model fixing with automatic imaging analysis and automatic mapping. communication of the ACM, 24(6):381 and 395, 1981.), but which requires at least three image coordinate points to be solved.
Disclosure of Invention
In view of the above, there is a need to provide a method, an apparatus, a computer device and a storage medium for calibrating a rotation relationship between a camera and an Inertial Measurement Unit (IMU) with high accuracy and high efficiency without requiring a known calibration apparatus, without restricting the motion of the camera, and without calculating the attitude of the camera in advance.
A method of calibrating a camera to inertial measurement unit rotational relationship, the method comprising:
acquiring a first image and a second image shot by a camera, and extracting affine matching point pairs of the first image and the second image; each pair of affine matching point pairs comprises a first homogeneous coordinate of a first image, a second homogeneous coordinate of an image point in a second image and an affine transformation matrix;
establishing a second relation between the homography matrix and the first and second homogeneous coordinates according to a first relation between the homography matrix of the first and second images and a rotation matrix and a translation vector for converting the first image into the second image;
establishing a third relationship of the affine transformation matrix and the rotation matrix based on the homography matrix according to the affine transformation matrix in the first image and the second image;
obtaining a polynomial equation set according to the second relation and the third relation; the order of the polynomial equation set is 6, and the unknowns are 3 rotation parameters and 3 translation parameters;
acquiring initial installation parameters between an inertia measurement unit and a camera, and performing first-order approximation on the polynomial equation set according to the initial installation parameters to obtain a simplified polynomial equation set with the order of 2;
and solving the simplified polynomial equation set by adopting a resolver to obtain a rotation relation matrix to be calibrated and a translation vector.
In one embodiment, the method further comprises the following steps: acquiring a plurality of matching interior point pairs in the first image and the second image;
calculating a reprojection error according to the matching interior point pairs and the homography matrix;
and constructing a conversion error cost function according to the reprojection error, and minimizing the error cost function by taking the corrected rotation matrix as an optimization parameter to obtain an optimized rotation matrix.
In one embodiment, the method further comprises the following steps: acquiring the homography of the first image and the second image as follows:
Figure 723244DEST_PATH_IMAGE001
wherein the content of the first and second substances,
Figure 289355DEST_PATH_IMAGE002
respectively representing a first homogeneous coordinate of an image point in the first image and a second homogeneous coordinate of an image point in the second image, H representing a homography matrix, R representing a rotation matrix, t representing a translation vector,dexpressed as the distance of the first image coordinate system to the 3D ground,
Figure 591023DEST_PATH_IMAGE003
representing a unit normal vector of the 3D ground in a first image coordinate system;
according to the homography, obtaining a first relation among a homography matrix, a rotation matrix and a translation vector, wherein the first relation is as follows:
Figure 420439DEST_PATH_IMAGE004
Figure 807558DEST_PATH_IMAGE005
wherein the content of the first and second substances,
Figure 177359DEST_PATH_IMAGE006
represents the rotation matrix calibrated for R, T represents the translation vector calibrated for T,
Figure 333534DEST_PATH_IMAGE007
representing the angular information output by the inertial measurement unit when the camera takes the first image,
Figure 458485DEST_PATH_IMAGE008
representing the angular information output by the inertial measurement unit when the camera takes the second image,
Figure 67321DEST_PATH_IMAGE009
representing a unit normal vector of the 3D ground in a first image coordinate system after calibration;
according to the first relation, establishing a second relation among the homography matrix, the first homogeneous coordinate and the second homogeneous coordinate as follows:
Figure 506392DEST_PATH_IMAGE010
wherein the content of the first and second substances,
Figure 720336DEST_PATH_IMAGE011
is that
Figure 688292DEST_PATH_IMAGE012
First, the
Figure 50003DEST_PATH_IMAGE013
Line and first
Figure 496028DEST_PATH_IMAGE014
Elements of a column
Figure 456156DEST_PATH_IMAGE015
。
In one embodiment, the method further comprises the following steps: obtaining affine matching point pairs in the first image and the second image
Figure 595013DEST_PATH_IMAGE016
(ii) a Wherein the content of the first and second substances,
Figure 444021DEST_PATH_IMAGE017
and
Figure 693737DEST_PATH_IMAGE018
is a corresponding point pair in the form of homogeneous coordinates in the first image and the second image, a represents an affine transformation matrix;
according to the homography matrix, establishing a third relation between the affine transformation matrix and the rotation matrix as follows:
Figure 882272DEST_PATH_IMAGE019
wherein the content of the first and second substances,
Figure 192031DEST_PATH_IMAGE020
is that
Figure 528334DEST_PATH_IMAGE021
To (1) a
Figure 909637DEST_PATH_IMAGE022
Line and first
Figure 749417DEST_PATH_IMAGE023
Column element
Figure 495656DEST_PATH_IMAGE024
，
Figure 788097DEST_PATH_IMAGE025
The direction of the first image is such that,
Figure 848457DEST_PATH_IMAGE026
is a direction in the second image;
Figure 542744DEST_PATH_IMAGE027
a projection of the depth is represented and,
Figure 459884DEST_PATH_IMAGE028
take a value of
Figure 505201DEST_PATH_IMAGE029
The last row of (2).
In one embodiment, the method further comprises the following steps: using Euler angles
Figure 228306DEST_PATH_IMAGE030
Parameterizing the rotation matrix to obtain a fourth relation:
Figure 777099DEST_PATH_IMAGE031
(ii) a Obtaining a polynomial equation set according to the second relation, the third relation and the fourth relation, wherein the polynomial equation set is:
Figure 130720DEST_PATH_IMAGE032
wherein the content of the first and second substances,
Figure 397753DEST_PATH_IMAGE033
。
in one embodiment, the method further comprises the following steps: obtaining initial setup parameters between an inertial measurement unit and a camera
Figure 534337DEST_PATH_IMAGE034
；
Obtaining the relationship between the initial installation parameters and the rotation matrix based on the rotation matrix of the small angle as follows:
Figure 468795DEST_PATH_IMAGE035
wherein the content of the first and second substances,
Figure 727738DEST_PATH_IMAGE036
is a rotation matrix based on small angles;
to pair
Figure 544384DEST_PATH_IMAGE037
A first order approximation is performed to yield:
Figure 546975DEST_PATH_IMAGE038
wherein the content of the first and second substances,
Figure 335939DEST_PATH_IMAGE039
is formed by three-dimensional vectors
Figure 500204DEST_PATH_IMAGE040
The formed anti-symmetric matrix is a matrix with a plurality of anti-symmetric matrixes,
Figure 945092DEST_PATH_IMAGE041
is an identity matrix;
according to
Figure 751374DEST_PATH_IMAGE042
The first relationship is transformed to obtain:
Figure 129266DEST_PATH_IMAGE043
according to the converted first relation, performing first order approximation on the polynomial equation set to obtain a simplified polynomial equation set with the order of 2:
Figure 568162DEST_PATH_IMAGE044
。
in one embodiment, the method further comprises the following steps: converting each solution of the optimized polynomial equation set into an output homography matrix;
and judging interior points between the matching point pairs of the images after the movement according to the output homography matrix.
A camera and inertial measurement unit rotational relationship calibration apparatus, the apparatus comprising:
the data acquisition module is used for acquiring a first image and a second image shot by the camera and extracting affine matching point pairs of the first image and the second image; each pair of affine matching point pairs comprises a first homogeneous coordinate of a first image, a second homogeneous coordinate of an image point in a second image and an affine transformation matrix;
the second relation establishment module is used for establishing a second relation between the homography matrix and the first homogeneous coordinate and the second homogeneous coordinate according to the first relation between the homography matrix of the first image and the second image and the rotation matrix and translation vector of the transformation from the first image to the second image;
a third relation establishing module, configured to establish a third relation between the affine transformation matrix and the rotation matrix based on the homography matrix according to the affine transformation matrix in the first image and the second image;
the equation set establishing module is used for obtaining a polynomial equation set according to the second relation and the third relation; the order of the polynomial equation set is 6, and the unknowns are 3 rotation parameters and 3 translation parameters;
the system comprises an equation set optimization module, a polynomial equation set optimization module and a camera, wherein the equation set optimization module is used for acquiring initial installation parameters between an inertia measurement unit and the camera and performing first-order approximation on the polynomial equation set according to the initial installation parameters to obtain a simplified polynomial equation set with the order of 2;
and the resolving module is used for solving the simplified polynomial equation set by adopting a resolver to obtain a rotation matrix to be calibrated and a translation vector.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring a first image and a second image shot by a camera, and extracting affine matching point pairs of the first image and the second image; each pair of affine matching point pairs comprises a first homogeneous coordinate of a first image, a second homogeneous coordinate of an image point in a second image and an affine transformation matrix;
establishing a second relation between the homography matrix and the first and second homogeneous coordinates according to a first relation between the homography matrix of the first and second images and a rotation matrix and a translation vector for converting the first image into the second image;
establishing a third relationship of the affine transformation matrix and the rotation matrix based on the homography matrix according to the affine transformation matrix in the first image and the second image;
obtaining a polynomial equation set according to the second relation and the third relation; the order of the polynomial equation set is 6, and the unknowns are 3 rotation parameters and 3 translation parameters;
acquiring initial installation parameters between an inertia measurement unit and a camera, and performing first-order approximation on the polynomial equation set according to the initial installation parameters to obtain a simplified polynomial equation set with the order of 2;
and solving the simplified polynomial equation set by adopting a resolver to obtain a rotation matrix to be calibrated and a translation vector.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring a first image and a second image shot by a camera, and extracting affine matching point pairs of the first image and the second image; each pair of affine matching point pairs comprises a first homogeneous coordinate of a first image, a second homogeneous coordinate of an image point in a second image and an affine transformation matrix;
establishing a second relation between the homography matrix and the first and second homogeneous coordinates according to a first relation between the homography matrix of the first and second images and a rotation matrix and a translation vector for converting the first image into the second image;
establishing a third relationship of the affine transformation matrix and the rotation matrix based on the homography matrix according to the affine transformation matrix in the first image and the second image;
obtaining a polynomial equation set according to the second relation and the third relation; the order of the polynomial equation set is 6, and the unknowns are 3 rotation parameters and 3 translation parameters;
acquiring initial installation parameters between an inertia measurement unit and a camera, and performing first-order approximation on the polynomial equation set according to the initial installation parameters to obtain a simplified polynomial equation set with the order of 2;
and solving the simplified polynomial equation set by adopting a resolver to obtain a rotation matrix to be calibrated and a translation vector.
According to the camera and inertia measurement unit rotation relation calibration method and device, the computer equipment and the storage medium, the solution of the calibration problem can be completed by introducing the affine matching point pairs and only one affine matching point pair, the calibration speed can be obviously improved, and calibration objects with known structures and other special equipment are not needed, so that the process is simple.
Drawings
FIG. 1 is a schematic flow chart of a method for calibrating the rotational relationship between a camera and an inertial measurement unit in one embodiment;
FIG. 2 is a schematic diagram of coordinate transformation in one embodiment;
FIG. 3 is a block diagram of an embodiment of a camera to inertial measurement unit rotational relationship calibration apparatus;
FIG. 4 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In one embodiment, as shown in fig. 1, there is provided a method for calibrating a rotational relationship between a camera and an inertial measurement unit, comprising the steps of:
and 102, acquiring a first image and a second image shot by the camera, and extracting affine matching point pairs of the first image and the second image.
Each pair of affine matching point pairs comprises a first homogeneous coordinate of the first image and a second homogeneous coordinate of the image point in the second image, and an affine transformation matrix.
And 104, establishing an equation second relation between the homography matrix and the first and second homogeneous coordinates according to the first relation between the homography matrix of the first and second images and the rotation matrix and translation vector of the transformation from the first image to the second image.
And 106, establishing a third relation between the affine transformation matrix and the rotation matrix based on the homography matrix according to the affine transformation matrix in the first image and the second image.
And 108, obtaining a polynomial equation set according to the second relation and the third relation.
The order of the polynomial equation set is 6, the unknowns are 3 rotation parameters and 3 translation parameters, and due to the complexity of the polynomial equation set, the polynomial equation set needs to be further optimized.
And step 110, acquiring initial installation parameters between the inertia measurement unit and the camera, and performing first-order approximation on the polynomial equation set according to the initial installation parameters to obtain a simplified polynomial equation set with the order of 2.
And 112, solving the simplified polynomial equation set by using a resolver to obtain a rotation matrix to be calibrated and a translation vector.
According to the camera and inertia measurement unit rotation relation calibration method, by introducing the affine matching point pairs, only one affine matching point pair can complete the solution of the calibration problem, the calibration speed can be obviously improved, and calibration objects with known structures and other special equipment are not needed, and the process is simple.
In one embodiment, the homography for obtaining the first image and the second image is:
Figure 31504DEST_PATH_IMAGE045
wherein the content of the first and second substances,
Figure 641477DEST_PATH_IMAGE046
respectively representing a first homogeneous coordinate of an image point in the first image and a second homogeneous coordinate of an image point in the second image, H representing a homography matrix, R representing a rotation matrix, t representing a translation vector,dexpressed as the distance of the first image coordinate system to the 3D ground,
Figure 139454DEST_PATH_IMAGE047
representing a unit normal vector of the 3D ground in a first image coordinate system;
according to the homography, obtaining a first relation among a homography matrix, a rotation matrix and a translation vector, wherein the first relation is as follows:
Figure 848784DEST_PATH_IMAGE048
Figure 65002DEST_PATH_IMAGE049
wherein the content of the first and second substances,
Figure 213087DEST_PATH_IMAGE050
represents the rotation matrix calibrated for R, T represents the translation vector calibrated for T,
Figure 565571DEST_PATH_IMAGE051
representing the angular information output by the inertial measurement unit when the camera takes the first image,
Figure 570436DEST_PATH_IMAGE052
representing the angular information output by the inertial measurement unit when the camera takes the second image,
Figure 273949DEST_PATH_IMAGE053
representing a unit normal vector of the 3D ground in a first image coordinate system after calibration;
according to the first relation, establishing a second relation among the homography matrix, the first homogeneous coordinate and the second homogeneous coordinate as follows:
Figure 960146DEST_PATH_IMAGE054
wherein the content of the first and second substances,
Figure 370398DEST_PATH_IMAGE055
is that
Figure 483848DEST_PATH_IMAGE056
First, the
Figure 674658DEST_PATH_IMAGE057
Line and first
Figure 898966DEST_PATH_IMAGE058
Elements of a column
Figure 553938DEST_PATH_IMAGE059
。
Specifically, as shown in fig. 2, the inertial measurement unit is mounted in conjunction with a camera, the movement of which includes translation and rotation.
Figure 838289DEST_PATH_IMAGE060
Respectively representing the camera coordinate system and the inertial measurement unit coordinate system, from
Figure 250816DEST_PATH_IMAGE061
To
Figure 278815DEST_PATH_IMAGE062
Is a rotational transformation matrix of
Figure 663660DEST_PATH_IMAGE063
。
Figure 384491DEST_PATH_IMAGE064
Representing the inertial measurement unit reference frame. From the view
Figure 284314DEST_PATH_IMAGE065
(first image) and
Figure 178320DEST_PATH_IMAGE066
(second image) to
Figure 479989DEST_PATH_IMAGE067
Can be expressed as
Figure 106142DEST_PATH_IMAGE068
And
Figure 430944DEST_PATH_IMAGE069
。
Figure 800746DEST_PATH_IMAGE070
is composed of
Figure 222500DEST_PATH_IMAGE071
To and
Figure 19554DEST_PATH_IMAGE072
and (5) coordinate systems with consistent directions. After transformation, only the translation component is needed to represent the aligned view
Figure 457751DEST_PATH_IMAGE073
And
Figure 365664DEST_PATH_IMAGE074
to be transformed between. Fig. 2 shows the original image pair on the left and the corrected image pair on the right.
Figure 641925DEST_PATH_IMAGE075
Is a unit normal vector of a plane
Figure 813143DEST_PATH_IMAGE076
Transformed into views relative to after calibration
Figure 909275DEST_PATH_IMAGE077
The value of (c). After transformation, the homography between the two images is as follows
Figure 886459DEST_PATH_IMAGE078
Wherein
Figure 17226DEST_PATH_IMAGE079
。
In one embodiment, affine matching point pairs in the first image and the second image are obtained
Figure 483979DEST_PATH_IMAGE080
(ii) a Wherein the content of the first and second substances,
Figure 801828DEST_PATH_IMAGE081
and
Figure 582702DEST_PATH_IMAGE082
is a corresponding point pair in the form of homogeneous coordinates in the first image and the second image, a represents an affine transformation matrix;
according to the homography matrix, establishing a third relation between the affine transformation matrix and the rotation matrix as follows:
Figure 771238DEST_PATH_IMAGE083
wherein the content of the first and second substances,
Figure 80997DEST_PATH_IMAGE084
is that
Figure 151721DEST_PATH_IMAGE085
To (1) a
Figure 205127DEST_PATH_IMAGE086
Line and first
Figure 372804DEST_PATH_IMAGE087
Column element
Figure 853464DEST_PATH_IMAGE088
，
Figure 677063DEST_PATH_IMAGE089
And
Figure 268581DEST_PATH_IMAGE090
is the first
Figure 431709DEST_PATH_IMAGE091
Directions in the individual images, i.e. the first two values of the homogeneous coordinates of the image points;
Figure 83271DEST_PATH_IMAGE092
a projection of the depth is represented and,
Figure 128587DEST_PATH_IMAGE093
take a value of
Figure 851692DEST_PATH_IMAGE094
The last row of (2).
Specifically, the local Affine transformation can be extracted by ASIFT (j. -m. Morel and g. Yu, "ASIFT: a New Framework for full affinity investigation Image company," SIAM Journal on Imaging Sciences 2, 438-:
Figure 666065DEST_PATH_IMAGE095
in one embodiment, Euler angles are used
Figure 754106DEST_PATH_IMAGE096
Parameterizing the rotation matrix to obtain a fourth relation:
Figure 21140DEST_PATH_IMAGE097
；
obtaining a polynomial equation set according to the second relation, the third relation and the fourth relation, wherein the polynomial equation set is as follows:
Figure 423302DEST_PATH_IMAGE098
wherein the content of the first and second substances,
Figure 92181DEST_PATH_IMAGE099
。
however, due to the complexity of the polynomial equation set, the maximum order of the Gr baby-ner base solver (Z. Kukelova, M. Bujnak, and T. Pajdla, "Automatic Generator of minor Problim solutions," Computer Vision-Eccv 2008, Pt Iii, Proceedings 5304, 302-. Therefore, there is a need to find a faster, more stable solution.
In practice, the approximate mounting relationship between the IMU and the camera may be measured manually or by an equipment manualAnd (4) obtaining. Thus, the rotational alignment relationship between the camera and the IMU may be expressed as an approximate rotation matrix resulting from multiplying the approximate mounting angle by a small angle based rotation matrix, thus in one embodiment, obtaining initial mounting parameters between the inertial measurement unit and the camera
Figure 351124DEST_PATH_IMAGE100
；
Based on the rotation matrix of the small angle, the relation between the obtained installation parameters and the rotation matrix is as follows:
Figure 105453DEST_PATH_IMAGE101
wherein the content of the first and second substances,
Figure 934476DEST_PATH_IMAGE102
is a rotation matrix based on small angles;
to pair
Figure 723440DEST_PATH_IMAGE103
A first order approximation is performed to yield:
Figure 887705DEST_PATH_IMAGE104
wherein the content of the first and second substances,
Figure 332593DEST_PATH_IMAGE105
is formed by three-dimensional vectors
Figure 873296DEST_PATH_IMAGE106
The formed anti-symmetric matrix is a matrix with a plurality of anti-symmetric matrixes,
Figure 516767DEST_PATH_IMAGE107
is an identity matrix;
according to
Figure 117512DEST_PATH_IMAGE108
To convert said first relationship to obtain a correctionHomography matrix:
Figure 908751DEST_PATH_IMAGE109
according to the converted first relation, performing first order approximation on the polynomial equation set to obtain a simplified polynomial equation set with the order of 2:
Figure 253145DEST_PATH_IMAGE110
。
the simplified polynomial equation set has a maximum order of 2 and a maximum solution number of 19, and may be solved faster and more stably by the Gr baby basis solver than the polynomial equation set.
In the RANSAC framework, the modified homography is used to convert each solution into a corresponding homography, which is used to determine inliers between pairs of matched points of the moving image, and to select the solution that yields the largest set of inliers as output. For each image pair, the rotation matrix from the camera coordinates to the IMU coordinates can be calculated as:
Figure 751122DEST_PATH_IMAGE111
wherein
Figure 726031DEST_PATH_IMAGE112
Is that
Figure 676670DEST_PATH_IMAGE113
Orthogonal form of (a). Furthermore, the motion of the camera can also be recovered, i.e. the rotation matrix and translation vector between the two views can be expressed as:
Figure 824754DEST_PATH_IMAGE114
Figure 177238DEST_PATH_IMAGE115
in the formula (I), the compound is shown in the specification,
Figure 447682DEST_PATH_IMAGE116
，
Figure 885617DEST_PATH_IMAGE117
calculated by the Grubner-based solver
Figure 837393DEST_PATH_IMAGE118
The value is obtained.
In one embodiment, a plurality of matching interior point pairs in the first image and the second image are obtained;
calculating a reprojection error according to the matching interior point pairs and the homography matrix;
and constructing a conversion error cost function according to the reprojection error, and minimizing the error cost function by taking the corrected rotation matrix as an optimization parameter to obtain an optimized rotation matrix.
In particular, for each image pair,
Figure 44383DEST_PATH_IMAGE119
and corresponding interior points
Figure 361095DEST_PATH_IMAGE120
Can be obtained by the single affine matching point pair method and RANSAC. For each image pair
Figure 286326DEST_PATH_IMAGE121
，
Figure 776213DEST_PATH_IMAGE122
And
Figure 431185DEST_PATH_IMAGE123
can be combined with
Figure 449957DEST_PATH_IMAGE124
The interior points are obtained together, and the reprojection error can be calculated
Figure 128063DEST_PATH_IMAGE125
Is calculated, wherein
Figure 156061DEST_PATH_IMAGE126
Is each image pair
Figure 540906DEST_PATH_IMAGE127
The restored homography matrix. The minimization of the total conversion error cost function is as follows:
Figure 996158DEST_PATH_IMAGE128
wherein
Figure 895981DEST_PATH_IMAGE129
Is a three parameter rotation estimate for optimization. The initial value is set to the average value of the calibration results obtained in the previous step.
Figure 462092DEST_PATH_IMAGE130
Is a view
Figure 593121DEST_PATH_IMAGE131
And
Figure 484854DEST_PATH_IMAGE132
the conversion between the first and the second,
Figure 606394DEST_PATH_IMAGE133
is the information from the IMU. Homography
Figure 179457DEST_PATH_IMAGE134
Is a transformation model which transforms the views
Figure 335632DEST_PATH_IMAGE135
To view in homogeneous image coordinates
Figure 398266DEST_PATH_IMAGE136
Corresponding image coordinates.
In order to reduce the influence of potential abnormal values, homography conversion error of points in image matching by Cauchy function is adopted
Figure 7102DEST_PATH_IMAGE137
And (3) weighting:
Figure 242911DEST_PATH_IMAGE138
wherein the Cauchy function
Figure 253593DEST_PATH_IMAGE139
The parameter may be set to an internal threshold of the RANSAC cycle.
In summary, the invention provides a method for calibrating the rotation relationship between the camera coordinate system and the inertial measurement unit coordinate system by directly utilizing the homography constraint of the image matching point pair, does not depend on calibration objects with known structures and other special equipment, does not need to calculate the camera attitude in advance, provides the minimum configuration solution for solving the rotation relationship between the camera and the inertial measurement unit, and improves the efficiency of eliminating the image matching point pair midrange value by the RANSAC algorithm.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 1 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 3, there is provided a camera and inertial measurement unit rotational relationship calibration apparatus comprising: a data acquisition module 302, a second relationship establishing module 304, a third relationship establishing module 306, an equation set establishing module 308, an equation set optimizing module 310, and a calculation module 312, wherein:
a data obtaining module 302, configured to obtain a first image and a second image captured by a camera, and extract an affine matching point pair of the first image and the second image; each pair of affine matching point pairs comprises a first homogeneous coordinate of a first image, a second homogeneous coordinate of an image point in a second image and an affine transformation matrix;
a second relation establishing module 304, configured to establish a second relation between the homography matrix and the first and second homogeneous coordinates according to a first relation between the homography matrix of the first and second images and a rotation matrix and a translation vector of a transformation from the first image to the second image;
a third relation establishing module 306, configured to establish a third relation between the affine transformation matrix and the rotation matrix based on the homography matrix according to the affine transformation matrix in the first image and the second image;
an equation set establishing module 308, configured to obtain a polynomial equation set according to the second relationship and the third relationship; the order of the polynomial equation set is 6, and the unknowns are 3 rotation parameters and 3 translation parameters;
the equation set optimization module 310 is configured to obtain initial installation parameters between the inertial measurement unit and the camera, and perform first-order approximation on the polynomial equation set according to the initial installation parameters to obtain a simplified polynomial equation set with an order of 2;
and the resolving module 312 is configured to solve the simplified polynomial equation set by using a resolver to obtain a rotation matrix to be calibrated and a translation vector.
In one embodiment, the method further comprises the following steps: the nonlinear optimization module is used for acquiring a plurality of matching interior point pairs in the first image and the second image; calculating a reprojection error according to the matching interior point pairs and the homography matrix; and constructing a conversion error cost function according to the reprojection error, and minimizing the error cost function by taking the corrected rotation matrix as an optimization parameter to obtain an optimized rotation matrix.
In one embodiment, the second relationship establishing module 304 is further configured to obtain a homography of the first image and the second image as follows:
Figure 487128DEST_PATH_IMAGE140
wherein the content of the first and second substances,
Figure 583260DEST_PATH_IMAGE141
respectively representing a first homogeneous coordinate of an image point in the first image and a second homogeneous coordinate of an image point in the second image, H representing a homography matrix, R representing a rotation matrix, t representing a translation vector,dexpressed as the distance of the first image coordinate system to the 3D ground,
Figure 498126DEST_PATH_IMAGE142
representing a unit normal vector of the 3D ground in a first image coordinate system;
according to the homography, obtaining a first relation among a homography matrix, a rotation matrix and a translation vector, wherein the first relation is as follows:
Figure 363314DEST_PATH_IMAGE143
Figure 767751DEST_PATH_IMAGE144
wherein the content of the first and second substances,
Figure 351179DEST_PATH_IMAGE145
represents the rotation matrix calibrated for R, T represents the translation vector calibrated for T,
Figure 194370DEST_PATH_IMAGE146
representing the angular information output by the inertial measurement unit when the camera takes the first image,
Figure 179643DEST_PATH_IMAGE147
representing the angular information output by the inertial measurement unit when the camera takes the second image,
Figure 489402DEST_PATH_IMAGE148
representing a unit normal vector of the 3D ground in a first image coordinate system after calibration;
according to the first relation, establishing a second relation among the homography matrix, the first homogeneous coordinate and the second homogeneous coordinate as follows:
Figure 763388DEST_PATH_IMAGE149
wherein the content of the first and second substances,
Figure 816795DEST_PATH_IMAGE150
is that
Figure 922154DEST_PATH_IMAGE151
First, the
Figure 402814DEST_PATH_IMAGE152
Line and first
Figure 288731DEST_PATH_IMAGE153
Elements of a column
Figure 880249DEST_PATH_IMAGE154
。
In one embodiment, the third relation establishing module 306 is configured to obtain affine matching point pairs in the first image and the second image
Figure 840115DEST_PATH_IMAGE155
(ii) a Wherein the content of the first and second substances,
Figure 960518DEST_PATH_IMAGE156
and
Figure 740255DEST_PATH_IMAGE157
is a corresponding point pair in the form of homogeneous coordinates in the first image and the second image, a represents an affine transformation matrix;
according to the homography matrix, establishing a third relation between the affine transformation matrix and the rotation matrix as follows:
Figure 401043DEST_PATH_IMAGE158
wherein the content of the first and second substances,
Figure 215415DEST_PATH_IMAGE159
is that
Figure 864309DEST_PATH_IMAGE160
To (1) a
Figure 131342DEST_PATH_IMAGE161
Line and first
Figure 595822DEST_PATH_IMAGE162
Column element
Figure 264701DEST_PATH_IMAGE163
，
Figure 461327DEST_PATH_IMAGE164
And
Figure 215656DEST_PATH_IMAGE165
is the first
Figure 218247DEST_PATH_IMAGE166
A direction in the individual image;
Figure 69529DEST_PATH_IMAGE167
a projection of the depth is represented and,
Figure 499373DEST_PATH_IMAGE168
take a value of
Figure 6578DEST_PATH_IMAGE169
Is the most important ofThe latter row.
The equation set establishing module 308 is further configured to employ the Euler angles
Figure 281701DEST_PATH_IMAGE170
Parameterizing the rotation matrix to obtain a fourth relation:
Figure 128434DEST_PATH_IMAGE171
；
obtaining a polynomial equation set according to the second relation and the third relation, including:
obtaining a polynomial equation set according to the second relation, the third relation and the fourth relation, wherein the polynomial equation set is:
Figure 463601DEST_PATH_IMAGE172
wherein the content of the first and second substances,
Figure 458102DEST_PATH_IMAGE173
。
in one embodiment, the equation set optimization module 310 is further configured to obtain mounting parameters between the inertial measurement unit and the camera
Figure 536916DEST_PATH_IMAGE174
；
Obtaining the relationship between the initial installation parameters and the rotation matrix based on the rotation matrix of the small angle as follows:
Figure 362790DEST_PATH_IMAGE175
wherein the content of the first and second substances,
Figure 400016DEST_PATH_IMAGE176
is a rotation matrix based on small angles;
to pair
Figure 350654DEST_PATH_IMAGE177
Performing a first order unfolding to yield:
Figure 436422DEST_PATH_IMAGE178
wherein the content of the first and second substances,
Figure 788906DEST_PATH_IMAGE179
is formed by three-dimensional vectors
Figure 997033DEST_PATH_IMAGE180
The formed anti-symmetric matrix is a matrix with a plurality of anti-symmetric matrixes,
Figure 434968DEST_PATH_IMAGE181
is an identity matrix;
according to
Figure 183481DEST_PATH_IMAGE182
The first relationship is transformed to obtain:
Figure 656051DEST_PATH_IMAGE183
according to the converted first relation, optimizing the polynomial equation set to obtain an optimized polynomial equation set with the order of 2 as follows:
Figure 35079DEST_PATH_IMAGE184
。
in one embodiment, the nonlinear optimization module is further configured to convert each solution of the optimized polynomial equation set into an output homography matrix; and judging interior points between the matching point pairs of the images after the movement according to the output homography matrix.
For the specific definition of the calibration device for the rotation relationship between the camera and the inertial measurement unit, reference may be made to the above definition of the calibration method for the rotation relationship between the camera and the inertial measurement unit, and details thereof are not repeated herein. The modules in the camera and inertial measurement unit rotational relationship calibration apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, a mobile phone, a drone, etc., and its internal structure diagram may be as shown in fig. 4. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method for calibrating the rotational relationship of a camera and an inertial measurement unit. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and when the computer equipment is a terminal or a mobile phone, the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
The technical personnel in the field know that the calculation method provided by the embodiment has small calculation amount, is suitable for equipment with limited calculation capacity, such as mobile phones, unmanned planes and the like, and has wide application prospect.
Those skilled in the art will appreciate that the architecture shown in fig. 4 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, a computer device is provided, comprising a memory storing a computer program and a processor implementing the steps of the method in the above embodiments when the processor executes the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method in the above-mentioned embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. A method for calibrating a camera to inertial measurement unit rotational relationship, the method comprising:
acquiring a first image and a second image shot by a camera, and extracting affine matching point pairs of the first image and the second image; each pair of affine matching point pairs comprises a first homogeneous coordinate of a first image, a second homogeneous coordinate of an image point in a second image and an affine transformation matrix;
establishing a second relation between the homography matrix and the first and second homogeneous coordinates according to a first relation between the homography matrix of the first and second images and a rotation matrix and a translation vector for converting the first image into the second image;
establishing a third relationship of the affine transformation matrix and the rotation matrix based on the homography matrix according to the affine transformation matrix in the first image and the second image;
obtaining a polynomial equation set according to the second relation and the third relation; the order of the polynomial equation set is 6, and the unknowns are 3 rotation parameters and 3 translation parameters;
acquiring initial installation parameters between an inertia measurement unit and a camera, and performing first-order approximation on the polynomial equation set according to the initial installation parameters to obtain a simplified polynomial equation set with the order of 2;
solving the simplified polynomial equation set by adopting a resolver to obtain a rotation matrix to be calibrated and a translation vector;
before obtaining a polynomial equation set from the second relationship and the third relationship, the method further comprises:
obtaining six equations according to the second relation and the third relation:
Figure 135721DEST_PATH_IMAGE001
wherein the content of the first and second substances,
Figure 831145DEST_PATH_IMAGE002
is that
Figure 400666DEST_PATH_IMAGE003
First, the
Figure 675790DEST_PATH_IMAGE004
Line and first
Figure 584840DEST_PATH_IMAGE005
The elements of the column are,
Figure 123269DEST_PATH_IMAGE006
h represents a homography matrix, and the affine matching point pairs are
Figure 852190DEST_PATH_IMAGE007
Wherein, in the step (A),
Figure 196584DEST_PATH_IMAGE008
and
Figure 694561DEST_PATH_IMAGE009
is a corresponding point pair in the form of homogeneous coordinates in the first image and the second image, a denotes an affine transformation matrix,
Figure 794104DEST_PATH_IMAGE010
is that
Figure 744743DEST_PATH_IMAGE011
To (1) a
Figure 892828DEST_PATH_IMAGE012
Line and first
Figure 245311DEST_PATH_IMAGE013
The elements of the column are, in turn,
Figure 391122DEST_PATH_IMAGE014
，
Figure 829057DEST_PATH_IMAGE015
and
Figure 515253DEST_PATH_IMAGE016
is the first
Figure 315719DEST_PATH_IMAGE017
The direction in the individual images is such that,
Figure 429168DEST_PATH_IMAGE018
，
Figure 619978DEST_PATH_IMAGE019
a projection of the depth is represented and,
Figure 844286DEST_PATH_IMAGE020
take a value of
Figure 109045DEST_PATH_IMAGE021
The last row of (2);
and obtaining a polynomial equation set according to the six equations.
2. The method of claim 1, further comprising:
acquiring a plurality of matching interior point pairs in the first image and the second image;
calculating a reprojection error according to the matching interior point pairs and the homography matrix;
and constructing a conversion error cost function according to the reprojection error, and minimizing the error cost function by taking the corrected rotation matrix as an optimization parameter to obtain an optimized rotation matrix.
3. The method of claim 1, wherein establishing a second relationship of the homography matrix with the first and second homogeneous coordinates according to a first relationship of the homography matrix of the first and second images with a rotation matrix and a translation vector of a first image to second image transformation comprises:
acquiring the homography of the first image and the second image as follows:
Figure 393396DEST_PATH_IMAGE022
wherein the content of the first and second substances,
Figure 71502DEST_PATH_IMAGE023
respectively representing a first homogeneous coordinate of an image point in the first image and a second homogeneous coordinate of an image point in the second image, H representing a homography matrix, R representing a rotation matrix, t representing a translation vector,drepresenting the distance of the first image coordinate system to the 3D ground,
Figure 99501DEST_PATH_IMAGE024
representing a unit normal vector of the 3D ground in a first image coordinate system;
according to the homography, obtaining a first relation between the homography matrix of the first image and the second image and the rotation matrix and the translation vector of the transformation from the first image to the second image, wherein the first relation is as follows:
Figure 110444DEST_PATH_IMAGE025
Figure 565697DEST_PATH_IMAGE026
wherein the content of the first and second substances,
Figure 465519DEST_PATH_IMAGE027
represents the rotation matrix calibrated for R, T represents the translation vector calibrated for T,
Figure 234892DEST_PATH_IMAGE028
representing the angular information output by the inertial measurement unit when the camera takes the first image,
Figure 536561DEST_PATH_IMAGE029
representing the angular information output by the inertial measurement unit when the camera takes the second image,
Figure 162714DEST_PATH_IMAGE030
representing a unit normal vector of the 3D ground in a first image coordinate system after calibration;
according to the first relation, establishing a second relation among the homography matrix, the first homogeneous coordinate and the second homogeneous coordinate as follows:
Figure 549833DEST_PATH_IMAGE031
wherein the content of the first and second substances,
Figure 247531DEST_PATH_IMAGE032
is that
Figure 138126DEST_PATH_IMAGE033
First, the
Figure 200760DEST_PATH_IMAGE034
Line and first
Figure 12858DEST_PATH_IMAGE035
The elements of the column are,
Figure 186351DEST_PATH_IMAGE036
。
4. the method of claim 3, wherein establishing a third relationship of the affine transformation matrix and the rotation matrix based on the homography matrix according to the affine transformation matrix in the first image and the second image comprises:
obtaining affine matching point pairs in the first image and the second image
Figure 462611DEST_PATH_IMAGE037
(ii) a Wherein the content of the first and second substances,
Figure 430567DEST_PATH_IMAGE038
and
Figure 854595DEST_PATH_IMAGE039
is a corresponding point pair in the form of homogeneous coordinates in the first image and the second image, a represents an affine transformation matrix;
according to the homography matrix, establishing a third relation between the affine transformation matrix and the rotation matrix as follows:
Figure 566200DEST_PATH_IMAGE040
wherein the content of the first and second substances,
Figure 696967DEST_PATH_IMAGE041
is that
Figure 835824DEST_PATH_IMAGE042
To (1) a
Figure 622514DEST_PATH_IMAGE043
Line and first
Figure 872230DEST_PATH_IMAGE044
The elements of the column are, in turn,
Figure 123083DEST_PATH_IMAGE045
，
Figure 760738DEST_PATH_IMAGE046
and
Figure 831462DEST_PATH_IMAGE047
is the first
Figure 150448DEST_PATH_IMAGE048
The direction in the individual images is such that,
Figure 990228DEST_PATH_IMAGE049
；
Figure 674150DEST_PATH_IMAGE050
a projection of the depth is represented and,
Figure 966591DEST_PATH_IMAGE051
take a value of
Figure 89268DEST_PATH_IMAGE052
The last row of (2).
5. The method of claim 4, wherein prior to deriving the system of polynomial equations from the second relationship and the third relationship, the method further comprises:
using Euler angles
Figure 609986DEST_PATH_IMAGE053
Parameterizing the rotation matrix to obtain a fourth relation:
Figure 261547DEST_PATH_IMAGE054
；
obtaining a polynomial equation set according to the second relation and the third relation, including:
obtaining a polynomial equation set according to the second relation, the third relation and the fourth relation, wherein the polynomial equation set is:
Figure 572442DEST_PATH_IMAGE055
wherein the content of the first and second substances,
Figure 967652DEST_PATH_IMAGE056
，
Figure 719707DEST_PATH_IMAGE057
。
6. the method of claim 5, wherein obtaining initial setup parameters between the inertial measurement unit and the camera, and performing a first order approximation of the polynomial equation set based on the initial setup parameters to obtain a simplified polynomial equation set of order 2 comprises:
obtaining initial setup parameters between an inertial measurement unit and a camera
Figure 807749DEST_PATH_IMAGE058
；
Obtaining the relationship between the initial installation parameters and the rotation matrix based on the rotation matrix of the small angle as follows:
Figure 340361DEST_PATH_IMAGE059
wherein the content of the first and second substances,
Figure 539261DEST_PATH_IMAGE060
is a rotation matrix based on small angles;
to pair
Figure 270457DEST_PATH_IMAGE061
A first order approximation is performed to yield:
Figure 529400DEST_PATH_IMAGE062
wherein the content of the first and second substances,
Figure 549309DEST_PATH_IMAGE063
is formed by three-dimensional vectors
Figure 489583DEST_PATH_IMAGE064
The formed anti-symmetric matrix is a matrix with a plurality of anti-symmetric matrixes,
Figure 12968DEST_PATH_IMAGE065
is an identity matrix;
according to
Figure 708392DEST_PATH_IMAGE066
The first relationship is transformed to obtain:
Figure 684438DEST_PATH_IMAGE067
according to the converted first relation, performing first order approximation on the polynomial equation set to obtain a simplified polynomial equation set with the order of 2:
Figure 553037DEST_PATH_IMAGE068
wherein the content of the first and second substances,
Figure 196508DEST_PATH_IMAGE069
。
7. the method of claim 2, wherein obtaining a plurality of matching interior point pairs in the first image and the second image comprises:
converting each solution of the optimized polynomial equation set into an output homography matrix;
and judging interior points between the matching point pairs of the images after the movement according to the output homography matrix.
8. A camera and inertial measurement unit rotational relationship calibration apparatus, said apparatus comprising:
the data acquisition module is used for acquiring a first image and a second image shot by the camera and extracting affine matching point pairs of the first image and the second image; each pair of affine matching point pairs comprises a first homogeneous coordinate of a first image, a second homogeneous coordinate of an image point in a second image and an affine transformation matrix;
the second relation establishment module is used for establishing a second relation between the homography matrix and the first homogeneous coordinate and the second homogeneous coordinate according to the first relation between the homography matrix of the first image and the second image and the rotation matrix and translation vector of the transformation from the first image to the second image;
a third relation establishing module, configured to establish a third relation between the affine transformation matrix and the rotation matrix based on the homography matrix according to the affine transformation matrix in the first image and the second image;
the equation set establishing module is used for obtaining a polynomial equation set according to the second relation and the third relation; the order of the polynomial equation set is 6, and the unknowns are 3 rotation parameters and 3 translation parameters;
the equation set simplifying module is used for acquiring initial installation parameters between the inertia measuring unit and the camera, and performing first-order approximation on the polynomial equation set according to the initial installation parameters to obtain a simplified polynomial equation set with the order of 2;
the resolving module is used for solving the simplified polynomial equation set by adopting a resolver to obtain a rotation matrix to be calibrated and a translation vector;
the equation set establishing module is further configured to obtain six equations according to the second relationship and the third relationship:
Figure 797253DEST_PATH_IMAGE070
wherein the content of the first and second substances,
Figure 729437DEST_PATH_IMAGE071
is that
Figure 808252DEST_PATH_IMAGE072
First, the
Figure 571808DEST_PATH_IMAGE073
Line and first
Figure 77876DEST_PATH_IMAGE074
The elements of the column are,
Figure 621990DEST_PATH_IMAGE075
h represents a homography matrix, and the affine matching point pairs are
Figure 504495DEST_PATH_IMAGE076
Wherein, in the step (A),
Figure 122558DEST_PATH_IMAGE077
and
Figure 65107DEST_PATH_IMAGE078
is a corresponding point pair in the form of homogeneous coordinates in the first image and the second image, a denotes an affine transformation matrix,
Figure 706303DEST_PATH_IMAGE079
is that
Figure 126920DEST_PATH_IMAGE080
To (1) a
Figure 865069DEST_PATH_IMAGE043
Line and first
Figure 712940DEST_PATH_IMAGE081
The elements of the column are, in turn,
Figure 467531DEST_PATH_IMAGE014
，
Figure 957418DEST_PATH_IMAGE082
and
Figure 284495DEST_PATH_IMAGE083
is the first
Figure 772108DEST_PATH_IMAGE084
The direction in the individual images is such that,
Figure 184635DEST_PATH_IMAGE085
，
Figure 212633DEST_PATH_IMAGE086
a projection of the depth is represented and,
Figure 722112DEST_PATH_IMAGE087
take a value of
Figure 177364DEST_PATH_IMAGE088
The last row of (2);
and obtaining a polynomial equation set according to the six equations.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
CN202110249005.8A 2021-03-08 2021-03-08 Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit Active CN112629565B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110249005.8A CN112629565B (en) 2021-03-08 2021-03-08 Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110249005.8A CN112629565B (en) 2021-03-08 2021-03-08 Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit

Publications (2)

Publication Number Publication Date
CN112629565A CN112629565A (en) 2021-04-09
CN112629565B true CN112629565B (en) 2021-06-01

Family

ID=75297655

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110249005.8A Active CN112629565B (en) 2021-03-08 2021-03-08 Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit

Country Status (1)

Country Link
CN (1) CN112629565B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113048985B (en) * 2021-05-31 2021-08-06 中国人民解放军国防科技大学 Camera relative motion estimation method under known relative rotation angle condition
CN114708331B (en) * 2022-02-24 2023-03-14 合肥的卢深视科技有限公司 Calibration method and device for depth camera, electronic equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110246194A (en) * 2019-06-25 2019-09-17 中国人民解放军国防科技大学 Method for quickly calibrating rotation relation between camera and inertia measurement unit
CN111429524A (en) * 2020-03-19 2020-07-17 上海交通大学 Online initialization and calibration method and system for camera and inertial measurement unit
CN111754586A (en) * 2020-06-28 2020-10-09 苏州臻迪智能科技有限公司 External parameter calibration method and device, external parameter calibration system and computer storage medium
CN112362084A (en) * 2020-11-23 2021-02-12 北京三快在线科技有限公司 Data calibration method, device and system

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109069067B (en) * 2015-12-16 2022-04-12 穆罕默德·R·马赫福兹 IMU calibration

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110246194A (en) * 2019-06-25 2019-09-17 中国人民解放军国防科技大学 Method for quickly calibrating rotation relation between camera and inertia measurement unit
CN111429524A (en) * 2020-03-19 2020-07-17 上海交通大学 Online initialization and calibration method and system for camera and inertial measurement unit
CN111754586A (en) * 2020-06-28 2020-10-09 苏州臻迪智能科技有限公司 External parameter calibration method and device, external parameter calibration system and computer storage medium
CN112362084A (en) * 2020-11-23 2021-02-12 北京三快在线科技有限公司 Data calibration method, device and system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Minimal Solutions for the Rotational Alignment of IMU-Camera Systems using Homography Constraints;Banglei Guan 等;《computer vision and image understanding》;20180312;第170卷;第79-91页 *
Rotational Alignment of IMU-camera Systems with 1-point RANSAC;Banglei Guan 等;《PRCV 2019: Pattern Recognition and Computer Vision》;20191031;第172-183页 *

Also Published As

Publication number Publication date
CN112629565A (en) 2021-04-09

Similar Documents

Publication Publication Date Title
CN110009681B (en) IMU (inertial measurement unit) assistance-based monocular vision odometer pose processing method
CN108871311B (en) Pose determination method and device
CN111415387B (en) Camera pose determining method and device, electronic equipment and storage medium
WO2021004416A1 (en) Method and apparatus for establishing beacon map on basis of visual beacons
CN112629565B (en) Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit
CN113409391B (en) Visual positioning method and related device, equipment and storage medium
US20090141043A1 (en) Image mosaicing apparatus for mitigating curling effect
CN112184824A (en) Camera external parameter calibration method and device
Yu et al. The effects of temperature variation on videometric measurement and a compensation method
CN109901123B (en) Sensor calibration method, device, computer equipment and storage medium
CN112446917B (en) Gesture determination method and device
CN112465877A (en) Kalman filtering visual tracking stabilization method based on motion state estimation
CN113436267B (en) Visual inertial navigation calibration method, device, computer equipment and storage medium
CN114063046A (en) Parameter calibration method and device, computer equipment and storage medium
US20230379422A1 (en) Method for vehicle hinge point calibration and corresponding calibration apparatus, computer device, and storage medium
CN111721283B (en) Precision detection method and device for positioning algorithm, computer equipment and storage medium
CN111445513A (en) Plant canopy volume obtaining method and device based on depth image, computer equipment and storage medium
CN108345463B (en) Three-dimensional measuring method and device based on robot, robot and storage medium
CN113628284B (en) Pose calibration data set generation method, device and system, electronic equipment and medium
CN113405532B (en) Forward intersection measuring method and system based on structural parameters of vision system
CN115205419A (en) Instant positioning and map construction method and device, electronic equipment and readable storage medium
CN111723826B (en) Method, device, computer equipment and storage medium for detecting precision of tracking algorithm
CN110232715B (en) Method, device and system for self calibration of multi-depth camera
Zhong et al. CalQNet-detection of calibration quality for life-long stereo camera setups
CN114063024A (en) Calibration method and device of sensor, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant