CN111699514A

CN111699514A - Calibration method and device for internal reference and relative attitude of camera, unmanned aerial vehicle and storage device

Info

Publication number: CN111699514A
Application number: CN201980010784.7A
Authority: CN
Inventors: 唐克坦; 林家荣; 张培科
Original assignee: SZ DJI Technology Co Ltd
Current assignee: SZ DJI Technology Co Ltd; Shenzhen Dajiang Innovations Technology Co Ltd
Priority date: 2019-05-30
Filing date: 2019-05-30
Publication date: 2020-09-22
Also published as: WO2020237574A1

Abstract

A calibration method and device for internal reference and relative attitude of a camera, an unmanned aerial vehicle and a storage device are provided, the calibration method for the internal reference comprises the following steps: acquiring an image shot by a camera on a calibration board, wherein a plurality of calibration objects which are randomly distributed are arranged on the calibration board (S11); identifying an image object of the labeled object in the image (S12); matching the identified image object of the calibration object with the calibration object on the calibration board (S13); a fitting operation is performed according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration plate to determine an internal reference of the camera (S14). By the method, the flexibility of the internal reference calibration of the camera can be improved.

Description

Calibration method and device for internal reference and relative attitude of camera, unmanned aerial vehicle and storage device

[ technical field ] A method for producing a semiconductor device

The application relates to the technical field of image processing, in particular to a calibration method and device for internal reference and relative attitude of a camera, an unmanned aerial vehicle and a storage device.

[ background of the invention ]

The internal reference calibration of the camera is an important basic technology for photogrammetry, robot visual navigation, various applications of computer vision and the like. The general internal reference calibration technology of the camera adopts a checkerboard calibration plate, firstly, the camera shoots the checkerboard calibration plate to obtain an image, then angular points in the image are extracted to be matched with the angular points on the checkerboard calibration plate, and then the internal reference of the camera is calibrated. However, when the checkerboard calibration board is used to calibrate the internal parameters of the camera, the checkerboard calibration board must be all within the shooting range of the camera, i.e. the camera cannot shoot only a part of the checkerboard calibration board, which reduces the flexibility of calibrating the internal parameters.

[ summary of the invention ]

The technical problem mainly solved by the application is to provide a calibration method and device for internal reference and relative attitude of a camera, an unmanned aerial vehicle and a storage device, and flexibility of calibration of the internal reference of the camera can be improved.

In order to solve the above technical problem, a first aspect of the present application provides an internal reference calibration method for a camera, including: acquiring an image shot by the camera on a calibration plate, wherein a plurality of calibration objects which are randomly distributed are arranged on the calibration plate; identifying an image object of a designated object in the image; matching the identified image object of the calibration object with the calibration object on the calibration plate; and performing fitting operation according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration plate to determine the internal reference of the camera.

In order to solve the above technical problem, a second aspect of the present application provides a method for calibrating a relative posture of a camera and an inertial measurement unit, including: acquiring external reference postures of the camera at multiple moments; acquiring the postures of the inertial measurement unit at a plurality of moments; and calibrating the relative postures of the camera and the inertial measurement unit according to the external reference postures of the camera at multiple moments and the postures of the inertial measurement unit at multiple moments.

In order to solve the above technical problem, a third aspect of the present application provides an internal reference calibration apparatus for a camera, including a processor and a memory, where the memory is used for storing program instructions; the processor executing the program instructions to: acquiring an image shot by the camera on a calibration plate, wherein a plurality of calibration objects which are randomly distributed are arranged on the calibration plate; identifying an image object of a designated object in the image; matching the identified image object of the calibration object with the calibration object on the calibration plate; and performing fitting operation according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration plate to determine the internal reference of the camera.

In order to solve the above technical problem, a fourth aspect of the present application provides an apparatus for calibrating a relative posture between a camera and an inertial measurement unit, including a processor and a memory, where the memory is used for storing program instructions; the processor executing the program instructions to: acquiring external reference postures of the camera at multiple moments; acquiring the postures of the inertial measurement unit at a plurality of moments; and calibrating the relative postures of the camera and the inertial measurement unit according to the external reference postures of the camera at multiple moments and the postures of the inertial measurement unit at multiple moments.

In order to solve the above technical problem, a fifth aspect of the present application provides an internal reference calibration system for a camera, including the camera and the internal reference calibration apparatus of the third aspect, wherein the camera is used for shooting a calibration board.

In order to solve the above technical problem, a sixth aspect of the present invention provides a system for calibrating a relative posture of a camera and an inertial measurement unit, comprising the camera, the inertial measurement unit and the apparatus of the fourth aspect, wherein the camera is used for shooting a calibration plate; the inertial measurement unit is used for measuring attitude data.

In order to solve the above technical problem, a seventh aspect of the present application provides an unmanned aerial vehicle, including the internal reference calibration system of the fifth aspect or the system for calibrating the relative attitude between the camera and the inertial measurement unit of the sixth aspect.

In order to solve the above technical problem, an eighth aspect of the present application provides a storage device storing program instructions that, when executed on a processor, perform the method of the first aspect or the second aspect.

According to the scheme, the calibration plate provided with the randomly distributed calibration objects is shot to obtain the image, the image objects of the calibration objects identified in the image are matched with the calibration objects on the calibration plate, the internal parameters of the camera are determined according to the positions of the image objects in the image and the positions of the corresponding matched calibration objects in the calibration plate, and the calibration objects of the calibration plate are randomly distributed, so that the calibration objects can be uniquely identified according to the distribution condition of the peripheral calibration objects of each calibration object, therefore, even if the calibration plate is partially shot, the calibration objects and the image objects in the image can be accurately matched, the effective and accurate calibration of the internal parameters of the camera is realized, the whole calibration plate is not required to be shot, and the efficiency and flexibility of internal parameter calibration are improved.

[ description of the drawings ]

FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a camera internal reference calibration method according to the present application;

FIG. 2 is a schematic illustration of a calibration plate employed in one application scenario of the present application;

FIG. 3 is a schematic diagram illustrating a matching relationship between a calibration object and an image object in an application scenario of the present application;

FIG. 4 is a schematic flowchart of step S13 in another embodiment of the camera internal reference calibration method of the present application;

FIGS. 5a-5c are schematic diagrams of calibration plates employed in various application scenarios of the present application;

FIG. 6 is a schematic flowchart of a camera calibration method according to another embodiment of the present disclosure;

FIG. 7 is a flowchart illustrating the step S64 in yet another embodiment of the camera calibration method according to the present invention;

FIG. 8 is a schematic flowchart of an embodiment of a method for calibrating a relative attitude of a camera and an inertial measurement unit according to the present application;

FIG. 9 is a schematic flowchart of step S83 in another embodiment of the method for calibrating the relative attitude of a camera and an inertial measurement unit according to the present application;

FIG. 10 is a schematic diagram of the trajectory path of the inertial measurement unit and the camera of the present application at a plurality of time instants;

FIG. 11 is a schematic structural diagram of an embodiment of an internal reference calibration apparatus of a camera according to the present application;

FIG. 12 is a schematic structural diagram of an embodiment of the apparatus for calibrating the relative attitude of a camera and an inertial measurement unit according to the present invention;

FIG. 13 is a schematic structural diagram of an embodiment of an internal reference calibration system of a camera according to the present application;

FIG. 14 is a schematic diagram illustrating an embodiment of a system for calibrating the relative attitude of a camera and an inertial measurement unit according to the present application;

fig. 15 is a schematic structural diagram of an embodiment of the drone of the present application;

fig. 16 is a schematic structural diagram of another embodiment of the drone of the present application;

FIG. 17 is a schematic structural diagram of an embodiment of a memory device according to the present application.

[ detailed description ] embodiments

The technical solutions in the embodiments of the present invention will be described clearly below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It will be understood that when an element is referred to as being "secured to" another element, it can be directly on the other element or intervening elements may also be present. When a component is referred to as being "connected" to another component, it can be directly connected to the other component or intervening components may also be present.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of an internal reference calibration method for a camera according to the present application, where the method may be applied to an unmanned aerial vehicle, and is specifically used for performing internal reference calibration on a camera configured on the unmanned aerial vehicle. The method specifically comprises the following steps:

s11: and acquiring an image shot by a camera on a calibration plate, wherein a plurality of calibration objects which are randomly distributed are arranged on the calibration plate.

Specifically, the execution subject of the method of this embodiment may be an internal reference calibration apparatus, and further, the execution subject may be a processor of the internal reference calibration apparatus, where the processor may be a general-purpose or special-purpose processor, and the processor may be one or more processors, which are not specifically limited herein. The internal reference calibration device can be configured on the unmanned aerial vehicle, in the process of internal reference calibration of the camera configured on the unmanned aerial vehicle, the camera on the unmanned aerial vehicle can shoot the calibration plate and output the shot image, and the internal reference calibration device can acquire the image output by the camera.

The calibration plate may be any calibration device having an image calibration function, and includes a plurality of calibration objects randomly distributed, so that the captured image correspondingly includes an image object of the calibration object, i.e., an image area representing the calibration object in the image.

Specifically, as shown in fig. 2, a plurality of calibration objects 21 are randomly distributed on the calibration board 20. The punctuation object 21 may be a circle or other shaped point. In addition, the calibration objects 21 on the calibration plate 20 may have the same size, and in some cases, the calibration objects 21 on the calibration plate 20 include at least two sizes of calibration objects, that is, at least two different size types of calibration objects, and for convenience of description, the two different size types of

calibration objects

211 and 212 are schematically illustrated here: the calibration plate 20 includes a base plate 22 and at least two size types of

calibration objects

211 and 212 disposed on the base plate 22. Optionally, the color of the outer ring and the inner part of the outer ring of at least one of the at least two size types of

calibration objects

211 and 212 are different, for example, the outer ring is black, and the inner part of the outer ring is white; or the outer ring is white and the inner part of the outer ring is black. Optionally, the color of the central part of at least one of the at least two size types of

calibration objects

211 and 212 is different from the color of the central part of the other of the at least two size types of calibration objects.

In the embodiment, the calibration plates with the randomly distributed calibration objects are used for realizing the internal reference calibration of the camera, compared with a chessboard calibration plate, the calibration plates with the randomly distributed calibration objects can realize the matching between the angular points and the image objects in the image by taking the whole image, and the calibration plates with the randomly distributed calibration objects can realize the matching between the calibration objects and the image objects in the image by partially taking the image of the calibration plates because the calibration object distribution around each calibration object can be uniquely identified.

S12: an image object of a target object in an image is identified.

Specifically, after acquiring an image obtained by shooting a calibration board, the internal reference calibration device identifies an image object of a calibration object from the image, wherein the image object is an image area of the shot calibration object in the image. Because the calibration object on the calibration plate is an obvious object, the internal reference calibration device can identify the image object of the calibration object from the image according to the characteristics of the calibration object. For example, when the calibration object of the calibration board is a dot, the internal reference calibration apparatus may extract a random point in the image by using a dot extraction (blob detector) algorithm. The dot extraction algorithm has higher precision than the checkerboard corner algorithm, so that the identification precision of the image object can be improved.

For convenience of explanation, in the following description, taking a calibration board as an example of a calibration board with two sizes of calibration objects randomly distributed, as shown in fig. 3, the internal reference calibration apparatus acquires an image 320 captured on the calibration board 310, and then can identify a plurality of image objects 321 from the image 320.

S13: and matching the identified image object of the calibration object with the calibration object on the calibration board.

Specifically, after the image object of the calibration object in the image is identified, it is necessary to establish a matching relationship between the detected image object and the calibration object on the calibration board, that is, to determine which calibration object on the calibration board corresponds to the detected image object. With continued reference to fig. 3, for the image 320 captured by the calibration board 310, a plurality of image objects 321 may be identified from the image 320, and each of the image objects 321 is matched with the calibration object 311 in the calibration board 310, and after the matching is completed, each of the image objects 321 establishes a one-to-one correspondence relationship with the calibration object 311 in the calibration board 310.

In certain embodiments, referring in conjunction with fig. 4, step S13 may include the following sub-steps:

s131: and determining the position characteristic parameters of the identified image object according to the position of the identified image object in the image.

Specifically, since the calibration objects on the calibration plate are randomly distributed, the distribution of the calibration objects around each calibration object is different, and therefore the calibration object can be uniquely identified according to the distribution of the calibration objects around each calibration object. Thus, the position characteristic parameter of the identified image object may be determined based on the position of the identified image object in the image and the position of one or more image objects surrounding the image object in the image.

S132: and matching the identified image object of the calibration object with the calibration object on the calibration board according to the identified position characteristic parameter of the image object and the prestored position characteristic parameter of the calibration object.

The internal reference calibration device may pre-store position characteristic parameters of a calibration object in a calibration board, where the position characteristic parameters of the calibration object may be pre-determined according to a position of the calibration object on the calibration board and positions of one or more calibration objects around the calibration object in the calibration board. The position feature parameter may be a feature vector or a hash value. Specifically, when the position characteristic parameter of the image object is the same as or similar to a pre-stored position characteristic parameter of one of the calibration objects, it may be determined that the image object and the calibration object are matched.

S14: and performing fitting operation according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration plate to determine the internal reference of the camera.

Specifically, the internal reference calibration apparatus may determine a position of the image object in the image, that is, a position where the image object is located in the image, where the position of the image object in the image may be a coordinate of the image object in an image coordinate system. In addition, the internal reference calibration device may determine a position of a calibration object matching the image object in the calibration board, and for convenience of description, the calibration object matching the image object may be simply referred to as a target calibration object. The position of the calibration object in the calibration board may be prestored in the internal reference calibration device, and after the target calibration object matching the image object is determined, the position of the target calibration object on the calibration board may be obtained from the prestored position of the calibration object in the calibration board. For example, for a calibration plate with randomly distributed dots, the position of the image object in the image may be the coordinates of the image area of the dots in the image coordinate system, and the position of the target calibration object on the calibration plate may be the position of the dots on the calibration plate.

After the position of the image object in the image and the position of the target calibration object on the calibration board are obtained, fitting operation can be performed according to the position of the image object in the image and the position of the target calibration object on the calibration board to determine internal parameters of the camera.

For example, for one frame image, the internal reference calibration device obtains image coordinates (u, v) of a plurality of image objects in the frame image and world coordinates (X, Y, Z) of a corresponding plurality of target calibration objects on a calibration board. And obtaining the homography matrix H of the frame image by using the image coordinates (u, v) of each image object in the frame image and the world coordinates (X, Y, Z) of the corresponding target calibration object on the calibration plate. And obtaining the internal reference of the camera, namely an internal reference matrix K by using the homography matrix H of the multi-frame image.

Specifically, the image coordinates (u, v) of the image object in the image and the world coordinates (X, Y, Z) of the corresponding target calibration object on the calibration plate satisfy the following relationship:

wherein the content of the first and second substances,

is the internal reference matrix of the camera, R, T is the rotation matrix and translation vector of the world coordinate system relative to the camera coordinate system, R ═ R₁r₂r₃]And α is an unknown scaling factor.

Taking the XY plane with Z0 on the world coordinate system as an example:

for each frame image, the image coordinate of the ith image object is (u)_i，v_i) Corresponding to the world coordinate (X) of the target calibration object on the calibration plate_i，Y_i,0). The coordinates of each set of image objects and target calibration objects constitute the following equation 2:

wherein the content of the first and second substances,

wherein

Representing the transpose of h.

Order to

The optimized objective function is then:

order to

Then equation (1) can be converted to the following equation:

for n sets of corresponding image objects and corresponding target calibration objects in one frame of image, n above equations (4) are obtained. For the n above equations (4), a least square solution thereof can be calculated as an optimal solution of the objective function (3). And correspondingly obtaining a homography matrix H in the frame image according to the optimal solution.

Obtaining a homography matrix H of the multi-frame images by using the method, and obtaining a group of constraint equations from the homography matrix H of each frame of images:

H＝K[r₁r₂T]

let B be K^-TK^-1Can be prepared by

Is shown as

Where B is a column of multidimensional, e.g., 6-dimensional, vectors made up of the elements in B, the above system of constraint equations can be expressed as the following equation (5):

if the above equation is established for each frame of image, multiple equations (5) can be obtained corresponding to multiple frames of images, and the least square solution can be solved to obtain the optimal B, so as to obtain the internal reference matrix K of the camera.

In the embodiment, an image is obtained by shooting the calibration plate provided with randomly distributed calibration objects, the image objects of the calibration objects identified in the image are matched with the calibration objects on the calibration plate, and then the internal parameters of the camera are determined according to the positions of the image objects in the image and the positions of the corresponding matched calibration objects in the calibration plate. In another embodiment, in order to improve the calibration efficiency, the calibration board photographed by the camera may include a plurality of calibration boards whose spatial postures are different from each other. In particular, each calibration plate is provided with randomly distributed calibration objects as described above. In some embodiments, a calibration plate of the plurality of calibration plates is disposed in connection with at least another calibration plate of the plurality of calibration plates. Wherein, the connection between the calibration plates can be fixed connection or movable connection. In some embodiments, the plurality of calibration plates 51 may be connected to form, but not limited to, a hinge (as shown in fig. 5 a), a funnel (as shown in fig. 5 b), a quadrangle (as shown in fig. 5 c), etc. It is understood that at least one calibration board in the calibration boards may be not connected to other calibration boards, and therefore, the connection relationship of the calibration boards is not limited herein.

Referring to fig. 6, for the case that the image captured by the camera includes the plurality of calibration plates, fig. 6 is a schematic flow chart of another embodiment of the camera internal reference calibration method according to the present application. The method can be applied to the unmanned aerial vehicle, and is particularly used for performing internal reference calibration on a camera configured on the unmanned aerial vehicle. The method specifically comprises the following steps:

s61: and acquiring an image shot by a camera on a calibration plate, wherein a plurality of calibration objects which are randomly distributed are arranged on the calibration plate.

S62: an image object of a target object in an image is identified.

Specifically, the specific descriptions of S61 and S62 may refer to the descriptions of S11 and S12 described above. In this embodiment, the camera captures a plurality of calibration plates to obtain the image, and further identifies the image object of the calibration object of each calibration plate in the image.

S63: and matching the identified image object of the calibration object of each calibration board with the calibration object on the calibration board.

The matching between the image object of the calibration object of each calibration board and the calibration object of the calibration board can be described with reference to S13, which is not described herein again.

S64: and performing fitting operation according to the position of the image object of the calibration object of each calibration plate in the image and the position of the calibration object of each calibration plate matched with the image object on the calibration plate to determine the internal parameters of the camera.

In the prior art, in order to improve the accuracy of calibrating the internal parameters of the camera, the camera needs to shoot the calibration board at a plurality of different positions, that is, the images shot by the multi-frame camera at different positions on the calibration board are used for calibrating the internal parameters of the camera, so that the efficiency of calibrating the internal parameters of the camera is reduced.

In order to improve the efficiency of internal reference calibration, the calibration plate shot by the camera is a plurality of calibration plates with different postures, and then the internal reference of the camera is obtained through fitting according to the positions of the image objects of the calibration plates with different postures in the image and the positions of the calibration objects on the calibration plates with different postures. By the method, the calibration plates with different postures are shot, relative to the shooting of the same calibration plate at different positions, the internal reference of the camera is obtained by matching the position of the image object of the calibration object of one calibration plate with the position of the calibration object of the calibration plate, and the efficiency of calibrating the internal reference can be improved. In addition, in some cases, the calibration object and the image object of the calibration board in multiple postures can be obtained by directly adopting one frame of image in the embodiment, and compared with the case that one calibration board is respectively photographed in different postures to obtain multiple frames of images so as to obtain the calibration object and the image object of the calibration board in multiple postures, the calibration method and the calibration device realize that one frame of image obtains multiple sets of calibration input data, and improve the calibration efficiency.

Specifically, referring to fig. 7, S64 specifically includes the following sub-steps:

s641: and determining the position of the calibration object of each calibration board in the world coordinate system according to the position of the calibration object of each calibration board matched with the image object on the calibration board.

For example, for the ith calibration Board Board_iThe jth calibration object thereof_iP_jOn the calibration Board Board_iHas the coordinates of_iP_j＝(x_j，y_j，0)^T。

After determining the world coordinate system, calibrating the Board_iPoint of_iP_jThe coordinates in the world coordinate system are_worldP_j＝R_i→world(x_j，y_j，0)^T+T_i→worldWherein R is_i→world，T_i→worldIs a calibration Board Board_iThe transformation parameters of the coordinate system to the world coordinate system (which may also be referred to as the external parameters of the calibration plate). Since the attitude between the calibration plates is fixed in this embodiment, R is_i→worldAnd T_i→worldIs constant during the calibration process. The coordinate system of a calibration plate can generally be selected as the world coordinate system, e.g. a calibration plate Board_iOn the XY plane with Z being 0 in the world coordinate system, in which case T_i→worldIs zero, R_i→worldIs an identity matrix. Of course, in other embodiments, the coordinate systems of the calibration boards are not used as the world coordinate system.

S642: and performing fitting operation according to the identified position of the image object of the calibration object on each calibration plate in the image and the position of the calibration object of each calibration plate in the world coordinate system to determine the internal parameters of the camera.

For example, for one frame of image, the internal reference calibration device is based on the identified image coordinates (u, v) of the image object of the calibration object on each calibration plate in the image and the world coordinates of the calibration object of each calibration plate in the world coordinate system_worldP_jAnd obtaining a homography matrix H of the frame image. And obtaining an internal parameter matrix K of the camera by using the homography matrix H of the multi-frame image.

Specifically, the image coordinates (u, v) of the image object in the image and the world coordinates of the corresponding target calibration object on the calibration plate_worldP_jThe following relationship is satisfied:

wherein the content of the first and second substances,

Referring to the description of step S14, the position of each identified image object of each calibration board in the frame image and the position of the matching calibration object may form an equation according to the relationship (6), so that a plurality of equations may be obtained, and a least square method may be used to obtain a homography matrix of the frame image, and further a homography matrix H of a plurality of frame images may be used to obtain an internal parameter matrix K of the camera. In this embodiment, the parameters can be obtained by using an LM nonlinear least squares method.

In the embodiment, because the postures of the calibration plates are fixed, the external parameters of the calibration plates are also unchanged, so that the constraint of the external parameters of the calibration plates is introduced in the internal parameter calibration process, the calibration robustness is improved, and the calibration precision can be improved.

Referring to fig. 8, fig. 8 is a schematic flowchart illustrating an embodiment of a method for calibrating a relative attitude between a camera and an inertial measurement unit according to the present invention. The method can be applied to a device comprising an inertial measurement device and a camera, such as a drone, and the following schematic description refers to a drone, and the camera and the inertial measurement device configured on the drone are calibrated for relative attitude. The method specifically comprises the following steps:

s81: the external reference postures of the camera at a plurality of moments are acquired.

The executing subject of the method of this embodiment may be a gesture calibration apparatus, and further, the executing subject may be a processor of the gesture calibration apparatus, where the processor may be a general-purpose or special-purpose processor, and where the processor may be one or more processors, which are not specifically limited herein. The attitude calibration device can be configured on the unmanned aerial vehicle and calibrates the relative attitude of the camera and the inertia measurement device configured on the unmanned aerial vehicle.

Specifically, the external reference posture of the camera can be obtained by the camera internal reference calibration process. For example, in the camera internal reference calibration process, a plurality of frames of images shot by a camera on a calibration board at a plurality of moments are obtained, a homography matrix H of each frame of image is obtained, and then an internal reference matrix K of the camera is obtained by using the homography matrices H of the images shot at the plurality of moments. The specific process of the internal reference calibration can refer to the embodiment of the internal reference calibration method. And obtaining the external reference posture of the camera at a certain moment by using the homography matrix H and the camera internal reference matrix K of the image at the certain moment.

For example, for time t_kOuter reference attitude matrix R_cam(t_k)：

At time t when the camera is acquired_kAfter the homography matrix H corresponding to the image obtained by shooting and the internal reference matrix K of the camera are obtained, the following formula is utilized to obtain the homography matrix H at the moment t of the camera_kOuter reference attitude matrix R_cam(t_k)：

H＝[h₁h₂h₃]＝αK[r₁r₂T]

B＝αK^-1H

This gives:

r₃＝r₁×r₂. Wherein h is₁，h₂Is

column

1, 2 of the homography matrix H; b₁，b₂，

b

₃1, 2, 3 columns of matrix B, respectively; r is₁，r₂，r₃External reference attitude matrix R_cam(t_k)

Column

1, 2, 3. Due to r₁Is the external reference attitude matrix R_cam(t_k) First column of (1), therefore | r₁|＝1，α＝|b₁|。

It is understood that the external reference pose of the camera may also be obtained by other means, such as the slovePnP algorithm, and is not limited herein.

S82: the attitude of the inertial measurement unit is acquired at a plurality of times.

In particular, the Inertial Measurement Unit (IMU) may be obtained directly from data of the relevant sensor device, such as an integrating gyroscope, at time t_kPosture R of_imu(t_k)。

In order to further improve the attitude accuracy of the inertial measurement unit, the measurement noise reduction technology of the inertial measurement unit can be used to obtain the attitude of the inertial measurement unit. For example, the bias parameters of the gyro sensor of the inertial measurement unit are estimated by standing the inertial measurement unit for a certain period of time (e.g., 5 to 7s), or the bias parameters are dynamically estimated by the extended Kalman filtering technique. For another example, the measurement data of the inertial measurement unit is low-pass filtered to reduce high-frequency noise. Therefore, the attitude estimation precision of the inertial measurement unit can be effectively improved.

S83: and calibrating the relative postures of the camera and the inertial measurement unit according to the external reference postures of the camera at multiple moments and the postures of the inertial measurement unit at multiple moments.

In this embodiment, the relative postures of the camera and the inertial measurement unit can be obtained by obtaining the external reference postures of the camera at a plurality of times and the postures of the inertial measurement unit at a plurality of times.

In some embodiments, in conjunction with fig. 9, the step S83 may include:

s831: and determining the external reference posture change of the camera at the adjacent moment in the multiple moments according to the external reference postures of the camera at the multiple moments.

For example, at time t when the camera is acquired_kOuter reference posture R_cam(t_k) To at adjacent time t_k+1Outer reference posture R_cam(t_k+1) Then, can be obtained at t_kAnd t_k+1External reference posture change at adjacent time

The following were used:

similarly, it can be obtained that the camera is at the adjacent time t₁And t₂Change of posture of external reference

Adjacent time t₂And t₃Change of posture of external reference

Adjacent time t₃And t₄Change of posture of external reference

And the external reference postures of the adjacent moments in the plurality of moments are changed.

S832: and determining the attitude change of the inertial measurement unit at the adjacent moment in the plurality of moments according to the attitude of the inertial measurement unit at the plurality of moments.

For example, at time t when obtaining the inertial measurement unit_kOuter reference posture R_imu(t_k) To at adjacent time t_k+1Outer reference posture R_imu(t_k+1) Then, the adjacent time t can be obtained_kAnd t_k+1Change of posture of external reference

The following were used:

similarly, the inertia measuring device can be obtained at the adjacent time t₁And t₂Change of posture of external reference

Adjacent time t₂And t₃Change of posture of external reference

Adjacent time t₃And t₄Change of posture of external reference

S833: and calibrating the relative postures of the camera and the inertial measurement unit according to the external reference posture change of the camera at the adjacent moment in a plurality of moments and the posture change of the inertial measurement unit at the adjacent moment in a plurality of moments.

For example, since the camera is rigidly connected to the inertial measurement unit, as shown in FIG. 10, from time t_kTo t_k+1There are two paths for calculating the external reference attitude matrix of the camera, and the two paths are specifically as follows:

wherein the content of the first and second substances,

is the camera from time t_kTo t_k+1The external reference posture transformation matrix of (1),

is an inertia measuring devicet_kTo t_k+1The attitude transformation matrix of (1).

From the two paths above, the following equation can be derived:

converting the above equation (7) to a quaternion expression, the following equation is obtained:

wherein the content of the first and second substances,

q_eare respectively

R_eThe corresponding quaternions, r (q), l (q), are right-product matrix (right-product matrix) and left-product matrix (left-product matrix) of quaternion q, respectively.

From the above equation (8), the following equation (9) can be obtained:

A_4×4q_e＝0 (9)

wherein the content of the first and second substances,

corresponding n groups obtained by using n groups of adjacent time

Data, construct on q_eThe system of linear equations (10) of (a),

A_4n×4q_e＝0 (10)

the above equation set is used as least square method problem, and q can be obtained by adopting pseudo-inverse solution or singular value solution_eI.e. obtaining the relative attitude R between the camera and the inertial measurement unit_e。

In this embodiment, the calibration of the relative attitude between the camera and the inertial measurement unit is realized by obtaining the external reference attitude of the camera at a plurality of times and the attitude of the inertial measurement unit at a plurality of times. In addition, internal reference calibration can be realized by utilizing a calibration plate provided with randomly distributed calibration objects, and then external reference postures of the camera can be obtained by utilizing calibrated internal reference of the camera and process data thereof. Further, the relative attitude calibration method may be performed during or after the execution of the above-mentioned internal reference calibration method, and may be implemented by the same or different devices.

Referring to fig. 11, fig. 11 is a schematic structural diagram of an embodiment of an internal reference calibration apparatus of the present application. In this embodiment, the internal reference calibration apparatus 110 includes a memory 111 and a processor 112 connected to each other.

Memory 111 may include both read-only memory and random access memory and provides instructions and data to processor 112. A portion of the memory 111 may also include non-volatile random access memory.

The Processor 112 may be a Central Processing Unit (CPU), and may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 111 is used to store program instructions.

A processor 112, calling the program instructions, and when the program instructions are executed, for: acquiring an image shot by the camera on a calibration plate, wherein a plurality of calibration objects which are randomly distributed are arranged on the calibration plate; identifying an image object of a designated object in the image; matching the identified image object of the calibration object with the calibration object on the calibration plate; and performing fitting operation according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration plate to determine the internal reference of the camera.

In some embodiments, the processor 112, when matching the identified image object of the calibration object with the calibration object on the calibration board, is specifically configured to: determining a position characteristic parameter of the identified image object according to the position of the identified image object in the image; and matching the identified image object of the calibration object with the calibration object on the calibration board according to the identified position characteristic parameter of the image object and the prestored position characteristic parameter of the calibration object.

Wherein the location characteristic parameter may include a hash value.

In some embodiments, the calibration plate comprises a plurality of calibration plates, wherein the plurality of calibration plates differ in spatial attitude. When matching the identified image object of the calibration object with the calibration object on the calibration board, the processor 112 is specifically configured to: and matching the identified image object of the calibration object of each calibration plate with the calibration object on the calibration plate. When performing fitting operation according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration board to determine the internal reference of the camera, the processor 112 is specifically configured to: and performing fitting operation according to the position of the image object of the calibration object of each calibration plate in the image and the position of the calibration object of each calibration plate matched with the image object on the calibration plate to determine the internal parameters of the camera.

In some embodiments, each of the plurality of calibration plates is disposed in connection with at least another one of the plurality of calibration plates.

In some embodiments, the processor 112 is specifically configured to, when performing a fitting operation according to the position of the image object of the calibration object of each calibration board in the image and the position of the calibration object of each calibration board matched with the image object on the calibration board to determine the internal reference of the camera: determining the position of the calibration object of each calibration plate in the world coordinate system according to the position of the calibration object of each calibration plate matched with the image object on the calibration plate; and performing fitting operation according to the identified position of the image object of the calibration object on each calibration plate in the image and the position of the calibration object of each calibration plate in the world coordinate system to determine the internal parameters of the camera.

In some embodiments, the calibration objects include calibration objects of at least two different size types.

In some embodiments, the calibration object comprises a dot.

The apparatus of this embodiment may be configured to implement the technical solution of the above-mentioned embodiment of the internal reference calibration method of the present application, and the implementation principle and the technical effect are similar, which are not described herein again.

Referring to fig. 12, fig. 12 is a schematic structural diagram of an embodiment of an apparatus for calibrating a relative attitude between a camera and an inertial measurement unit according to the present application. In this embodiment, the apparatus 120 includes a memory 121 and a processor 122 connected to each other.

Memory 121 may include both read-only memory and random access memory, and provides instructions and data to processor 122. A portion of memory 121 may also include non-volatile random access memory.

The Processor 122 may be a Central Processing Unit (CPU), and may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 121 is used to store program instructions.

A processor 122, calling the program instructions, and when the program instructions are executed, for: acquiring external reference postures of the camera at multiple moments; acquiring the postures of the inertial measurement unit at a plurality of moments; and calibrating the relative postures of the camera and the inertial measurement unit according to the external reference postures of the camera at multiple moments and the postures of the inertial measurement unit at multiple moments.

In some embodiments, the processor 122 is specifically configured to, when calibrating the relative poses of the camera and the inertial measurement unit according to the poses of the camera and the inertial measurement unit at multiple times,: determining the external reference posture change of the camera at the adjacent moment in a plurality of moments according to the external reference postures of the camera at the plurality of moments; determining the attitude change of the inertial measurement unit at adjacent moments in a plurality of moments according to the attitudes of the inertial measurement unit at the moments; and calibrating the relative postures of the camera and the inertial measurement unit according to the external reference posture change of the camera at the adjacent moment in a plurality of moments and the posture change of the inertial measurement unit at the adjacent moment in a plurality of moments.

In some embodiments, the processor 122 is further configured to: acquiring an image shot by the camera on a calibration plate, wherein a plurality of calibration objects which are randomly distributed are arranged on the calibration plate; identifying an image object of a designated object in the image; matching the identified image object of the calibration object with the calibration object on the calibration plate; and acquiring the external reference postures of the camera at a plurality of moments according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration plate.

Further, when matching the identified image object of the calibration object with the calibration object on the calibration board, the processor 122 is specifically configured to: determining the position characteristic parameters of the identified image objects according to the positions of the image objects in the image; and matching the detected image object of the calibration object with the calibration object on the calibration board according to the determined position characteristic parameter and the pre-stored position characteristic parameter of the calibration object.

Wherein the location characteristic parameter may include a hash value.

In some embodiments, the calibration plate may include a plurality of calibration plates, wherein the plurality of calibration plates are different in spatial attitude from each other.

The device of this embodiment, the device 120, may be configured to execute the technical solution of the method for calibrating the relative posture between the camera and the inertial measurement unit in this application, and the implementation principle and the technical effect are similar, which are not described herein again.

Referring to fig. 13, fig. 13 is a schematic structural diagram of an embodiment of an internal reference calibration system of a camera according to the present application. The detection system 130 includes a camera 1301 and an internal reference calibration apparatus 1302 connected to each other. The camera 1301 is used for shooting the calibration plate to obtain an image. The internal reference calibration device 1302 is the internal reference calibration device described in the above embodiments, and is not described herein again.

Referring to fig. 14, fig. 14 is a schematic structural diagram of an embodiment of a system for calibrating a relative attitude of a camera and an inertial measurement unit according to the present application. The detection system 140 includes a device 1402 for calibrating the relative pose of the camera and the inertial measurement device, and a camera 1401 and an inertial measurement device 1403 connected to the device 1402. The camera 1401 is used to photograph the calibration board to obtain an image. The inertial measurement unit 1403 is used to measure attitude data. The device 1402 for calibrating the relative posture of the camera and the inertial measurement unit is the device for calibrating the relative posture of the camera and the inertial measurement unit described in the above embodiments, and is not described herein again.

Please refer to fig. 15, fig. 15 is a schematic structural diagram of an embodiment of the unmanned aerial vehicle according to the present application. In this embodiment, the unmanned aerial vehicle includes an internal reference calibration system of a camera, where the internal reference calibration system may specifically be as described in the above system embodiments, and includes an internal reference calibration device 1501 and a camera 1502.

Further, the drone may further comprise a carrying device 1503, wherein the carrying device 1503 is used for carrying the camera 1502. Optionally, the unmanned aerial vehicle is further provided with functional circuits such as a visual sensor and an inertia measuring device according to actual needs.

In an embodiment where the drone is further provided with an inertial measurement device, the drone may further include a system to calibrate the relative pose of the camera and the inertial measurement device. As shown in fig. 16, the system for calibrating the relative attitude of the camera and the inertial measurement unit in the drone may specifically be as described in the above system embodiments, and includes a unit 1601 for calibrating the relative attitude of the camera and the inertial measurement unit, a camera 1602, and an inertial measurement unit 1604. Further, the drone may also include a carrier 1603, where the carrier 1603 may be used to carry the camera 1602 and the inertial measurement device 1604.

In some embodiments, the drone may be a rotorcraft and the camera 1502/1602 may be a primary camera of the drone. The carriage 1503/1603 may be a two or three axis pan/tilt head.

Referring to fig. 17, fig. 17 is a schematic structural diagram of a memory device according to an embodiment of the present application. In this embodiment, the storage device 170 stores program instructions 1701, and when the program instructions 1701 run on a processor, the technical solution of the above-described method embodiment of the present application is executed.

The storage device 170 may be a medium that can store computer instructions, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the program instructions, and the server may send the stored program instructions to other devices for operation or may self-operate the stored program instructions.

In addition, the calibration of the relative attitude between the camera and the inertial measurement unit can be realized by obtaining the external reference attitude of the camera at a plurality of moments and the attitude of the inertial measurement unit at a plurality of moments. The calibration plate provided with the randomly distributed calibration objects is used for realizing the internal reference calibration, and further the external reference attitude of the camera is obtained by utilizing the calibrated internal reference of the camera and the process data thereof.

In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program instructions.

The above embodiments are merely examples and are not intended to limit the scope of the present disclosure, and all modifications, equivalents, and flow charts using the contents of the specification and drawings of the present disclosure or those directly or indirectly applied to other related technical fields are intended to be included in the scope of the present disclosure.

Claims

1. An internal reference calibration method of a camera is characterized by comprising the following steps: acquiring an image shot by the camera on a calibration plate, wherein a plurality of calibration objects which are randomly distributed are arranged on the calibration plate;

identifying an image object of a designated object in the image;

matching the identified image object of the calibration object with the calibration object on the calibration plate;

and performing fitting operation according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration plate to determine the internal reference of the camera.

2. The method of claim 1, wherein matching the image object of the identified calibration object with the calibration object on the calibration board comprises:

determining a position characteristic parameter of the identified image object according to the position of the identified image object in the image;

and matching the identified image object of the calibration object with the calibration object on the calibration board according to the identified position characteristic parameter of the image object and the prestored position characteristic parameter of the calibration object.

3. The method of claim 2, wherein the location characteristic parameter comprises a hash value.

4. The method according to any one of claims 1 to 3, wherein the calibration plate comprises a plurality of calibration plates, wherein the plurality of calibration plates are different in spatial attitude,

the matching the identified image object of the calibration object with the calibration object on the calibration board comprises:

matching the identified image object of the calibration object of each calibration plate with the calibration object on the calibration plate;

the determining the internal reference of the camera by performing fitting operation according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration plate comprises:

and performing fitting operation according to the position of the image object of the calibration object of each calibration plate in the image and the position of the calibration object of each calibration plate matched with the image object on the calibration plate to determine the internal parameters of the camera.

5. The method of claim 4, wherein each of the plurality of calibration plates is disposed in connection with at least another one of the plurality of calibration plates.

6. The method according to claim 4 or 5,

the determining the internal parameters of the camera by performing fitting operation according to the position of the image object of the calibration object of each calibration board in the image and the position of the calibration object of each calibration board matched with the image object on the calibration board comprises:

determining the position of the calibration object of each calibration plate in the world coordinate system according to the position of the calibration object of each calibration plate matched with the image object on the calibration plate;

and performing fitting operation according to the identified position of the image object of the calibration object on each calibration plate in the image and the position of the calibration object of each calibration plate in the world coordinate system to determine the internal parameters of the camera.

7. A method according to any of claims 1-6, characterized in that said calibration objects comprise at least two different size types of calibration objects.

8. The method according to any of claims 1-7, wherein the calibration object comprises a dot.

9. A method of calibrating the relative pose of a camera and an inertial measurement unit, comprising:

acquiring external reference postures of the camera at multiple moments;

acquiring the postures of the inertial measurement unit at a plurality of moments;

and calibrating the relative postures of the camera and the inertial measurement unit according to the external reference postures of the camera at multiple moments and the postures of the inertial measurement unit at multiple moments.

10. The method of claim 9, wherein calibrating the relative pose of the camera and the inertial measurement unit based on the pose of the camera and the pose of the inertial measurement unit at the plurality of times comprises:

determining the external reference posture change of the camera at the adjacent moment in a plurality of moments according to the external reference postures of the camera at the plurality of moments;

determining the attitude change of the inertial measurement unit at adjacent moments in a plurality of moments according to the attitudes of the inertial measurement unit at the moments;

and calibrating the relative postures of the camera and the inertial measurement unit according to the external reference posture change of the camera at the adjacent moment in a plurality of moments and the posture change of the inertial measurement unit at the adjacent moment in a plurality of moments.

11. The method according to claim 9 or 10,

acquiring an image shot by the camera on a calibration plate, wherein a plurality of calibration objects which are randomly distributed are arranged on the calibration plate;

identifying an image object of a designated object in the image;

and acquiring the external reference postures of the camera at a plurality of moments according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration plate.

12. The method of claim 11, wherein matching the image object of the identified calibration object with the calibration object on the calibration board comprises:

determining the position characteristic parameters of the identified image objects according to the positions of the image objects in the image;

and matching the detected image object of the calibration object with the calibration object on the calibration board according to the determined position characteristic parameter and the pre-stored position characteristic parameter of the calibration object.

13. The method of claim 12, wherein the location characteristic parameter comprises a hash value.

14. The method according to any one of claims 11-13, wherein the calibration plate comprises a plurality of calibration plates, wherein the plurality of calibration plates differ in spatial attitude.

15. The method of claim 14, wherein each of the plurality of calibration plates is disposed in connection with at least another one of the plurality of calibration plates.

16. An internal reference calibration device of a camera is characterized by comprising a processor and a memory, wherein,

the memory to store program instructions;

the processor executing the program instructions to:

identifying an image object of a designated object in the image;

17. The apparatus of claim 16,

when the processor matches the identified image object of the calibration object with the calibration object on the calibration board, the processor is specifically configured to:

18. The apparatus of claim 17, wherein the location characteristic parameter comprises a hash value.

19. The apparatus according to any one of claims 16-18, wherein the calibration plate comprises a plurality of calibration plates, wherein the plurality of calibration plates are spatially oriented differently from one another,

when the processor performs fitting operation according to the position of the image object in the image and the position of the calibration object matched with the image object on the calibration board to determine the internal reference of the camera, the processor is specifically configured to:

20. The apparatus of claim 19, wherein each of the plurality of calibration plates is configured to be coupled to at least one other of the plurality of calibration plates.

21. The apparatus of claim 19 or 20,

when the processor performs fitting operation according to the position of the image object of the calibration object of each calibration board in the image and the position of the calibration object of each calibration board matched with the image object on the calibration board to determine the internal reference of the camera, the processor is specifically configured to:

22. An arrangement according to any of claims 16-21, characterized in that said calibration objects comprise at least two different size types of calibration objects.

23. An apparatus according to any of claims 16-22, wherein the calibration object comprises a dot.

24. A device for calibrating the relative attitude of a camera and an inertial measurement unit is characterized by comprising a processor and a memory, wherein,

the memory to store program instructions;

the processor executing the program instructions to:

acquiring external reference postures of the camera at multiple moments;

25. The apparatus of claim 24, wherein the processor, when calibrating the relative pose of the camera and the inertial measurement unit based on the pose of the camera and the pose of the inertial measurement unit at the plurality of times, is specifically configured to:

26. The apparatus of claim 24 or 25, wherein the processor is further configured to:

identifying an image object of a designated object in the image;

27. The apparatus of claim 26, wherein the processor, when matching the identified image object of the calibration object with the calibration object on the calibration board, is specifically configured to:

28. The apparatus of claim 27, wherein the location characteristic parameter comprises a hash value.

29. The apparatus of any one of claims 26-28, wherein the calibration plate comprises a plurality of calibration plates, wherein the plurality of calibration plates are spatially oriented differently from one another.

30. The apparatus of claim 29, wherein each of the plurality of calibration plates is configured to be coupled to at least one other of the plurality of calibration plates.

31. An internal reference calibration system of a camera, comprising a camera and an internal reference calibration apparatus according to any one of claims 16-23,

the camera is used for shooting the calibration plate.

32. A system for calibrating the relative pose of a camera and an inertial measurement unit, comprising a camera, an inertial measurement unit and a device according to any one of claims 24-30,

the camera is used for shooting the calibration plate;

the inertial measurement unit is used for measuring attitude data.

33. A drone comprising the internal reference calibration system of claim 31 or the system of calibrating the relative attitude of the camera and inertial measurement unit of claim 32.

34. A storage device storing program instructions for performing the method of any of claims 1-8, or the method of any of claims 9-15, when the program instructions are run on a processor.