WO2020181409A1

WO2020181409A1 - Capture device parameter calibration method, apparatus, and storage medium

Info

Publication number: WO2020181409A1
Application number: PCT/CN2019/077475
Authority: WO
Inventors: 熊策; 徐彬; 周游
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-03-08
Filing date: 2019-03-08
Publication date: 2020-09-17
Also published as: CN111316325B; CN111316325A

Abstract

Embodiments of the invention provide a capture device parameter calibration method, apparatus, and storage medium. The method comprises: using a first capture device and a second capture device among capture devices to respectively capture, at a first time point, a first image and a second image comprising a target object, and determining first depth information of the target object at the first time point; using the first capture device and the second capture device to respectively capture, at a second time point, a third image and a fourth image comprising the target object, and determining second depth information of the target object at the second time point; and calibrating a rotation relationship and a displacement relationship between the first capture device and the second capture device according to the first depth information, the second depth information, and a change in orientation of a movable platform between the first time point and the second time point. The invention introduces an additional constraint, and reduces the effect of an outlier on parameter calibration, thereby improving the accuracy and efficiency of parameter calibration between a first capture device and a second capture device.

Description

Camera parameter calibration method, equipment and storage medium

Technical field

The embodiments of the present invention relate to the field of unmanned aerial vehicles, and in particular to a method, equipment and storage medium for calibration of parameters of a photographing device.

Background technique

In the prior art, an intelligent movable platform is usually provided with a shooting device, such as a binocular vision module. The binocular vision module can not only provide image information of the target object to the movable platform, but also provide its depth information. So as to provide a richer decision-making basis for intelligent control. The movable platform can be drones, autonomous vehicles, auxiliary driving devices, driving recorders, smart electric vehicles, scooters, balance vehicles, multi-camera smartphones, etc.

Due to external factors such as changes in temperature and humidity, or vibration of the movable platform, it is difficult for the binocular vision module to maintain a stable state on the movable platform. Even if the binocular vision module of the movable platform has been accurately calibrated at the factory, as the movable platform is continuously used, the binocular vision module may undergo minor deformations, resulting in the binocular vision module. The pre-calibrated parameters between the binoculars may no longer be accurate, affecting the control of the movable platform.

However, in the prior art, the accuracy and efficiency of calibration of the parameters of the camera on the movable platform are low, and it is usually necessary to make a special calibration board to perform calibration through manual operation.

Summary of the invention

The embodiments of the present invention provide a method, equipment and storage medium for parameter calibration of a photographing device, so as to improve the accuracy and efficiency of parameter calibration between the first photographing device and the second photographing device in the photographing device.

The first aspect of the embodiments of the present invention is to provide a method for calibrating parameters of a photographing device. The photographing device is configured to be mounted on a movable platform. The photographing device includes at least a first photographing device and a second photographing device. The method includes :

Acquiring a first image and a second image including a target object captured by the first camera and the second camera at the first moment, and determining the first depth information of the target object;

Acquiring a third image and a fourth image including the target object captured by the first photographing device and the second photographing device at a second time, and determining second depth information of the target object;

Acquiring the pose change of the movable platform between the first moment and the second moment;

The parameters of the photographing device are calibrated according to the first depth information, the second depth information, and the pose change.

A second aspect of the embodiments of the present invention is to provide a movable platform, the movable platform is equipped with a camera, the camera at least includes a first camera and a second camera, the mobile platform includes a memory and processor;

The memory is used to store program codes;

The processor calls the program code, and when the program code is executed, is used to perform the following operations:

Acquiring a first image and a second image including a target object respectively captured by the first camera and the second camera at the first moment, and determining the first depth information of the target object;

A third aspect of the embodiments of the present invention is to provide a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the method described in the first aspect.

The camera parameter calibration method, equipment and storage medium provided by this embodiment, through the first camera and the second camera in the camera, the first image and the second image including the target object are respectively captured at the first moment to determine the target The first depth information of the object at the first moment, and the third image and the fourth image including the target object captured by the first camera and the second camera at the second time respectively, determine the second depth information of the target object at the second time Depth information, based on the first depth information, the second depth information, and the position and attitude changes of the movable platform between the first moment and the second moment, calibrate the rotation and displacement relationships between the first camera and the second camera Compared with calibrating the rotation relationship and displacement relationship between the first imaging device and the second imaging device only according to the first depth information and the second depth information, the constraint item is added, and the influence of outliers on the parameter calibration is reduced, The accuracy and efficiency of parameter calibration between the first camera and the second camera are improved.

Description of the drawings

In order to explain the technical solutions in the embodiments of the present invention more clearly, the following will briefly introduce the drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

FIG. 1 is a flowchart of a method for calibrating camera parameters provided by an embodiment of the present invention;

Figure 2 is a schematic diagram of a drone provided by an embodiment of the present invention;

Figure 3 is a schematic diagram of an application scenario provided by an embodiment of the present invention;

4 is a schematic diagram of another application scenario provided by an embodiment of the present invention;

Figure 5 is a schematic diagram of yet another application scenario provided by an embodiment of the present invention;

FIG. 6 is a schematic diagram of another application scenario provided by an embodiment of the present invention;

FIG. 7 is a schematic diagram of Rolling Shutter provided by an embodiment of the present invention;

Fig. 8 is a schematic structural diagram of a movable platform provided by an embodiment of the present invention.

Reference signs:

20: main camera; 21: first camera; 22: second camera;

30: target object; 31: first image; 32: second image;

33: third image; 34: fourth image; 40: image;

41: image; 42: image; 51: image; 52: image;

70: movable platform; 71: first camera; 72: second camera;

73: memory; 74: processor.

detailed description

The technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or a central component may also exist. When a component is considered to be "connected" to another component, it can be directly connected to another component or there may be a centered component at the same time.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the present invention. The terms used in the description of the present invention herein are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The term "and/or" as used herein includes any and all combinations of one or more related listed items.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

The embodiment of the present invention provides a method for parameter calibration of a photographing device. FIG. 1 is a flowchart of a method for calibrating camera parameters provided by an embodiment of the present invention. The photographing device is mounted on a movable platform, and may include being installed on the mobile platform. The photographing device includes at least a first photographing device and a second photographing device. Optionally, the movable platform includes a drone or a vehicle. This embodiment takes a drone as an example for schematic description. The drone is equipped with a camera. The camera at least includes a first camera and a second camera. Optionally, the camera is a dual camera of the drone. The eye system is composed of a first camera and a second camera. As shown in FIG. 2, the drone includes a main camera 20 and a binocular system, and the binocular system includes a first camera 21 and a second camera 22. Specifically, the first photographing device 21 may be the left-eye camera of the drone, and the second photographing device 22 may be the right-eye camera of the drone. It can be understood that this is only a schematic description, and does not limit the specific shape and structure of the drone.

As shown in Figure 1, the method for calibrating the parameters of the camera in this embodiment may include:

Step S101: Acquire a first image and a second image including a target object captured by the first camera and the second camera at the first moment, and determine first depth information of the target object.

In this embodiment, the first photographing device 21 and the second photographing device 22 can photograph the same target object at the same time. As shown in Figure 3, 30 represents the target object in the three-dimensional space, 31 represents the first image captured by the first camera 21 at time t1 that includes the

target object

30, and 32 represents the target captured by the second camera 22 at time t1 and includes the target The second image of the object 30. The processor in the drone can obtain the first image 31 taken by the first camera 21 at time t1 and the second image 32 taken by the second camera 22 at time t1, and use triangulation measurement to determine the target object 30 The depth information and the triangulation error. Here, the depth information of the target object 30 at time t1 is recorded as the first depth information, and the triangulation error at time t1 is recorded as the first error information.

It can be understood that the depth information of the target object 30 can be determined according to the depth information of the three-dimensional points on the target object 30. As shown in FIG. 3, the point P represents any three-dimensional point on the target object 30. Specifically, the depth information of the point P can be determined according to the three-dimensional position information of the point P in the three-dimensional space. For example, the three-dimensional position information of the point P in the three-dimensional space can be calculated by the triangulation method. Optionally, the three-dimensional space can be a world coordinate system.

As shown in Figure 4, G represents the coordinate origin of the world coordinate system, C ₀ , C ₁ , and C ₂ represent the coordinate origin of the camera coordinate system when the camera is in three different poses. Image 40, image 41 and image 42 are in turn Images taken when the camera is in three different poses. It can be understood that the camera can shoot the same target object in different poses. The camera can be the main camera, the first camera or the second camera of the drone. As shown in Figure 4, the same three-dimensional point on the target object, such as the mapping point of point P in different images, may have different positions in the corresponding images. For example, the mapping point of point P in image 40 is p0, point P The mapping point in the image 41 is p1, and the mapping point of the point P in the image 42 is p2. The positions of p0, p1, and p2 in the corresponding images may be different.

According to the conversion relationship between the world coordinate system and the pixel plane coordinate system, the three-dimensional coordinates (x _w , y _w , z _w ) of the three-dimensional point on the target object in the world coordinate system and the mapping point of the three-dimensional point in the image can be obtained. The position information in the image, such as the relationship of pixel coordinates (μ, ν), is specifically shown in the following formula (1):

Among them, z _c represents the coordinates of the three-dimensional point on the Z axis of the camera coordinate system. K represents the internal parameters of the camera, R represents the rotation matrix of the camera coordinate system relative to the world coordinate system, T represents the translation matrix of the camera coordinate system relative to the world coordinate system, and R and T are the external parameters of the camera. In this embodiment, the internal parameter K of the camera is a known quantity. Optional,

_{_{Wherein, α x = fm x, α}} y = fm y, f represents a focal length of the camera; _x m and _y m is the image corresponding to the x-axis, the number of pixels per unit distance in the y-axis direction. γ is the distortion parameter between the x-axis and the y-axis. μ ₀ , v ₀ is the position of the optical center of the camera in the pixel plane coordinate system. According to the above formula (1), it can be seen that when K, (μ, ν), z _c , R and T are known, the three-dimensional coordinates of the three-dimensional point on the target object in the world coordinate system (x _w , y _w ,z _w ).

However, the theoretical projection point of the three-dimensional point on the target object in the image may not be exactly the same as the projection point actually observed in the image. For example, theoretically the projection point of the point P on the normalized plane of the C ₀ stand is denoted as p′ ₀ .

among them,

Indicates the rotation matrix of the camera coordinate system relative to the world coordinate system when the coordinate origin of the camera coordinate system is C ₀ .

Indicates the translation matrix of the camera coordinate system relative to the world coordinate system when the coordinate origin of the camera coordinate system is C ₀ . In addition, the rotation matrix of the camera coordinate system when the coordinate origin is C ₁ relative to the camera coordinate system when the coordinate origin is C ₀ can also be recorded as

The translation matrix of the camera coordinate system when the coordinate origin is C ₁ relative to the camera coordinate system when the coordinate origin is C ₀ is recorded as

The rotation matrix of the camera coordinate system when the coordinate origin is C ₂ relative to the camera coordinate system when the coordinate origin is C ₁ is recorded as

The translation matrix of the camera coordinate system when the coordinate origin is C ₂ relative to the camera coordinate system when the coordinate origin is C ₁ is recorded as

For example, the projection point of the point P actually observed in the image 40 is denoted as p ₀ , p ₀ =[u ₀ ,v ₀ ] ^T. Ideally, p′ ₀ = p ₀ , but in actual situations, the two are not the same. The errors generated by p′ ₀ and p ₀ are recorded as reprojection errors. Here, the points can be determined by the following formula (2) The three-dimensional coordinates of P in the world coordinate system (x _w ,y _w ,z _w ):

, N represents the number of seats or the number of the image shown in Figure _{_{4, (u i, v i)}} T i-th imaging device captures an image of the point P _{_{(x w, y w, z}} w) The projected pixel coordinates. As shown in Figure 4, for three different camera positions C ₀ , C ₁ , C ₂ , the position of the point P calculated by the projection points p0, p1 and p2 in the image may not be the same position. Therefore, the above formula can be used ，Use the optimization problem to obtain the three-dimensional coordinates of point P (x _w , y _w , z _w ).

At a certain moment, according to the mapping point of the point P in the images captured by the first camera and the second camera, the external parameters between the first camera and the second camera, the internal parameters of the first camera, and The internal parameters of the second camera can be calculated using the triangulation measurement method to obtain the depth information and triangulation error of the three-dimensional point P. For example, [Z,Cost]=triangulate(p _L ,p _R ,R _LR ,t _LR ,K _L ,K _R ), where triangulate represents the triangulation measurement method, and p _L represents the point P taken by the first camera The pixel coordinates of the mapping point in the image, p _R represents the pixel coordinates of the mapping point of the point P in the image captured by the second camera, R _LR represents the rotation relationship between the first camera and the second camera, t _LR It represents the displacement relationship between the first camera and the second camera, K _L represents the internal parameter of the first camera, and K _R represents the internal parameter of the second camera. Z represents the depth information of the three-dimensional point P calculated by the triangulation measurement method, and Cost represents the triangulation error, which can specifically be the error of the depth information of the three-dimensional point P, as shown in Figure 5, the point P is a three-dimensional space A three-dimensional point in p1 and p2 are the mapping points of point P in two different images (for example, image 51 and image 52). The two different images can be the first camera and the second camera at the same time taking pictures. The epipolar line corresponding to the point p1 is in the image 52. When the epipolar line is searched for the epipolar line, the point found is p2'. Due to the error between p2' and p2, the depth of the three-dimensional point P after the triangulation exists For a certain error, such as PP' shown in Figure 5, the error is the triangulation error.

As shown in FIG. 3, at time t1, the pixel coordinates of the projection point of the point P actually observed from the first image 31 in the first image 31 is p11, and the projection point of the point P actually observed from the second image 32 The pixel coordinate in the second image 32 is p12. The external parameters between the first camera and the second camera are denoted as R _LR and t _LR , where R _LR represents the rotational relationship between the first camera and the second camera, and the rotational relationship is specifically the first The rotation matrix of the camera coordinate system of the imaging device relative to the camera coordinate system of the second imaging device, or the rotation matrix of the camera coordinate system of the second imaging device relative to the camera coordinate system of the first imaging device. t _LR represents the displacement relationship between the first camera and the second camera. The displacement relationship is specifically the translation matrix of the camera coordinate system of the first camera relative to the camera coordinate system of the second camera, or the second camera The translation matrix of the camera coordinate system relative to the camera coordinate system of the first camera. The internal parameter of the first camera is marked as K _L , and the internal parameter of the second camera is marked as K _R. K _L and K _R are fixed values. In this embodiment, calibration is not required, and R _LR and t _LR are the original Examples of parameters to be calibrated.

As shown in Fig. 3, at time t1, according to p11, p12, R _LR , t _LR , K _L and K _R , the depth information and triangulation error of the three-dimensional point P can be calculated by the triangulation measurement method.

As shown in FIG. 3, O1 represents the optical center of the first camera, and O2 represents the optical center of the second camera. At time t1, the depth information of the point P can be determined based on the first camera or the second camera. For example, the depth information of the point P is determined with the first camera as a reference, the depth information of the point P determined with the first camera as the reference at t1 is recorded as z1, and the triangulation error at time t1 is recorded as Cost1. Correspondingly, [z1,Cost1]=triangulate(p11,p12,R _LR ,t _LR ,K _L ,K _R ).

Step S102: Acquire a third image and a fourth image including the target object captured by the first camera and the second camera at a second moment, and determine second depth information of the target object.

As shown in FIG. 3, 33 represents the third image including the target object 30 captured by the first camera 21 at time t2, and 34 represents the fourth image including the target object 30 captured by the first camera 21 at time t2. The processor in the drone can obtain the third image 33 taken by the first camera 21 and the fourth image 34 taken by the second camera 22 at time t2, and use the triangulation method to determine the depth information and triangulation of the point P Error, here, the depth information at point P at time t2 is recorded as second depth information, and the triangulated error at time t2 is recorded as second error information.

As shown in Fig. 3, at time t2, the pixel coordinate of the projection point of the point P actually observed from the third image 33 in the third image 33 is p21, and the projection point of the point P actually observed from the fourth image 34 The pixel coordinate in the fourth image 34 is p22. The depth information of the point P determined on the basis of the first camera at time t2 is recorded as z2, and the triangulation error at time t2 is recorded as Cost2. Correspondingly, [z2,Cost2]=triangulate(p21,p22,R _LR , t _LR ,K _L ,K _R ).

Step S103: Obtain the pose change of the movable platform between the first moment and the second moment.

In this embodiment, between time t1 and time t2, the pose of the drone may change. Here, the pose change of the drone includes a position change or a posture change. In other words, between time t1 and time t2, the drone may move and/or rotate. When the drone moves between t1 and t2, the speedometer information can be used to determine the travel distance of the drone between t1 and t2. When the drone rotates between time t1 and time t2, the attitude change of the drone can be measured by the inertial measurement unit on the drone.

For example, taking the position change of the drone as an example, between t1 and t2, the movement distance of the drone is d, correspondingly, the movement distance of the first camera 21 or the second camera 22 is also d ,As shown in Figure 6. It can be understood that when the attitude of the drone changes between time t1 and time t2, the movement distance of the first camera 21 and the movement distance of the second camera 22 may be different.

Step S104: Calibrate the parameters of the camera according to the first depth information, the second depth information, and the pose change.

Optionally, the camera parameters include: external parameters between the first camera and the second camera. Optionally, the external parameters between the first camera and the second camera include: a rotation relationship and a displacement relationship between the first camera and the second camera.

In this embodiment, the first depth information may be the depth information of the point P determined on the basis of the first camera at time t1, or the depth information of the point P determined on the basis of the second camera at time t1. Similarly, the second depth information may be the depth information of the point P determined on the basis of the first camera at time t2, or the depth information of the point P determined on the basis of the second camera at time t2.

In the case of taking the first camera as the reference, optionally, according to the depth information of the point P determined on the basis of the first camera at time t1, and the depth information of the point P determined on the basis of the first camera at time t2 , And the movement distance of the first imaging device 21 between time t1 and time t2, calibrating the rotation relationship and displacement relationship between the first imaging device and the second imaging device.

In the case of using the second camera as a reference, optionally, according to the depth information of the point P determined on the basis of the second camera at time t1, and the depth information of the point P determined on the basis of the second camera at time t2 , And the movement distance of the second imaging device 22 between time t1 and time t2, calibrating the rotation relationship and displacement relationship between the first imaging device and the second imaging device.

Optionally, the calibrating the parameters of the photographing device according to the first depth information, the second depth information, and the pose change includes: according to the first depth information and the second depth information And the geometric constraints between the pose changes and calibrate the camera parameters.

As shown in FIG. 6, z1 represents the depth information of the point P determined on the basis of the first imaging device at time t1, and z2 represents the depth information of the point P determined on the basis of the first imaging device at time t2. d represents the movement distance of the first camera between time t1 and time t2. z1, z2, and d constitute the three sides of the triangle. According to the geometric constraints between the three sides of the triangle, that is, the sum of the two sides is greater than the third side, and the difference between the two sides is less than the third side, the first camera and the The rotation relationship and displacement relationship between the second camera. Therefore, when z1, z2, and d are all accurate, the relationship between the three should satisfy the following geometric constraints, which are specifically shown in the following formula (3):

|d-z2|＜z1＜|d+z2| (3)

Optionally, the calibration of the camera parameters according to the geometric constraints between the first depth information, the second depth information, and the pose change includes: according to the first depth information, the The geometric constraints between the second depth information and the pose change are used to determine target error information; and the camera parameters are calibrated according to the target error information.

For example, after determining the geometric constraint shown in the above formula (3), further, according to the geometric constraint, the following formula (4) is obtained:

Among them, b represents the baseline distance between the first camera and the second camera, when z1 represents the depth information of the point P determined on the basis of the first camera at t1, and z2 represents the depth information determined on the basis of the first camera When the point P is the depth information at time t2, f represents the focal length of the first camera.

In some embodiments, the first depth information and the second depth information are determined based on the first photographing device; the first depth information, the second depth information, and the The geometric constraints between the pose changes and the determination of target error information include: according to the first depth information, the second depth information, and the geometric constraints between the pose changes, the first camera, and The distance information between the second imaging devices and the focal length of the first imaging device determine the target error information.

For example, the target error information is recorded as Cost3, and Cost3 can be determined according to formula (4), and Cost3 is specifically shown in formula (5) as follows:

Among them, z1 represents the depth information of the point P determined on the basis of the first imaging device at time t1, and z2 represents the depth information of the point P determined on the basis of the first imaging device at time t2. d represents the movement distance of the first camera between time t1 and time t2. b represents the baseline distance between the first camera and the second camera. f represents the focal length of the first camera. When there is an error in one or more of the values of z1, z2, or d, the relationship between the three of them does not meet the constraints of formula (4), that is, there may be the first two situations as in formula (5) At this time, a target error information can be given to indicate this error. As shown in formula (5), when the relationship between the three meets the aforementioned constraints, the target error information can be set to zero.

In other embodiments, the first depth information and the second depth information are determined on the basis of the second camera; the first depth information, the second depth information, and the The geometric constraints between the pose changes and the determination of target error information include: according to the first depth information, the second depth information, and the geometric constraints between the pose changes, the first camera The distance information from the second imaging device and the focal length of the second imaging device determine the target error information.

For example, z1' indicates the depth information of the point P determined on the basis of the second camera at t1, z2' indicates the depth information of the point P determined on the basis of the second camera at t2, and d'indicates the depth information at time t1 The movement distance of the second camera between t2 and t2. z1', z2', and d'constitute the three sides of the triangle. According to the geometric constraint between z1', z2', and d', the relationship described in formula (3) is obtained, and further, the same is obtained according to the geometric constraint In the relationship described in formula (4), and similarly to the target error information described in formula (5), in the target error information, b represents the baseline distance between the first camera and the second camera, f Indicates the focal length of the second camera.

After the target error information Cost3 is determined, the rotation relationship and displacement relationship between the first imaging device and the second imaging device can be calibrated according to the target error information.

Optionally, the calibrating the parameters of the shooting device according to the target error information includes: determining a cost function according to the target error information; and calibrating the parameters of the shooting device according to the cost function.

For example, Cost1, Cost2, and Cost3 constitute the cost function Cost, and Cost is specifically shown in the following formula (6):

Cost=[Cost1,Cost2,Cost3] (6)

Further, according to the cost function, the rotation relationship and the displacement relationship between the first imaging device and the second imaging device are determined.

Optionally, the calibrating the parameters of the photographing device according to the cost function includes: optimizing the solution of the cost function, and determining the parameters of the photographing device that can minimize the second norm of the cost function.

For example, by solving the optimization problem described in the following formula (7), the rotation relationship and the displacement relationship between the first camera and the second camera are determined:

Wherein, R _LR represents the rotation relationship between the first camera and the second camera, and t _LR represents the displacement relationship between the first camera and the second camera. ‖Cost‖ ² represents the second norm of the cost function Cost. Specifically, adjust the parameters R _LR and t _LR so that

The smallest when

The R _LR and t _LR corresponding to the minimum hour are the final calibration parameters.

It is understandable that when the triangulation error Cost1 at time t1, the triangulation error Cost2 at t2 and the target error information Cost3 are optimized here, the least squares norm shown in formula (7) may not be used. The method of using other optimization methods is not limited here.

It is understandable that the movable platform not only includes two shooting devices, such as a first shooting device and a second shooting device, but also more shooting devices. When the movable platform includes more shooting devices, this embodiment The method for calibrating the parameters of the shooting device can be applied to the calibration of external parameters between any two shooting devices among the plurality of shooting devices.

In this embodiment, the first camera and the second camera on the movable platform respectively take the first image and the second image including the target object at the first moment to determine the first depth information of the target object at the first moment, and According to the first photographing device and the second photographing device respectively photographing the third image and the fourth image including the target object at the second time, the second depth information of the target object at the second time is determined, according to the first depth information and the second depth Information and the position and posture changes of the movable platform between the first moment and the second moment to calibrate the rotation relationship and displacement relationship between the first camera and the second camera, compared to only based on the first depth information and the second camera. Two depth information calibrate the rotation relationship and displacement relationship between the first camera and the second camera, increase the constraint item, reduce the influence of outliers on the parameter calibration, and improve the impact on the first camera and the second camera The accuracy and efficiency of parameter calibration.

The embodiment of the present invention provides a method for parameter calibration of a photographing device. On the basis of the foregoing embodiment, the first camera and the second camera may be a global shutter (Global Shutter) camera, or the first and the second camera may be a rolling shutter (Rolling Shutter) camera. In the images captured by Global Shutter cameras, the exposure time of each row of pixels is the same.

Optionally, the position change is determined according to the time difference between the first time and the second time, and the moving speed of the movable platform between the first time and the second time of.

For example, when the first camera and the second camera are both Global Shutter cameras, the movement distance d in the above embodiment can be expressed as d=(t2-t1)*speed, which can be the movable platform at time t1 And the movement speed within t2 time.

Because in the image taken by the Rolling Shutter camera, the exposure time of each row of pixels is different. As shown in Figure 7, a certain frame of image includes N rows of pixels, and each row starts to be exposed at a different time point. For example, the exposure start time of the first line is recorded as Start1, and the exposure start time of the second line is recorded as Start2. The start time of the third line of exposure is recorded as Start3, and so on. Optionally, the exposure time length of each row is the same, that is, the time interval between the exposure start time and the exposure end time of each row is the same. In addition, except for the first row, the time interval between the exposure start time of each row and the exposure start time of the previous row is the same. For example, the time interval between Start1 and Start2 is the same as the time interval between Start2 and Start3.

Therefore, when the first camera and the second camera are Rolling Shutter cameras, at different times, such as t1 and t2, the mapping points of the same three-dimensional point P in different images are located in different rows of the corresponding images. As shown in FIG. 3, at time t1, the mapping point of the three-dimensional point P in the first image 31 is p11. At time t2, the mapping point of the three-dimensional point P in the third image 33 is p21. The line where p11 is located in the first image 31 and the line where p21 is located in the third image 33 are different, so the exposure time of p11 and p21 may be different. As a result, the exposure time of the target object in the first image 31 and the third image 33 is different.

Or, at time t1, the mapping point of the three-dimensional point P in the second image 32 is p12. At time t2, the mapping point of the three-dimensional point P in the fourth image 34 is p22. The line where p12 is located in the second image 32 and the line where p22 is located in the fourth image 34 are different, so the exposure time of p12 and p22 may be different. As a result, the exposure time of the target object in the second image 32 and the fourth image 34 is different.

Optionally, the position change is based on the time difference between the first time and the second time, the exposure time difference of the target object in the first image and the third image, and the The moving speed of the movable platform between the first moment and the second moment is determined.

For example, when the first camera and the second camera are Rolling Shutter cameras, the same 3D point P may be in different exposure lines due to the movement of the mobile platform in the image, resulting in the same 3D point P at different times. The exposure time in the image is also different. The movement distance d in the above embodiment can be compensated according to the difference in the exposure time of the target object in the first image 31 and the third image 33. Optionally, the compensated d can be expressed as d=(t2- t1+Δt)*speed, where Δt represents the difference in the exposure time of the target object in the first image 31 and the third image 33.

Optionally, the position change is based on the time difference between the first time and the second time, the exposure time difference of the target object in the second image and the fourth image, and the The moving speed of the movable platform between the first moment and the second moment is determined.

For example, when the first photographing device and the second photographing device are Rolling Shutter cameras, the movement distance d in the above embodiment can be calculated according to the difference in the exposure time of the target object in the second image 32 and the fourth image 34. Compensation, optionally, the compensated d can be expressed as d=(t2-t1+Δt)*speed, where Δt represents the difference value of the exposure time of the target object in the second image 32 and the fourth image 34.

This embodiment compensates for the movement distance of the movable platform through the different exposure time of the target object in the images captured by the same camera at different times, reduces the influence of the rolling shutter camera on the parameter calibration of the camera, and further improves the shooting The accuracy of the device parameter calibration also enables the camera parameter calibration method to be applicable to a rolling shutter camera, which improves the application range of the camera parameter calibration method.

The embodiment of the present invention provides a movable platform. FIG. 8 is a schematic structural diagram of a movable platform provided by an embodiment of the present invention. As shown in FIG. 8, the movable platform 70 includes at least a first camera 71 and a second camera 72, a memory 73 and a processor 74. The memory is used to store program code; the processor 74 calls the program code, and when the program code is executed, is used to perform the following operations: acquire the first camera and the second camera at the first moment The first image and the second image of the target object captured by the device, and the first depth information of the target object is determined; the first image and the second image captured by the first camera and the second camera at the second moment include all The third image and the fourth image of the target object, and determine the second depth information of the target object; acquire the pose change of the movable platform between the first moment and the second moment; The first depth information, the second depth information, and the pose change are used to calibrate the camera parameters.

Optionally, the processor 74 is specifically configured to calibrate the parameters of the photographing device according to the first depth information, the second depth information, and the pose change: The geometric constraint between the second depth information and the pose change is used to calibrate the camera parameters.

Optionally, the processor 74 is specifically configured to calibrate the camera parameters according to the geometric constraints between the first depth information, the second depth information, and the pose change: The geometric constraints between the first depth information, the second depth information, and the pose change determine target error information; and the camera parameters are calibrated according to the target error information.

Optionally, when calibrating the parameters of the camera according to the target error information, the processor 74 is specifically configured to: determine a cost function according to the target error information; and calibrate the camera according to the cost function parameter.

Optionally, when calibrating the parameters of the photographing device according to the cost function, the processor 74 is specifically configured to: optimize the cost function to determine the smallest two-norm of the cost function The camera parameters.

Optionally, the camera parameters include: external parameters between the first camera and the second camera.

Optionally, the external parameters between the first camera and the second camera include: a rotation relationship and a displacement relationship between the first camera and the second camera.

Optionally, the first depth information and the second depth information are determined based on the first camera; the processor 74 is based on the first depth information, the second depth information, and the When determining the target error information, the geometric constraints between the pose changes are specifically used to: according to the first depth information, the second depth information, and the geometric constraints between the pose changes, the first The distance information between a photographing device and the second photographing device and the focal length of the first photographing device determine the target error information.

Optionally, the first depth information and the second depth information are determined based on the second camera; the processor 74 is based on the first depth information, the second depth information, and the When determining the target error information, the geometric constraints between the pose changes are specifically used to: according to the first depth information, the second depth information, and the geometric constraints between the pose changes, the first The distance information between a photographing device and the second photographing device and the focal length of the second photographing device determine the target error information.

Optionally, the posture change includes a position change or a posture change.

Optionally, the movable platform includes a drone or a vehicle.

The specific principles and implementation manners of the movable platform provided by the embodiment of the present invention are similar to the foregoing embodiment, and will not be repeated here.

In addition, this embodiment also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the method for calibrating the camera parameters described in the foregoing embodiment.

In the several embodiments provided by the present invention, it should be understood that the disclosed device and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware, or may be implemented in the form of hardware plus software functional units.

The above-mentioned integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor execute the method described in the various embodiments of the present invention. Part of the steps. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code .

Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, only the division of the above-mentioned functional modules is used as an example. In practical applications, the above-mentioned functions can be allocated by different functional modules as required, namely, the device The internal structure is divided into different functional modules to complete all or part of the functions described above. For the specific working process of the device described above, reference may be made to the corresponding process in the foregoing method embodiment, which is not repeated here.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: The technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be equivalently replaced; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. range.

Claims

A method for parameter calibration of a photographing device, characterized in that the photographing device is used to be mounted on a movable platform, the photographing device includes at least a first photographing device and a second photographing device, and the method includes:

Acquiring a first image and a second image including a target object respectively captured by the first camera and the second camera at the first moment, and determining the first depth information of the target object;

Acquiring a third image and a fourth image including the target object captured by the first photographing device and the second photographing device at a second time, and determining second depth information of the target object;

Acquiring the pose change of the movable platform between the first moment and the second moment;

The parameters of the photographing device are calibrated according to the first depth information, the second depth information, and the pose change.
The method according to claim 1, wherein the calibrating the parameters of the photographing device according to the first depth information, the second depth information, and the pose change comprises:

Calibration of the camera parameters according to the geometric constraints between the first depth information, the second depth information, and the pose change.
The method according to claim 2, wherein the calibrating the parameters of the shooting device according to the geometric constraints between the first depth information, the second depth information, and the pose change comprises:

Determine target error information according to the geometric constraints between the first depth information, the second depth information, and the pose change;

According to the target error information, the camera parameters are calibrated.
The method according to claim 3, wherein the calibration of the parameters of the photographing device according to the target error information comprises:

Determine a cost function according to the target error information;

According to the cost function, the camera parameters are calibrated.
The method according to claim 4, wherein the calibrating the parameters of the photographing device according to the cost function comprises:

Optimize the cost function to determine the parameters of the photographing device that can minimize the second norm of the cost function.
The method according to any one of claims 1 to 5, wherein the photographing device parameters comprise: external parameters between the first photographing device and the second photographing device.
7. The method according to claim 6, wherein the external parameters between the first photographing device and the second photographing device comprise:

The rotation relationship and the displacement relationship between the first imaging device and the second imaging device.
The method according to any one of claims 3-7, wherein the first depth information and the second depth information are determined on the basis of the first photographing device;

The determining target error information according to the geometric constraints between the first depth information, the second depth information, and the pose change includes:

According to the first depth information, the second depth information and the geometric constraints between the pose changes, the distance information between the first camera and the second camera, and the first The focal length of the photographing device determines the target error information.
7. The method according to any one of claims 3-7, wherein the first depth information and the second depth information are determined on the basis of the second photographing device;

The determining target error information according to the geometric constraints between the first depth information, the second depth information, and the pose change includes:

According to the first depth information, the second depth information and the geometric constraints between the pose changes, the distance information between the first camera and the second camera, and the second The focal length of the photographing device determines the target error information.
The method according to any one of claims 1-9, wherein the pose change includes a position change or a posture change.
The method according to claim 10, wherein the position change is based on the time difference between the first moment and the second moment, and the movable platform between the first moment and the The speed of movement between the second moment is determined.
The method according to claim 10, wherein the position change is based on the time difference between the first moment and the second moment, and the target object is in the first image and the third moment. The exposure time difference in the image and the moving speed of the movable platform between the first moment and the second moment are determined.
The method according to claim 10, wherein the position change is based on the time difference between the first moment and the second moment, and the target object is in the second image and the fourth moment. The exposure time difference in the image and the moving speed of the movable platform between the first moment and the second moment are determined.
The method according to any one of claims 1-13, wherein the movable platform comprises a drone or a vehicle.
A movable platform, characterized in that the movable platform is equipped with a camera, the camera includes at least a first camera and a second camera, and the mobile platform includes a memory and a processor;

The memory is used to store program codes;

The processor calls the program code, and when the program code is executed, is used to perform the following operations:

Acquiring a first image and a second image including a target object respectively captured by the first camera and the second camera at the first moment, and determining the first depth information of the target object;

Acquiring a third image and a fourth image including the target object captured by the first photographing device and the second photographing device at a second time, and determining second depth information of the target object;

Acquiring the pose change of the movable platform between the first moment and the second moment;

The parameters of the photographing device are calibrated according to the first depth information, the second depth information, and the pose change.
The mobile platform of claim 15, wherein the processor uses the first depth information, the second depth information, and the pose change to calibrate the parameters of the shooting device. in:

Calibration of the camera parameters according to the geometric constraints between the first depth information, the second depth information, and the pose change.
The movable platform of claim 16, wherein the processor calibrates the camera device according to the geometric constraints between the first depth information, the second depth information, and the pose change When parameter, it is specifically used for:

Determine target error information according to the geometric constraints between the first depth information, the second depth information, and the pose change;

According to the target error information, the camera parameters are calibrated.
The mobile platform according to claim 17, wherein the processor is specifically configured to: when calibrating the parameters of the photographing device according to the target error information:

Determine a cost function according to the target error information;

According to the cost function, the camera parameters are calibrated.
The mobile platform according to claim 18, wherein the processor is specifically configured to: when calibrating the parameters of the photographing device according to the cost function:

An optimized solution is performed on the cost function, and the photographing device parameters that can minimize the two norm of the cost function are determined.
The movable platform according to any one of claims 15-19, wherein the parameters of the photographing device comprise: external parameters between the first photographing device and the second photographing device.
22. The movable platform of claim 20, wherein the external parameters between the first camera and the second camera include:

The rotation relationship and the displacement relationship between the first imaging device and the second imaging device.
The movable platform according to any one of claims 17-21, wherein the first depth information and the second depth information are determined on the basis of the first camera;

When the processor determines target error information according to the geometric constraints between the first depth information, the second depth information, and the pose change, it is specifically configured to:

According to the first depth information, the second depth information and the geometric constraints between the pose changes, the distance information between the first camera and the second camera, and the first The focal length of the photographing device determines the target error information.
The movable platform according to any one of claims 17-21, wherein the first depth information and the second depth information are determined on the basis of the second camera;

When the processor determines target error information according to the geometric constraints between the first depth information, the second depth information, and the pose change, it is specifically configured to:

According to the first depth information, the second depth information and the geometric constraints between the pose changes, the distance information between the first camera and the second camera, and the second The focal length of the photographing device determines the target error information.
The movable platform according to any one of claims 15-23, wherein the change in position and posture includes a change in position or a change in posture.
The movable platform according to claim 24, wherein the position change is based on the time difference between the first time and the second time, and the time difference between the movable platform at the first time and the The movement speed between the second moments is determined.
The movable platform according to claim 24, wherein the position change is based on the time difference between the first moment and the second moment, and the target object in the first image and the The exposure time difference in the third image and the moving speed of the movable platform between the first moment and the second moment are determined.
The movable platform of claim 24, wherein the position change is based on the time difference between the first moment and the second moment, and the target object is in the second image and the The exposure time difference in the fourth image and the moving speed of the movable platform between the first moment and the second moment are determined.
The movable platform according to any one of claims 15-27, wherein the movable platform comprises a drone or a vehicle.
A computer-readable storage medium, characterized in that a computer program is stored thereon, and the computer program is executed by a processor to implement the method according to any one of claims 1-14.