CN108447097A - Depth camera scaling method, device, electronic equipment and storage medium - Google Patents

Abstract

The embodiment of the invention discloses a kind of depth camera scaling method, device, electronic equipment and storage medium, wherein this method includes：Synchronous acquisition image, each depth camera are respectively provided with corresponding label at least two depth cameras in control panorama depth camera system during the motion；Obtain pose when at least one first label camera acquires each frame image；If the second label camera occurs with the first label camera, historical angel is overlapping, is overlapped corresponding image according to historical angel and pose when the first label camera acquires each frame image, calculate the second label camera and the first label camera synchronization relative pose.The embodiment of the present invention is based on pre-set first label camera in panorama depth camera system, during acquiring image when camera system is in movement, is overlapped using historical angel, determines relative pose between camera, realize the quick online self-calibration of polyphaser.Without demarcating object, without having fixed big visual field overlapping range between adjacent cameras.

Description

Depth camera calibration method, device, electronic equipment and storage medium

technical field

Embodiments of the present invention relate to machine vision technology, and in particular to a depth camera calibration method, device, electronic equipment, and storage medium.

Background technique

With the development of robot navigation, virtual reality and augmented reality technology, RGB-D cameras (that is, depth cameras) are widely used in robot navigation, static scene reconstruction and dynamic human body reconstruction. RGB-D camera combines traditional RGB camera and depth camera, which has the advantages of high precision, small size, large amount of information, passivity and rich information.

At present, due to the limited Field of View (FoV) of a single RGB-D camera, using a single RGB-D camera for navigation cannot obtain all the surrounding information at the same time, and the small field of view also has certain limitations for 3D reconstruction, such as dynamic object reconstruction However, it is often impossible to obtain full-body or multi-object models, and for example, it is impossible to reconstruct dynamic objects and static objects in the field of view at the same time.

If a multi-camera system is used, precise extrinsic calibration of the cameras is required. At present, the camera calibration method based on specific calibration objects is generally used, which is relatively cumbersome and requires a large overlapping range of fields of view between adjacent cameras, which cannot adapt to the situation where the overlapping range of fields of view is small. However, there is a calibration method based on Simultaneous Localization And Mapping (SLAM for short), which uses the camera to move according to a preset trajectory to collect images, and then processes the images offline for calibration, which cannot quickly and simultaneously calibrate 360° panoramic RGB-D cameras online .

Contents of the invention

Embodiments of the present invention provide a depth camera calibration method, device, electronic equipment, and storage medium, so as to realize real-time online panoramic multi-camera self-calibration without human intervention.

In a first aspect, an embodiment of the present invention provides a method for calibrating a depth camera, including:

Controlling at least two depth cameras in the panoramic depth camera system to acquire images synchronously during motion, wherein each depth camera sets a corresponding label;

Obtain the pose of at least one first label camera when capturing each frame of image;

If the second tag camera overlaps with the first tag camera in terms of historical viewing angles, calculate the relationship between the second tag camera and the The relative pose of the first labeled camera at the same moment.

In the second aspect, the embodiment of the present invention also provides a depth camera calibration device, including:

A camera control module, configured to control at least two depth cameras in the panoramic depth camera system to acquire images synchronously during motion, wherein each depth camera is provided with a corresponding label;

A pose acquisition module, configured to acquire poses of at least one first label camera when each frame of image is collected;

The relative pose calculation module is used to overlap the corresponding image according to the historical view angle and the pose of each frame image collected by the first tag camera when the second tag camera overlaps with the first tag camera. , calculating the relative pose of the second tag camera and the first tag camera at the same moment.

In a third aspect, an embodiment of the present invention also provides an electronic device, including:

one or more processors;

memory for storing one or more programs;

A panoramic depth camera system, including at least two depth cameras, the at least two depth cameras cover a panoramic field of view for collecting images;

When the one or more programs are executed by the one or more processors, the one or more processors implement the depth camera calibration method according to any embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the depth camera calibration method as described in any embodiment of the present invention is implemented.

In the embodiment of the present invention, based on the preset first tag camera in the panoramic depth camera system, during the process of capturing images while the panoramic depth camera system is moving, the second tag camera and the first tag camera's historical angle of view overlap are used to determine the second The relative pose of the tag camera to the first tag camera enables fast online self-calibration of multiple cameras. This method does not require a calibration object, and does not require a fixed large field of view overlapping range between adjacent cameras. When building a camera system, there are small or no overlapping viewing angles between adjacent cameras. The movement of the panoramic depth camera system makes different The camera collects images with overlapping historical perspectives, which can be calibrated accordingly. This method has a small amount of calculation and can be calibrated online based on the CPU.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a flow chart of a depth camera calibration method provided by Embodiment 1 of the present invention;

FIG. 2 is a schematic diagram of a panoramic depth camera system provided by Embodiment 1 of the present invention;

FIG. 3 is a flow chart of a depth camera calibration method provided by Embodiment 2 of the present invention;

FIG. 4 is a flow chart of a depth camera calibration method provided by Embodiment 3 of the present invention;

Fig. 5 is a flow chart of obtaining the pose of the first label camera provided by Embodiment 4 of the present invention;

FIG. 6 is a structural block diagram of a depth camera calibration device provided in Embodiment 5 of the present invention;

FIG. 7 is a schematic structural diagram of an electronic device provided by Embodiment 6 of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, only some structures related to the present invention are shown in the drawings but not all structures.

Embodiment one

Fig. 1 is a flow chart of the depth camera calibration method provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation of multi-camera self-calibration. Calibration in this embodiment refers to calculating the relative pose between cameras. The method can be performed by a depth camera calibration device or electronic equipment, and the depth camera calibration device can be implemented by software and/or hardware, such as by a central processing unit (Central Processing Unit, referred to as CPU), and the control of the camera is completed by the CPU and calibration; further, the device can be integrated in portable mobile electronic equipment. As shown in Figure 1, the method specifically includes:

S101. Control at least two depth cameras in the panorama depth camera system to acquire images synchronously during motion, where each depth camera sets a corresponding tag.

Wherein, the panoramic depth camera system (may be simply referred to as the camera system) includes at least two depth cameras (RGB-D cameras), and the at least two depth cameras cover a 360-degree panoramic field of view. In practical applications, the camera size and number of cameras to be used can be selected according to specific requirements, and each camera is fixed on a platform (such as a rigid structure part) according to the camera size and number of cameras to meet the field of view coverage requirements. Complete the construction of the panoramic depth camera system. Specifically, the required number of cameras is determined according to the field of view of a single camera and the required field of view. The sum of the fields of view of all cameras needs to be greater than the required field of view. For example, taking a single camera with the same viewing angle as an example, the number of cameras n needs to satisfy n×Fov>α, Fov represents the field of view of a single camera, and α represents the required field of view of the built camera system, for example, if α=360°, the horizontal field of view of a single camera is 65°, and the vertical field of view is 80°, n=5 or n=6 can be selected, and considering the requirement of the vertical field of view, 6 cameras can be used. Arrange the cameras reasonably according to the size and number of cameras used. For example, RGB with a length of 10-15 cm, a width of 3-5 cm, a height of 3-5 cm, and a resolution of 640×480 -D camera as an example, select a regular hexagonal prism with a base length of 5 cm as the axis, and fix the camera lens on the axis through the fixing piece, as shown in Figure 2. It should be noted that when building a camera system, in combination with specific system usage requirements, adjacent cameras may have no viewing angle overlap or a small viewing angle overlap, for example, a viewing angle overlap of one or two degrees.

The depth cameras in the panoramic depth camera system collect images synchronously, and the synchronization method can be hardware synchronization or software synchronization. Specifically, hardware synchronization refers to using a signal (such as a rising edge signal) to simultaneously trigger all cameras to capture images at the same time; software method refers to stamping each image with a time stamp when caching the images captured by each camera, and the one with the closest time stamp The image is considered to be the image collected at the same time, that is, a buffer area is maintained, and the image frame with the closest time stamp of each camera is output each time.

When performing camera self-calibration online, the panoramic depth camera system needs to be in motion to collect images as observation information. For example, use a robot equipped with a panoramic depth camera system or a user to hold a panoramic depth camera system to move freely in a conventional room, and ensure that the rotational movement is increased as much as possible during the movement, so as to increase the historical viewing angle overlap between the cameras during the movement and obtain more observations information for calibration.

Each depth camera has a corresponding attribute label, which is used to distinguish the role of the camera. For example, the first label indicates that the camera is a reference camera, and the second label indicates that the camera is a non-reference camera. For simplicity, the specific value of the label can be 0 Or 1, such as 0 for non-reference camera and 1 for reference camera.

S102. Acquire a pose when at least one first label camera captures each frame of image.

Among them, the camera role can be determined according to the camera label. There is at least one reference camera among the at least two depth cameras included in the panoramic depth camera system. Specifically, any camera in the system can be preset as the reference camera, and the first tag camera is the reference camera. It should be noted that if multiple reference cameras are set in advance, the relative poses between the reference cameras need to be accurately known. Considering the accuracy of calibration, generally one reference camera can be set in advance. For the reference camera in the camera system, during the entire calibration process, it is necessary to obtain the pose of the reference camera in real time when it captures each frame of image. Pose change, pose specifically includes position (translation matrix T) and attitude (rotation matrix R), involving six degrees of freedom X, Y, Z, α, β, and γ, where X, Y, and Z are parameters of the camera position, α, β and γ are the parameters of the camera pose.

S103, if the second tag camera overlaps with the first tag camera in terms of historical angles of view, calculate the position and orientation of the second tag camera and the first tag camera at the same time according to the image corresponding to the historical angle of view overlap and the pose of the first tag camera when each frame of image is collected. The relative pose at each moment.

Among them, with the movement of the panorama depth camera system, each second tag camera will overlap with the first tag camera in terms of historical angle of view, so that the relative pose of each second tag camera and the first tag camera at the same time can be obtained, that is, The relative poses between the cameras are obtained, and the real-time online camera self-calibration process is completed.

It should be noted that during the entire calibration process, the depth cameras to be calibrated in the panoramic depth camera system are all in motion, and these moving cameras collect images synchronously in real time, and maintain key frames under each camera. For the reference camera, single-camera pose estimation is performed according to each frame of images collected to obtain the camera pose when each frame of image is collected; at the same time, for non-reference cameras, single-camera pose estimation is not performed, and the acquisition Each frame of the image is used as the observation information to determine whether there is any overlapping of the historical angle of view with any reference camera. If it overlaps, the relative pose of the non-reference camera and the corresponding reference camera at the same time can be calculated.

The depth camera calibration method in this embodiment is based on the first tag camera preset in the panoramic depth camera system, and uses the historical perspectives of the second tag camera and the first tag camera during the process of capturing images while the panoramic depth camera system is moving. Overlapping, determining the relative pose of the second tag camera to the first tag camera, enables fast online self-calibration of multiple cameras. This method does not require a calibration object, and does not require a fixed large field of view overlapping range between adjacent cameras. When building a camera system, there are small or no overlapping viewing angles between adjacent cameras. The movement of the panoramic depth camera system makes different The camera collects images with overlapping historical perspectives, which can be calibrated accordingly. This method has a small amount of calculation and can be calibrated online based on the CPU. The depth camera calibration method of the embodiment of the present invention is applicable to application backgrounds such as indoor robot navigation or three-dimensional scene reconstruction.

Embodiment two

On the basis of the foregoing embodiments, this embodiment further optimizes the determination of overlapping historical view angles and the calculation of relative poses between cameras.

For each depth camera to be calibrated in the camera system, a frame of image is collected, and it is determined whether the frame of image is a key frame, so as to determine the overlap of historical viewing angles. If subsequent loopback optimization is performed, key frames are also required. Specifically, while obtaining the pose of each frame image captured by at least one first label camera, it may also include: respectively matching the feature points of the current frame image captured by each depth camera with its previous key frame to obtain two frames The conversion relationship matrix between images; if the conversion relationship matrix is greater than or equal to the preset conversion threshold, determine that the current frame image is a key frame under the corresponding depth camera, and store the key frame.

Among them, the first frame of image captured by each depth camera is a key frame by default, and for each frame of image captured subsequently, it is determined whether the frame image is a key frame by comparing the frame image with the nearest key frame under the camera frame. The preset conversion threshold is set in advance according to the motion situation when the depth camera captures images. For example, if the pose changes greatly when the camera captures two adjacent frames of images, the preset conversion threshold is set larger. The feature point matching can adopt the existing matching algorithm, for example, based on the fast feature point extraction and description (Oriented FAST and Rotated BRIEF, referred to as ORB) algorithm (sparse algorithm) to perform feature matching on the color image collected by the RGB-D camera, or Dense registration using the direct method.

Fig. 3 is a flow chart of the depth camera calibration method provided by Embodiment 2 of the present invention. As shown in Fig. 3, the method includes:

S301. Control at least two depth cameras in the panorama depth camera system to acquire images synchronously during motion, where each depth camera sets a corresponding tag.

S302. Acquire a pose when at least one first label camera captures each frame of image.

S303, matching the feature points of the current frame image collected by the second tag camera with the historical key frame of the at least one first tag camera; if there is a historical key frame and the current frame image reaches the matching threshold, determine the second tag camera and the corresponding The first tab camera has history view overlap.

Wherein, every time the second tag camera collects a new frame of image, it performs feature point matching with the key frame stored under the first tag camera (that is, the historical key frame). If the matching threshold is reached, it is considered that there is an overlap of historical perspectives. The feature point matching can use the existing matching algorithm, for example, based on the sparse ORB algorithm to perform feature matching on the color image collected by the RGB-D camera, or use the direct method for dense registration. The matching threshold may be a preset number of matching feature points.

S304, remove the abnormal data in the corresponding relationship between the current frame image collected by the second label camera and the feature point corresponding to the historical key frame, and calculate the relative positional relationship between the current frame image and the corresponding historical key frame according to the remaining feature point corresponding relationship;

S305, calculating the transformation relationship between the pose when the second tag camera captures the current frame image and the pose when the first tag camera captures the corresponding historical key frame according to the relative positional relationship;

S306. Calculate the relative pose of the second tag camera and the first tag camera at the current frame time according to the transformation relationship and the poses of the first tag camera from collecting the corresponding historical key frame to collecting the current frame image.

Among them, the Random Sample Consensus (RANSAC) algorithm can be used to remove abnormal data, and the relative positional relationship between two frames of images with overlapping viewing angles can be calculated according to the remaining feature point correspondence; according to the relative positional relationship between images , the corresponding camera pose transformation relationship can be obtained. Since the pose of the first tag camera is always estimated when each frame of image is captured, the pose of the first tag camera between the historical key frame corresponding to the historical viewing angle overlap and the current frame is known, so the historical viewing angle can be deduced The relative pose of the overlapping second tag camera and the first tag camera at the current frame moment (that is, the moment when historical viewing angles overlap). The relative pose also includes six degrees of freedom involving the rotation matrix R and the translation matrix T.

Exemplarily, the panoramic depth camera system includes three depth cameras A, B and C, wherein camera A is a first label camera, and cameras B and C are second label cameras. The three cameras collect images synchronously. When the 10th frame is collected, the 10th frame of camera B and the 5th frame (historical key frame) of camera A overlap in historical perspective. Point matching can calculate the transformation relationship between the pose of the 10th frame captured by camera B and the pose of the 5th frame captured by camera A. Since camera A has been doing pose estimation, the poses of the 5th to 10th frames are recorded, and the relative poses of camera B and camera A at the 10th frame can be obtained by deriving frame by frame.

The depth camera calibration method of this embodiment uses historical key frames to determine that the historical viewing angles of the second tagged camera and the first tagged camera overlap, and calculates the pose transformation relationship between the second tagged camera and the first tagged camera through two frames of images with overlapping viewing angles , and then using the pose of the first tag camera between the corresponding historical key frame and the current frame under the first tag camera, the relative pose of the second tag camera and the first tag camera at the same time can be deduced. The calculation is simple and can be Guaranteed fast online calibration.

Optionally, in order to obtain a more accurate relative pose between cameras and improve calibration accuracy, after calculating the relative pose of the second tag camera and the first tag camera at the same moment, the calculated relative pose Global optimization of the pose, specifically including: if there is a key frame in the current frame image synchronously collected by the depth camera whose relative pose has been calculated, then the key frame will be optimized according to the key frame and the historical key frame under the depth camera whose relative pose has been calculated. Loopback detection; if the loopback is successful (that is, a matching historical keyframe is found), optimize and update the relative pose between the corresponding depth cameras according to the keyframe and the corresponding historical keyframe.

Among them, the loopback detection is to judge whether the depth camera has moved to a place that has been reached or a place that has a large overlap with the historical perspective. Loopback detection is performed for the key frame rate. For each frame of image captured by the depth camera, it is necessary to determine whether the frame image is a key frame. If so, perform loopback detection. If not, wait for the arrival of the next key frame to perform loopback detection. . Among them, the judgment of the key frame can be to match the feature points of each frame of image collected by the depth camera with the previous key frame corresponding to the camera to obtain the conversion relationship matrix between the two frames of images; if the conversion relationship matrix is greater than or equal to the preset If the conversion threshold is set, the current frame image is determined to be the key frame under the camera.

Since the relative poses between cameras are involved, the comparison object of the loopback detection in this embodiment is the historical keyframes under the depth camera whose relative poses have been calculated in the camera system, rather than just comparing its own historical keyframes. If the loopback Successfully, according to the current key frame and the corresponding historical key frame, the relative pose between the corresponding depth cameras is optimized and updated to reduce the cumulative error and improve the accuracy of the relative pose between the cameras.

Specifically, the loopback detection of the current keyframe and the historical keyframe can be performed by matching the ORB feature points of the current keyframe and the historical keyframe. If the matching degree is high, the loopback is successful. Preferably, the current keyframe can be The degree of matching with the historical keyframes, select one or more historical keyframes with a high matching degree to optimize and update the relative poses of the corresponding cameras. It should be noted that if the matched historical key frame belongs to the depth camera itself, then the own pose of the depth camera is optimized according to the current key frame and the historical key frame. The optimization update process of the above relative pose can be started after the relative pose of a pair of cameras in the camera system is obtained, and the calculated relative pose between the cameras is updated along with the process of camera movement to collect images. If the camera system The relative poses between all the cameras in are calculated, and when the preset conditions are met (for example, the number of optimizations of the relative poses between the cameras reaches the preset number, or the preset error requirements are met), the optimization update is stopped, and the obtained The final more accurate calibration parameters.

Exemplarily, still taking the panoramic depth camera system including three depth cameras A, B and C as an example, after calculating the relative pose of camera B and camera A at the same time, after determining whether there is a historical angle of view between camera C and camera A While overlapping (to determine the relative poses of the two), the relative poses between cameras B and A are also optimized and updated according to the key frames collected by camera A and/or camera B. If the relative pose of camera C and camera A at the same time is obtained, the relative pose of camera B and camera C can be further deduced, thereby obtaining the relative pose between the cameras, and then optimized and updated. For the current frame moment, the three cameras capture corresponding images synchronously. If it is judged that the image a captured by camera A and the image b captured by camera B are key frames, and the image c captured by camera C is not a key frame, then the key frames a and b performs loopback detection, both of which are successful in loopback, and optimize and update according to keyframes a and b respectively. The relative pose calculation method during the optimization update process is the same as the initial relative pose calculation method, and will not be repeated here.

In this embodiment, the calculated relative poses between the cameras can be optimized and updated in real time according to the images collected by the cameras, so as to improve the accuracy of camera calibration.

Embodiment Three

The depth camera calibration method in each of the above embodiments uses the preset first label camera in the panoramic depth camera system as a reference to obtain the relative poses between the cameras. On the basis of the foregoing embodiments, this embodiment provides another depth camera calibration method to further speed up the online calibration speed. FIG. 4 is a flow chart of a depth camera calibration method provided by Embodiment 3 of the present invention. As shown in FIG. 4, the method includes:

S401. Control at least two depth cameras in the panorama depth camera system to acquire images synchronously during motion, where each depth camera sets a corresponding tag.

S402. Acquire the pose of at least one first tag camera when each frame of image is captured.

S403, if the second tag camera overlaps with the first tag camera in terms of historical angles of view, calculate the position and orientation of the second tag camera and the first tag camera in the same The relative pose at each moment.

S404. Modify the tag of the second tag camera to the first tag.

That is to say, after the relative pose is calculated, the second tag camera whose historical viewing angle overlaps with the first tag camera is included in the benchmark category to expand the benchmark range.

S405, repeatedly performing the operation of acquiring the pose of at least one first tag camera when each frame of image is captured, calculating the relative pose of the second tag camera with overlapping historical perspectives and the corresponding first tag camera at the same moment, and modifying the tag ( That is, S402 to S404) are repeatedly executed until the at least two depth cameras do not include the second tag camera. The tags of the depth camera in the camera system are all changed to the first tag, which means that each second tag camera has calculated its relative pose with other cameras and obtained the calibration result.

In this embodiment, after obtaining the relative pose between the second tag camera and the first tag camera by using the overlap of historical viewing angles, the second tag camera is also used as a reference to expand the reference range and increase the occurrence of overlapping historical viewing angles of other second tag cameras. probability, so that the calibration speed can be further accelerated.

For the determination of the key frame, the determination of the overlap of historical view angles, the calculation of the relative pose, and the optimization update of the relative pose, please refer to the description of the foregoing embodiments, and details are not repeated here.

Embodiment four

On the basis of the above-mentioned embodiments, this embodiment further optimizes "obtaining the pose of each frame of images captured by at least one first label camera" to increase the calculation speed. Fig. 5 is a flow chart of obtaining the pose of the first label camera provided by Embodiment 4 of the present invention. As shown in Fig. 5, the method includes:

S501. For each first label camera, perform feature extraction on each frame of image collected by the first label camera, to obtain at least one feature point of each frame of image.

Among them, the feature extraction of the image is to find some pixel points (that is, feature points) with iconic features in the frame image, for example, it can be a corner point, texture, and edge pixel point in a frame image. The ORB algorithm may be used for feature extraction of each frame of image to find at least one feature point in the frame of image.

S502. Perform feature point matching on two adjacent frames of images to obtain a corresponding relationship of feature points between the two adjacent frames of images.

Considering that the frequency of image acquisition by the camera is relatively fast during the motion process, part of the content of two adjacent frames of images collected by the same camera is the same, so there is a certain correspondence between the feature points corresponding to the two frames of images. Sparse ORB feature registration or direct dense registration can be used to obtain the feature point correspondence between two adjacent frames of images, that is, to obtain the feature point correspondence between two adjacent frames of images.

Specifically, take a feature point between two adjacent frames of images as an example, assuming that the feature points X1 and X2 representing the same texture feature in the two frames of images are located at different positions of the two frames of images, represented by H(X1, X2) The Hamming distance between two feature points X1, X2, XOR operation is performed on the two feature points, and the number of the result is 1, which is used as the Hamming distance of a feature point between two adjacent frames of images (that is, the feature point correspondence).

S503, remove the abnormal data in the feature point correspondence, and calculate the nonlinear term in J(ξ) ^T J(ξ) through the linear component including the second-order statistics of the remaining feature points and the nonlinear component including the camera pose Perform multiple iterative calculations for δ=-(J(ξ) ^T J(ξ)) ^-1 J(ξ) ^T r(ξ) to solve the pose when the reprojection error r(ξ) is less than the preset threshold. Specifically, the Gauss-Newton method can be used for iterative calculation. Preferably, the pose at which the reprojection error is minimized may be calculated.

where r(ξ) represents the vector containing all reprojection errors, J(ξ) is the Jacobian matrix of r(ξ), ξ represents the Lie algebra of the camera pose, and δ represents the increment of r(ξ) at each iteration. Quantity; R _i represents the rotation matrix of the camera when capturing the i-th frame image; R _j represents the rotation matrix of the camera when capturing the j-th frame image; Indicates the kth feature point on the i-th frame image; Represents the kth feature point on the jth frame image; C _{i, j} represents the set of correspondence between the i-th frame image and the j-th frame image; ||C _i,j ||-1 represents the i-th frame image The number of corresponding relations with the feature points of the j-th frame image; [] _× means vector product; ||C _i,j || means take the norm of C _i,j .

Furthermore, the non-linear term The expression is:

in, Represents linear components; r _il ^T and r _jl represent nonlinear components, r _il ^T is the l-th row in the rotation matrix R _i , r _jl is the transpose of the l-th row in the rotation matrix R _j , l=0,1 , 2 (this embodiment counts from 0 based on the idea of programming, and l=0 means the first row of the matrix, which is commonly referred to as, and so on).

Specifically, there will be unqualified abnormal data in the corresponding relationship between the feature points between the two adjacent frames of images obtained in S502. feature points, and if they are subjected to the matching operation of S502, an abnormal corresponding relationship will appear. Preferably, the abnormal data can be removed using the RANSAC algorithm, and the remaining feature point correspondences are expressed as in, Indicates the corresponding relationship between the kth feature point between the i-th frame image and the j-th frame image; j=i-1.

Calculating the camera pose is to solve the nonlinear least squares problem between two frames of images with the following formula as the cost function:

Wherein, E represents the reprojection error of the i-th frame image in Euclidean space compared to the j-th frame image (referring to the previous frame image in this embodiment); T _i represents the pose when the camera captures the i-th frame image (according to the foregoing The explanation of the camera pose shows that it actually refers to the pose change of the i-th frame image relative to the previous frame image), T _j represents the pose of the camera when the j-th frame image is collected; N represents the total frames collected by the camera number; Indicates the kth feature point on the i-th frame image The homogeneous coordinates of Indicates the kth feature point on the jth frame image homogeneous coordinates of . It should be noted that when i and k take the same value, and means the same point, the difference is that are the local coordinates, are homogeneous coordinates.

In order to speed up the calculation rate, this embodiment does not directly calculate the cost function of formula (2), but calculates J(ξ ) Nonlinear term in ^T J(ξ) Perform multiple iterative calculations on δ=-(J(ξ) ^T J(ξ))- ¹ J(ξ) ^T r(ξ) to solve the pose when the reprojection error is less than the preset threshold. by nonlinear terms The expression of When calculating, the fixed linear part between the two frames of images Considering W as a whole to calculate, it does not need to calculate according to the number of feature point correspondences, which reduces the complexity of camera pose calculation and enhances the real-time performance of camera pose calculation.

The derivation process of formula (1) is described below, and the principle of reducing algorithm complexity is analyzed in combination with the derivation process.

In Euclidean space, the camera pose T _i =[R _i /t _i ] when the camera captures the i-th frame of image, in fact, T _i refers to when the camera captures the i-th frame of image relative to the acquisition of the j-th frame of image (this embodiment The pose transformation matrix when the middle finger is on the previous frame image), including the rotation matrix R _i and the translation matrix t _i . The rigid transformation T _i in the Euclidean space is represented by the Lie algebra ξ _i on the SE3 space, that is, ξ _i also represents the camera pose when the camera captures the i-th frame image, and T(ξ _i ) maps the Lie algebra ξ _i is T _i in Euclidean space.

For each feature point correspondence Its reprojection error is:

The reprojection error in Euclidean space in formula (1) can be expressed as E(ξ)=||r(ξ)||, where r(ξ) represents the vector containing all reprojection errors, namely:

T(ξ _i )P _i ^k can be expressed as (for simplicity, ξ _i is omitted below):

in, represents the lth row in the rotation matrix R _i ; t _il represents the lth element in the translation vector t _i , l=0,1,2.

in, Represents the Jacobian matrix corresponding to the feature point correspondence between the i-th frame image and the j-th frame image; m represents the m-th feature point correspondence.

is a 6×6 square matrix, representation matrix the transposition of The expression is as follows:

Among them, I _3×3 represents a 3×3 identity matrix. According to formula (6) and formula (7), The four non-zero 6×6 sub-matrices in are: Below to As an example, the other three non-zero sub-matrices are also calculated similarly, and will not be repeated here.

Among them, combining formula (5) can get:

Will Expressed as W, combined with formula (5), the nonlinear term in formula (10) can be Simplified to formula (1), the structural term in this nonlinear term is linearized to W. Although for structural items in terms of is nonlinear, but after the above analysis, All non-zero elements in are linearly related to the second-order statistics of the structural items in C _i,j, and the second-order statistics of the structural items are and That is, the sparse matrix The second-order statistics for the structural terms in C _i,j are element-wise linear.

It should be noted that each correspondence The Jacobian matrices of are composed of geometric terms ξ _i , ξ _j and structural terms Decide. For all correspondences in the same frame pair C _i,j , their corresponding Jacobian matrices share the same geometric terms but have different structural terms. For a frame pair C _i,j , calculate When , the existing algorithm depends on the number of feature point correspondences in C _i,j , but this embodiment can efficiently calculate It is only necessary to calculate the second-order statistics W of the structural items, and it is not necessary for each corresponding relationship to involve the relevant structural items in the calculation, that is, The four non-zero sub-matrices in can be calculated with complexity O(1) instead of complexity O(||C _i,j ||).

Therefore, the sparse matrices JTJ and JTr required in the iterative step of nonlinear Gauss-Newton optimization of δ=-(J(ξ) ^T J(ξ)) ^-1 J(ξ) ^T r(ξ) can be of complexity O (M) Efficient calculation, instead of the original computational complexity O(N _coor ), N _coor represents the total number of all feature point correspondences of all frame pairs, and M represents the number of frame pairs. Generally, O(N _coor ) is about 300 in sparse matching, and about 10000 in dense matching, which is much larger than the number M of frame pairs.

After the above derivation, in the process of camera pose calculation, for each frame pair, calculate W, and then calculate formulas (1), (10, (9), (8) and (6) to obtain Furthermore, iterative calculation can be used to obtain ξ when r(ξ) is minimum.

Optionally, in order to obtain a more accurate pose of the first tag camera, after acquiring at least one pose of the first tag camera when each frame of image is captured, a globally consistent optimization update can be performed on the acquired pose, specifically including : If the current frame image captured by the first tag camera is a key frame, perform loopback detection based on the current key frame and the historical key frames of the first tag camera; if the loopback is successful, check the acquired first tag camera according to the current key frame The pose is globally consistent and optimally updated.

That is to say, for each frame of image collected by the first tag camera, it is necessary to judge whether the frame of image is a key frame, if so, perform loop detection, if not, wait for the arrival of the next key frame to perform loop detection. Wherein, the judgment of the key frame can be that each frame of image collected by the first label camera is matched with the previous key frame corresponding to the camera to obtain the conversion relationship matrix between the two frames of images; if the conversion relationship matrix is greater than or is equal to the preset conversion threshold, then the current frame image is determined to be the key frame under the camera.

Globally consistent optimization update means that during the calibration process, with the movement of the camera, when the depth camera moves to the place it has reached or has a large overlap with the historical perspective, the current frame image is consistent with the captured image, instead of Phenomena such as interleaving and aliasing occur. Loopback detection is based on the current observation of the depth camera to judge whether the camera has moved to a place it has reached or has a large overlap with the historical perspective. If the loopback is successful, the pose of the first label camera is globally consistent and optimized according to the current key frame. update to reduce the cumulative error.

Specifically, the loopback detection of the current keyframe and the historical keyframe can be performed by matching the ORB feature points of the current keyframe and the historical keyframe. If the matching degree is high, the loopback is successful. Preferably, the current keyframe can be According to the degree of matching with the historical keyframes, one or more historical keyframes with high matching degree are selected for globally consistent optimization update of the camera pose.

Preferably, the globally consistent optimization update of the camera pose is to solve the following It is the problem of minimizing the transformation error between the current keyframe of the cost function and all historical keyframes with high matching degree. Among them, E(T ₁ ,T ₂ ,…,T _N-1 |T _i ∈ SE3,i∈[1,N-1]) represents all frame pairs (any historical matching key frame and the current key frame are one frame pair) conversion error; N represents the number of historical key frames that match the current key frame; E _{i, j} represents the conversion error between the i-th frame and the j-th frame, and the conversion error is the reprojection error.

Specifically, in the process of optimizing and updating the camera pose, it is necessary to keep the relative pose of the non-key frame and its corresponding key frame unchanged. The specific optimization update algorithm can use the existing BA algorithm or the method in S503 In order to improve the optimization speed, details will not be repeated here. Similarly, the algorithm of this embodiment (ie, the method in S503 ) can also be used for the calculation and optimization process of the relative pose between cameras.

In this embodiment, the camera pose is iteratively calculated through the linear component including the second-order statistics of feature points and the nonlinear component including the camera pose. When calculating, the fixed linear part Considering W as a whole for calculation, the complexity of camera pose calculation is reduced, the real-time performance of camera pose calculation is enhanced, and the hardware requirements are low. Applying the above algorithm to the process of solving pose and back-end optimization can obtain fast and globally consistent calibration parameters.

It should be noted that the embodiment of the present invention can realize pose estimation and optimization based on the process and principle of SLAM. The pose estimation is realized by the front-end visual odometry thread, and the pose optimization is realized by the back-end loop detection and optimization algorithm. For example, using The existing bundle adjustment (Bundle Adjustment, BA for short) algorithm or the algorithm in this embodiment.

In the process of SLAM, the following operations are performed according to the collected images: the pose estimation and optimization of the first label camera, the relative pose between the cameras is calculated by using the view overlap, and the calculated relative pose is optimized. These operations can be performed concurrently. Through globally consistent SLAM, the pose of each camera is optimized and the relative pose between the calculated cameras is continuously updated, while a local map and a globally consistent global map can be maintained to accommodate regular RGB-D cameras for indoor robot navigation or Application background of 3D scene reconstruction. The map in SLAM refers to the movement trajectory of the camera in the world coordinate system and the position in the world coordinate system of the key frames observed in the movement trajectory. If the camera system is rigidly deformed due to physical impact, in this embodiment, only the calibration program needs to be started for quick re-calibration without rearranging the calibration objects.

Embodiment five

This embodiment provides a depth camera calibration device, which can be used to execute the depth camera calibration method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. The device may be realized by means of hardware and/or software, such as by a CPU. For technical details not exhaustively described in this embodiment, refer to the depth camera calibration method provided in any embodiment of the present invention. Control signals and images need to be transmitted between the depth camera calibration device and the panoramic depth camera system. There are many communication methods between the two, for example, communication through serial ports, network cables, etc. way of communication. As shown in FIG. 6 , the device includes: a camera control module 61 , a pose acquisition module 62 and a relative pose calculation module 63 .

A camera control module 61, configured to control at least two depth cameras in the panoramic depth camera system to acquire images synchronously during motion, wherein each depth camera is provided with a corresponding label;

A pose acquisition module 62, configured to acquire poses when at least one first tag camera captures each frame of image;

The relative pose calculation module 63 is used to calculate the first tag camera according to the image corresponding to the historical view overlap and the pose when each frame of image is collected by the first tag camera when the second tag camera and the first tag camera have overlapping historical perspectives. The relative pose of the second tag camera and the first tag camera at the same moment.

Optionally, the above-mentioned device may also include:

The label modification module is used to modify the label of the second label camera to the first label after the relative pose calculation module 63 calculates the relative pose of the second label camera and the first label camera at the same moment;

The operation execution module is used to repeatedly execute the acquisition of the pose of at least one first tag camera when each frame of image is captured, calculate the relative pose of the second tag camera with overlapping historical perspectives and the corresponding first tag camera at the same time, and modify The operation of the tag (that is, repeatedly execute the operations of the pose acquisition module 62, the relative pose calculation module 63 and the tag modification module) until at least two depth cameras of the panoramic depth camera system do not include the second tag camera.

Optionally, the above-mentioned device may further include: a key frame determination module, configured to obtain the pose of each frame image captured by at least one first tag camera at the same time as the pose acquisition module 62. Match the feature points of the image with its previous key frame to obtain the transformation relationship matrix between the two frames of images; if the transformation relationship matrix is greater than or equal to the preset conversion threshold, determine the current frame image as the key frame under the corresponding depth camera, and Store the keyframe, specifically store the keyframe and the depth camera it belongs to.

Optionally, the above-mentioned device may further include: an angle-of-view overlapping determination module, configured to combine the current frame image captured by the second tag camera with the above-mentioned at least A historical key frame of the first tag camera is matched with feature points; if there is a historical key frame and the current frame image reaches the matching threshold, it is determined that the historical angle of view of the second tag camera overlaps with the corresponding first tag camera.

Optionally, the relative pose calculation module 63 includes:

The relative positional relationship calculation unit is used to remove the abnormal data in the corresponding relationship between the current frame image collected by the second label camera and the corresponding historical key frame, and calculate the current frame image and the corresponding historical key frame according to the remaining feature point corresponding relationship relative positional relationship;

A transformation relationship calculation unit, used to calculate the transformation relationship between the pose when the second tag camera captures the current frame image and the pose when the first tag camera captures the corresponding historical key frame according to the above-mentioned relative positional relationship;

The relative pose calculation unit is used to calculate the relative position between the second tag camera and the first tag camera at the current frame moment according to the above-mentioned transformation relationship and the poses of the first tag camera from collecting the corresponding historical key frame to collecting the current frame image. pose.

Optionally, the pose acquisition module 62 includes:

A feature extraction unit, configured to perform feature extraction on each frame of image collected by the first tag camera for each first tag camera, to obtain at least one feature point of each frame of image;

A feature matching unit is used to perform feature point matching on two adjacent frames of images to obtain the corresponding relationship of feature points between the two adjacent frames of images;

The iterative calculation unit is used to remove abnormal data in the feature point correspondence, and calculates J(ξ) in ^T J(ξ) through the linear component including the second-order statistics of the remaining feature points and the nonlinear component including the camera pose. non-linear term Perform multiple iterative calculations on δ=-(J(ξ) ^T J(ξ)) ^-1 J(ξ) ^T r(ξ) to solve the pose when the reprojection error is less than the preset threshold;

where r(ξ) represents the vector containing all reprojection errors, J(ξ) is the Jacobian matrix of r(ξ), ξ represents the Lie algebra of the camera pose, and δ represents the increment of r(ξ) at each iteration. Quantity; R _i represents the rotation matrix of the camera when capturing the i-th frame image; R _j represents the rotation matrix of the camera when capturing the j-th frame image; Represents the kth feature point on the i-th frame image; Represents the kth feature point on the jth frame image; C _{i, j} represents the set of correspondence between the i-th frame image and the j-th frame image; ||C _i,j ||-1 represents the i-th frame image The number of corresponding relations with the feature points of the j-th frame image; [] _× means vector product; ||C _i,j || means take the norm of C _i,j .

Furthermore, the non-linear term The expression is:

in, Represents linear components; r _il ^T and r _jl represent nonlinear components, r _il ^T is the l-th row in the rotation matrix R _i , r _jl is the transpose of the l-th row in the rotation matrix R _j , l=0,1 ,2.

Optionally, the above-mentioned device may also include:

The loop closure detection module is used to obtain at least one pose of each frame image captured by the first label camera, if the current frame image collected by the first label camera is a key frame, then according to the current key frame and the first label camera. Historical keyframes for loopback detection;

The optimization update module is used to perform globally consistent optimization update on the obtained first label camera pose according to the current key frame when the loopback is successful.

Optionally, the above-mentioned loop closure detection module is also used to calculate the relative pose of the second tag camera and the first tag camera at the same time after the relative pose calculation module 63 calculates the synchronous acquisition of the depth camera of the relative pose. If there is a key frame in the current frame of the image, the loopback detection is performed based on the keyframe and the historical keyframe under the depth camera whose relative pose has been calculated; Keyframes and corresponding historical keyframes are optimized to update the relative poses between corresponding depth cameras.

It is worth noting that in the above embodiment of the depth camera calibration device, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, The specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present invention.

Embodiment six

This embodiment provides an electronic device, including: one or more processors, a memory, and a panoramic depth camera system. Among them, the memory is used to store one or more programs. The panoramic depth camera system includes at least two depth cameras, and the at least two depth cameras cover the panoramic field of view and are used for collecting images. When the one or more programs are executed by the one or more processors, the one or more processors implement the depth camera calibration method according to any embodiment of the present invention.

FIG. 7 is a schematic structural diagram of an electronic device provided by Embodiment 6 of the present invention. Figure 7 shows a block diagram of an exemplary electronic device suitable for use in implementing embodiments of the present invention. The electronic device 712 shown in FIG. 7 is only an example, and should not limit the functions and scope of use of this embodiment of the present invention.

As shown in FIG. 7, electronic device 712 takes the form of a general-purpose computing device. Components of electronic device 712 may include, but are not limited to: one or more processors (or called processing unit 716 ), system memory 728 , and bus 718 connecting different system components (including system memory 728 and processing unit 716 ).

Bus 718 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus structures. These architectures include, by way of example, but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect ( PCI) bus.

Electronic device 712 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 712 and include both volatile and nonvolatile media, removable and non-removable media.

System memory 728 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 730 and/or cache memory 732 . Electronic device 712 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 734 may be used to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 7, commonly referred to as a "hard drive"). Although not shown in FIG. 7, a disk drive for reading and writing to removable nonvolatile disks (e.g., "floppy disks") may be provided, as well as for removable nonvolatile optical disks (e.g., CD-ROM, DVD-ROM or other optical media) CD-ROM drive. In these cases, each drive may be connected to bus 718 through one or more data media interfaces. System memory 728 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present invention.

Programs/utilities 740 may be stored, for example, in system memory 728 as a set (at least one) of program modules 742 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include the realization of the network environment. Program modules 742 generally perform the functions and/or methodologies of the described embodiments of the invention.

The electronic device 712 may also communicate with one or more external devices 714 (e.g., a keyboard, pointing device, display 724, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 712, and/or communicate with Any device (eg, network card, modem, etc.) that enables the electronic device 712 to communicate with one or more other computing devices. Such communication may occur through input/output (I/O) interface 722 . Moreover, the electronic device 712 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and/or a public network such as the Internet) through the network adapter 720 . As shown, network adapter 720 communicates with other modules of electronic device 712 via bus 718 . It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with electronic device 712, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives And data backup storage system, etc.

The processing unit 716 executes various functional applications and data processing by running the programs stored in the system memory 728 , such as implementing the depth camera calibration method provided by the embodiment of the present invention.

The electronic device 712 may further include: a panoramic depth camera system 750, including at least two depth cameras, the at least two depth cameras cover a panoramic field of view, and are used for collecting images. The panoramic depth camera system 750 is connected to the processing unit 716 and the system memory 728 . The depth cameras included in the panoramic depth camera system 750 can capture images under the control of the processing unit 716 . Specifically, the panoramic depth camera system 750 can be embedded and installed in an electronic device.

Optionally, one or more processors are central processing units; the electronic device is a portable mobile electronic device, such as a mobile robot, a drone, a three-dimensional visual interaction device (such as VR glasses or a wearable helmet) or an intelligent terminal (such as phone or tablet), etc.

Embodiment seven

This embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the depth camera calibration method according to any embodiment of the present invention is implemented.

The computer storage medium in the embodiments of the present invention may use any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of computer readable storage media include: electrical connections with one or more leads, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), Erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a data signal carrying computer readable program code in baseband or as part of a carrier wave. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. .

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including - but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out the operations of the present invention may be written in one or more programming languages, or combinations thereof, including object-oriented programming languages—such as Java, Smalltalk, C++, and conventional Procedural programming language—such as "C" or a similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).

The serial numbers of the above embodiments are for description only, and do not represent the advantages and disadvantages of the embodiments.

Those of ordinary skill in the art should understand that each module or each operation of the above-mentioned embodiments of the present invention can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed on a network formed by multiple computing devices , optionally, they can be implemented with executable program codes of computer devices, so that they can be stored in storage devices and executed by computing devices, or they can be made into individual integrated circuit modules, or multiple of them Each module or operation is implemented as a single integrated circuit module. As such, the present invention is not limited to any specific combination of hardware and software.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention, and the present invention The scope is determined by the scope of the appended claims.

Claims (13)

1. A depth camera calibration method, characterized in that, comprising: Controlling at least two depth cameras in the panoramic depth camera system to acquire images synchronously during motion, wherein each depth camera sets a corresponding label; Obtain the pose of at least one first label camera when capturing each frame of image; If the second tag camera overlaps with the first tag camera in terms of historical viewing angles, calculate the relationship between the second tag camera and the The relative pose of the first labeled camera at the same moment. 2. The method according to claim 1, wherein, after calculating the relative pose of the second tag camera and the first tag camera at the same moment, further comprising: modifying the label of the second label camera to the first label; Repeat the operation of obtaining the pose of at least one first tag camera when each frame of image is captured, calculating the relative pose of the second tag camera with overlapping historical perspectives and the corresponding first tag camera at the same moment, and modifying the tag until all The at least two depth cameras do not include the second label camera. 3. The method according to claim 1, characterized in that, while acquiring at least one first label camera to capture the pose of each frame image, it also includes: Match the feature points of the current frame image collected by each depth camera with its previous key frame to obtain the transformation relationship matrix between the two frames of images; If the conversion relationship matrix is greater than or equal to a preset conversion threshold, determine that the current frame image is a key frame under the corresponding depth camera, and store the key frame. 4. The method according to claim 1, wherein, before calculating the relative pose of the second tag camera and the first tag camera at the same moment, further comprising: performing feature point matching on the current frame image collected by the second label camera and the historical key frames of the at least one first label camera; If there is a historical key frame and the current frame image reaches a matching threshold, it is determined that the historical angle of view of the second tag camera overlaps with the corresponding first tag camera. 5. The method according to claim 1, characterized in that, according to the overlapping images corresponding to the historical view angles and the poses of each frame image captured by the first label camera, the distance between the second label camera and the first label camera is calculated. The relative pose of the tag camera at the same moment, including: Remove the abnormal data in the current frame image collected by the second label camera and the feature point correspondence of the corresponding historical key frame, and calculate the relative positional relationship between the current frame image and the corresponding historical key frame according to the remaining feature point correspondence ; Calculate the transformation relationship between the pose when the second tag camera captures the current frame image and the pose when the first tag camera captures the corresponding historical key frame according to the relative positional relationship; Calculate the relative position of the second tag camera and the first tag camera at the current frame moment according to the transformation relationship and the positions of the first tag camera from collecting the corresponding historical key frame to collecting the current frame image posture. 6. The method according to claim 1, wherein obtaining the pose of each frame image collected by at least one first label camera comprises: For each first label camera, feature extraction is performed on each frame of image collected by the first label camera to obtain at least one feature point of each frame of image; Carrying out feature point matching on two adjacent frames of images to obtain the corresponding relationship of feature points between the two adjacent frames of images; Remove the abnormal data in the feature point correspondence, and calculate the nonlinear term in J(ξ) ^T J(ξ) through the linear component including the second-order statistics of the remaining feature points and the nonlinear component including the camera pose Perform multiple iterative calculations on δ=-(J(ξ) ^T J(ξ)) ^-1 J(ξ) ^T r(ξ) to solve the pose when the reprojection error is less than the preset threshold; where r(ξ) represents the vector containing all reprojection errors, J(ξ) is the Jacobian matrix of r(ξ), ξ represents the Lie algebra of the camera pose, and δ represents the increment of r(ξ) at each iteration. Quantity; R _i represents the rotation matrix of the camera when capturing the i-th frame image; R _j represents the rotation matrix of the camera when capturing the j-th frame image; Represents the kth feature point on the i-th frame image; Represents the kth feature point on the jth frame image; C _{i, j} represents the set of correspondence between the i-th frame image and the j-th frame image; ||C _i,j ||-1 represents the i-th frame image The number of corresponding relations with the feature points of the j-th frame image; [] _× means vector product; ||C _i,j || means take the norm of C _i,j . 7. The method according to claim 6, wherein the nonlinear term The expression is: in, Represents linear components; r _il ^T and r _jl represent nonlinear components, r _il ^T is the l-th row in the rotation matrix R _i , r _jl is the transpose of the l-th row in the rotation matrix R _j , l=0,1 ,2. 8. The method according to claim 1 or 6, wherein, after acquiring at least one first label camera to capture the pose of each frame of image, it also includes: If the current frame image collected by the first label camera is a key frame, then perform loop closure detection according to the current key frame and the historical key frames of the first label camera; If the loopback is successful, a globally consistent optimization update is performed on the acquired pose of the first label camera according to the current key frame. 9. The method according to claim 1 or 2, wherein after calculating the relative pose of the second tag camera and the first tag camera at the same moment, further comprising: If there is a key frame in the current frame image synchronously collected by the depth camera that has calculated the relative pose, then perform loopback detection according to the key frame and the historical key frames under the depth camera that has calculated the relative pose; If the loopback is successful, optimize and update the relative poses between corresponding depth cameras according to the keyframes and corresponding historical keyframes. 10. A depth camera calibration device, characterized in that, comprising: A camera control module, configured to control at least two depth cameras in the panoramic depth camera system to acquire images synchronously during motion, wherein each depth camera is provided with a corresponding label; A pose acquisition module, configured to acquire poses of at least one first label camera when each frame of image is collected; The relative pose calculation module is used to overlap the corresponding image according to the historical view angle and the pose of each frame image collected by the first tag camera when the second tag camera overlaps with the first tag camera. , calculating the relative pose of the second tag camera and the first tag camera at the same moment. 11. An electronic device, characterized in that it comprises: one or more processors; memory for storing one or more programs; A panoramic depth camera system, including at least two depth cameras, the at least two depth cameras cover a panoramic field of view for collecting images; When the one or more programs are executed by the one or more processors, the one or more processors implement the depth camera calibration method according to any one of claims 1-9. 12. The electronic device according to claim 11, wherein the one or more processors are central processing units; and the electronic device is a portable mobile electronic device. 13. A computer-readable storage medium, on which a computer program is stored, wherein when the program is executed by a processor, the depth camera calibration method according to any one of claims 1-9 is implemented.