CN116681827A - A defect-free 3D point cloud reconstruction method and device based on multi-surveillance cameras and point cloud fusion - Google Patents

Abstract

The application provides a defect-free three-dimensional point cloud reconstruction method and a device thereof based on multi-monitoring camera and point cloud fusion, wherein the method comprises the following steps: establishing a cubic mirror coordinate system by using a theodolite, and simultaneously measuring targets by using a plurality of monitoring cameras and the theodolite to indirectly obtain a conversion matrix of the coordinate system of the monitoring cameras and the cubic mirror coordinate system, so as to realize global calibration of the monitoring cameras; shooting an object picture by using the calibrated monitoring camera, and reconstructing a three-dimensional point cloud; and carrying out point cloud preprocessing on the plurality of point clouds, realizing point cloud coarse registration by using a camera global calibration result, realizing point cloud fine registration by using a traditional point cloud registration algorithm, and fusing the registered point clouds into a defect-free three-dimensional point cloud. The method has high global calibration precision of the camera, and the point cloud rough registration and the point cloud fine registration have high speed and high robustness, so that the three-dimensional reconstruction precision of the object can be greatly improved, and the object point cloud without defects is obtained.

Description

A defect-free 3D point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion and its device

technical field

The present application relates to the field of camera calibration and point cloud processing, in particular to a defect-free three-dimensional point cloud reconstruction method and device based on multi-surveillance cameras and point cloud fusion.

Background technique

Object point cloud reconstruction is of great significance for the study of object surface topography, terrain analysis, 3D visualization and other fields. The traditional object point cloud reconstruction method is based on a single camera or a pair of binocular cameras. Due to the limited viewing angle, the reconstructed point cloud has holes and defects; if you choose to change the viewing angle to shoot, you need to use the point cloud registration algorithm for registration. However, most existing point cloud coarse registration algorithms do not have a very high accuracy rate when dealing with objects with inconspicuous features, and most of the fine registration depends on the results of coarse registration, so it is not easy to obtain high accuracy stably. 3D point cloud of objects with flawless precision.

Different from the reconstruction of 3D point cloud by moving supplementary shots with a single camera, the 3D reconstruction using multiple surveillance cameras at multiple viewing angles has higher accuracy. In industry, the camera global calibration using theodolite and cube mirror has the advantage of high precision, so it is often used to calibrate the camera used in aerospace projects. Since the global calibration of the camera does not need to be performed every time it is used, it only needs to be calibrated once after production and processing, and then calibrated again in a regular period to correct the error. Originating from cameras, this makes multi-camera systems widely used in numerous industrial fields.

Contents of the invention

This application provides a defect-free 3D point cloud reconstruction method and device based on multi-surveillance cameras and point cloud fusion. Calibration accuracy, and point cloud coarse registration and fine registration have faster speed and higher robustness, which can greatly improve the accuracy of 3D reconstruction of objects and obtain defect-free object point clouds.

According to some embodiments, the present application adopts the following technical solutions:

A defect-free 3D point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion, including:

Use the theodolite to establish the cubic mirror coordinate system;

Use multiple monitoring cameras and theodolite to measure the target at the same time, and indirectly obtain the transformation matrix between the coordinate system of the multiple monitoring cameras and the coordinate system of the cubic mirror, and realize the global calibration of the multiple monitoring cameras;

Use the calibrated surveillance camera to take pictures of objects and reconstruct 3D point clouds;

Perform point cloud preprocessing on multiple point clouds;

Use the camera global calibration results to achieve coarse point cloud registration;

Use traditional point cloud registration algorithms to achieve fine point cloud registration; and

The registered point clouds are fused into a defect-free 3D point cloud.

Further, use the theodolite to measure the angle at both ends of a standard rod with a known length, and self-align the theodolite with the cube mirror, record the angle at this time, then measure the angle of the crosshairs on the corresponding surface of the cube mirror, and calculate the coordinate system of the theodolite Transformation matrix to and from the cubic mirror coordinate system.

Further, the internal and external parameters of the camera are obtained by shooting the target with multiple surveillance cameras, and then the angle of some corners of the target is measured with the theodolite, and the conversion matrix between the camera coordinate system and the theodolite coordinate system is calculated, and then the conversion matrix between the camera coordinate system and the cubic mirror coordinate system is calculated, The cubic mirror coordinate system is used as the world coordinate system.

Further, the globally calibrated camera is used to shoot the object, the parallax principle is used to reconstruct the 3D point cloud binocularly, and the MVS method is used to reconstruct the 3D point cloud monocularly.

Further, the redundant part of the reconstructed point cloud is cropped, the point cloud is filtered, and down-sampled.

Furthermore, the coordinate transformation of the preprocessed point cloud is carried out by using the transformation matrix between the camera and the world coordinate system calibrated globally by the surveillance camera, and the coarse registration of the point cloud is realized.

Further, the AAICP algorithm with high speed and high precision is used to fine-register the point cloud after rough registration.

Further, the registered point clouds are merged into a large point cloud, and duplicate points are judged to be deleted to obtain a defect-free 3D point cloud of the object.

A defect-free 3D point cloud reconstruction device based on multi-surveillance cameras and point cloud fusion, including:

The three-dimensional coordinate measurement module of the theodolite is configured to measure the angle of any point in the space and calculate its three-dimensional coordinates;

The camera global calibration module is configured to obtain the internal and external parameters of each monitoring camera and the conversion matrix between the camera coordinate system and the world coordinate system;

The multi-camera 3D point cloud fusion module is configured to perform point cloud preprocessing, and realize coarse point cloud registration according to the results of camera global calibration, then use traditional algorithms for fine point cloud registration, and finally combine multiple point clouds Fusion into a complete and defect-free object 3D point cloud.

Invention effect

This application is based on multi-surveillance cameras shooting from multiple perspectives, and uses theodolite and cube mirrors to globally calibrate the multi-surveillance cameras, which has the advantage of high calibration accuracy, and the accuracy of the 3D point cloud reconstructed by the calibrated cameras is also very high. At the same time, using the results of camera global calibration for coarse registration of point clouds, compared with traditional point cloud registration algorithms, it has higher accuracy, speed and robustness. This application uses the Anderson acceleration ICP point cloud fine registration algorithm, which has the advantages of high registration accuracy, fast registration speed and good robustness.

In this application, the position of the binocular camera is fixed, and the movement of the monocular camera is controlled by a high-precision mechanical arm. Therefore, the object can move at will, multi-camera shooting at the same time and then parallel processing, and then through fast point cloud registration and fusion, the real-time performance is high. Users can move or replace objects along the way, and observe the defect-free 3D point cloud of the object in real time on the user interface.

Description of drawings

The accompanying drawings constituting a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application, and do not constitute an improper limitation of the present application.

Fig. 1 is the flow chart of the defect-free three-dimensional point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion provided by an embodiment of the present application;

Fig. 2 is the schematic diagram of the principle of using two theodolites to measure the angle of any point in space to solve its three-dimensional coordinate system;

Fig. 3 is to use two theodolites and cube mirror two mutual vertical plane self-collimation and measure corresponding surface crosshair angle to solve the schematic diagram of the principle of the conversion matrix of theodolite coordinate system and cube mirror coordinate system;

Figure 4 is a diagram of the relationship between the target world coordinate system, the camera coordinate system and the pixel coordinate system when the monocular camera parameters are calibrated;

Figure 5 is a schematic diagram of the distortion that exists when the camera is used;

Fig. 6 is a flow chart of binocular camera stereo calibration;

Fig. 7 is a comparison diagram before (a) and (b) after stereo correction of the target image by the binocular camera;

Figure 8 is a diagram of the epipolar geometric relationship when solving the coordinates of the monocular point cloud;

Fig. 9 is a schematic diagram of the division principle of the KD tree used when judging repeated points in point cloud fusion;

Fig. 10 is an effect diagram of defect-free 3D point cloud reconstruction of objects using the defect-free 3D point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion provided by this application, in which stones and soil are taken as examples.

Detailed ways

The concept, specific structure and technical effects of the present application will be clearly and completely described below in conjunction with the embodiments and drawings, so as to fully understand the purpose, features and effects of the present application. Apparently, the described embodiments are only some of the embodiments of this project, not all of them.

At present, the method of using the camera to reconstruct the 3D point cloud of the object has defects such as holes due to the limitation of the shooting angle, or the point cloud reconstructed from multiple perspectives is not effective in rough registration due to the inconspicuous characteristics of the object, which makes it difficult to obtain high-precision and defect-free images. Object 3D point cloud.

For the problems referred to above, the embodiment of the present application is by utilizing the theodolite to measure the angle at both ends of a known standard rod, and the theodolite and the cube mirror are self-collimated, record the angle at this time, and then measure the crosshairs on the corresponding surface of the cube mirror. Angle, calculate the transformation matrix between the theodolite coordinate system and the cubic mirror coordinate system; then use the multi-surveillance camera to shoot the target to obtain the internal and external parameters of the camera, and then use the theodolite to measure the corner angle of the target part, calculate the transformation matrix between the camera coordinate system and the theodolite coordinate system, and then Calculate the transformation matrix between the camera coordinate system and the world coordinate system; then use the globally calibrated camera to shoot the object, use the parallax principle to reconstruct the 3D point cloud binocularly, and use the MVS method to reconstruct the 3D point cloud monocularly; then reconstruct the redundant part of the point cloud Crop, filter the point cloud, and perform downsampling; then use the transformation matrix between the camera and the world coordinate system calibrated globally by the surveillance camera to perform coordinate transformation on the preprocessed point cloud to achieve coarse registration of the point cloud; then use the speed The fast and high-precision AAICP algorithm performs fine registration on the point cloud after rough registration; finally, the registered point cloud is merged into a large point cloud, and duplicate points are judged to be deleted to obtain a three-dimensional object without defects point cloud.

Example 1:

As shown in Figure 1, the defect-free 3D point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion provided by the embodiment of the present application includes the following steps:

Step 101, use the theodolite to measure the angles at both ends of a standard rod with known length, and self-align the theodolite with the cube mirror, record the angle at this time, then measure the angle of the crosshairs on the corresponding surface of the cube mirror, and calculate the coordinate system of the theodolite The transformation matrix with the cubic mirror coordinate system is as follows:

(1) Use the built-in self-collimating laser of the theodolite to self-collimate with the reflective plane of the cube mirror, and measure the angle at this time; then use two latitude and longitude for mutual aiming, so that the lasers emitted by the two theodolites are mutually collimated, and measure the angle at this time as the initial angle ;

(2) As shown in Figure 2, the two theodolites can calculate the three-dimensional coordinates by measuring the angle of any point in space, but first, the baseline length of the two theodolites in the horizontal direction needs to be known. Use the theodolite to measure the angle at both ends of a standard rod with known length, and substitute it into the three-dimensional coordinate solution equation and the distance equation L ² =(x _T2 -x _T1 ) ² +(y _T2 -y _T1 ) ² +(z _T2 -z _T1 ) ² can solve the baseline length of the theodolite, and then can solve the three-dimensional coordinates of any point;

The theodolite baseline length b can be calculated by the following formula:

Among them, L is the length of the baseline of the two theodolites in the horizontal direction, and α _A1 , β _A1 , α _A2 , and β _A2 are the horizontal and vertical angles of the two ends of the standard bar, respectively.

After obtaining the baseline length of the theodolite, the coordinates of any point in space in the theodolite coordinate system can be calculated by the following formula:

(3) As shown in Figure 3, according to the angle after self-collimation of the two surfaces of the theodolite and the cube mirror, the theodolite coordinate system is rotated, and the x-axis of the theodolite coordinate system is first rotated around the y-axis by a horizontal angle, and then around the z-axis Rotate the vertical angle, at this time, the x-axis is parallel to the x-axis and y-axis of the cubic mirror coordinate system respectively, the cubic mirror coordinate system can be established, and the rotation matrix between the theodolite coordinate system and the cubic mirror coordinate system can also be solved:

Among them, x _M , y _M , z _M are the vectors of the three axes of the cubic mirror coordinate system in the theodolite coordinate system, is the coordinate rotation matrix from the cubic mirror coordinate system to theodolite coordinate system.

(4) Use the theodolite to measure the crosshair angle of the cube mirror corresponding to the self-collimation surface, and combine it with the coordinate transformation equation, the translation vector between the theodolite coordinate system and the cube mirror coordinate system can be solved, and finally the rotation matrix The coordinate transformation matrix between the theodolite coordinate system and the cubic mirror coordinate system can be obtained by combining with the translation vector.

Since the cubic mirror is a cube whose side length is known, assuming that half of its side length is a, the crosshair points of the two surface centers of the cube mirror corresponding to theodolite T ₁ and theodolite T ₂ are respectively P and Q, as shown in the figure 3, then the homogeneous coordinates of P and Q in the cubic mirror coordinate system are _PM ＝[a 0 0 1] ^T and Q _M ＝[0 a 0 1] ^T respectively. Then the coordinates of these two points in the theodolite coordinate system satisfy the following formula:

Assuming that two theodolites measure the angles of the crosshairs of the cube mirror as (α _AP , β _AP ) and (α _BQ , β _BQ ), respectively, theodolite T ₁ and point P satisfy the following relationship:

Similarly, theodolite T ₂ and point Q satisfy the following relationship:

Wherein, h is the distance between theodolite T ₂ and theodolite T ₁ in the vertical direction, assuming that theodolite T ₁ and theodolite T ₂ are mutually aiming at each other, and the angle in the vertical direction of theodolite T ₁ is β _AB , then h is solved as follows:

h=btanβ _AB

The following equations can be obtained by combining the coordinate conversion equation and the distance equation, and the translation vector between the theodolite coordinate system and the cubic mirror coordinate system can be solved:

Among them, x _PT and x _QT are the coordinates of the crosshair point on the self-collimation surface of the theodolite and the cube mirror in the x-axis direction of the theodolite coordinate system, and T ₁ , T ₂ , T ₃ are the elements in the translation vector to be obtained .

Step 102, use multiple monitoring cameras to shoot the target to obtain the internal and external parameters of the camera, then use the theodolite to measure the corner angle of the target, calculate the conversion matrix between the camera coordinate system and the theodolite coordinate system, and then calculate the conversion matrix between the camera coordinate system and the world coordinate system, specifically as follows:

(1) Use multiple surveillance cameras to capture targets with constantly changing positions, and extract the target corners in the image. For binocular cameras, it is also necessary to match the corners of the left and right images;

(2) Since the internal parameters of each camera remain unchanged, and the external parameters change with each target position, a set of equations can be established to solve the internal and external parameters of the camera through multiple target pictures, the target world coordinate system, and the camera coordinate system The relationship between and the pixel coordinate system is shown in Figure 4;

The equations of each corner point on the target and internal parameters can be established by the following formula:

Among them, dx and dy respectively represent the real size of a pixel on the CCD in the x and y directions, f is the focal length of the camera, s is the tilt coefficient of the camera, generally 0, u ₀ and v ₀ are the pixel coordinates of the principal point of the camera coordinates under the system, Among them, f _x , f _y , u ₀ , and v ₀ are internal camera parameters to be calibrated.

(3) For a binocular camera, after obtaining the internal and external parameters of a single camera, it is also necessary to simultaneously solve the coordinate transformation matrix between the left and right cameras according to the matched angular position;

Since the extrinsic parameter matrix of the left and right cameras corresponding to each target image has been obtained in the single target timing, for any point P _W on the target, the coordinate value of the point in the left and right camera coordinate system can be obtained by the following formula:

in, and /> are the coordinate rotation matrix and coordinate translation matrix from the target world coordinate system to the left and right camera coordinate systems, respectively. Each image corresponds to the above four matrix parameters. By combining the equations of multiple images, the coordinate transformation matrix between the left and right camera coordinate systems can be obtained as follows:

Due to errors in single-target positioning, the R and T matrices obtained by using the left and right camera extrinsic parameter matrices corresponding to each image may not be completely equal. Therefore, the error must be calculated by back-projection, and iteratively find the R and T corresponding to the minimum error. The optimal solution of the matrix.

(4) When the camera is actually used, there is distortion as shown in Figure 5. When the internal and external parameters of the camera are calibrated, the distortion coefficient of the camera can be obtained, and then the image needs to be de-distorted, and the pixel coordinates of the image are converted into imaging plane coordinates according to the internal parameters System coordinates, substituting the modified coordinates into the distortion correction equation, and calculating the new pixel coordinates from the obtained results, and performing integer interpolation on the non-integer coordinates to obtain the distortion-corrected image;

Distortion correction can be achieved by the following formula:

Among them, the radial coordinate (x _u , y _u ) are the coordinates of the undistorted image captured by the ideal camera on the image plane, (x _d , y _d ) are the coordinates of the actual image point on the image plane, (x _c , y _c ) is the coordinates of the principal point of the camera on the image plane; /> is the distortion radius, k ₁ and k ₂ are radial distortion coefficients, p ₁ and p ₂ are tangential distortion coefficients.

(5) For binocular cameras, it is also necessary to rotate the coordinate systems of the left and right cameras so that the coordinate axes of the two coordinate systems are parallel to each other, and the coordinate planes are coplanar, as follows:

After the stereo calibration of the binocular camera is completed, the coordinate rotation matrix R and the coordinate translation matrix T between the left and right cameras can be obtained, as shown in Figure 6(a); as shown in Figure 6(b), the left camera coordinate system along the rotation matrix R Rotate half of R in the positive direction, and record the rotation matrix as R _h1 ; as shown in Figure 6(c), then rotate the right camera coordinate system by half of R in the opposite direction of the rotation matrix R, and record the rotation matrix as R _h2 ; After half the rotation of the camera, the coordinate system of the left camera and the coordinate system of the right camera are parallel but not coplanar, as shown in Figure 6(d); as shown in Figure 6(e), at this time, a point P in the space is between the coordinate systems of the left and right cameras The coordinate relationship is P _L =P _R +R _h2 T; rotate the parallel but non-coplanar left and right camera coordinate systems R _rec to be parallel to the correction coordinate system O _rec x _rec y _rec z _rec , so that the left and right camera coordinate systems are parallel and co-planar As shown in Figure 6(f), the x _rec axis is parallel to the line connecting the origins of the left and right camera coordinate systems, and the x _rec O _rec z _rec plane is parallel to the Z _L O _L O _R Z _R plane.

For the correction coordinate system, let t=R _h2 T, the projection of the coordinate axis of the correction coordinate system in the camera coordinate system can be established as: y _rec =[0 0 1] ^T ×x _rec , z _rec =x _rec ×y _rec , so the coordinate rotation matrix between the correction coordinate system and the camera coordinate system is R _rec =[x _rec y _rec z _rec ] ^T . The resulting left and right camera rotation matrix is:

Substituting all the left and right camera image point coordinates into the above formula, the image after stereo correction of the binocular camera can be obtained. The result is shown in Figure 7, (a) is the target image before stereo correction, and (b) is the target image after stereo correction.

(6) Use the theodolite to measure part of the corner angle of the target, solve its coordinates in the theodolite coordinate system, combine it with the coordinates of the corner points in the camera coordinate system obtained during camera parameter calibration, and use the SVD decomposition method to solve Coordinate transformation matrix between the camera coordinate system and theodolite coordinate system;

Construct the following matrix for multiple corner points:

in, and /> is the average value of coordinates of m known coordinates in the camera coordinate system and theodolite coordinate system. By SVD decomposition of H, two orthonormal matrices and a diagonal matrix can be obtained:

U·Δ·V ^T ＝SVD(H)

Among them, U is the normalized eigenvector of H ^T matrix, V is the normalized eigenvector of H ^T H matrix, Δ is a diagonal matrix containing H ^T matrix and H ^T H matrix eigenvalues .

Calculate the sign of the determinant |V U ^T |, if it is greater than 0, the coordinate rotation matrix is If it is less than 0, change the sign of any column of V, and then substitute it into the above formula to solve the coordinate rotation matrix. Then you can use the coordinate values of a point P in the two coordinate systems to solve the coordinate translation matrix as />

(7) Multiply the coordinate transformation matrix between the camera coordinate system and the theodolite coordinate system and the coordinate transformation matrix between the theodolite coordinate system and the cubic mirror coordinate system to obtain the camera coordinate system and the cubic mirror coordinate system (world coordinate system ), the coordinate transformation matrix between So far, the global calibration of multi-surveillance cameras is completed.

Step 103, using the globally calibrated camera to photograph the object, using the parallax principle to reconstruct the 3D point cloud binocularly, and using the MVS method to reconstruct the 3D point cloud monocularly, as follows:

(1) Use each surveillance camera after global calibration to shoot objects at the same time;

(2) For the binocular camera, use the parallax principle to solve the disparity map of the left and right images, and reconstruct the 3D point cloud according to the disparity map;

(3) For the monocular camera, extract the key points of the image and perform key point matching, then use the key points to reconstruct the sparse point cloud, and then use the MVS method to reconstruct the dense point cloud.

Step 104, clipping the excess part of the reconstructed point cloud, filtering the point cloud, and performing downsampling, as follows:

(1) By judging the main part, set the three-coordinate axis threshold or the spherical area to cut off the redundant part;

(2) Use bilateral filtering, Gaussian filtering, conditional filtering, straight-through filtering, random sampling consistency (RANSAC) filtering and other filtering methods for point clouds to remove outliers and noise points;

(3) In order to improve the speed of point cloud registration, it is necessary to downsample the point cloud, divide the point cloud into multiple voxel grids according to the same width, keep the point closest to the center point in each voxel grid, and remove the rest points, so that the number of point cloud points is greatly reduced but the overall characteristics remain unchanged.

Step 105, using the transformation matrix between the camera and the world coordinate system calibrated globally by the monitoring camera to perform coordinate transformation on the preprocessed point cloud to realize the coarse registration of the point cloud, as follows:

(1) Solve the coordinates of the monocular camera reconstruction point cloud in the monocular camera coordinate system according to the image taken by the monocular camera;

For images taken at two different positions, it is first necessary to solve the coordinate transformation matrix between the two camera positions, which can be solved by using the epipolar geometric relationship. As shown in Figure 8, C and C' are the optical centers of the two cameras, and the image points of any point M in the space on the image planes of the two cameras are m and m'respectively; since any point M ₁ on CM is in The image points on the image surface I are all m, and the image points on the image surface I' are all on the epipolar line l' _m . Similarly, the image points of any point _M'1 on the image surface I on C'M They are all on the epipolar line l _m ; and the epipolar lines on the two image planes must pass through the intersection e and e' of the camera optical center line CC' and the corresponding image plane. Since m and m' are the image points on the two image planes, the matching process between the corresponding feature points in the two images can be changed from two-dimensional to one-dimensional by using the epipolar geometric relationship, reducing the amount of calculation and cost of matching time.

Assume that the coordinates of the spatial point M in the two camera coordinate systems are X=[xyz] ^T and X′=[x′ y′ z′] ^T respectively, and the pixel coordinates in the two images are p=[uv 1] ^T and p′=[u′ v′ 1] ^T , then the coordinates on the two normalized image planes are and /> The coordinate transformation matrix between the two camera coordinate systems satisfies X=RX′+T, then it satisfies in the image plane coordinate system:

Since z and z' are unknown parameters, in order to eliminate their influence, cross-multiply T on the left side of both ends of the above formula at the same time, and get:

Then multiply both sides of the above formula to the left at the same time get:

due to vector perpendicular to vector T and /> Therefore /> with /> The result of dot product is 0, so we can get:

Among them, E=T×R is the essential matrix. Considering that the image plane coordinate system and the pixel coordinate system are related through the internal reference matrix K, it can be known from the above formula that the pixel coordinates satisfy:

p ^T K ^-T T×RK ^-1 p'=p ^T Fp'=0

Wherein, F=K ^-T T×RK ^-1 is the fundamental matrix. Since the internal reference matrix K is known, it is only necessary to solve the six parameters in the coordinate transformation matrix. Use the feature point matching algorithm to find the matching feature points in the two images, use multiple pairs of feature points to construct the above equation, and then use the least square method to obtain the coordinate transformation matrix parameters.

After obtaining the coordinate transformation matrix, you can use triangulation to estimate the depth of the spatial point, and then solve the coordinates of the spatial point in the monocular camera coordinate system. Pair Cross multiply the left side of both ends at the same time /> can get:

Substituting the pixel coordinates into the above formula can get:

z'K ^-1 p×RK ^-1 p'+K ^-1 p×T＝0

Since only z' is unknown in the above formula, z' can be solved through the above formula, and z can be solved similarly. The pixel coordinates and the coordinates in the camera coordinate system satisfy zp=KX, that is, the coordinates in the camera coordinate system can be calculated. Therefore, the coordinates of the spatial point in the monocular camera coordinate system can be calculated by using the epipolar geometric relationship and triangulation.

(2) Solve the coordinates of the binocular camera reconstruction point cloud in the binocular camera coordinate system according to the images captured by the binocular camera;

Since the relative positions of the two cameras of the binocular camera are fixed, compared with the point cloud coordinate solution of the monocular camera, the step of calculating the coordinate transformation matrix between the two cameras is omitted. Both the left and right eyes of the binocular camera satisfy the internal parameter matrix equation, and considering that the coordinate transformation matrix between the left and right cameras can be obtained through the matched corner points, after matching the feature points of the left and right images, the following equations can be obtained:

In the above formula, there are six unknowns x _l , y _l , z _l , x _r , y _r , and z _r , and there are seven equations in total. Therefore, the coordinates of the spatial point in the binocular camera coordinate system can be directly solved.

(3) Use the coordinate transformation matrix between each camera coordinate system calibrated globally and the world coordinate system to transform the coordinates of the point clouds solved in steps (1) and (2) in their respective camera coordinate systems, and all are unified In the world coordinate system, the point cloud coarse registration is realized.

The accuracy of coarse registration in this step depends on the global calibration accuracy of the camera. Since theodolite and cubic mirror are used for calibration, the calibration accuracy is very high, and the corresponding point cloud coarse registration accuracy is also high. good conditions.

Step 106, use the AAICP algorithm with high speed and high precision to perform fine registration on the point cloud after rough registration, as follows:

(1) Randomly select a point set p _i ∈ P in the source point cloud P;

(2) Estimate the corresponding point set q _i ∈ Q from the target point cloud Q, so that ‖q _i -p _i ‖=min;

(3) Use the SVD decomposition method for the two point sets to solve the coordinate transformation matrix between the two point clouds;

(4) Transform p _i using the coordinate transformation matrix calculated in (3), and obtain a new corresponding point set p′ _i ={p′ _i =R p _i +T, p _i ∈ P};

(5) Calculate the average distance between p′ _i and q _i

(6) Set the distance threshold, if d is less than the given distance threshold or the number of iterations is greater than the maximum number of iterations, then stop the iterative process, the coordinate transformation matrix calculated at this time is the optimal coordinate transformation matrix, otherwise return to (2) to continue the iteration ;

(7) Accelerate the convergence process of ICP according to Anderson's accelerated thinking, and greatly improve the speed of point cloud registration without losing accuracy.

Step 107, merging the registered point clouds into a large point cloud, and judging the repeated points to delete to obtain a defect-free 3D point cloud of the object, as follows:

(1) Directly add and sum multiple registered point clouds;

(2) Construct the KD tree for the point cloud after the addition, and the three-dimensional point constructs the KD tree as shown in Figure 9;

(3) Traverse all the points in the point cloud one by one. For a certain point, start from the root node, access the KD number downwards, compare the dimension value corresponding to the point with the median of the dimension corresponding to the node, if it is less than the median number, then visit the left subtree, otherwise visit the right subtree until the leaf node is accessed, calculate the distance between the point and the data stored in the leaf node, if the distance is less than the radius threshold, then perform mark;

(4) Perform a backtracking operation, visit from the leaf node upwards, and judge whether there are nodes whose distance is less than the threshold. If so, enter the node and continue to visit the leaf node to mark the point whose distance is less than the threshold. In the subsequent traversal During the process, if a marked point is encountered, it will be skipped directly;

(5) Delete the marked points in the large point cloud, and only keep the unmarked points, and the point cloud fusion is completed.

Example 2:

A defect-free 3D point cloud reconstruction device based on multi-surveillance cameras and point cloud fusion, including:

The three-dimensional coordinate measurement module of the theodolite is configured to measure the angle of any point in the space and calculate its three-dimensional coordinates;

The camera global calibration module is configured to obtain the internal and external parameters of each monitoring camera and the transformation matrix between the camera coordinate system and the world coordinate system;

The multi-camera 3D point cloud fusion module is configured to perform point cloud preprocessing, and realize coarse point cloud registration according to the results of camera global calibration, then use traditional algorithms for fine point cloud registration, and finally combine multiple point clouds Fusion into a complete and defect-free object 3D point cloud.

The effect of the method proposed in this application is shown in Figure 10, where (a) and (b) are the source point cloud and target point cloud, and (c) and (d) are the top view and side view of the AAICP point cloud fine registration results. Compared with traditional methods, the method proposed in this application has higher 3D point cloud reconstruction accuracy and speed, and has better robustness. This application uses theodolite and cubic mirror to calibrate multi-surveillance cameras, and obtains the internal and external parameters of the cameras more accurately; uses the calibration results to achieve point cloud coarse registration. Compared with the traditional method using point cloud coarse registration algorithm, the speed Faster and better robustness; and use the AAICP algorithm to achieve more accurate and faster point cloud fine registration, thereby improving the real-time performance of the system, and can reconstruct the point cloud of objects in the industry without defects in real time and study the surface morphology of objects , terrain analysis, and 3D visualization play an important role.

The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still implement the foregoing embodiments Modifications to the technical solutions described in the examples, or equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the application, and should be included in the Within the protection scope of this application.

Claims (9)

1. A defect-free three-dimensional point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion, characterized in that, comprising: Use the theodolite to establish the cubic mirror coordinate system; Use multiple monitoring cameras and theodolite to measure the target at the same time, and indirectly obtain the transformation matrix between the coordinate system of the multiple monitoring cameras and the coordinate system of the cubic mirror, and realize the global calibration of the multiple monitoring cameras; Use the calibrated surveillance camera to take pictures of objects and reconstruct 3D point clouds; Perform point cloud preprocessing on multiple point clouds; Use the camera global calibration results to achieve coarse point cloud registration; Use traditional point cloud registration algorithms to achieve fine point cloud registration; and The registered point clouds are fused into a defect-free 3D point cloud. 2. a kind of defect-free three-dimensional point cloud reconstruction method based on multi-monitoring cameras and point cloud fusion as claimed in claim 1, is characterized in that, Use the theodolite to measure the angle at both ends of a standard rod with a known length, and self-align the theodolite and the cube mirror, record the angle at this time, then measure the angle of the crosshairs on the corresponding surface of the cube mirror, and calculate the coordinate system of the theodolite and the cube mirror The transformation matrix of the coordinate system. 3. a kind of defect-free three-dimensional point cloud reconstruction method based on multi-monitoring cameras and point cloud fusion as claimed in claim 2, is characterized in that, Use multiple monitoring cameras to shoot the target to obtain the internal and external parameters of the camera, and then use the theodolite to measure the corner angle of the target, calculate the conversion matrix between the camera coordinate system and the theodolite coordinate system, and then calculate the conversion matrix between the camera coordinate system and the cubic mirror coordinate system, where the cubic The mirror coordinate system acts as the world coordinate system. 4. a kind of defect-free three-dimensional point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion as claimed in claim 1, is characterized in that, Use the globally calibrated camera to shoot the object, use the parallax principle to reconstruct the 3D point cloud binocularly, and use the MVS method to reconstruct the 3D point cloud monocularly. 5. a kind of defect-free three-dimensional point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion as claimed in claim 4, is characterized in that, Clip the excess part of the reconstructed point cloud, filter the point cloud, and perform downsampling. 6. a kind of defect-free three-dimensional point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion as claimed in claim 5, is characterized in that, The coordinate transformation of the preprocessed point cloud is carried out by using the transformation matrix between the camera and the world coordinate system calibrated globally by the surveillance camera, and the coarse registration of the point cloud is realized. 7. a kind of defect-free three-dimensional point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion as claimed in claim 6, is characterized in that, Use the AAICP algorithm with high speed and high precision to fine-register the point cloud after rough registration. 8. a kind of defect-free three-dimensional point cloud reconstruction method based on multi-surveillance cameras and point cloud fusion as claimed in claim 7, is characterized in that, The registered point clouds are merged into a large point cloud, and duplicate points are judged to be deleted to obtain a defect-free 3D point cloud of the object. 9. A defect-free three-dimensional point cloud reconstruction device, which is a defect-free three-dimensional point cloud reconstruction device based on the defect-free three-dimensional point cloud reconstruction method based on multi-monitoring cameras and point cloud fusion according to claims 1 to 8, characterized in that, The defect-free three-dimensional point cloud reconstruction device includes: The three-dimensional coordinate measurement module of the theodolite is configured to measure the angle of any point in the space and calculate its three-dimensional coordinates; The camera global calibration module is configured to obtain the internal and external parameters of each monitoring camera and the transformation matrix between the camera coordinate system and the world coordinate system; The multi-camera 3D point cloud fusion module is configured to perform point cloud preprocessing, and realize coarse registration of point clouds according to the results of camera global calibration, then use traditional algorithms for fine registration of point clouds, and finally combine multiple point clouds Fusion into a complete and defect-free object 3D point cloud.