CN109544599B - A 3D Point Cloud Registration Method Based on Camera Pose Estimation - Google Patents

Abstract

The invention discloses a three-dimensional point cloud registration method based on camera pose estimation. By organically combining the three-dimensional modeling process problem and the three-dimensional point cloud automatic registration problem, the process of the multi-view three-dimensional modeling process is optimized, thereby The problem of automatic registration of 3D point cloud is transformed into the problem of relative position and pose estimation of cameras from adjacent viewpoints. Moreover, compared with the traditional point cloud registration method, the present invention makes full use of the internal parameters of the camera to simplify the process of obtaining the corresponding point set, and improves the speed of obtaining. Information to filter the corresponding point set, so as to effectively avoid the influence of "abnormal points" on the 3D point cloud matching results, speed up the convergence speed of the matching process and improve the matching accuracy.

Description

A 3D Point Cloud Registration Method Based on Camera Pose Estimation

technical field

The invention relates to the field of reverse engineering, in particular to a three-dimensional point cloud registration method based on camera pose estimation.

Background technique

With the advancement of Made in China 2025 and the "Industry 4.0" plan of various countries in the world, 3D modeling technology has developed rapidly and has rapidly penetrated into many manufacturing fields. Since the launch of Microsoft's Kinect depth sensor, Intel's realsense, and Apple's iphone X, 3D modeling technology has rapidly swept the consumer electronics field. With the rapid development of 3D modeling technology, it has been widely used in many industries and fields.

Obtaining 3D information of objects by measuring is the most accurate method for obtaining spatial geometric information of objects in the field of 3D modeling. The collection of 3D points obtained by 3D measurement is usually called a point cloud. Affected by shadows and occlusions, most 3D measuring instruments cannot obtain complete object topography information through only one measurement. One measurement can only collect 3D data of a certain part of the surface of an object from a certain viewing angle. To complete the 3D measurement of the entire surface of the object, it is necessary to fuse the 3D measurement structure of the object from different perspectives. Since the 3D data collected multiple times are not in the same coordinate system, in order to complete the fusion of multi-view 3D point cloud data, it is necessary to The three-dimensional point clouds obtained from various perspectives are unified into the same coordinate system, and this process is called point cloud registration. The registration of multi-view 3D point clouds is the basis for the subsequent stitching and fusion of 3D point cloud data. The speed and accuracy of point cloud registration directly affect the quality of 3D modeling and the application effect of 3D point cloud data in actual business processes.

In the field of point cloud registration, the most classic and used method is the closest point iteration method ICP algorithm proposed by Besl et al. (Besl P J, Mckay N D.A method for registration of 3D shapes. IEEE), which uses Newton iteration or search method to find two groups of The closest point pair corresponding to the point cloud is iterated using the Euclidean distance as the objective function to obtain a three-dimensional rigid body transformation. Due to the high accuracy of the ICP algorithm, it quickly became the mainstream algorithm in multi-view point cloud registration. With the wide application of the ICP algorithm, researchers have done many detailed studies on the algorithm. Most of the currently applicable methods are the development and improvement of this method.

The Normal Distribution Transform (The Three-Dimensional NormalDistributions Transform-an Efficient Representation for Registration SurfaceAnalysis and Loop Detection) proposed by Martin Magnusson is applied to the statistical model of three-dimensional points, using standard optimization techniques to determine the optimal value between two point clouds Matching, because it does not utilize the feature calculation and matching of each corresponding point in the registration process, so the processing time is faster than the closest point iteration method.

However, the above two types of existing methods both consider the point cloud registration problem from the 3D point cloud data itself, and do not effectively utilize the information in the process of single-view 3D data acquisition. A method for unifying point cloud fusion and stitching to consider the complete surface modeling process of the entire object is reported.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a new three-dimensional point cloud registration method, which can effectively utilize the information in the single-view three-dimensional data acquisition process to improve the point cloud registration accuracy and speed of the three-dimensional point cloud.

In order to achieve the above-mentioned purpose of the invention, the present invention provides the following technical solutions:

A three-dimensional point cloud registration method based on camera pose estimation, which includes the following steps:

Step S01: read a new set of three-dimensional modeling images, and can complete three-dimensional modeling based on the read three-dimensional modeling images;

Step S02: extracting a reliable region on the three-dimensional modeling image to obtain a reliable map;

Step S03: According to the reliable map and the calibrated measurement system parameter information, calculate the three-dimensional coordinates Pts _u,v corresponding to each pixel point (u,v) in the reliable area, and calculate all the three-dimensional coordinates calculated by the three-dimensional coordinates Pts u,v Pts _{u, v} form the 3D point cloud {Pts _cur } of the current viewing angle;

Step S04: extracting the depth information feature and the color information feature of the three-dimensional point cloud {Pts _cur } of the current viewing angle to obtain a multi-feature {Features _cur } composed of the spatial domain feature and the color domain feature;

Step S05: use the transformation of the position and attitude of the camera to represent the relative motion of the object under test and the camera device, and initialize the relative pose RT _relative between the camera of the previous viewing angle and the current viewing angle, wherein the displacement parameter T is 0, and the attitude parameter R is the unit array;

Step S06: Turn the previous viewing angle to the current viewing angle, and calculate the pixel coordinates (u′ corresponding to each three-dimensional data point in the three-dimensional point cloud {Pts _last } of the previous viewing angle according to the internal parameters and relative pose of the camera. ,v'), and find the pixel coordinates (u,v) _nearest with the pixel coordinates (u',v') with the smallest Euclidean distance in the reliable area of the current 3D modeling image, and the 3D point cloud from the above perspective {Pts _last } The three-dimensional point corresponding to the pixel coordinates (u', v') in {Pts _cur } and the three-dimensional point corresponding to the pixel coordinates (u, v) _nearest in {Pts cur } form a corresponding point-to-point pair Pair (u, v), and use all The corresponding point-to-point pairs constitute the corresponding point-point set {Pairs};

Step S07: Based on the similarity measure, filter the corresponding point set {Pairs} to obtain a credible corresponding point set {Pairs _relability };

Step S08: Based on the credible corresponding point set {Pairs _relability }, calculate the rotation and translation matrix RT _accmu between the corresponding points, and apply the RT _accmu to the relative pose RT _relative between the camera of the previous viewing angle and the current viewing angle through matrix operation , and use the calculation result to update the relative pose RT _relative ;

Step S09: judge whether the iteration termination condition is met; if so, go to step 10, otherwise go to step S06 again;

Step S10: According to the specific reason for the iterative termination in step S09, determine whether the three-dimensional point cloud {Pts _cur } of the current viewing angle is a key frame, if it is a key frame, then go to step 11, if it is not a key frame, then go to step S16;

Among them, the method of judging whether the three-dimensional point cloud {Pts _cur } of the current viewing angle is a key frame is as follows:

(1) If the 3D measurement of the current perspective is the first measurement, the 3D point cloud of the current perspective is the key frame by default;

(2) If the 3D point cloud {Pts _cur } of the current perspective is successfully registered with the 3D point cloud {Pts _last } of the previous perspective, and the 3D measurement of the current perspective and the 3D measurement of the previous key frame have been separated by N _{KeyFrameInterval} times 3D If the relative pose RT _relative of the measurement or the 3D measurement of the current perspective and the relative pose of the 3D point cloud of the previous key frame exceeds a certain threshold, then the 3D point cloud {Pts _cur } of the current perspective is the first type of key frame;

(3) If the registration of the 3D point cloud {Pts _cur } of the current perspective and the 3D point cloud {Pts _last } of the previous perspective is unsuccessful, the 3D point cloud of the current perspective is the second type of key frame;

Step S11: determine the type of the current key frame, if the current key frame is the first type of key frame, then go to step S12, if the current key frame is the second type of key frame, then go to step S13;

Step S12: take all the three-dimensional point clouds {Pts _all } that have been registered as the three-dimensional point cloud {Pts _last } of the previous viewing angle, and take the relative pose RT _relative updated in step S08 as the initial value of the relative pose, Execute steps S05 to S09, and substitute the updated relative pose RT _relative into step S16 after completing the three-dimensional point cloud registration;

Step S13: Using the search method for the best corresponding point set based on the affine invariance theory, the multi-component features {Features _cur } of the three-dimensional point cloud {Pts _cur } of the current viewing angle and all the three-dimensional point clouds {Pts _all that have been registered } The corresponding multi-features {Features _all } are searched for the best corresponding point set, and the best corresponding point set {KeyPairs _feature } is obtained;

Step S14: According to the best corresponding point set {KeyPairs _feature }, calculate the relative pose RT relative between the three-dimensional point cloud {Pts _cur } of the current viewing angle and all the three-dimensional point clouds {Pts _all } that have been registered, and set the relative pose RT _relative . The calculated relative pose RT _relative is used as the coarse registration result;

Step S15: Determine whether the point cloud registration in steps S12 to S14 is successful; if the registration is successful, go to step S16, otherwise go to step S17;

Step S16: apply the relative pose RT _relative calculated in step S14 to all the three-dimensional point clouds {Pts _all } that have been registered, and then add the three-dimensional point cloud {Pts cur } of the current viewing angle to the three-dimensional point cloud {Pts _cur } that has been registered. In all 3D point clouds {Pts _all }, and use RT _relative to update the relative pose of the previous key frame;

Step S17: discard the 3D point cloud {Pts _cur } data of the current viewing angle;

Step S18: determine whether the three-dimensional data acquisition is completed; if not, then go to step S01 to perform the registration of the three-dimensional point cloud data of the next viewing angle.

According to a specific embodiment, in the three-dimensional point cloud registration method based on the camera pose estimation of the present invention, the method of judging whether the registration is successful is: judging whether the mean value and variance of the distances between all the corresponding points in the corresponding point set meet the requirements ; or, determine whether the feature similarity statistics of the corresponding points in the corresponding point set meet the requirements.

According to a specific embodiment, in the three-dimensional point cloud registration method based on camera pose estimation of the present invention, the three-dimensional modeling image is a single-frame or multi-frame random coded image, or a striped structured light image.

According to a specific embodiment, in the three-dimensional point cloud registration method based on camera pose estimation of the present invention, when the three-dimensional modeling image is a randomly encoded image, the average sub-region grayscale fluctuation is used as the reliability evaluation index, and obtain the reliable map; wherein,

Let the average sub-region grayscale fluctuation be:

Among them, N(u,v,H,V) represents the neighborhood of size Hx V around the pixel (u,v), Sp (μ,ν) represents the grayscale fluctuation of the _p -th sub-neighborhood; pixel (u,v) ), the sub-region grayscale fluctuation S(μ,ν) of the 3-neighborhood is:

为像素(u,v)所对应的所有三维建模图像的3邻域的灰度均值。Wherein, I _i is the i-th three-dimensional modeling image, N is the number of three-dimensional modeling images captured by a single camera to complete a three-dimensional modeling,

is the gray mean of all 3-neighborhoods of all 3D modeling images corresponding to pixel (u, v).

According to a specific embodiment, in the three-dimensional point cloud registration method based on camera pose estimation of the present invention, when the three-dimensional modeling image is the three-dimensional modeling of striped structured light, the modulation degree of the stripes is used as the reliability evaluation index, And the reliable map is obtained; wherein, the modulation degree of the nth fringe is:

Among them, I ₀ (x, y) is the background light intensity, C ₀ (x, y) is the contrast of the stripes, and N is the number of phase shift steps of the stripe structured light.

According to a specific embodiment, in the three-dimensional point cloud registration method based on camera pose estimation of the present invention, the three-dimensional modeling result is:

Pts3D=Calc3D(ImgVec,R,SysInfo)

Among them, Calc3D is a modeling function, ImgVec is a modeling image, R is the reliability graph, and SysInfo is a system parameter.

According to a specific embodiment, in the three-dimensional point cloud registration method based on camera pose estimation of the present invention, in the step S04, the obtained multivariate features are:

Pts3D _feature = feature(Pts3D,ImgVec,Radius,FeaID)

Among them, Radius is the size of the surrounding neighborhood selected when calculating the feature of the pixel (u, v), FeaID is the combined form of the feature, Pts3D is the 3D modeling result; ImgVec is the modeling image.

According to a specific embodiment, in the three-dimensional point cloud registration method based on the camera pose estimation of the present invention, in the step S07, based on the similarity measure, the method of screening the corresponding point set {Pairs} is as follows: : keep the corresponding point pairs whose similarity measure exceeds a certain threshold, and delete the rest of the corresponding point pairs; among them, the similarity measure is:

Smilarity _Pair = w ₁ Similarity _Color (Pair)+w ₂ Similarity _Geometry (Pair)

Among them, Similarity _Color (Pair) is the color feature similarity measure of the corresponding point-to-point pair Pair (u, v), Similarity _Geometry (Pair) is the corresponding point-to-point pair Pair (u, v) The similarity measure of geometric features, w ₁ , w ₂ are the weights of color features and geometric features, respectively.

Further, the similarity measure of the color feature is the covariance of the color feature of the corresponding point-to-point pair Pair(u,v) in the color domain; the similarity measure of the geometric feature is the geometry of the corresponding point-to-point pair Pair(u,v) in the space domain. covariance of features.

Compared with the prior art, the beneficial effects of the present invention are: the three-dimensional point cloud registration method based on the camera pose estimation of the present invention, by organically combining the three-dimensional modeling process problem and the three-dimensional point cloud automatic registration problem, to many The process of 3D modeling of the perspective is optimized, so that the automatic registration of 3D point clouds can be transformed into the relative position and attitude estimation of cameras in adjacent perspectives. Moreover, compared with the traditional point cloud registration method, the present invention makes full use of the internal parameters of the camera to simplify the process of obtaining the corresponding point set, and improves the speed of obtaining. Information to filter the corresponding point set, so as to effectively avoid the influence of "abnormal points" on the 3D point cloud matching results, speed up the convergence speed of the matching process and improve the matching accuracy. In particular, the present invention is particularly suitable for the application of acquiring multi-view three-dimensional point clouds through the relative motion of the target object and the three-dimensional data acquisition device. In the case where the difference between adjacent angles is so large that the classical point cloud registration method fails, it still has the advantages of Strong adaptability.

Description of drawings:

FIG. 1 is a schematic flow chart of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with test examples and specific embodiments. However, it should not be construed that the scope of the above-mentioned subject matter of the present invention is limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

As shown in FIG. 1 , the three-dimensional point cloud registration method based on camera pose estimation of the present invention includes the following steps:

Step S01: read a new set of three-dimensional modeling images, and can complete three-dimensional modeling based on the read three-dimensional modeling images; specifically, when the present invention is implemented, the read three-dimensional modeling images are a single frame or Randomly encoded images of multiple frames, or striped structured light images. Of course, the three-dimensional modeling image read by the present invention may also be other forms of encoded images, as long as the three-dimensional modeling can be realized under the support of a certain algorithm.

Step S02 : extracting a reliable region from the three-dimensional modeling image to obtain a reliable map.

Specifically, when the 3D modeling image is a randomly coded image, the average sub-region grayscale fluctuation is used as the reliability evaluation index to obtain the reliability map; wherein,

Let the average sub-region grayscale fluctuation be:

Among them, N(u,v,H,V) represents the neighborhood of size Hx V around the pixel (u,v), Sp (μ,ν) represents the grayscale fluctuation of the _p -th sub-neighborhood; pixel (u,v) ), the sub-region grayscale fluctuation S(μ,ν) of the 3-neighborhood is:

为像素(u,v)所对应的所有三维建模图像的3邻域的灰度均值。当然，也可以选择其它尺度的邻域代替3邻域。Wherein, I _i is the i-th three-dimensional modeling image, N is the number of three-dimensional modeling images captured by a single camera to complete a three-dimensional modeling,

is the gray mean of all 3-neighborhoods of all 3D modeling images corresponding to pixel (u, v). Of course, other scales of neighborhoods can also be selected to replace the 3-neighborhood.

Specifically, when the three-dimensional modeling image is the three-dimensional modeling of striped structured light, the modulation degree of the stripes is used as the reliability evaluation index, and the reliability map is obtained. For example, taking N-step phase shift of fringe structured light as an example, a group of phase-shift fringes are projected onto the surface of the object to be measured, and the intensity distribution of the n-th phase-shift fringe on the surface of the object can be expressed as:

I _n (x,y)=I ₀ (x,y){1+C ₀ (x,y)cos[Φ(x,y)+2πn/N]}

Among them, I ₀ (x, y) is the background light intensity, C ₀ (x, y) is the contrast of the fringes, Φ(x, y) is the phase distribution of the fringes, then the modulation degree of the fringes is expressed as

Simplified available

可以作为三维建模图像的可靠性评价标准。It can be seen that the modulation degree M(x,y) of the stripes is proportional to I ₀ (x,y)C ₀ (x,y), and there is only a difference between the two scale factors

It can be used as a reliability evaluation standard for 3D modeling images.

During specific implementation, when the reliability index is greater than a certain threshold, it is judged that the three-dimensional modeling result corresponding to the pixel (u, v) is credible; otherwise, it is regarded as not credible and set to 0. Setting a threshold value can exclude unreliable areas in the 3D modeling image, so that credible effective pixels can be formed into an effective area that can effectively perform 3D modeling.

Step S03: According to the reliable map and the calibrated measurement system parameter information, calculate the three-dimensional coordinates Pts _u,v corresponding to each pixel point (u,v) in the reliable area, and calculate all the three-dimensional coordinates Pts u,v from the calculated three-dimensional coordinates Pts _u,v Forms a 3D point cloud {Pts _cur } for the current viewing angle.

Specifically, in order to facilitate the data parallelism in the subsequent 3D point cloud registration process to improve the calculation speed, the organization form of the 3D point cloud data can be analogous to a color image, which is a rectangle with the same size as the input 3D modeling image. The data element is a three-dimensional space coordinate composed of three data of x, y, and z.

The 3D modeling results are:

Pts3D=Calc3D(ImgVec,R,SysInfo)

Among them, Calc3D is the modeling function, ImgVec is the modeling image, R is the reliability graph, and SysInfo is the system parameter.

Step S04: Perform depth information feature extraction and color information feature extraction on the three-dimensional point cloud {Pts _cur } of the current viewing angle to obtain a multi-dimensional feature {Features _cur } composed of spatial domain features and color domain features.

In the implementation process, in order to achieve parallel acceleration, various features can be selected and combined according to the actual measurement situation. For example, the depth information feature in the spatial domain is composed of normals, the normal statistical distribution of the spatial neighborhood, the topological structure of the points in the spatial neighborhood, the distance statistical distribution of the points in the spatial neighborhood, and the chromaticity statistical distribution of the points in the spatial neighborhood. It can be the weighting of the above multiple geometric features, wherein the points participating in the feature statistics can be all the points in the spatial neighborhood, or can be some points in the spatial neighborhood. The selection scheme of some points in the spatial neighborhood can be selected in various ways, such as selection in the "cross" type, "m" type scheme, random selection, and so on.

When extracting color features, the image carrying color information is first converted to the YUV or HSV color space, and the color features do not consider the brightness information so as to exclude the brightness change factors caused by the shooting angle of view and the surface shape of the object.

Specifically, the multivariate features are:

Pts3D _feature = feature(Pts3D,ImgVec,Radius,FeaID)

Among them, Radius is the size of the surrounding neighborhood selected when calculating the feature of the pixel (u, v), FeaID is the combined form of the feature, Pts3D is the 3D modeling result; ImgVec is the modeling image. The combined form of the features is a single color information feature or a depth information feature, or a color information feature and a depth information feature.

Step S05: use the transformation of the position and attitude of the camera to represent the relative motion of the object under test and the camera device, and initialize the relative pose RT _relative between the camera of the previous viewing angle and the current viewing angle, wherein the displacement parameter T is 0, and the attitude parameter R is the unit array.

Step S06: Turn the previous viewing angle to the current viewing angle, and calculate the pixel coordinates (u′ corresponding to each three-dimensional data point in the three-dimensional point cloud {Pts _last } of the previous viewing angle according to the internal parameters and relative pose of the camera. ,v'), and find the pixel coordinates (u,v) _nearest with the pixel coordinates (u',v') with the smallest Euclidean distance in the reliable area of the current 3D modeling image, and the 3D point cloud from the above perspective {Pts _last } The three-dimensional point corresponding to the pixel coordinates (u', v') in {Pts _cur } and the three-dimensional point corresponding to the pixel coordinates (u, v) _nearest in {Pts cur } form a corresponding point-to-point pair Pair (u, v), and use all The corresponding point-to-point pairs constitute the corresponding point-to-point set {Pairs}.

Specifically, according to the external parameters of the camera at the moment and the internal parameters of the camera calibrated by the event, each valid point in the 3D point cloud data {Pts _last } measured from the previous perspective is projected to the current perspective in an imaging manner according to the camera model. Next, obtain the imaging pixel coordinates (u', v') of each valid point under the current viewing angle. The organization form of the three-dimensional point cloud of the current viewing angle in step S03 is a matrix form.

The camera model and the imaging of three-dimensional points on the pixel plane coordinate system through perspective projection according to the camera model are basic knowledge in the field of the industry, which will not be repeated in this patent.

Step S07: Based on the similarity measure, filter the corresponding point set {Pairs} to obtain a credible corresponding point set {Pairs _relability }. There are shadows and occlusions in the measurement process, and the corresponding point-to-point pairs formed in these areas are unreliable. Therefore, the method of filtering the corresponding point set {Pairs} is: retain the corresponding point pairs whose similarity measure exceeds a certain threshold, and delete the remaining corresponding point pairs; wherein, the similarity measure is:

Smilarity _Pair = w ₁ Similarity _Color (Pair)+w ₂ Similarity _Geometry (Pair)

Among them, Similarity _Color (Pair) is the color feature similarity measure of the corresponding point pair Pair (u, v), Similarity _Geometry (Pair) is the corresponding point pair Pair (u, v) The similarity measure of geometric features, w ₁ , w ₂ are the weights of color features and geometric features, respectively.

Further, the similarity measure of the color feature is the covariance of the color feature of the corresponding point-to-point pair Pair(u,v) in the color domain; the similarity measure of the geometric feature is the geometry of the corresponding point-to-point pair Pair(u,v) in the space domain. covariance of features.

Step S08: Calculate the rotation and translation matrix RT _accmu between the corresponding points based on the credible corresponding point set {Pairs _relability }, and apply the RT _accmu to the relative pose RT _relative between the camera of the previous view and the current view through matrix operations, and Use the result of this calculation to update the relative pose RT _relative .

Step S09: determine whether the iteration termination condition is satisfied; if it is satisfied, go to Step 10, otherwise, go to Step S06 again. Specifically, the termination condition includes the number of iterations and the judgment of the convergence state of the iterative process, among which, the convergence state of the iterative process refers to the matrix difference obtained by two adjacent iterations, and the setting of the iteration termination condition belongs to the prior art, and this patent does not Repeat.

Step S10: According to the specific reason for the iterative termination in step S09, determine whether the three-dimensional point cloud {Pts _cur } of the current viewing angle is a key frame, if it is a key frame, then go to step 11, if it is not a key frame, then go to step S16;

Among them, the method of judging whether the three-dimensional point cloud {Pts _cur } of the current viewing angle is a key frame is as follows:

(1) If the 3D measurement of the current perspective is the first measurement, the 3D point cloud of the current perspective is the key frame by default;

(2) If the 3D point cloud {Pts _cur } of the current perspective is successfully registered with the 3D point cloud {Pts _last } of the previous perspective, and the 3D measurement of the current perspective and the 3D measurement of the previous key frame have been separated by N _{KeyFrameInterval} times 3D If the relative pose RT _relative of the measurement or the 3D measurement of the current perspective and the relative pose of the 3D point cloud of the previous key frame exceeds a certain threshold, then the 3D point cloud {Pts _cur } of the current perspective is the first type of key frame;

(3) If the registration of the 3D point cloud {Pts _cur } of the current perspective and the 3D point cloud {Pts _last } of the previous perspective is unsuccessful, the 3D point cloud of the current perspective is the second type of key frame.

Step S11: Determine the type of the current key frame, if the current key frame is the first type of key frame, then go to step S12, if the current key frame is the second type of key frame, then go to step S13.

Among them, the judgment method of the type of the current key frame is: if the registration of the 3D point cloud of the current perspective and the 3D point cloud of the previous perspective is successful, and the current key frame and the previous key frame are at least N _{KeyFrameInterval} frames or the current perspective and the previous key frame. If the relative position T _KeyFrame and attitude N _{KeyFrameRotate} of the frame's perspective exceed a certain threshold, the current key frame is the first type of key frame; if the registration of the 3D point cloud of the current view and the 3D point cloud of the previous view is unsuccessful, the current key frame is The second type of keyframe.

Step S12: take all the three-dimensional point clouds {Pts _all } that have been registered as the three-dimensional point cloud {Pts _last } of the previous viewing angle, and take the relative pose RT _relative updated in step S08 as the initial value of the relative pose, Steps S05 to S09 are executed, and after the three-dimensional point cloud registration is completed, the updated relative pose RT _relative is substituted into step S16.

Step S13: Using the search method for the best corresponding point set based on the affine invariance theory, the multi-component features {Features _cur } of the three-dimensional point cloud {Pts _cur } of the current viewing angle and all the three-dimensional point clouds {Pts _all that have been registered } The corresponding multi-features {Features _all } are searched for the best corresponding point set, and the best corresponding point set {KeyPairs _feature } is obtained.

Step S14: According to the best corresponding point set {KeyPairs _feature }, calculate the relative pose RT relative between the three-dimensional point cloud {Pts _cur } of the current viewing angle and all the three-dimensional point clouds {Pts _all } that have been registered, and set the relative pose RT _relative . The calculated relative pose RT _relative is used as the coarse registration result.

Step S15: Determine whether the point cloud registration in steps S12 to S14 is successful; if the registration is successful, go to step S16, otherwise go to step S17.

Step S16: apply the relative pose RT _relative calculated in step S14 to all the three-dimensional point clouds {Pts _all } that have been registered, and then add the three-dimensional point cloud {Pts cur } of the current viewing angle to the three-dimensional point cloud {Pts _cur } that has been registered. All 3D point clouds {Pts _all }, and use RT _relative to update the relative pose of the previous key frame; the relative pose is the position of the previous key frame relative to the newly added 3D point cloud {Pts _cur } of the current view. posture.

Step S17: Discard the 3D point cloud {Pts _cur } data of the current viewing angle.

Step S18: determine whether the three-dimensional data acquisition is completed; if not, then go to step S01 to perform the registration of the three-dimensional point cloud data of the next viewing angle.

In the present invention, the method of judging whether the registration is successful is: judging whether the mean value and variance of the corresponding point distances of the corresponding point set meet the requirements; or, judging whether the feature similarity statistics of the corresponding points in the corresponding point set meet the requirements.

The present invention is based on the three-dimensional point cloud registration method based on the camera pose estimation. By organically combining the three-dimensional modeling process problem and the three-dimensional point cloud automatic registration problem, the process of the multi-view three-dimensional modeling process is optimized, so that the three-dimensional point cloud The problem of automatic registration is transformed into the problem of relative position and pose estimation of cameras from adjacent viewpoints. Moreover, compared with the traditional point cloud registration method, the present invention makes full use of the internal parameters of the camera to simplify the process of obtaining the corresponding point set, and improves the speed of obtaining. Information to filter the corresponding point set, so as to effectively avoid the influence of "abnormal points" on the 3D point cloud matching results, speed up the convergence speed of the matching process and improve the matching accuracy. In particular, the present invention is particularly suitable for the application of acquiring multi-view three-dimensional point clouds through the relative motion of the target object and the three-dimensional data acquisition device. In the case where the difference between adjacent angles is so large that the classical point cloud registration method fails, it still has the advantages of Strong adaptability.

Claims (9)

1. A three-dimensional point cloud registration method based on camera pose estimation is characterized by comprising the following steps:

step S01: reading a new set of three-dimensional modeling images, and performing three-dimensional modeling based on the read three-dimensional modeling images;

step S02: extracting a reliable region of the three-dimensional modeling image to obtain a reliable image;

step S03: according to the reliable graph and the calibrated parameter information of the measuring system, calculating the three-dimensional coordinates Pts corresponding to each point pixel point (u, v) in the reliable area_u,vAnd from all said three-dimensional coordinates Pts calculated_u,vThree-dimensional point cloud { Pts) forming current view angle_cur}；

Step S04: three-dimensional point cloud { Pts) of current view angle_curCarrying out depth information characteristic extraction and color information characteristic extraction to obtain multivariate characteristics { Features formed by space domain characteristics and color domain characteristics_cur}；

Step S05: the relative movement of the measured target and the camera piece is expressed by adopting the transformation of the position and the posture of the camera, and the relative pose RT between the camera at the previous visual angle and the camera at the current visual angle is initialized_relativeWherein the displacement parameter T is 0, and the attitude parameter R is a unit matrix;

step S06: the previous visual angle is converted to the current visual angle, and the three-dimensional point cloud { Pts) of the previous visual angle is calculated according to the internal parameters and the relative pose of the camera_lastAnd finding the pixel coordinate (u, v) with the minimum Euclidean distance in the reliable area of the current three-dimensional modeling image according to the pixel coordinate (u ', v') corresponding to each three-dimensional data point in the three-dimensional modeling image_nearestThree-dimensional point cloud { Pts) from the above perspective_lastThe three-dimensional point corresponding to the pixel coordinate (u ', v') in (f) and (Pts)_curCoordinates of middle pixel (u, v)_nearestThe corresponding three-dimensional points form a corresponding point Pair Pair (u, v), and all the corresponding point Pairs form a corresponding point set { Pairs };

step S07: based on similarity measurement, screening the corresponding point set { Pairs } to obtain a credible corresponding point set { Pairs }_relability}；

Step S08: based on the set of credible corresponding points { Pairs_relabilityAnd calculating a rotation translation matrix RT between corresponding points_accmuThe RT is_accmuActing on relative pose RT between previous visual angle and current visual angle camera through matrix operation_relativeAnd updating the relative pose RT using the calculation result_relative；

Step S09: judging whether an iteration termination condition is met; if yes, go to step 10, otherwise, go to step S06 again;

step S10: according to the specific reason of the iteration termination of the step S09, the three-dimensional point cloud { Pts) of the current view angle is judged_curJudging whether the frame is a key frame, if so, entering the step 11, otherwise, entering the step S16;

wherein, the three-dimensional point cloud { Pts of the current view angle is judged_curThe way of judging whether the key frame is as follows:

(1) if the three-dimensional measurement of the current visual angle is the first measurement, the default of the three-dimensional point cloud of the current visual angle is the key frame;

(2) if the three-dimensional point cloud { Pts of the current view angle_curAnd the three-dimensional point cloud of the previous view { Pts }_lastRegistering successfully, and the three-dimensional measurement of the current visual angle is separated from the three-dimensional measurement of the previous key frame by N_{KeyFrameInterval}Relative pose RT of secondary three-dimensional measurement or three-dimensional measurement of current view angle_relativeIf the relative pose of the three-dimensional point cloud of the previous key frame exceeds a certain threshold, the three-dimensional point cloud { Pts) of the current view angle_curThe key frames are of a first type;

(3) if the three-dimensional point cloud { Pts of the current view angle_curAnd the three-dimensional point cloud of the previous view { Pts }_lastIf the registration is unsuccessful, the current view three-dimensional point cloud is a second type key frame;

step S11: judging the type of the current key frame, if the current key frame is the first type key frame, entering the step S12, if the current key frame is the second type key frame, entering the step S13;

step S12: all three-dimensional point clouds { Pts with completed registration_allAs the three-dimensional point cloud { Pts) of the previous view angle_lastAnd at the relative pose RT updated through step S08_relativeExecuting the steps S05 to S09 for the initial value of the relative pose, and finishing the registration of the three-dimensional point cloud and then updating the relative pose RT_relativeStep S16 is substituted;

step S13: adopting an optimal corresponding point set searching method based on affine invariance theory to carry out three-dimensional point cloud { Pts) of the current view angle_curMultivariate Features of { Features }_curAnd all three-dimensional point clouds (Pts) with completed registration_allCorresponding multivariate characteristics { Features }_allFinding the optimal corresponding point set to obtain the optimal corresponding point set { KeyPairs }_feature}；

Step S14: according to the optimal corresponding point set { KeyPairs_featureAnd calculating three-dimensional point cloud { Pts) of the current view angle_curWith all three-dimensional point clouds { Pts) having completed registration_allRelative pose RT between_relativeAnd calculating the relative pose RT_relativeAs a result of the coarse registration;

step S15: judging whether the point cloud registration of the steps S12 to S14 is successful; if the registration is successful, the step S16 is entered, otherwise, the step S17 is entered;

step S16: the relative pose RT calculated in the step S14 is compared with_relativeActing on all three-dimensional point clouds { Pts) with completed registration_allThen three-dimensional point cloud { Pts) of the current view angle_curAdd to all three-dimensional point clouds { Pts) that have completed registration_allIn (b), and use RT_relativeUpdating the relative pose of the last key frame;

step S17: discarding the three-dimensional point cloud { Pts) of the current view angle_curData;

step S18: judging whether three-dimensional data acquisition is finished or not; if not, the process proceeds to step S01, and registration of the three-dimensional point cloud data of the next view angle is performed.

2. The three-dimensional point cloud registration method based on camera pose estimation according to claim 1, wherein the mode for judging whether the registration is successful is as follows: judging whether the mean value and the variance of the distances between all corresponding points in the corresponding point set meet the requirements or not; or judging whether the feature similarity statistics of the corresponding points in the corresponding point set meets the requirement or not.

3. The camera pose estimation-based three-dimensional point cloud registration method according to claim 1, wherein the three-dimensional modeling image is a single-frame or multi-frame randomly coded image or a striped structured light image.

4. The three-dimensional point cloud registration method based on camera pose estimation according to claim 3, wherein when the three-dimensional modeling image is a random coding image, the reliability map is obtained by taking average sub-region gray level fluctuation as a reliability evaluation index; wherein,

let the average subregion grayscale fluctuation be:

where N (u, V, H, V) represents a neighborhood of H x V around the pixel (u, V), S_p(mu, ν) represents the grayscale fluctuation of the p-th sub-neighborhood; the sub-region grayscale fluctuation S (μ, ν) of the 3 neighborhood of the pixel (u, v) is:

wherein, I_iFor the ith three-dimensional modeling image, N is the number of three-dimensional modeling images taken by a single camera to complete one three-dimensional modeling,

for all three-dimensional modelled images to which pixel (u, v) correspondsThe gray level mean of 3 neighborhoods.

5. The three-dimensional point cloud registration method based on camera pose estimation according to claim 3, wherein when the three-dimensional modeling image is a fringe structured light three-dimensional modeling, the modulation degree of the fringes is used as a reliability evaluation index to obtain the reliable image; the modulation degree of the nth stripe is as follows:

wherein, I₀(x, y) is background light intensity, C₀(x, y) is the contrast of the fringes, and N represents the number of phase-shift steps of the structured light of the fringes.

6. The camera pose estimation-based three-dimensional point cloud registration method according to claim 4 or 5, wherein the three-dimensional modeling result is:

Pts3D＝Calc3D(ImgVec,R,SysInfo)

wherein Calc3D is a modeling function, ImgVec is a modeling image, R is the reliability map, and SysInfo is a system parameter.

7. The three-dimensional point cloud registration method based on camera pose estimation according to claim 4 or 5, wherein in the step S04, the obtained multivariate features are:

Pts3D_feature＝feature(Pts3D,ImgVec,Radius,FeaID)

wherein Radius is the size of the selected surrounding neighborhood when calculating the characteristics of the pixel (u, v), FeaID is the combination form of the characteristics, and Pts3D is the three-dimensional modeling result; ImgVec is the modeled image.

8. The three-dimensional point cloud registration method based on camera pose estimation according to claim 1, wherein in the step S07, based on the similarity measure, the corresponding point set { Pairs } is screened by: corresponding point pairs with similarity measurement exceeding a certain threshold value are reserved, and the rest corresponding point pairs are deleted; wherein the similarity measure is:

Smilarity_Pair＝w₁Similarity_Color(Pair)+w₂Similarity_Geometry(Pair)

wherein, the Similarity_Color(Pair) is a color feature Similarity metric, Similarity, for Pair (u, v) for the corresponding point Pair_Geometry(Pair) is a geometric feature similarity metric, w, for Pair (u, v) for the corresponding point Pair₁、w₂The weights of the color feature and the geometric feature, respectively.

9. The camera pose estimation based three-dimensional point cloud registration method of claim 8, wherein the similarity measure of color features is the covariance of the color features of the corresponding point Pair Pair (u, v) in the color domain; the similarity measure of the geometric features is the covariance of the geometric features of the corresponding point Pair Pair (u, v) in the spatial domain.

Images