CN111080627A

CN111080627A - 2D +3D large airplane appearance defect detection and analysis method based on deep learning

Info

Publication number: CN111080627A
Application number: CN201911321821.4A
Authority: CN
Inventors: 汪俊; 郭向林; 刘元朋; 李红卫
Original assignee: Nanjing University of Aeronautics and Astronautics
Current assignee: Nanjing University of Aeronautics and Astronautics
Priority date: 2019-12-20
Filing date: 2019-12-20
Publication date: 2020-04-28
Anticipated expiration: 2039-12-20
Also published as: CN111080627B

Abstract

The invention discloses a 2D +3D large airplane appearance defect detection and analysis method based on deep learning, which is characterized by comprising the following steps of: collecting multi-view 2D images and 3D point cloud data; acquiring a complete airplane point cloud model through registration; extracting image and point cloud feature points, and performing 2D-3D correspondence according to feature matching; estimating the pose of the camera according to the 2D-3D corresponding relation; according to the camera pose, the assignment of texture colors of the 2D image to the 3D point cloud is realized; determining point cloud semantic segmentation according to the point cloud color and coordinate information; and (5) performing detection and analysis on the shape defects of the airplane according to the point cloud semantic segmentation. The method is based on a 2D +3D large airplane appearance defect detection and analysis method for deep learning, utilizes a vision sensor device and an optical three-dimensional detection system measurement technology to process and analyze the collected 2D +3D data, can accurately and effectively detect and extract the appearance defects on the large airplane, has reasonable conception, and can be automatically applied in scenes such as airplane safety inspection and the like in practice.

Description

2D +3D large airplane appearance defect detection and analysis method based on deep learning

Technical Field

The invention relates to the fields of deep learning, computer vision, graphics and the like, in particular to a method for detecting and analyzing appearance defects of a large airplane.

Background

The traditional Non-destructive testing (NDT) technology utilizes the characteristics of sound, light, magnetism and electricity, detects whether defects or Non-uniformity exist in the detected object on the premise of not damaging or influencing the use performance of the detected object, gives information such as the size, position, property and quantity of the defects, further judges the current technical state (such as qualification or Non-qualification, residual service life and the like) of the detected object, and has an important position in the aspect of defect detection of large airplanes. However, the conventional non-destructive inspection has the characteristics of high inspection cost and low speed, and although the inspection rate of volume defects (pores, slag, tungsten inclusions, burn-through, undercut, flash, pits, etc.) is high, the inspection of the volume defects (lack of penetration, lack of fusion, cracks, etc.) is easy to miss if the photographing angle is not proper.

At present, the detection and analysis of the shape defects of large-size airplanes by using a computer vision technology based on a crawling robot is one of mainstream nondestructive detection technologies. The technique utilizes a "flat crawler" mobile robot with suction cup feet to crawl the skin of the aircraft and simultaneously locate and analyze the defects. Specifically, rivet detection is performed using a computer vision algorithm using four cameras mounted on the mobile robot, and the robot spine axis is aligned with the rivet line so as to be positioned at the correct location. However, the technology is unstable due to the influence of illumination, the structural design requirement is high, the calibration is complicated, and the missing rate is high. Therefore, automatic non-destructive inspection has not been successful in visual inspection.

With the development of digital measurement technology, an engineering technical method for detecting the appearance of a large-size airplane in a shutdown state by using a laser tracker and a laser scanner is realized. The large airplane appearance detection based on the laser tracker is a non-contact type mapping and detecting method under the complete machine halt state, which is provided for large-size airplanes or special aircrafts, the size of a mapping object is very large, the span exceeds 40m, the three-view size is 46m multiplied by 42m multiplied by 14m, the precision requirement is high, the content to be measured is very various, the method not only comprises the whole appearance mapping of key components such as a nose, a fuselage, a wing, an engine cabin, a horizontal tail, a vertical tail and the like, but also needs to perform the appearance mapping of different positions of each movable wing surface under different configuration states such as cruising, takeoff, landing and the like. However, the 3D point cloud data acquired by the laser tracker has its own problems, such as sparseness of points, and the effective sensing distance for the algorithm does not exceed 10m, although the accuracy is high. In addition, the method is directly based on 3D point cloud, and the difficulty is that no visual features enable tasks such as tracking and positioning to be not as direct as in vision. The visual advantage is that the amount of information contained is enormous and can provide a large number of visual features.

From the above analysis, it can be seen that the vision and laser trackers should not be two techniques in conflict, but rather have advantages and disadvantages. With the development of computer vision and machine learning level, the computer vision can replace human eyes to identify, position and measure targets, and is applied to many industrial detection problems.

Disclosure of Invention

The invention aims to provide a 2D +3D large airplane appearance defect detection and analysis method based on deep learning, which combines visual sensor equipment and an optical three-dimensional detection system and completes the detection and analysis of the large airplane appearance defect by using the acquired data so as to fill the blank in the prior art.

The technical scheme provided by the invention is as follows:

A2D +3D large airplane appearance defect detection and analysis method is characterized by comprising the following steps:

s1, respectively acquiring images and point clouds of a large-size airplane from a plurality of stations by utilizing a PTZ camera and a laser tracker which are installed on a mobile robot to form multi-view 2D images and 3D point cloud data;

s2, acquiring a complete airplane point cloud model through 3D point cloud registration;

s3, respectively extracting 2D image and 3D point cloud characteristic points, and performing 2D-3D correspondence according to characteristic matching;

s4, estimating the pose of the camera according to the corresponding relation of 2D-3D;

s5, according to the pose of the camera, the assignment of the texture color of the 2D image to the 3D point cloud is achieved, and the 3D point cloud with texture information is obtained;

s6, performing semantic segmentation on the 3D point cloud with the texture information;

and S7, performing defect analysis on the large airplane according to the semantic segmentation result.

On the basis of the above scheme, a further improved or preferred scheme further comprises:

further, in the step S5, mapping the 3D point cloud to an image space according to the estimated camera pose; then, for the correctly matched 2D-3D characteristic point pairs, assigning the color information of the 2D characteristic points to the corresponding 3D characteristic points; for the unmatched 2D-3D feature point pairs, selecting the nearest 2D feature point, and assigning color information to the 3D feature points; for other non-feature points, interpolation is used to obtain color information.

Further, in the step S6, performing self-supervised semantic segmentation according to the textured 3D point cloud constructed in the step S5, wherein the process includes the following steps:

s6.1. sequence generation: establishing a spherical neighborhood which takes any point x in the 3D point cloud with the texture information as a center and has a certain central radius, sequencing all points in the spherical neighborhood according to a z coordinate value, then randomly extracting (k-1) points from the spherical neighborhood, wherein the points have smaller z values compared with the point x, and the (k-1) points and the last point x form a z-order sequence with the length of k;

s6.2, repeating the step S6.1, and generating a plurality of z-sequence sequences for each point in the point cloud;

s6.3, self-supervision characteristic learning: with (x)₁，x₂，…，x_k) Represents any z-sequence with length of k, and uses the front (k-1) point (x) of the z-sequence₁，x₂，...，x_k-1) Predicting the next point x_κUsing a subsequence (x) of length (k-1)₁，x₂，…，x_k-1) Predicting the displacement x_k-x_k-1。

The input of the self-supervision characteristic learning network structure is a three-dimensional point ordered sequence (x) with the length of (k-1)₁，x₂，…，x_k-1) The output is the displacement to the next point, x_k-x_k-1Using multiple spatial coding layers to encode each point x_iEncoding into a high-dimensional vector V_iI is more than or equal to 1 and less than or equal to k-1, and the spatial coding layer consists of 1D convolution, batch normalization and a ReLU activation function; then, the high-dimensional vector sequence (v)₁，v₂，...，v_k-1) Sending to a multilayer Recurrent Neural Network (RNN); finally, the RNN hidden state is transformed into the 3D output y, the spatial displacement estimate required to reach the next point in the sequence, using the full connectivity layer.

Further, the step S7 includes a S7.1 defect detecting process and a S7.2 defect characterizing process, where the S7.1 defect detecting process includes:

a. smoothing and resampling by a mobile least square algorithm, obtaining 3D point cloud data of the component through semantic segmentation, and reconstructing a curved surface by using high-order polynomial interpolation;

b. further estimating the normal and curvature of the curved surface based on a moving least squares method;

c. the component is divided into two parts, a damaged region and a non-damaged region, by using a region growing algorithm:

selecting random points from different regions as seed points, gradually increasing until the whole point cloud covering the component is covered, testing the angle between the normal of the neighborhood point and the normal of the current seed point for each seed point, adding the current neighborhood point to a seed point set if the angle is smaller than a certain threshold, outputting a group of clusters corresponding to each seed point set, regarding one cluster as a part of points on the same smooth surface, combining the clusters, and finally marking the defect region on the component by a visualization method.

Further, the step S2 includes:

s2.1, performing initial registration based on the global measurement field;

and S2.2, carrying out fine registration based on graph optimization on the basis of the initial registration.

And S2.1, calculating the global coordinates of target points arranged around the airplane by using a laser tracker through a self-calibration distance measurement method, and constructing a global measurement field of the whole airplane.

And S2.2, establishing an undirected graph model for optimization by converting the overlapping area between the cloud and the point cloud of each station into the weight of nodes and edges in the graph, and finishing the precise registration of the whole point cloud by iteratively searching and closing a newly generated ring.

Further, the step S3 includes:

s3.1, extracting a group of feature points on the image by using a 2D SIFT detector;

s3.2, extracting a group of feature points of the point cloud of the large airplane after the optimized registration by using the 3D ISS;

and S3.3, obtaining the 2D-3D corresponding relation by utilizing the triple deep neural network to jointly learn the image and the point cloud feature point descriptor according to the two groups of feature points extracted in the steps S31 and S32.

Has the advantages that:

the invention relates to a 2D +3D large airplane appearance defect detection and analysis method based on deep learning, which utilizes a vision sensor device and an optical three-dimensional detection system measurement technology to process and analyze collected 2D +3D data, can accurately and effectively detect and extract appearance defects on a large airplane, has reasonable conception, and can realize automatic application in scenes such as airplane safety inspection and the like in practice.

Drawings

FIG. 1 is a flow chart of aircraft appearance defect detection and analysis according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a global measurement field construction according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a triple depth network structure according to an embodiment of the present invention;

FIG. 4 is a schematic view of a space-filling curve according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of an auto-supervised feature learning according to an embodiment of the present invention;

FIG. 6 is a schematic illustration of an aircraft profile defect analysis implementing an embodiment of the invention;

FIG. 7 is a schematic diagram of an aircraft profile defect detection process embodying embodiments of the present invention;

FIG. 8 is a schematic diagram of an aircraft profile defect depth estimation implementing an embodiment of the invention.

FIG. 9 is a diagram illustrating the size and direction of a detected defect in accordance with an embodiment of the present invention.

Detailed Description

The embodiment relates to a method for detecting and analyzing 2D +3D large airplane appearance defects based on deep learning, which mainly comprises the steps of jointly learning a 2D image and a 3D point cloud feature point descriptor through a triple (three-way) neural network; obtaining 2D-3D feature matching pairs by calculating Euclidean distance similarity matrixes between the 2D and 3D feature descriptors; then, estimating the pose of the camera by a PnP method by utilizing the geometric relation between the 2D-3D matching pairs; projecting the 3D point cloud to an image space by using the estimated camera pose to obtain a 3D point cloud with texture; carrying out self-supervision semantic segmentation on the 3D point cloud with the texture; finally, each part obtained by semantic segmentation is subjected to defect analysis.

To further clarify the technical solution and design principle of the present invention, the following detailed description is made with reference to the accompanying drawings.

As shown in fig. 1, a method for detecting and analyzing a shape defect of a 2D +3D large aircraft based on deep learning includes the following steps:

s1, acquiring 2D images and 3D point cloud data of a single station through a PTZ camera and a laser tracker which are installed on a mobile robot, and then shooting images and acquiring 3D point cloud data by the mobile robot from different visual angles;

s2, acquiring complete point cloud data of the airplane through 3D registration

In the field of industrial measurement, limited by the size of an airplane and mutual shielding among all parts, point cloud data measured by a laser tracker of each station only comprises partial airplane appearance and is positioned under a self-measurement coordinate system. In order to obtain the final complete data of the airplane, point cloud data in different coordinate systems need to be unified to a global coordinate system through a point cloud registration method. Based on the above factors, in the present embodiment, for the multi-station 3D point cloud data obtained in step S1, a global measurement field is first established to perform initial registration; and then, based on the initial registration, carrying out fine registration based on graph optimization to provide an aircraft complete point cloud model for subsequent processing steps.

Step S2 specifically includes:

step S2.1:

a. constructing a global measurement field: the point cloud of each station is directly registered by only depending on the traditional method, and the obtained result precision hardly meets the requirement of the appearance detection of the large airplane. Therefore, to improve the overall accuracy, the present embodiment constructs a global measurement field of the entire region to be scanned by the laser tracker.

As shown in FIG. 2, FIG. 2 includes a station T with 1 target point P (x, y, z) and 3 trackers₁(0，0，0)、T₂(X₂，0，0)、T₃(X₃，Y₃0), by T₁The measuring coordinate system of the tracker on one of the stations is a reference according to T₂Determining X-axis at the second tracker site, and combining with T₃The third tracker site establishes the XOY plane. Regarding the coordinates of each target point and the tracker as unknown parameters, the distance between the tracker site and the target point is a known parameter, and regarding the coordinates as coordinates, the distance measurement value of the same target point P at different site visual angles through the tracker is constructed, and an equation set with 6 unknown parameters is constructed:

in the above formula, d₁For the first station T of the tracker₁Distance to target point P, d₂For the second site T of the tracker₂Distance to target point P, d₃For the third station T of the tracker₃The distance from the target point P to the target point P can be used for solving the coordinates of the target point in the target point P, and the problem of constructing the global measuring field of the tracker can be converted into a problem of solving a linear equation set by a method of increasing the number of the tracker stations and the number of the target points.

In practical situations, the position of the coordinate system is arbitrarily selected, A tracker sites and B target points are arranged, and the transfer parameters can be obtained as long as the following relations are met:

AB≥3(A+B)

by the self-calibration method, the construction of a field global measuring field can be rapidly and accurately finished, meanwhile, extra data are provided by the number of redundant tracker detection sites or targets, and the overall measuring precision can be improved through overall adjustment.

b. Initial registration: after the global measurement field is established, any one station tracker data can be converted to a global coordinate system through a common target point. Similarly, the point cloud data measured by the laser scanner at a certain station view angle can also be converted into a global coordinate system by registering the target point in the point cloud with the target point of the same name measured by the tracker at the station.

By p_iCoordinates representing subscripted target points of the global coordinate system, q_iThe coordinate of the target point under the local coordinate system of the laser tracker is represented, and the set P ═ { P can be obtained₁，p₂，…，p_nQ ═ Q₁，q₂，…，q_nP as the target point cloud, Q as the source point cloud, P_iAs points in the point cloud P, q_iDetermining the rigid body orientation relation of two point clouds by using a least square method, wherein the points in the point cloud Q are n and the point number of the point cloud is n:

as long as more than 3 target points exist, the point cloud data can be quickly and roughly registered to the constructed global coordinate system, and the initial registration work of the whole data is completed.

In the formula (1.1), R and t are respectively a rotation matrix and a translation matrix corresponding to the two groups of point clouds. Obtaining a translation matrix by performing partial derivation on t in (1.1)

Wherein

The gravity centers of two sets of point sets P and Q are respectively. By translation, the new coordinates of the points in the two clouds can be represented as:

that is to say will

Registration is performed as an initial translation vector of the two point cloud.

Equation (1.1) can be simplified as:

to minimize the objective function, it is necessary to

Performing SVD (singular value decomposition) on H ═ U Λ V^TU is a matrix formed by singular vectors, V is an inverse matrix of U, and Λ is a diagonal matrix formed by singular values. When R ═ VU^TEquation (1.2) takes the minimum value, and the best rotation matrix is obtained. And then, the optimal rotation matrix is utilized to complete the rough registration of the point clouds P and Q.

In the above formula, the superscript T represents the matrix transpose.

S2.2, fine registration based on graph optimization;

because the registration of each pair of point clouds has a certain slight deviation compared with an ideal result, for the multi-view point cloud registration condition, if the point clouds are linearly registered from the first station in sequence, a large registration error is likely to occur between the first station and the last station, so that the inconsistency of the whole registration result is caused, which is the closed-loop problem to be solved by the multi-view point cloud registration of the airplane.

The embodiment adopts a graph optimization method to select a proper registration order so as to eliminate error accumulation. The point cloud of each visual angle is taken as a node in the graph, and adjacent nodes with overlapping relation are connected by edges to form an undirected graph of the multi-visual angle point cloud. Through a graph theory optimization method, a plurality of end-to-end connected nodes are selected in an iteration mode to serve as a ring and are closed, so that a new node is formed until no residual nodes exist in the graph, and the global optimization registration work of the large airplane point cloud is completed.

S3, extracting the feature points of the 2D image and the 3D point cloud, and performing 2D-3D correspondence according to the feature points of the 2D image and the 3D point cloud, wherein the method specifically comprises the following steps:

s3.1, extracting a group of feature points on an image by utilizing a 2D SIFT detector

1≤n≤N；

S3.2, extracting a group of characteristic points from the point cloud of the large airplane subjected to optimized registration by using 3D ISS

1≤m≤M；

Here, N and M represent 2D and 3D key point numbers extracted from the 2D image and the 3D point cloud, respectively. SIFT and ISS are mature feature point extraction algorithms.

S3.3, jointly learning a feature point descriptor of the image and the point cloud by using a triple deep neural network according to the two groups of feature points extracted in the steps S31 and S32;

first, a set of local patches (Patch) centered around each 2D and 3D keypoints is created (descriptors include not only keypoints, but also bags)Including points around the keypoint that contribute to it), the deep Triplet network sums μ and

each 2D and 3D keypoint in (a) is mapped to the same high-dimensional feature space to jointly learn a corresponding descriptor set, represented as

And

d is the dimension of the descriptor. The occurrence of n here and in steps 2.1, 3.1 etc. is used only to indicate an unspecified amount in general and does not mean that the amounts represented are necessarily equal.

Then, taking the triplet as input: one reference sample (anchor/reference positive case), one homogeneous sample (positive case), and one heterogeneous sample (negative case). Triplets constitute two types of descriptor pairs: a matching descriptor pair and a non-matching descriptor pair. The Triplet network maximizes the similarity of matched pairs and minimizes the similarity of unmatched pairs by training the pairwise similarity loss function. Expressing its learning objective in similarity distance is: the similarity distance between matching feature descriptor pairs is much smaller than the similarity distance of non-matching descriptor pairs, thus establishing a 2D-3D correspondence between Φ and Ψ.

As shown in fig. 3, the image and point cloud local patches are sent to the network as positive and negative examples of anchor image patches. Representing input triplets as

The triple network consists of three branches, wherein one branch learns the 2D image feature point descriptor G (x)_I；θ_I)：x_I→ p, image patch x_IMapping to descriptor p; the other two branches have shared weight and learn the descriptor F (x) of the feature point of the 3D point cloud_M；θ_M)：x_M→ q, local small block x of point cloud_MMapped to its descriptor q, theta_I、θ_MIs the network weight. Through TThe riplet loss function implements the similarity between the joint learning image and the point cloud feature points. And finally, optimizing the Triplet network parameters by using a random gradient descent method. Image descriptor function G (x)_I；θ_I) The design is that a VGG convolutional neural network is followed by a complete connection layer to extract a 2D image small block key point descriptor. A global average pooling layer is applied over the feature map of convolution 4. Connecting two fully connected layers at the end of the network outputs the desired descriptor dimensions. 3D feature point descriptor function F (x)_M；θ_M) The method is designed into a PointNet network to extract point cloud local small descriptor. The network is trained using Triplet losses, hence the anchor point

Example of harmony

Similar distance between matched pairs

Far less than anchor point

Negative example of

Similar distances between non-matching pairs

Namely, it is

Triple loss Using a weighted Softmargin function

Wherein d ═ d_pos-d_negSuch a loss function may enable the deep network to speed up convergence.

S4, estimating the pose of the camera by using the feature point matching pairs according to the 2D-3D corresponding relation;

the specific process of step S4 is as follows:

s4.1, matching the feature points according to the 2D and 3D feature points and the descriptors extracted in the step S3.3 to finally obtain 2D-3D feature point matching pairs;

specifically, a similarity measurement matrix of each pair of 2D/3D feature descriptors is calculated according to Euclidean distances between feature vectors, then 3D feature points of each 2D image key point are sequenced according to the similarity measurement, and the top 8 nearest 3D feature points can be selected as matching pairs.

S4.2, acquiring more than three groups of feature matching pairs obtained according to the step S4.1, estimating the camera pose according to a PnP algorithm, and eliminating matching pairs with matching errors through a Random Sample Consensus (RANSAC) algorithm;

s5, realizing assignment from 2D texture colors to 3D point clouds according to the pose information of the camera;

firstly, mapping the 3D point cloud to an image space according to the camera pose estimated in the step S4.2; then, for the correctly matched 2D-3D characteristic point pairs, assigning the color information of the 2D characteristic points to the corresponding 3D characteristic points; for the unmatched 2D-3D feature point pairs, selecting the nearest 2D feature point, and assigning color information to the 3D feature points; for other non-feature points, interpolation is used to obtain color information.

S6, performing self-supervision semantic segmentation according to the 3D point cloud with texture constructed in the step S5, wherein the specific process is as follows:

s6.1. sequence generation: specifically, as shown in fig. 4, for any point x in the textured 3D point cloud, S is used_r(x) Representing a spherical neighborhood centered at x with a radius r. For spherical neighborhood S_r(x) All points in (A) are sorted according to z-coordinate value and then from (S)_r(x) Randomly extracts (k-1) points which have smaller z values than x, and the (k-1) points plus the last x form a z-order sequence with the length of k.

S6.2, in order to capture various local structures, repeating the step S6.1 to generate a plurality of z-sequence sequences for each point x in the point cloud.

S6.3. Self-supervision feature learning: with (x)₁，x₂，…，x_k) Represents any z-ordered sequence with length of k, and the embodiment uses the front (k-1) point (x) of the z-ordered sequence₁，x₂，...，x_k-1) Predicting the next point x_k. To stabilize the feature learning process, its equivalent task is learned: using a subsequence (x) of length (k-1)₁，x₂，…，x_k-1) Predicting the displacement x_k-x_k-1. The z-order sequence provides a stable structure to learn unstructured point clouds.

The embodiment includes a spatial coding layer, and the sub-supervised feature learning network structure is shown in fig. 5: the input is a three-dimensional ordered sequence of points (x) of length (k-1)₁，x₂，…，x_k-1) The output is the displacement to the next point, x_k-x_k-1. Using multiple spatial coding layers to encode each point x_iEncoding into high dimensional vectors v_iThe spatial coding layer consists of 1D convolution, batch normalization and a ReLU activation function; then, the high-dimensional vector sequence (v)₁，v₂，...，v_k-1) Sending to a multi-layer Recurrent Neural Network (RNN); finally, the RNN hidden state is transformed into the 3D output y, the spatial displacement estimate required to reach the next point in the sequence, using the full connectivity layer.

S7, performing defect analysis on each segmented part according to the semantic segmentation constructed in the step S6, and specifically comprising the following steps:

s7.1, defect detection: as shown in fig. 7, the method comprises four steps: firstly, smoothing the point cloud by a Moving Least Squares (MLS) algorithm; next, estimating the normal and curvature of each point in the point cloud; further utilizing the normal and curvature information, and dividing the point cloud into a defect area and a non-defect area by using a region growing algorithm; finally, the defect area is marked by using a visualization method.

The process of step S7.1 specifically is:

a. the 3D point cloud is smoothed and resampled by a Moving Least Squares (MLS) algorithm. The surface is reconstructed by interpolation of a high-order polynomial, and a mathematical model of the surface is described as follows:

given a higher order polynomial function f:

and a set of points S ═ c_i，f_i|f(c_i)＝f_iTherein of

Point c_iThe moving least squares approximation of the spherical neighborhood is defined as the error functional:

in order to be a weighted least-squares solution,

theta is called a weighting function, and a Gaussian function is used in the present embodiment

h represents the average distance.

b. Further estimating the normal and curvature of the curved surface based on a moving least square method; given a query point p_qAnd its neighborhood P^KDetermining points x and normal vectors n by least squares plane fitting algorithm_xThe indicated tangent plane S. g_i∈P^KThe distance to the plane S is defined as: d_i＝(g_i-x)·n_x，d_iS is a least squares plane corresponding to 0, where

The center of mass of the lens. Minimum eigenvalue lambda₀Corresponding feature vector v₀As a normal n_xAn approximate estimate of (c). The curvature is estimated by eigenvalues of the covariance matrix:

wherein λ₀＝min(λ_{j＝０，1，2}). The above steps are repeated and the normal and curvature are estimated for each point.

c. The region growing algorithm is used to divide the part into two parts, a damaged region and a non-damaged region. Firstly, selecting random points from different areas as seed points; then, grow gradually until the entire point cloud is covered. For region growing, a rule is needed to check the homogeneity of the region after each growing step, pick the points that satisfy the surface normal and curvature smoothness constraints, and add them to the current set of seed points. For each seed point, testing the angle between the normal of the neighborhood point and the normal of the current seed point, and if the angle is smaller than a certain threshold value, adding the current neighborhood point to the seed point set. In this way, the algorithm outputs a set of clusters, where each cluster is a set of points that are considered to be part of the same smooth surface. Finally, the defect area is marked by using a visualization method.

S7.2, defect characterization: using the S7.1 defect detection results, the size and depth of the defect are estimated. The purpose of this process is to extract and display the three most important pieces of information for defects: size (bounding box), maximum depth, and direction of defect. Specifically, comprise

a. Extracting the lowest point: for each point a in the defect area_iAs shown in FIG. 8, by Δ z (a)_i)＝z_P(ideal)-z(a_i) Estimate its distance from the ideal plane p_idealThe height difference of (2). If | Δ z (a)_i) If | is lower than the predefined threshold, then consider a_iNot the defect point. The lowest point of a defect is determined by the maximum value of all points in the defect area, i.e. max Δ z (a)_i) L, and Δ z (a)_i) Determines whether the defect is an indentation or a protrusion. When Δ z (a)_i) Detect a dent for positive, when Δ z (a)_i) A protrusion is detected when negative.

b. Size and orientation of the defect: for the defect area, in order to display the size and direction of the defect, a directional bounding box is constructed using Principal Component Analysis (PCA), i.e., a minimum rectangular area containing the defect area is found. In the present embodiment, first, the centroid of the defective region is calculated; then, PCA algorithm is applied to determine twoCoordinate system e of spindle assembly_ξFinally, continue along e_ξSearching for an endpoint. These points together constitute the directional bounding box of the defect, the result of which is shown in fig. 9.

The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the foregoing description only for the purpose of illustrating the principles of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the invention as defined by the appended claims, specification, and equivalents thereof.

Claims

1. A2D +3D large airplane appearance defect detection and analysis method based on deep learning is characterized by comprising the following steps:

2. The method for detecting and analyzing the appearance defects of the 2D +3D large airplane based on the deep learning as claimed in claim 1, wherein in the step S5, the 3D point cloud is mapped to the image space according to the estimated camera pose; then, for the correctly matched 2D-3D characteristic point pairs, assigning the color information of the 2D characteristic points to the corresponding 3D characteristic points; for the unmatched 2D-3D feature point pairs, selecting the nearest 2D feature point, and assigning color information to the 3D feature points; for other non-feature points, interpolation is used to obtain color information.

3. The method for detecting and analyzing the appearance defects of the 2D +3D large airplane based on the deep learning as claimed in claim 1 or 2, wherein in the step S6, the self-supervised semantic segmentation is performed according to the textured 3D point cloud constructed in the step S5, and the process comprises the following steps:

s6.1. generating a z-order sequence, namely establishing a spherical neighborhood which takes any point x in the 3D point cloud with the texture information as a center and has a certain central radius, sequencing all points in the spherical neighborhood according to a z coordinate value, and then randomly extracting (k-1) points from the spherical neighborhood, wherein the points have smaller z values compared with the point x, and the (k-1) points and the last point x form a z-order sequence with the length of k;

s6.3, self-supervision characteristic learning by (x)₁，x₂，…，x_k) Represents any z-sequence with length of k, and uses the front (k-1) point (x) of the z-sequence₁，x₂，...，x_k-1) Predicting the next point x_kUsing a subsequence (x) of length (k-1)₁，x₂，…，x_k-1) Predicting the displacement x_k-x_k-1。

4. The method for detecting and analyzing the appearance defects of 2D +3D large airplanes based on deep learning as claimed in claim 3, wherein the input of the self-supervision feature learning network structure is a three-dimensional ordered point sequence (x) with the length of (k-1)₁，x₂，…，x_k-1) The output is the displacement to the next point, x_k-x_k-1Using multiple spatial coding layers to encode each point x_iEncoding into high dimensional vectors v_iI is more than or equal to 1 and less than or equal to k-1, and the space coding layer is formed by 1D convolution and batch normalizationComposition of the activation function of change and ReLU; then, the high-dimensional vector sequence (v)₁，v₂，...，v_k-1) Sending to a multilayer Recurrent Neural Network (RNN); finally, the RNN hidden state is transformed into the 3D output y, the spatial displacement estimate required to reach the next point in the sequence, using the full connectivity layer.

5. The method for detecting and analyzing the appearance defects of the large 2D +3D airplane based on the deep learning of claim 1, wherein the step S7 includes a S7.1 defect detection process and a S7.2 defect characterization process, wherein,

s7.1, the defect detection process comprises the following steps:

6. The method for detecting and analyzing the appearance defect of the 2D +3D large airplane based on the deep learning as claimed in claim 1, wherein the step S2 includes:

s2.1, performing initial registration based on the global measurement field;

7. The method for detecting and analyzing the appearance defects of the 2D +3D large airplane based on the deep learning as claimed in claim 6, wherein the step S2.1 is to use a laser tracker to calculate the global coordinates of target points arranged around the airplane by a self-calibration distance measurement method, and to construct the global measurement field of the whole airplane.

8. The method for detecting and analyzing the appearance defects of the 2D +3D large aircraft based on the deep learning as claimed in claim 6, wherein in the step S2.2, an undirected graph model for optimization is established by converting the overlapping area between the cloud and the point cloud of each station into the weight of nodes and edges in the graph, and the fine registration of the whole point cloud is completed by iteratively searching and closing a newly generated ring.

9. The method for detecting and analyzing the appearance defect of the 2D +3D large airplane based on the deep learning as claimed in claim 1, wherein the step S3 includes: