WO2021179593A1

WO2021179593A1 - Deep learning-based three-dimensional pipeline reconstruction method, system, medium, and apparatus

Info

Publication number: WO2021179593A1
Application number: PCT/CN2020/120385
Authority: WO
Inventors: 屠长河; 程莉莉; 魏卓; 孙铭超; 辛士庆; 安德劳; 李扬彦; 陈宝权
Original assignee: 山东大学
Priority date: 2020-03-10
Filing date: 2020-10-12
Publication date: 2021-09-16
Also published as: CN111429563A; CN111429563B

Abstract

The present disclosure provides a deep learning-based three-dimensional pipeline reconstruction method, a system, a medium, and an apparatus, pertaining to the technical field of three-dimensional pipeline reconstruction. The method comprises: learning features of a point cloud by using a deep learning method, and at least obtaining a category of a component associated with points, a radius of the component associated with the points, and a direction vector of the points; calculating axis points by using the radius of the component associated with the points and the direction vector of the points, and performing, by incorporating a category label of the component associated with the points, clustering on the axis points to obtain candidate instances; obtaining, by using a graph-based method, a connection relationship between the different candidate instances, and constituting, by using the component as a node, a structure of a graph; and replacing the node in the graph with an actual three-dimensional component model to complete reconstruction of an entire pipeline. The present disclosure addresses the low accuracy issue of existing three-dimensional pipeline reconstruction, and reduces the complexity problems of general pipeline reconstruction to that of a combination of component detection and model fitting problems, thereby achieving accurate three-dimensional pipeline reconstruction.

Description

Pipeline 3D reconstruction method, system, medium and equipment based on deep learning

Technical field

The present disclosure relates to the technical field of pipeline three-dimensional reconstruction, and in particular to a pipeline three-dimensional reconstruction method, system, medium and equipment based on deep learning.

Background technique

The statements in this section merely provide background technology related to the present disclosure, and do not necessarily constitute prior art.

High-quality 3D models of power plants, petrochemical plants, and other factories are critical in many applications, including disaster simulation, monitoring, and execution training. Industrial bases are built according to specific plans, usually combined with 3D CAD models. However, building a complete and accurate three-dimensional model is a difficult task. In addition, these models may not exist in older facilities or may not reflect the current appearance of the site. Today, modern laser scanners can capture three-dimensional surfaces and geometric figures with high precision, generating dense point cloud samples. However, in a three-dimensional pipeline scene, capturing surface geometry is particularly challenging.

The inventors of the present disclosure found that due to the importance and universality of the function of the pipeline, it is the main structure of many industrial sites. They are thin structures defined by long cylinders organized in dense and complex structures. Although pipes are only cylindrical in basic shape, which can be easily defined as their axis and radius, they often contain additional components such as flanges, valves, air inlets, elbows, tees, etc. Therefore, the small surface of the pipeline, the serious self-occlusion caused by the complex structure, partial missing, insufficient sampling and other problems can easily cause errors in the three-dimensional scanning and reconstruction of the pipeline.

At present, a common method of point cloud 3D pipeline reconstruction is based on geometric processing and fitting.

The key to this type of method is to find the position of the radius and axis of the cylinder. The commonly used method is to use RanSac, Hough transform and other methods to perform fitting to detect the cylinder, mainly to restore the cylindrical pipe in the industrial plant.

Some technicians have proposed to simplify the problem of 3D factory reconstruction to the detection problem of a 2D circle after the pipeline is projected onto a plane. However, this method is limited to pipes that are perpendicular or parallel to the ground. This type of work only detects straight pipes, while joints such as elbows cannot be restored automatically. Although the improved Hough transform realizes the automatic detection of the cylinder parameters in the point cloud, and then reconstructs the connection relationship between the cylinders to form a continuous network, and finally uses the smart factory 3D (SP3D) to process the data to reconstruct the entire pipeline, but the connection The recovery of the relationship is based on the pre-defined several connection rules for speculation, which has a high degree of uncertainty. Moreover, this method has local problems, and it is seldom able to reconstruct a complete model of thermal power plant with accurate connectivity. Bottom-up primitive fitting techniques are also sensitive to noise and outliers because they lack global and content perception considerations. Although cylindrical is usually the main geometric shape for these locations, the actual data contains a large number of other structures, such as flanges, valves, air inlets, elbows, tees, etc., as shown in Figure 1. Commercial software EdgeWise can also be used to interactively reconstruct pipeline operations, however, these software products usually require a lot of manual work.

Summary of the invention

In order to solve the deficiencies of the prior art, the present disclosure provides a pipeline 3D reconstruction method, system, medium and equipment based on deep learning, which reduces the complexity of common pipeline reconstruction problems to a combination of component detection and model fitting problems. The accurate three-dimensional reconstruction of the pipeline is realized.

In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

The first aspect of the present disclosure provides a pipeline 3D reconstruction method based on deep learning.

A pipeline 3D reconstruction method based on deep learning, including the following steps:

Obtain the point cloud data of the pipeline, use the deep learning method to learn the characteristics of the point cloud, at least get the category of the component to which the point belongs, the radius of the component to which the point belongs, and the direction vector of the point;

Use the radius of the component to which the point belongs and the direction vector of the point to calculate the axis point, and combine the category label of the component to which the point belongs to cluster the axis points to obtain candidate instances;

Use graph-based methods to obtain the connection relationship between different candidate instances, and use components as nodes to form the structure of the graph;

Replace the nodes in the figure with actual three-dimensional component models to complete the entire pipeline reconstruction.

The second aspect of the present disclosure provides a pipeline 3D reconstruction system based on deep learning, including:

The point cloud learning module is configured to: obtain the point cloud data of the pipeline, use the deep learning method to learn the characteristics of the point cloud, and at least obtain the category of the component to which the point belongs, the radius of the component to which the point belongs, and the direction vector of the point;

The candidate instance acquisition module is configured to calculate the axis point by using the radius of the component to which the point belongs and the direction vector of the point, and combine the category label of the component to which the point belongs to cluster the axis point to obtain the candidate instance;

The graph structure component module is configured to: use graph-based methods to obtain the connection relationship between different candidate instances, and use components as nodes to form the structure of the graph;

The pipeline reconstruction module is configured to replace the nodes in the figure with the actual three-dimensional component model to complete the entire pipeline reconstruction.

The third aspect of the present disclosure provides a medium on which a program is stored, and when the program is executed by a processor, the steps in the pipeline 3D reconstruction method based on deep learning as described in the first aspect of the present disclosure are realized.

The fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and capable of running on the processor, and the processor executes the program as described in the first aspect of the present disclosure. The steps in the deep learning-based pipeline 3D reconstruction method are to reconstruct the pipeline 3D model.

Compared with the prior art, the beneficial effects of the present disclosure are:

1. The method, system, medium, and electronic equipment described in the present disclosure reduce the complexity of common pipeline reconstruction problems to a combination of component detection and model fitting problems, and are highly robust and realize accurate three-dimensional reconstruction of pipelines.

2. The methods, systems, media, and electronic devices described in the present disclosure use a combination of clustering and graphs to filter detection results and generate a class diagram global pipeline model, which effectively prevents the generation of training sets and the design of training networks Wait for the error caused by a priori detection.

3. The methods, systems, media, and electronic devices described in the present disclosure embed the initial unreliable local prior detection into a processing framework that considers global attributes and semantic structure, so as to better understand the industrial structure. The cloud reconstructs the complete pipeline structure to achieve more accurate pipeline 3D reconstruction.

4. In the method, system, medium, and electronic device of the present disclosure, the radius and direction vector adopt a weight-sharing framework for regression calculation, which can have better accuracy and convergence.

Description of the drawings

Fig. 1 is a schematic diagram of an existing pipeline structure provided in the background art.

FIG. 2 is a schematic flowchart of a pipeline 3D reconstruction method based on deep learning provided in Embodiment 1 of the disclosure.

FIG. 3 is a schematic diagram of the radius and point direction vector of the component provided in Embodiment 1 of the disclosure.

FIG. 4 is a schematic diagram of the structure of the network framework provided by Embodiment 1 of the disclosure.

FIG. 5 is a schematic diagram of the result of the noisy point cloud and the predicted component category provided by Embodiment 1 of the disclosure.

FIG. 6 is a schematic diagram of the original point cloud provided by Embodiment 1 of the disclosure and the corresponding axis points calculated through prediction features.

FIG. 7 is a schematic diagram of skeletons of different types of components provided in Embodiment 1 of the disclosure.

FIG. 8 is a schematic diagram of reconstruction of the synthetic scene provided in Embodiment 1 of the disclosure.

FIG. 9 is a schematic diagram of comparison of reconstruction results provided by Embodiment 1 of the present disclosure and other methods.

FIG. 10 is a schematic diagram of reconstruction results under point clouds with different missing degrees provided in Embodiment 1 of the disclosure.

FIG. 11 is a schematic diagram of the reconstruction result under the real point cloud data provided in Embodiment 1 of the disclosure.

Detailed ways

It should be pointed out that the following detailed descriptions are all exemplary and are intended to provide further description of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present disclosure belongs.

It should be noted that the terms used here are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprising" and/or "including" are used in this specification, they indicate There are features, steps, operations, devices, components, and/or combinations thereof.

In the case of no conflict, the embodiments in the application and the features in the embodiments can be combined with each other.

Example 1:

In the pipeline design, the pipeline scene is assembled by pipeline components and pipeline supports. In this embodiment, due to the complexity of the problem, pipe components are mainly considered, and supporting parts such as floors and fences are ignored.

Choose six types of components as basic parts: pipes, flanges, elbows, reducers, tees and crosses. Use an extra label to mark points that do not belong to these six types of components. In this embodiment, the category of the component is learned, and therefore, the complexity of the ordinary pipeline reconstruction problem is reduced to a combination of component detection and model fitting problems.

In this embodiment, a priori-based learning method is adopted, and a deep learning network is trained to learn candidate features of 3D point clouds. The prior detection of generating training set and designing training network usually has errors, so a combination of clustering and graph technology is used to filter the detection results and generate a class diagram global pipeline model. Embed the initial unreliable local prior detection into a processing framework that considers global attributes and semantic structure.

As shown in Figure 2, Embodiment 1 of the present disclosure provides a pipeline 3D reconstruction method based on deep learning. Given a scanned point cloud of a pipeline, the method completes the reconstruction in four steps:

(1) Use deep learning technology to learn the characteristics of the point cloud: the category c of the component that the point belongs to, the radius r of the component that the point belongs to, the direction vector o of the point; the category of the component to which the point belongs can assign the point to the corresponding predefined component Category

In piping design, the category and radius of the component can uniquely determine the shape of the component. Since pipeline reconstruction pays more attention to the geometric characteristics of the pipeline, the shape information of the component is obtained by detecting the category and radius of the component. According to the pipeline design standard, the radius of the component is a predetermined discrete value. As mentioned above, six types of components are selected as basic parts: pipes, flanges, reducers, elbows, tees and crosses, and an additional type label is added to distinguish non-component points.

Therefore, the characteristics of each point (p∈P) are predicted by learning methods: the type c of the component to which the point belongs, the radius r of the component to which the point belongs, and the direction vector o of the point. Since the type and radius of the component to which a point belongs are discrete, classification is used to predict, and the discrete category prediction has better accuracy than the regression of continuous values; the direction vector of the point is continuous, so regression is used for prediction. Using the direction vector and radius of the point, the position of the axis point corresponding to the scanning point can be calculated.

(1-1) Generation of training set

This embodiment implements a pipeline generator to simulate a similar real scene model, and trains the network on the synthesized pipeline model. In order to simulate a real pipeline scene, a synthetic pipeline is generated by assembling components.

The specific steps are as follows:

A random skeleton diagram is first generated in the set scene range. For each diagram node, the type, radius and orientation of the components are randomly assembled to obtain the entire pipeline scene. After that, the virtual scanning library is used to sample the surface of the pipeline to simulate the scanning point cloud. Combining the labels required for network learning, for each scan point, get the type of component it belongs to, the radius of the component and the direction vector of the point, and generate the groundtruth about the scan point. The type label of the component is 0, 1, 2, 3, 4, 5, 6, 0-straight pipe, 1-flange, 2-elbow, 3-tee, 4-four-way, 5-reducing pipe , 6-noise point; the radius range of the component is 0.365-4.6, and there are 23 different sizes in total, that is, there are 23 radius labels, respectively 0.365,...,4.6.

(1-2) Network design

As described above, this embodiment uses network learning to predict the category c of the component to which the point belongs, the radius r of the component to which the point belongs, and the direction vector o of the point. Use PointCNN convolutional neural network to achieve classification and regression tasks. The input of the network is the scanned point cloud, each point contains (x, y, z) coordinate information, and the output is the above three tags. As shown in Figure 3, r represents the radius of the component to which the point belongs, and o represents the direction vector of the point.

The specific network framework is shown in Figure 4. The input of the network is a point cloud P containing location information. Use the network in the upper left corner to predict the component category of each point. The network contains four layers of convolution, four layers of deconvolution, and two layers of MLP to obtain 7 channel feature maps. And followed by a softmax activation layer. Then use the classifier to filter out noise, that is, points that do not belong to the six types of components.

Then a multi-task network is defined to handle classification and regression. The network in the upper right corner is used to predict the radius of the component to which each point belongs, and the network in the lower right corner is used to predict the orientation. The range of the radius is 0.365-4.6 meters, with a total of 23 sizes. Therefore, the branch classified in the upper right corner is subjected to four-layer convolution, four-layer deconvolution and two-layer MLP output 23 channel feature vectors, followed by a softmax layer, and the regression branch is four-layer convolution, four-layer deconvolution and two-layer MLP The feature vectors of the 3 channels are output, that is, the corresponding direction vectors, in which the convolutional layer and the deconvolutional layer of the two branches are shared.

In this embodiment, multi-task training is performed to train the entire network at the same time. In the classification task, the cross-entropy loss function is used; in the regression task, the L2loss is used, because the radius and direction vector are related to the offset vector from the scanning point to the component axis. These two tasks use a weight-sharing framework, which can be better The accuracy and convergence.

(1-3) Test steps

Enter the point cloud with coordinate information, and get the category label c of the component to which each point belongs through the classification network; then use this category label to filter out points of non-predefined component types, here is the point with category label c=6; As input, the radius r of the component to which each point belongs and the direction vector o of the point are obtained through the multi-task network;

The test result is shown in Figure 5. The left side is the input point cloud with noise, and the right side is the result of the predicted component category. In the figure on the right, different color depths represent different types of components.

(2) Calculate the axis point by using the radius r of the component to which the point belongs and the direction vector o of the point, and combine the category label c of the component to which the point belongs to cluster the axis points to obtain candidate instances;

The specific steps are:

(2-1) Calculate the axis point corresponding to the scan point

According to the network prediction, the component category of each point, the radius of the component and the direction vector of the point are obtained according to the network prediction, that is, the specific expression of the point is:

_{L = {(p 1, c} 1, r 1, o 1), (p 2, c 2, r 2, o 2), ... (p n, c n, r n, o n)};

First use the predicted component radius and the direction vector of the point to calculate the point on the axis corresponding to each point;

For the point p _i , using the predicted component radius r _i and the direction vector o _i , use the following formula to calculate the corresponding axis point a _i :

a _i = p _i +o _i ×r _i ;

The specific expression of the axis point is:

L ₂ ={(a ₁ ,c ₁ ,r ₁ ),(a ₂ ,c ₂ ,r ₂ ),...(a _n ,c _n ,r _n )};

Where c _i ∈ {0,1,2,3,4,5}, where 0 represents straight pipe, 1 represents flange, 2 represents elbow pipe, 3 represents tee, 4 represents cross, and 5 represents reducer; r _i ∈{0,1,2,3,...,21,22}; where 0-22 correspond to different radii.

(2-2) Use clustering algorithm DBSCAN to cluster axis points

Use the type label c _i to cluster the axis points using the clustering algorithm DBSCAN.

The specific steps are:

Take out the axis points with the type label c _i =0, use the clustering algorithm to cluster to obtain the straight-pipe candidate instance set; similarly take out the points with the category labels 1, 2, 3, 4, 5 and cluster them respectively Obtain the flange instance set, the elbow instance set, the tee instance set, the four-way instance set, and the reducer instance set. So far, all the candidate instance sets in the scene can be obtained. You can know the instance number where each axis point is located, and you can also know the axis point corresponding to each instance.

As shown in Figure 6, the left image is the original point cloud, and the right image is the corresponding axis point calculated by the prediction feature. Similarly, points of different color depths represent different component types.

(3) Use graph-based methods to obtain the connection relationship between different component instances. The process includes adding and deleting instances to obtain a reasonable instance. The instances and their connection relationships are finally represented in the form of a graph;

According to the characteristics learned by the network, the corresponding axis points and candidate component instance sets are obtained through the second step of processing. This step mainly obtains the connection relationship between different components. By iteratively finding the longest path of the minimum spanning tree, the overall skeleton of the pipeline is obtained, and then the connection relationship between the components is initially obtained. Due to a certain error in network prediction, this embodiment uses rules to optimize the graph structure to obtain a reasonable connection relationship between instances. Finally, the overall skeleton is obtained according to the connection relationship between the examples.

The specific steps are:

(3-1) For each candidate component instance, a candidate skeleton is obtained.

The axis points of the template skeleton and the instance are matched using the ICP algorithm to obtain the candidate skeleton of the instance. The skeleton of the template is represented by endpoints and lines. In order to match the template to a proper position, first encrypt the points on the template skeleton according to the axis points of the example, so that the number of points on the template skeleton is the same as the axis points of the example. Then use the matching algorithm to place the skeleton of the template in a suitable position to obtain the candidate skeleton of the example.

(3-2) MST_D gets the overall frame.

Given the axis point,

L ₂ ={(a ₁ ,c ₁ ,r ₁ ),(a ₂ ,c ₂ ,r ₂ ),...(a _n ,c _n ,r _n )};

The overall framework is obtained by iteratively extracting the longest path of the minimum spanning tree. And use this connection relationship to guide the connection relationship between the instances.

Specific steps are as follows:

Initialize graph G;

For each axis point a, calculate k nearest neighbors, and the distance is less than ε, where k=20; ε=0.7; neighbors that meet the conditions are connected to this axis point by an edge to form a graph G;

Calculate the minimum spanning tree forest T;

foreach t ∈ T do

Calculate the longest path d in t

Add d to D

Remove the node in d from L

Update T

So far, the connection relationship of all points is obtained, which constitutes the framework of the pipeline.

(3-3) According to the clustering situation, the initial connection relationship between the instances is obtained.

In this step, the connection relationship between the axis points obtained above and the instance information obtained by clustering are used to obtain the connection relationship between the instances;

The pipeline frame represented by D is composed of multiple acyclic paths d, and d is a path composed of axis points. Traverse each path d and combine the clustering results to obtain the connection relationship between the instances on the path d.

Specific steps are as follows:

First, get all the instances on the path according to the instance label corresponding to each axis point;

Then get the starting and ending position of each instance on the path;

Get the connection relationship between the instances on this path according to the start and end positions; if the start and end intervals of one instance are within the interval of another instance, delete this instance; if the start and end of the two instances If the interval coincidence degree is higher than the threshold ε, where ε=0.98, the instances with few points will be deleted;

After processing all acyclic paths in turn, the initial relationship between all instances can be obtained.

(3-4) Optimize the graph architecture according to the rules to make the connection relationship between the components reasonable.

According to observations, different types of components have specific neighbor relationships. This step combines the template information of each component instance obtained in step (3-1), and uses these connection rules to optimize the graph architecture to obtain accurate connection relationships between instances. The rules are as follows:

Direct management: There are at most two neighbors, and the directions of the neighbor instances need to be the same;

Flange: There are at most two neighbors, and the orientation of the neighbor instances needs to be the same;

Bend: There are at most two neighbors, and the normal directions of the two neighboring endpoints connected to this component are perpendicular to each other;

Tee: There are three neighbors, and the three neighboring endpoints adjacent to this component need to form a three-way layout;

Four-way: There are four neighbors, and the four neighboring endpoints adjacent to this component need to form a four-way layout;

If the rule is not met, it is determined that the component category label predicted by the network is wrong, and then the label is marked;

Finally, the detection updates the connection relationship between the instances according to the mark. If the direction vectors of the adjacent end points are in a vertical state, an elbow instance is added between the two instances.

(3-5) Calculate the skeleton based on the connection relationship and the axis point.

The connection relationship between the instances is obtained above, and the template skeleton corresponding to the instance point cloud is known. Next, the connection relationship is used to optimize the position of the template skeleton again; now the final instance and the connection relationship between the instances can be obtained, taking the instance as Nodes represent the connection relationship of instances in the form of graphs, and there are edges between adjacent instances.

(3-6) Obtain the overall frame based on the example skeleton.

(4) Replace the nodes in the figure with actual three-dimensional component models to complete the entire pipeline reconstruction.

The technology described in this embodiment reconstructs the local structure that follows the connection rules and the semantic relationship in the pipeline. The results show that the method can better reconstruct the complete pipeline structure from the industrial structure point cloud.

Each axis point has a predicted radius. According to the radius of these points, the radius with the most votes is taken as the radius of the component instance; the type, radius, and position of the template corresponding to the instance are known. Now use the template instead of the figure. The nodes complete the reconstruction of the entire pipeline.

(5) Results display

Figure 8 shows the reconstruction of the composite scene, from left to right are the input point cloud, the axis points marked with the color of the component category, the skeleton of the pipeline, and the reconstructed pipeline.

As shown in Figure 9, from left to right: the input point cloud, the skeleton obtained by the method of this embodiment, the reconstruction result of the method of this embodiment, the skeleton (L1-medial skeleton of point cloud) obtained by Huang et al., Liu The reconstruction result of the method (Cylinder detection in large-scale point cloud of pipeline plant) (simplifying the 3D factory reconstruction problem to the detection problem of the 2D circle after the pipeline is projected to the plane), the reconstruction result and groundtruth obtained by the EdgeWise software (Correct reconstruction result).

The comparison includes four scenes of different scales and complexity, from simple small scenes to complex large scenes. When the density of the point cloud and the complexity of the pipeline increase, the methods of Huang et al. and Liu et al. lose components, and the method of Liu et al. cannot reconstruct some joints such as elbows, but can only reconstruct straight pipes. The commercial software EdgeWise requires a lot of manual interaction to complete the reconstruction. The longer the interaction time, the better the results can be obtained, but it is a very time-consuming task.

In order to evaluate the robustness of the method, the point cloud with added noise and different sparsity levels was tested. This experiment simulates the problems that can occur in real pipeline scenes: self-occlusion, high noise and missing caused by weak lighting and reflections.

Starting from a dense scan, gradually increase the noise and sparsity of each point by controlling the parameters of the virtual scan. The degree of sparsity is controlled by the number of scanning cameras and the number of viewing angles of each camera. Then add different degrees of Gaussian noise by adjusting the Gaussian parameters. Figure 9 shows that the scan sparsity and noise levels increase from top to bottom, and the density is 100%, 80%, 65%, and 50%, respectively.

Fig. 11 is the reconstruction result under real point cloud data, from left to right are the input point cloud, the reconstruction result of the method in this embodiment, the reconstruction result of Liu et al., and the reconstruction result of the EdgeWise software.

Example 2:

Embodiment 2 of the present disclosure provides a pipeline 3D reconstruction system based on deep learning, including:

The point cloud learning module is configured to learn the characteristics of the point cloud by using a deep learning method to at least obtain the category of the component to which the point belongs, the radius of the component to which the point belongs, and the direction vector of the point;

The graph structure component module is configured to: use graph-based methods to obtain the connection relationship between different component instances, and use the components as nodes to form the structure of the graph;

The specific working method of the three-dimensional reconstruction system is the same as the three-dimensional reconstruction method described in Embodiment 1, and will not be repeated here.

Example 3:

Embodiment 3 of the present disclosure provides a medium on which a program is stored, and when the program is executed by a processor, the steps in the method for 3D reconstruction of a pipeline based on deep learning as described in Embodiment 1 of the present disclosure are realized.

Example 4:

Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and capable of running on the processor. The processor executes the program as described in Embodiment 1 of the present disclosure. The steps in the deep learning-based pipeline 3D reconstruction method are to reconstruct the pipeline 3D model.

The above descriptions are only preferred embodiments of the present disclosure and are not used to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

A pipeline 3D reconstruction method based on deep learning, which is characterized in that it comprises the following steps:

Obtain the point cloud data of the pipeline, use the deep learning method to learn the characteristics of the point cloud, at least get the category of the component to which the point belongs, the radius of the component to which the point belongs, and the direction vector of the point;

Use the radius of the component to which the point belongs and the direction vector of the point to calculate the axis point, and combine the category label of the component to which the point belongs to cluster the axis points to obtain candidate instances;

Use graph-based methods to obtain the connection relationship between different candidate instances, and use components as nodes to form the structure of the graph;

Replace the nodes in the figure with actual three-dimensional component models to complete the entire pipeline reconstruction.
The pipeline 3D reconstruction method based on deep learning according to claim 1, wherein the PointCNN convolutional neural network is used to realize the feature learning of the point cloud, and the category of the component to which the point belongs, the radius of the component to which the point belongs, and the direction of the point are output. vector.
The pipeline 3D reconstruction method based on deep learning according to claim 1, wherein the calculation method of the axis point is specifically:

a i ＝p i +o i ×r i

Among them, p i is the scanning point, r i is the radius of the component to which the point belongs, and o i is the direction vector of the point;

or,

Using the category label of the component to which the point belongs, clustering the axis points using the clustering algorithm DBSCAN to obtain the set of candidate instances in the scene, and then obtain the instance number where each axis point is located and the axis point corresponding to each instance.
The pipeline 3D reconstruction method based on deep learning according to claim 1, characterized in that the connection relationship between different component instances is obtained by a graph-based method, specifically:

For each candidate component instance, a candidate skeleton is obtained;

Obtain the overall skeleton of the pipeline by iteratively finding the longest path of the minimum spanning tree;

According to the clustering situation, get the initial connection relationship between the instances;

Optimize the graph architecture according to the rules to make the connection relationship between the components reasonable;

Calculate the skeleton based on the connection relationship and axis points;

Obtain the overall frame based on the example skeleton.
The pipeline 3D reconstruction method based on deep learning according to claim 4, characterized in that, for each candidate component instance, a candidate skeleton is obtained, specifically:

The skeleton of the template is represented by endpoints and lines. First, the number of points on the template skeleton is encrypted according to the axis points of the example, so that the number of points on the template skeleton is the same as the number of axis points of the example;

The ICP matching algorithm is used to place the skeleton of the template in a suitable position, and the candidate skeleton of the example is obtained.
The pipeline 3D reconstruction method based on deep learning according to claim 4, characterized in that, according to the clustering situation, the initial connection relationship between the instances is obtained, specifically:

Obtain all instances on the path according to the instance label corresponding to each axis point, and get the starting and ending position of each instance on the path;

Get the connection relationship between the instances on this path according to the start and end positions; if the start and end intervals of one instance are within the interval of another instance, delete this instance; if the start and end of the two instances If the interval coincidence degree is higher than the threshold, the instances with few points will be deleted;

Process all acyclic paths in turn to obtain the initial relationship between all instances;

or,

The graph architecture is optimized according to the rules to make the connection relationship between the components reasonable. The specific rules are:

Direct management: There are at most two neighbors, and the directions of the neighbor instances need to be the same;

Flange: There are at most two neighbors, and the orientation of the neighbor instances needs to be the same;

Bend: There are at most two neighbors, and the normal directions of the two neighboring endpoints connected to this component are perpendicular to each other;

Tee: There are three neighbors, and the three neighboring endpoints adjacent to this component need to form a three-way layout;

Four-way: There are four neighbors, and the four neighboring endpoints adjacent to this component need to form a four-way layout;

If the rules are not met, it is determined that the component category label predicted by the network is wrong, and then the label is marked; the connection relationship between the instances is updated according to the label. If the direction vector of the adjacent end point is vertical, a curve is added between the two instances. Tube instance

or,

Calculate the skeleton based on the connection relationship and the axis points, specifically:

Use the instance connection relationship to optimize the position of the template skeleton again to obtain the final instance and the connection relationship between the instances. The instance is the node, and the connection relationship of the instance is represented in the form of a graph. There are edges between adjacent instances.
The pipeline 3D reconstruction method based on deep learning according to claim 1, wherein each axis point has a predicted radius, and the radius with the most votes is taken as the radius of the component instance according to the radius of these points;

According to the type, radius and position of the template corresponding to the obtained instance, the template is used to replace the nodes in the figure to complete the reconstruction of the entire pipeline.
A pipeline 3D reconstruction system based on deep learning, which is characterized in that it includes:

The point cloud learning module is configured to: obtain the point cloud data of the pipeline, use the deep learning method to learn the characteristics of the point cloud, and at least obtain the category of the component to which the point belongs, the radius of the component to which the point belongs, and the direction vector of the point;

The candidate instance acquisition module is configured to calculate the axis point by using the radius of the component to which the point belongs and the direction vector of the point, and combine the category label of the component to which the point belongs to cluster the axis point to obtain the candidate instance;

The graph structure component module is configured to: obtain the connection relationship between different candidate instances using a graph-based method, and use components as nodes to form the structure of the graph;

The pipeline reconstruction module is configured to replace the nodes in the figure with the actual three-dimensional component model to complete the entire pipeline reconstruction.
A medium having a program stored thereon, wherein the program is executed by a processor to implement the steps in the deep learning-based pipeline 3D reconstruction method according to any one of claims 1-7.
An electronic device, comprising a memory, a processor, and a program stored on the memory and running on the processor, wherein the processor executes the program when the program is executed as described in any one of claims 1-7 The steps in the deep learning-based pipeline 3D reconstruction method are to reconstruct the pipeline 3D model.