CN116258835A - Method and system for 3D reconstruction of point cloud data based on deep learning - Google Patents

Method and system for 3D reconstruction of point cloud data based on deep learning Download PDF

Info

Publication number
CN116258835A
CN116258835A CN202310483537.7A CN202310483537A CN116258835A CN 116258835 A CN116258835 A CN 116258835A CN 202310483537 A CN202310483537 A CN 202310483537A CN 116258835 A CN116258835 A CN 116258835A
Authority
CN
China
Prior art keywords
point cloud
point
points
vector
cloud data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310483537.7A
Other languages
Chinese (zh)
Other versions
CN116258835B (en
Inventor
石小川
马超
张典
李烺琛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University WHU
Original Assignee
Wuhan University WHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University WHU filed Critical Wuhan University WHU
Priority to CN202310483537.7A priority Critical patent/CN116258835B/en
Publication of CN116258835A publication Critical patent/CN116258835A/en
Application granted granted Critical
Publication of CN116258835B publication Critical patent/CN116258835B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20Finite element generation, e.g. wire-frame surface description, tesselation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/082Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/70Denoising; Smoothing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • G06T7/246Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/42Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10028Range image; Depth image; 3D point clouds
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02TCLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00Road transport of goods or passengers
    • Y02T10/10Internal combustion engine [ICE] based vehicles
    • Y02T10/40Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Computational Linguistics (AREA)
  • Computer Graphics (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Geometry (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

The invention provides a point cloud data three-dimensional reconstruction method and system based on deep learning, comprising the following steps: step 1, aiming at an object to be scanned, carrying out multiple scanning from a plurality of different angles by utilizing a three-dimensional laser scanner to obtain a plurality of point cloud data of the same object to be scanned; step 2, preprocessing the collected multiple point cloud data, including removing discrete points and recovering noise points to correct positions; step 3, calculating a transformation relation according to the source point cloud and the target point cloud for any two pieces of point cloud data, aligning the source point cloud to the coordinate system where the target point cloud is located, thereby realizing the registration of the point clouds and finally obtaining a complete point cloud image; and 4, carrying out surface reconstruction on the unstructured point cloud image to obtain a triangular grid model of the point cloud data, and carrying out optimization and rendering to obtain a three-dimensional model approaching to a real object. The invention can convert the real non-matter cultural heritage into a digital model which is permanently stored in a computer.

Description

基于深度学习的点云数据三维重建方法和系统Method and system for 3D reconstruction of point cloud data based on deep learning

技术领域Technical Field

本发明涉及信息技术领域,主要面向实物类非物质文化遗产,提出一种运用深度学习方法,通过去噪、配准和表面重构三步,将点云数据构建为三维建模的自动化方法和系统。The present invention relates to the field of information technology, and is mainly aimed at physical intangible cultural heritage. It proposes an automated method and system for constructing point cloud data into three-dimensional modeling through three steps of denoising, registration and surface reconstruction using a deep learning method.

背景技术Background Art

由于文物具有易破损性,且大部分文物都存在自然损坏现象(氧化、腐蚀、生锈等),对文物的实物保护需要投入大量的资源。近年来,随着激光扫描技术和计算机技术的不断发展,数字化文物保护得到了越来越多的应用,如文物三维建模、虚拟现实、数字文物修复等。其中,文物三维建模将文物的实物转化为可以永久存储在计算机中的数字模型,具有成本低、效率高、易展示等优势,是目前主流的数字化文化保护方式。文物三维建模需要用到点云,点云是激光扫描得到的原始数据,它是空间中点的集合,每个点除了包含点的三维坐标信息,还包含颜色、法向量、强度值、时间等信息。Since cultural relics are fragile and most of them are subject to natural damage (oxidation, corrosion, rust, etc.), the physical protection of cultural relics requires a lot of resources. In recent years, with the continuous development of laser scanning technology and computer technology, digital cultural relic protection has been increasingly applied, such as cultural relic 3D modeling, virtual reality, digital cultural relic restoration, etc. Among them, cultural relic 3D modeling converts the physical objects of cultural relics into digital models that can be permanently stored in computers. It has the advantages of low cost, high efficiency, and easy display. It is the current mainstream digital cultural protection method. Cultural relic 3D modeling requires point cloud, which is the raw data obtained by laser scanning. It is a collection of points in space. In addition to the 3D coordinate information of each point, each point also contains information such as color, normal vector, intensity value, time, etc.

在扫描点云数据建模方面,现有的点云LOD模型建立技术存在着一些不足之处,比如内存消耗大、处理时间长和稳定性差等问题,难以处理海量点云数据,且生成的三维模型效果不佳。为了解决这些问题,需要采用一种更高效、更稳定的点云处理技术。在这方面,深度学习技术提供了一种很好的解决方案。深度学习具有处理大型数据集的高效性、提取特征的自主性,非常适合点云数据处理。深度学习技术在图像、语音、自然语言处理等领域的成功应用,也为点云处理提供了一种新的思路。基于深度学习的点云处理技术能够实现点云的高效处理和建模,为非物质文化遗产的保护和传承提供了有力的支持。In terms of scanning point cloud data modeling, the existing point cloud LOD model building technology has some shortcomings, such as high memory consumption, long processing time and poor stability. It is difficult to process massive point cloud data, and the generated 3D model is not good. In order to solve these problems, a more efficient and stable point cloud processing technology is needed. In this regard, deep learning technology provides a good solution. Deep learning has the efficiency of processing large data sets and the autonomy of extracting features, which is very suitable for point cloud data processing. The successful application of deep learning technology in image, speech, natural language processing and other fields also provides a new idea for point cloud processing. Point cloud processing technology based on deep learning can realize efficient processing and modeling of point clouds, providing strong support for the protection and inheritance of intangible cultural heritage.

然而点云数据的特殊性使得传统的深度学习模型不能直接应用于点云数据的处理。点云数据是一种特殊的三维数据表示方法,它由大量的离散点组成,每个点包含三个坐标值以及其他可能的属性值,如颜色、法向量等。与传统的二维图像数据相比,点云数据的特殊性主要表现在两个方面,它具有数据无序性和仿射变化无关性。首先,一团点云数据中有很多个点数据,这些点在点云文件里无论以什么顺序出现,它们指代的信息并不改变,相反,一张图片里的点在图像里已经按照固有的顺序排列好了。其次,点云数据存在于三维空间中,具有仿射变化无关性即旋转不变性,这导致用于二维图像的传统的卷积神经网络无法用于点云数据。由于这些特殊性质,导致传统的卷积神经网络模型难以直接处理点云数据,并且也一直没有深度处理点云数据的标准流程。However, the particularity of point cloud data makes it impossible for traditional deep learning models to be directly applied to the processing of point cloud data. Point cloud data is a special three-dimensional data representation method, which consists of a large number of discrete points, each of which contains three coordinate values and other possible attribute values, such as color, normal vector, etc. Compared with traditional two-dimensional image data, the particularity of point cloud data is mainly manifested in two aspects. It has data disorder and affine change independence. First, there are many point data in a group of point cloud data. No matter in what order these points appear in the point cloud file, the information they refer to does not change. On the contrary, the points in a picture are arranged in the image according to the inherent order. Secondly, point cloud data exists in three-dimensional space and has affine change independence, that is, rotation invariance, which makes the traditional convolutional neural network used for two-dimensional images unable to be used for point cloud data. Due to these special properties, it is difficult for traditional convolutional neural network models to directly process point cloud data, and there has been no standard process for deep processing of point cloud data.

发明内容Summary of the invention

为了解决现有技术的不足之处,有效对3D点云数据进行建模,本发明提出了一套点云数据三维建模方法,包括了去噪、配准和表面重构三个步骤流程,并通过改进的神经网络模型,实现用深度学习技术对点云数据进行处理以得到最终的三维网格模型。In order to address the shortcomings of the prior art and effectively model 3D point cloud data, the present invention proposes a set of point cloud data three-dimensional modeling methods, including three steps of denoising, alignment and surface reconstruction, and through an improved neural network model, it realizes the use of deep learning technology to process point cloud data to obtain the final three-dimensional mesh model.

为了达到上述目的,本发明提供的技术方案是:基于深度学习的点云数据三维重建方法,包括如下步骤:In order to achieve the above object, the technical solution provided by the present invention is: a three-dimensional reconstruction method of point cloud data based on deep learning, comprising the following steps:

步骤1,针对被扫描物体,利用三维激光扫描仪从多个不同的角度进行多次扫描,得到对于同一个待扫描物体的多幅点云数据;Step 1: Scan the object to be scanned multiple times from multiple different angles using a three-dimensional laser scanner to obtain multiple point cloud data for the same object to be scanned;

步骤2,对采集的多幅点云数据进行预处理,包括去除离散点和将噪声点恢复到正确的位置上;Step 2: preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position;

步骤3,对于任意两幅点云数据,根据源点云和目标点云计算一个变换关系,将源点云对齐至目标点云所处的坐标系下,从而实现点云的配准,最终得到一个完整的点云图像;Step 3: For any two point cloud data, a transformation relationship is calculated based on the source point cloud and the target point cloud, and the source point cloud is aligned to the coordinate system of the target point cloud, so as to achieve point cloud registration and finally obtain a complete point cloud image;

步骤4,对非结构化的点云图像进行表面重构,得到点云数据的三角网格模型,再经过优化和渲染,得到逼近于实物的三维模型。Step 4: reconstruct the surface of the unstructured point cloud image to obtain a triangular mesh model of the point cloud data, and then optimize and render it to obtain a three-dimensional model that is close to the real object.

进一步的,步骤2中,首先进行局部划分,将每幅点云数据分成多个局部patch,每个局部patch以一个中心点为中心,包含距离中心r范围内的所有点;Furthermore, in step 2, local division is first performed to divide each point cloud data into multiple local patches, each local patch is centered on a central point and contains all points within a range r from the center;

其次,将每个局部patch输入到四元空间转移网络,学习到四维向量,通过这个四维向量,将原始点云迁移到一个新的位姿状态,然后通过一个全连接层改变位姿状态的维度,使用多层感知机网络提取逐个点云的特征,并对每个点云的特征向量做逆变换,还原到原本的位姿;在获得处理完的逐点特征向量后,对一个patch中的每个点的特征向量进行融合得到一个局部patch的特征向量;Secondly, each local patch is input into the quaternion space transfer network to learn a four-dimensional vector. Through this four-dimensional vector, the original point cloud is transferred to a new posture state, and then the dimension of the posture state is changed through a fully connected layer. The multi-layer perceptron network is used to extract the features of each point cloud, and the feature vector of each point cloud is inversely transformed to restore it to the original posture. After obtaining the processed point-by-point feature vector, the feature vector of each point in a patch is fused to obtain the feature vector of a local patch.

最后对融合的特征向量使用若干全连接层进行回归处理,获得最终结果。Finally, the fused feature vector is regressed using several fully connected layers to obtain the final result.

进一步的,通过改变最后一个全连接层的通道数量实现离散点的去除和噪声点的恢复,对离群点的判定:最后一个全连接层的通道数量设定为1,过程描述函数

Figure SMS_2
Figure SMS_4
Figure SMS_6
表示该点为离群点的概率,自行设置阈值
Figure SMS_8
,即
Figure SMS_10
时,判定
Figure SMS_12
为离群点,将其去除,所有离群点的集合记为
Figure SMS_13
;对噪声点判定:最后一个全连接层的通道数量设定为3,过程描述函数
Figure SMS_1
Figure SMS_3
Figure SMS_5
表示学习到的每个点的偏移向量,修正后的点即为
Figure SMS_7
Figure SMS_9
表示原始的噪声点云,
Figure SMS_11
表示原始的噪声点云中的一个点。Furthermore, by changing the number of channels in the last fully connected layer, the removal of discrete points and the recovery of noise points are achieved. For the determination of outliers, the number of channels in the last fully connected layer is set to 1, and the process description function
Figure SMS_2
,
Figure SMS_4
,
Figure SMS_6
Indicates the probability that the point is an outlier, and sets the threshold by yourself
Figure SMS_8
,Right now
Figure SMS_10
When
Figure SMS_12
is an outlier, remove it, and the set of all outliers is recorded as
Figure SMS_13
; For noise point determination: the number of channels in the last fully connected layer is set to 3, and the process description function
Figure SMS_1
,
Figure SMS_3
,
Figure SMS_5
Represents the offset vector of each learned point, and the corrected point is
Figure SMS_7
,
Figure SMS_9
represents the original noisy point cloud,
Figure SMS_11
Represents a point in the original noisy point cloud.

进一步的,步骤3中任意两幅点云数据进行配准的具体实现方式如下;Furthermore, the specific implementation method of registering any two point cloud data in step 3 is as follows;

首先,分别对源点云和目标点云进行特征提取,得到两个全局特征,然后,将这两个全局特征进行拼接得到两个点云的合并特征,所述拼接操作是为了让源点云和目标点云产生关联,以便在后续的处理中能够更好地进行匹配和对齐;First, feature extraction is performed on the source point cloud and the target point cloud respectively to obtain two global features. Then, the two global features are spliced to obtain the merged features of the two point clouds. The splicing operation is to associate the source point cloud and the target point cloud so that they can be better matched and aligned in subsequent processing.

接下来,将合并特征作为输入,通过五个全连接层来计算点云之间的变换关系,最后的输出是一个七维的向量,其中前三位是平移向量,后四位是单位四元数,将单位四元数转换为旋转矩阵,根据输出的平移向量和旋转矩阵将源点云进行平移和旋转操作,即将源点云对齐至目标点云所处的坐标系下;Next, the merged features are used as input, and the transformation relationship between point clouds is calculated through five fully connected layers. The final output is a seven-dimensional vector, of which the first three are translation vectors and the last four are unit quaternions. The unit quaternion is converted into a rotation matrix. The source point cloud is translated and rotated according to the output translation vector and rotation matrix, that is, the source point cloud is aligned to the coordinate system of the target point cloud.

最后,将对齐后的源点云和目标点云进行融合。Finally, the aligned source point cloud and target point cloud are fused.

进一步的,步骤4中,通过神经网络分辨出点云的表面形状点,形状外点和形状内点,而后对表面形状点上的每一帧点云进行体素表面重构,得到点云数据的三角网格模型,形状外点和形状内点不参与重构;表面重构的具体实现方式如下;Furthermore, in step 4, the surface shape points, shape external points and shape internal points of the point cloud are distinguished by a neural network, and then the voxel surface is reconstructed for each frame of the point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data, and the shape external points and shape internal points do not participate in the reconstruction; the specific implementation method of the surface reconstruction is as follows;

首先初始化一个空三角形网格,然后执行以下操作:First initialize an empty triangle mesh, then do the following:

41,获取当前体素在(x, y, z)处的立方体cube,包含8个顶点;41. Get the cube of the current voxel at (x, y, z), which contains 8 vertices;

42,获取当前立方体cube的边界集合,该边界集合为所有占据概率在[0.5-tau,0.5+tau]区间内的点的集合,tau为设置的阈值;42. Get the boundary set of the current cube cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau], where tau is the set threshold;

43,对于集合中的每个点,执行以下操作:43. For each point in the set, do the following:

i.选取顶点v1和v2,其中p1<0.5, p2>0.5;i. Select vertices v1 and v2, where p1<0.5, p2>0.5;

ii.通过插值函数插值(v1, v2, p1, p2)获得一个新的占据概率等于0.5的插值点vertex;ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5;

iii.将插值点vertex添加到三角形网格M中;iii. Add the interpolation point vertex to the triangle mesh M;

44,构建当前立方体cube的三角形面列表,该列表包含当前cube中占据概率为0.5的点和刚生成的vertex点;44. Build a triangular face list of the current cube, which contains points with an occupation probability of 0.5 in the current cube and the newly generated vertex points;

45,对于每个三角形面,将其添加到三角形网格M中,当所有三角形面都已经添加到三角形网格中,算法结束;45. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends.

进一步的,神经网络学习过程如下;Further, the neural network learning process is as follows;

(1)潜在向量编码:使用点卷积方法首先为每个输入点

Figure SMS_14
产生一个潜在向量
Figure SMS_15
,编码器
Figure SMS_16
由点云卷积网络实现,只需要改变最后一层的通道数来控制向量
Figure SMS_17
的维度,其中
Figure SMS_18
为经过配准后的点云中的点,n表示输入点的个数;(1) Latent vector encoding: Use the point convolution method to first encode each input point
Figure SMS_14
Generate a latent vector
Figure SMS_15
, encoder
Figure SMS_16
Implemented by the point cloud convolutional network, only the number of channels in the last layer needs to be changed to control the vector
Figure SMS_17
The dimensions of
Figure SMS_18
is the point in the registered point cloud, and n represents the number of input points;

(2)相对潜在向量编码:给定一个任意查询点

Figure SMS_20
,查询点
Figure SMS_21
来自输入点,构造一组点集邻居
Figure SMS_24
,它包括离
Figure SMS_26
最近的k个点,设点集邻居中的点为
Figure SMS_28
,而后用查询点
Figure SMS_30
相对于
Figure SMS_32
的局部坐标
Figure SMS_19
,对每个点集邻居
Figure SMS_22
中的点
Figure SMS_23
的潜在向量
Figure SMS_25
进行增强,这些增强的潜在向量经过
Figure SMS_27
处理得到相对潜在向量
Figure SMS_29
,其中
Figure SMS_31
为级联操作;通过以上操作对每一个输入点都完成一次查询;(2) Relative latent vector encoding: Given an arbitrary query point
Figure SMS_20
, query point
Figure SMS_21
From the input point, construct a set of point set neighbors
Figure SMS_24
, which includes
Figure SMS_26
The nearest k points, let the points in the point set neighbors be
Figure SMS_28
, and then use the query point
Figure SMS_30
Relative to
Figure SMS_32
The local coordinates of
Figure SMS_19
, for each point set neighbor
Figure SMS_22
Points in
Figure SMS_23
The latent vector
Figure SMS_25
These enhanced latent vectors are enhanced by
Figure SMS_27
Processing to obtain the relative potential vector
Figure SMS_29
,in
Figure SMS_31
It is a cascade operation; through the above operation, a query is completed for each input point;

(3)特征加权:通过学习

Figure SMS_34
个独立的线性层,由
Figure SMS_36
个相应的权重向量
Figure SMS_38
参数化,产生
Figure SMS_40
个相对权重
Figure SMS_41
Figure SMS_43
表示点乘操作,权重总和为1;最后通过softmax将相对权重
Figure SMS_44
变为权重
Figure SMS_33
,并平均为
Figure SMS_35
,查询点
Figure SMS_37
处的特征向量
Figure SMS_39
由相邻点
Figure SMS_42
的相对潜在向量
Figure SMS_45
加权求和得到:
Figure SMS_46
;(3) Feature weighting: through learning
Figure SMS_34
independent linear layers, consisting of
Figure SMS_36
The corresponding weight vector
Figure SMS_38
Parameterization, generating
Figure SMS_40
Relative weight
Figure SMS_41
,
Figure SMS_43
represents the dot multiplication operation, the sum of the weights is 1; finally, the relative weights are converted to
Figure SMS_44
Becomes weight
Figure SMS_33
, and the average is
Figure SMS_35
, query point
Figure SMS_37
The eigenvector at
Figure SMS_39
By adjacent points
Figure SMS_42
The relative potential vector
Figure SMS_45
The weighted summation yields:
Figure SMS_46
;

(4)解码为占据概率:线性层

Figure SMS_47
将特征向量
Figure SMS_48
解码为占据得分
Figure SMS_49
,这是一个维度为2的全连接层输出,然后通过softmax转换为占用概率
Figure SMS_50
;(4) Decoding into occupancy probability: linear layer
Figure SMS_47
The feature vector
Figure SMS_48
Decoded as occupancy score
Figure SMS_49
, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax
Figure SMS_50
;

(5)体素表面重构:得到各个点的占用概率

Figure SMS_51
后,将占用概率
Figure SMS_52
为0.5
Figure SMS_53
tau的点云当作一个体素格子进行重构。(5) Voxel surface reconstruction: obtain the occupancy probability of each point
Figure SMS_51
After that, the occupation probability
Figure SMS_52
0.5
Figure SMS_53
The point cloud of tau is reconstructed as a voxel grid.

进一步的,步骤1中采用的三维激光扫描仪包括激光脉冲扫描仪和相位测距扫描仪。Furthermore, the three-dimensional laser scanner used in step 1 includes a laser pulse scanner and a phase ranging scanner.

本发明还提供基于深度学习的点云数据三维重建系统,包括如下模块:The present invention also provides a point cloud data 3D reconstruction system based on deep learning, comprising the following modules:

点云获取模块,针对被扫描物体,利用三维激光扫描仪从多个不同的角度进行多次扫描,得到对于同一个待扫描物体的多幅点云数据;The point cloud acquisition module uses a 3D laser scanner to perform multiple scans from multiple different angles on the object to be scanned, thereby obtaining multiple point cloud data for the same object to be scanned;

预处理模块,用于对采集的多幅点云数据进行预处理,包括去除离散点和将噪声点恢复到正确的位置上;The preprocessing module is used to preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position;

配准模块,对于任意两幅点云数据,根据源点云和目标点云计算一个变换关系,将源点云对齐至目标点云所处的坐标系下,从而实现点云的配准,最终得到一个完整的点云图像;The registration module calculates a transformation relationship between the source point cloud and the target point cloud for any two point cloud data, aligns the source point cloud to the coordinate system of the target point cloud, thereby realizing point cloud registration and finally obtaining a complete point cloud image;

重构模块,用于对非结构化的点云图像进行表面重构,得到点云数据的三角网格模型,再经过优化和渲染,得到逼近于实物的三维模型。The reconstruction module is used to reconstruct the surface of unstructured point cloud images to obtain a triangular mesh model of the point cloud data, and then obtain a three-dimensional model that is close to the real object through optimization and rendering.

进一步的,预处理模块中,首先进行局部划分,将每幅点云数据分成多个局部patch,每个局部patch以一个中心点为中心,包含距离中心r范围内的所有点;Furthermore, in the preprocessing module, local division is first performed to divide each point cloud data into multiple local patches, each local patch is centered on a central point and contains all points within a range r from the center;

其次,将每个局部patch输入到四元空间转移网络,学习到四维向量,通过这个四维向量,将原始点云迁移到一个新的位姿状态,然后通过一个全连接层改变位姿状态的维度,使用多层感知机网络提取逐个点云的特征,并对每个点云的特征向量做逆变换,还原到原本的位姿;在获得处理完的逐点特征向量后,对一个patch中的每个点的特征向量进行融合得到一个局部patch的特征向量;Secondly, each local patch is input into the quaternion space transfer network to learn a four-dimensional vector. Through this four-dimensional vector, the original point cloud is transferred to a new posture state, and then the dimension of the posture state is changed through a fully connected layer. The multi-layer perceptron network is used to extract the features of each point cloud, and the feature vector of each point cloud is inversely transformed to restore it to the original posture. After obtaining the processed point-by-point feature vector, the feature vector of each point in a patch is fused to obtain the feature vector of a local patch.

最后对融合的特征向量使用若干全连接层进行回归处理,获得最终结果,具体实现方式为:Finally, the fused feature vector is regressed using several fully connected layers to obtain the final result. The specific implementation method is as follows:

通过改变最后一个全连接层的通道数量实现离散点的去除和噪声点的恢复,对离群点的判定:最后一个全连接层的通道数量设定为1,过程描述函数

Figure SMS_55
Figure SMS_56
Figure SMS_58
表示该点为离群点的概率,自行设置阈值
Figure SMS_61
,即
Figure SMS_63
时,判定
Figure SMS_65
为离群点,将其去除,所有离群点的集合记为
Figure SMS_66
;对噪声点判定:最后一个全连接层的通道数量设定为3,过程描述函数
Figure SMS_54
Figure SMS_57
Figure SMS_59
表示学习到的每个点的偏移向量,修正后的点即为
Figure SMS_60
Figure SMS_62
表示原始的噪声点云,
Figure SMS_64
表示原始的噪声点云中的一个点。By changing the number of channels in the last fully connected layer, the discrete points can be removed and the noise points can be restored. For the determination of outliers, the number of channels in the last fully connected layer is set to 1. The process description function
Figure SMS_55
,
Figure SMS_56
,
Figure SMS_58
Indicates the probability that the point is an outlier, and sets the threshold by yourself
Figure SMS_61
,Right now
Figure SMS_63
When
Figure SMS_65
is an outlier, remove it, and the set of all outliers is recorded as
Figure SMS_66
; For noise point determination: the number of channels in the last fully connected layer is set to 3, and the process description function
Figure SMS_54
,
Figure SMS_57
,
Figure SMS_59
Represents the offset vector of each learned point, and the corrected point is
Figure SMS_60
,
Figure SMS_62
represents the original noisy point cloud,
Figure SMS_64
Represents a point in the original noisy point cloud.

进一步的,重构模块中,通过神经网络分辨出点云的表面形状点,形状外点和形状内点,而后对表面形状点上的每一帧点云进行体素表面重构,得到点云数据的三角网格模型,形状外点和形状内点不参与重构;表面重构的具体实现方式如下;Furthermore, in the reconstruction module, the surface shape points, shape external points and shape internal points of the point cloud are distinguished by a neural network, and then the voxel surface is reconstructed for each frame of the point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data, and the shape external points and shape internal points do not participate in the reconstruction; the specific implementation method of the surface reconstruction is as follows;

首先初始化一个空三角形网格,然后执行以下操作:First initialize an empty triangle mesh, then do the following:

41,获取当前体素在(x, y, z)处的立方体cube,包含8个顶点;41. Get the cube of the current voxel at (x, y, z), which contains 8 vertices;

42,获取当前立方体cube的边界集合,该边界集合为所有占据概率在[0.5-tau,0.5+tau]区间内的点的集合,tau为设置的阈值;42. Get the boundary set of the current cube cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau], where tau is the set threshold;

43,对于集合中的每个点,执行以下操作:43. For each point in the set, do the following:

i.选取顶点v1和v2,其中p1<0.5, p2>0.5;i. Select vertices v1 and v2, where p1<0.5, p2>0.5;

ii.通过插值函数插值(v1, v2, p1, p2)获得一个新的占据概率等于0.5的插值点vertex;ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5;

iii.将插值点vertex添加到三角形网格M中;iii. Add the interpolation point vertex to the triangle mesh M;

44,构建当前立方体cube的三角形面列表,该列表包含当前cube中占据概率为0.5的点和刚生成的vertex点;44. Build a triangular face list of the current cube, which contains points with an occupation probability of 0.5 in the current cube and the newly generated vertex points;

45,对于每个三角形面,将其添加到三角形网格M中,当所有三角形面都已经添加到三角形网格中,算法结束;45. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends.

其中神经网络学习过程如下;The neural network learning process is as follows:

(1)潜在向量编码:使用点卷积方法首先为每个输入点

Figure SMS_67
产生一个潜在向量
Figure SMS_68
,编码器
Figure SMS_69
由点云卷积网络实现,只需要改变最后一层的通道数来控制向量
Figure SMS_70
的维度,其中
Figure SMS_71
为经过配准后的点云中的点,n表示输入点的个数;(1) Latent vector encoding: Use the point convolution method to first encode each input point
Figure SMS_67
Generate a latent vector
Figure SMS_68
, encoder
Figure SMS_69
Implemented by the point cloud convolutional network, only the number of channels in the last layer needs to be changed to control the vector
Figure SMS_70
The dimensions of
Figure SMS_71
is the point in the registered point cloud, and n represents the number of input points;

(2)相对潜在向量编码:给定一个任意查询点

Figure SMS_73
,查询点
Figure SMS_75
来自输入点,构造一组点集邻居
Figure SMS_77
,它包括离
Figure SMS_79
最近的k个点,设点集邻居中的点为
Figure SMS_81
,而后用查询点
Figure SMS_83
相对于
Figure SMS_85
的局部坐标
Figure SMS_72
,对每个点集邻居
Figure SMS_74
中的点
Figure SMS_76
的潜在向量
Figure SMS_78
进行增强,这些增强的潜在向量经过
Figure SMS_80
处理得到相对潜在向量
Figure SMS_82
,其中
Figure SMS_84
为级联操作;通过以上操作对每一个输入点都完成一次查询;(2) Relative latent vector encoding: Given an arbitrary query point
Figure SMS_73
, query point
Figure SMS_75
From the input point, construct a set of point set neighbors
Figure SMS_77
, which includes
Figure SMS_79
The nearest k points, let the points in the point set neighbors be
Figure SMS_81
, and then use the query point
Figure SMS_83
Relative to
Figure SMS_85
The local coordinates of
Figure SMS_72
, for each point set neighbor
Figure SMS_74
Points in
Figure SMS_76
The latent vector
Figure SMS_78
These enhanced latent vectors are enhanced by
Figure SMS_80
Processing to obtain the relative potential vector
Figure SMS_82
,in
Figure SMS_84
It is a cascade operation; through the above operation, a query is completed for each input point;

(3)特征加权:通过学习

Figure SMS_87
个独立的线性层,由
Figure SMS_88
个相应的权重向量
Figure SMS_90
参数化,产生
Figure SMS_93
个相对权重
Figure SMS_94
Figure SMS_97
表示点乘操作,权重总和为1;最后通过softmax将相对权重
Figure SMS_99
变为权重
Figure SMS_86
,并平均为
Figure SMS_89
,查询点
Figure SMS_91
处的特征向量
Figure SMS_92
由相邻点
Figure SMS_95
的相对潜在向量
Figure SMS_96
加权求和得到:
Figure SMS_98
;(3) Feature weighting: through learning
Figure SMS_87
independent linear layers, consisting of
Figure SMS_88
The corresponding weight vector
Figure SMS_90
Parameterization, generating
Figure SMS_93
Relative weight
Figure SMS_94
,
Figure SMS_97
represents the dot multiplication operation, the sum of the weights is 1; finally, the relative weights are converted to
Figure SMS_99
Becomes weight
Figure SMS_86
, and the average is
Figure SMS_89
, query point
Figure SMS_91
The eigenvector at
Figure SMS_92
By adjacent points
Figure SMS_95
The relative potential vector
Figure SMS_96
The weighted summation yields:
Figure SMS_98
;

(4)解码为占据概率:线性层

Figure SMS_100
将特征向量
Figure SMS_101
解码为占据得分
Figure SMS_102
,这是一个维度为2的全连接层输出,然后通过softmax转换为占用概率
Figure SMS_103
;(4) Decoding into occupancy probability: linear layer
Figure SMS_100
The feature vector
Figure SMS_101
Decoded as occupancy score
Figure SMS_102
, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax
Figure SMS_103
;

(5)体素表面重构:得到各个点的占用概率

Figure SMS_104
后,将占用概率
Figure SMS_105
为0.5
Figure SMS_106
tau的点云当作一个体素格子进行重构。(5) Voxel surface reconstruction: obtain the occupancy probability of each point
Figure SMS_104
After that, the occupation probability
Figure SMS_105
0.5
Figure SMS_106
The point cloud of tau is reconstructed as a voxel grid.

与现有技术相比,本发明的优点和有益效果:Compared with the prior art, the advantages and beneficial effects of the present invention are:

(1)通过局部划分的思路,在卷积神经网络模型中引入了对称函数和空间转移网络,使得深度学习模型能够适应点云数据的无序性和仿射变化无关性,使其适用于点云数据的去噪,提升了点云数据的质量,以实现更好的三维模型构建效果。(1) Through the idea of local division, symmetric functions and spatial transfer networks are introduced into the convolutional neural network model, so that the deep learning model can adapt to the disorder and affine change independence of point cloud data, making it suitable for point cloud data denoising and improving the quality of point cloud data to achieve better 3D model construction effect.

(2)通过多层感知机MLP和点的潜在向量相对编码方法对点云数据进行特征提取,使得点云数据的稀疏性对特征提取的影响减少,能更精准地提取点云的局部形状信息。(2) Feature extraction of point cloud data is performed through a multi-layer perceptron (MLP) and a point latent vector relative encoding method, which reduces the impact of the sparsity of point cloud data on feature extraction and enables more accurate extraction of the local shape information of the point cloud.

(3)引入了占据概率概念以及基于attention机制的特征加权方法进行点云表面形状确定,解决传统三维模型构建方法的效率低、可扩展性不强和生成精度不高等问题。(3) The concept of occupancy probability and a feature weighting method based on the attention mechanism are introduced to determine the surface shape of the point cloud, solving the problems of low efficiency, poor scalability and low generation accuracy of traditional 3D model construction methods.

本发明创建了一套利用深度学习技术把点云数据建模为三维模型的自动化系统,将实物类非物质文化遗产转化为可以永久存储在计算机中的数字模型,以实现非物质文化遗产的保护和传承。The present invention has created an automated system that uses deep learning technology to model point cloud data into three-dimensional models, converting physical intangible cultural heritage into digital models that can be permanently stored in computers, so as to achieve the protection and inheritance of intangible cultural heritage.

附图说明BRIEF DESCRIPTION OF THE DRAWINGS

图1为本发明的流程图。FIG. 1 is a flow chart of the present invention.

图2为本发明中去噪模块的架构图。FIG. 2 is a schematic diagram of the denoising module of the present invention.

图3为本发明中表面重构模块的示意图。FIG. 3 is a schematic diagram of a surface reconstruction module in the present invention.

具体实施方式DETAILED DESCRIPTION

下面结合附图和实施例对本发明的技术方案作进一步说明。The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

如图1所示,本发明提供的基于深度学习的点云数据三维重建方法,包括如下步骤:As shown in FIG1 , the method for three-dimensional reconstruction of point cloud data based on deep learning provided by the present invention comprises the following steps:

步骤一,点云数据获取Step 1: Point cloud data acquisition

获取点云,实际上就是获取物体表面采样点的三维坐标。获取点云数据最常用的设备,根据测距原理的不同,可将三维激光扫描仪分为:Acquiring point cloud actually means obtaining the three-dimensional coordinates of sampling points on the surface of an object. The most commonly used equipment for acquiring point cloud data can be divided into three types according to different distance measurement principles:

(1)激光脉冲扫描仪,基于脉冲测距法。激光发生器发出光脉冲,打在被测物体表面并反射回来,由扫描仪上的接收器所接收。光速是已知的,由传播时间可以计算得出扫描仪和采样点之间的距离,再结合扫描仪的位置以及扫描的角度,就可以准确计算出每一个采样点的三维坐标。这种基于时间测量的脉冲测距法,需要精度很高的时钟系统。它的特点是测距范围广、使用场景灵活但精度相对较低,多用于远距离的扫描测量中。(1) Laser pulse scanner, based on pulse ranging method. The laser generator emits a light pulse, which hits the surface of the object to be measured and reflects back, and is received by the receiver on the scanner. The speed of light is known, and the distance between the scanner and the sampling point can be calculated from the propagation time. Combined with the position of the scanner and the scanning angle, the three-dimensional coordinates of each sampling point can be accurately calculated. This pulse ranging method based on time measurement requires a very accurate clock system. It is characterized by a wide ranging range and flexible usage scenarios, but relatively low accuracy, and is mostly used in long-distance scanning measurements.

(2)相位测距扫描仪,基于相位测距法。相位测距法本质上也是一种脉冲测距法,只不过它对于距离的计算是基于光波的相位差而不是基于时间的。由于光也是一种波,发射光和接收光之间存在相位差,通过这个相位差可以间接地计算光的传播时间,从而计算出距离。它有更高的扫描精度,一般用于中等距离或者近距离的扫描测量中,精度可达毫米级。(2) Phase ranging scanner, based on phase ranging method. Phase ranging method is essentially a pulse ranging method, except that its distance calculation is based on the phase difference of light waves rather than time. Since light is also a wave, there is a phase difference between the emitted light and the received light. Through this phase difference, the propagation time of light can be indirectly calculated, and thus the distance can be calculated. It has higher scanning accuracy and is generally used for medium-distance or short-distance scanning measurements, with an accuracy of up to millimeter level.

在本发明的应用场景中,扫描对象为非遗文物实体或举办非遗文化活动的场所,采用的是地面固定激光扫描系统和相位测距扫描仪结合使用的扫描方式。In the application scenario of the present invention, the scanning object is an intangible cultural heritage entity or a place for holding intangible cultural heritage activities, and a scanning method that combines a ground-fixed laser scanning system and a phase ranging scanner is adopted.

在实际的扫描场景中,由于待扫描物体体积较大(如建筑物)、存在遮挡物或是扫描仪本身的扫描角度有限,单次扫描并不能获取完整的物体表面的点云数据,需要使用多个扫描站在多个不同的角度进行多次扫描,于是就得到了对于同一个待扫描物体的多幅点云数据。In actual scanning scenarios, due to the large size of the object to be scanned (such as a building), the presence of obstructions, or the limited scanning angle of the scanner itself, a single scan cannot obtain the point cloud data of the complete object surface. Multiple scanning stations are required to perform multiple scans at multiple different angles, thus obtaining multiple point cloud data for the same object to be scanned.

步骤二,去噪模块Step 2: Denoising module

在点云采集的原始过程中,由于受到采集环境(光线、空气中的水汽、设备抖动、遮挡物)、扫描设备(设备故障、扫描精度低)和被扫描物体(物体表面有细小的凹凸)的影响,扫描得到的点云中通常存在一些噪声点,这些噪声点不仅增加了点云的数据量,还影响了后期点云处理步骤的精确度,从而影响到最终的建模效果。因此处理工作的第一步就是要去除这些噪声点。In the original process of point cloud acquisition, due to the influence of the acquisition environment (light, water vapor in the air, equipment jitter, obstructions), scanning equipment (equipment failure, low scanning accuracy) and scanned objects (small bumps on the surface of the object), there are usually some noise points in the scanned point cloud. These noise points not only increase the amount of point cloud data, but also affect the accuracy of the subsequent point cloud processing steps, thereby affecting the final modeling effect. Therefore, the first step in the processing work is to remove these noise points.

本模块的初始输入为一帧点云数据,点云数据在被输入到网络之前,首先需要进行局部划分,将整个点云分成多个局部patch,每个局部patch以一个中心点为中心,包含距离中心r范围内的所有点。这样可以将点云数据划分为多个较小的块,方便网络学习。The initial input of this module is a frame of point cloud data. Before the point cloud data is input into the network, it needs to be locally divided first, dividing the entire point cloud into multiple local patches. Each local patch is centered on a central point and contains all points within a range of r from the center. In this way, the point cloud data can be divided into multiple smaller blocks, which is convenient for network learning.

接下来,使用四元空间转移网络,学习到了四维向量,通过这个向量,将原始点云迁移到一个新的位姿状态,以保证模型对特定空间的仿射变化无关性。然后,使用多层感知机网络提取逐个点的特征,并对每个点的特征向量做类似的逆变换,到达原本的位姿。为了保证模型能够适应点云数据的无序性,使用对称函数对各点特征进行特征融合,将各点特征合并成为一个局部patch的特征向量。Next, the quaternion space transfer network is used to learn a four-dimensional vector, through which the original point cloud is transferred to a new posture state to ensure that the model is independent of affine changes in a specific space. Then, a multi-layer perceptron network is used to extract the features of each point, and a similar inverse transformation is performed on the feature vector of each point to reach the original posture. In order to ensure that the model can adapt to the disorder of the point cloud data, a symmetric function is used to fuse the features of each point and merge the features of each point into a feature vector of a local patch.

得到了融合的结果后,使用若干全连接层处理融合的局部特征向量,获得网络输出的最终结果。需要注意的是,由于点云数据中常常包含离群点和噪声点,可以通过改变最后一个回归层的通道数量,以适应所需输出的大小,同时增强模型的鲁棒性。After obtaining the fusion result, several fully connected layers are used to process the fused local feature vectors to obtain the final result of the network output. It should be noted that since point cloud data often contains outliers and noise points, the number of channels in the last regression layer can be changed to adapt to the required output size and enhance the robustness of the model.

上述过程用公式表示为:

Figure SMS_107
Figure SMS_108
是一帧点云数据中的每一个点。
Figure SMS_109
是希望通过训练得到的目标函数,
Figure SMS_110
是对称函数,
Figure SMS_111
是特征提取函数。基本思路就是对各个元素(即点云中的各个点)使用
Figure SMS_112
分别处理,再送入对称函数
Figure SMS_113
中处理,以实现排列不变性。The above process is expressed as follows:
Figure SMS_107
,
Figure SMS_108
It is each point in a frame of point cloud data.
Figure SMS_109
is the objective function we hope to obtain through training,
Figure SMS_110
is a symmetric function,
Figure SMS_111
is a feature extraction function. The basic idea is to use
Figure SMS_112
Process them separately and then send them to the symmetric function
Figure SMS_113
to achieve permutation invariance.

本发明所要去除的点包括两类,一种称为离群点,它是远离点云主体部分的点,可以将它直接去除,另一种称为噪声点,实际上是由正确点发生偏移而形成的,因此去除它并不是将其删除,而是在对它上施加一个逆偏移向量,将其恢复到正确的位置上。整个去噪过程:

Figure SMS_114
The points to be removed by the present invention include two types: one is called outlier points, which are points far away from the main part of the point cloud and can be directly removed; the other is called noise points, which are actually formed by the deviation of the correct points. Therefore, removing them does not mean deleting them, but applying an inverse deviation vector to them to restore them to the correct position. The whole denoising process:
Figure SMS_114

其中

Figure SMS_115
表示通过去噪模块后预测的点云结果,
Figure SMS_116
表示去噪模块预测出的离群点的集合,
Figure SMS_117
表示原始的噪声点云,
Figure SMS_118
表示原始的噪声点云中的一个点,
Figure SMS_119
表示学习到的每个点的偏移向量。具体过程如下:in
Figure SMS_115
Represents the predicted point cloud result after passing through the denoising module,
Figure SMS_116
represents the set of outliers predicted by the denoising module,
Figure SMS_117
represents the original noisy point cloud,
Figure SMS_118
represents a point in the original noise point cloud,
Figure SMS_119
Represents the offset vector of each point learned. The specific process is as follows:

(1)局部划分。为了能够把握点云数据的局部特征,需要将一整个点云分为多个patch,给定一个点云

Figure SMS_120
,一个局部patch以点
Figure SMS_121
为中心,包含距离中心r范围内的所有点,记为
Figure SMS_122
,假设其中有
Figure SMS_123
个点。(1) Local division. In order to grasp the local features of point cloud data, it is necessary to divide the entire point cloud into multiple patches.
Figure SMS_120
, a local patch is composed of points
Figure SMS_121
As the center, it includes all points within the range of r from the center, recorded as
Figure SMS_122
, assuming that there is
Figure SMS_123
points.

(2)逐点特征提取。初始输入为一帧的全部点云数据的集合,每一个点云表示为一个

Figure SMS_124
维向量,其中N代表点的数量,3对应XYZ坐标。如图2所示,首先将数据按照上一步分好的patch,共有
Figure SMS_126
个,每次输入一个patch,通过一个预训练好的四元空间转移网络(STN1),输出一个位姿变换的四维向量(旋转)的参数:
Figure SMS_129
,通过这个操作,保证了模型对特定空间的仿射变化无关性,并且将原始点云迁移到一个新的有利于网络学习的位姿状态:
Figure SMS_130
,而后通过一个全连接层(FNN1)更改维度便于特征提取,然后使用多层感知机网络提取逐个点的特征并处理:
Figure SMS_132
,最后也要对逐点的特征向量做类似的逆变换,到达原本的位姿,因此还需要引入一个四元空间转移网络(STN2)和一个全连接层(FNN2):
Figure SMS_134
,注意这里的
Figure SMS_136
Figure SMS_125
都共有
Figure SMS_127
个,因为他们是
Figure SMS_128
Figure SMS_131
个点的特征向量,但是
Figure SMS_133
Figure SMS_135
的维度不同。(2) Point-by-point feature extraction. The initial input is a set of all point cloud data of a frame, and each point cloud is represented by a
Figure SMS_124
dimensional vector, where N represents the number of points and 3 corresponds to the XYZ coordinates. As shown in Figure 2, first divide the data into patches according to the previous step.
Figure SMS_126
Each time a patch is input, it is passed through a pre-trained four-element spatial transfer network (STN 1 ) to output the parameters of a four-dimensional vector (rotation) of pose transformation:
Figure SMS_129
Through this operation, the model is guaranteed to be independent of affine changes in a specific space, and the original point cloud is migrated to a new pose state that is conducive to network learning:
Figure SMS_130
, and then a fully connected layer (FNN 1 ) is used to change the dimension for feature extraction, and then a multi-layer perceptron network is used to extract and process the features of each point:
Figure SMS_132
Finally, a similar inverse transformation is performed on the point-by-point feature vector to reach the original position, so a quaternion space transfer network (STN 2 ) and a fully connected layer (FNN 2 ) are also introduced:
Figure SMS_134
, note that
Figure SMS_136
and
Figure SMS_125
All in common
Figure SMS_127
because they are
Figure SMS_128
middle
Figure SMS_131
The eigenvectors of points, but
Figure SMS_133
and
Figure SMS_135
The dimensions are different.

(3)特征融合。网络在获得处理完的逐点特征向量后,对一个patch中的每个点的特征向量求和(即通过对称函数实现),这样就合并成为一个patch的特征向量,不论输入点的顺序如何,加法的交换律保证了合并后的特征向量都是一样的,即保证模型能够适应点云数据的无序性,同时它保留了更多的信息:

Figure SMS_137
Figure SMS_138
Figure SMS_139
的维度相同。(3) Feature fusion. After obtaining the processed point-by-point feature vectors, the network sums the feature vectors of each point in a patch (i.e., implemented through a symmetric function), thus merging them into a feature vector of a patch. Regardless of the order of the input points, the commutative law of addition ensures that the merged feature vectors are the same, which ensures that the model can adapt to the disorder of point cloud data, while retaining more information:
Figure SMS_137
,
Figure SMS_138
and
Figure SMS_139
The dimensions are the same.

(4)回归模块。得到了特征融合的结果后,对融合的特征向量使用若干全连接层(

Figure SMS_141
)处理,获得网络输出的最终结果。可以通过改变最后一个回归层(即全连接层)的通道数量,以适应所需输出的大小。对离群点判定,回归层的通道数量设定为1,表示该点为离群点的概率,过程描述函数
Figure SMS_142
Figure SMS_144
,可自行设置阈值
Figure SMS_146
,即
Figure SMS_147
时,判定
Figure SMS_148
为离群点,将其去除,所有离群点的集合记为
Figure SMS_149
;对噪声点判定,回归层的通道数量设定为3,表示应对该点施加的逆偏移向量,过程描述函数
Figure SMS_140
Figure SMS_143
,修正后的点即为
Figure SMS_145
。(4) Regression module. After obtaining the result of feature fusion, the fused feature vector is processed using several fully connected layers (
Figure SMS_141
) to obtain the final result of the network output. The number of channels in the last regression layer (i.e., the fully connected layer) can be changed to adapt to the required output size. For outlier determination, the number of channels in the regression layer is set to 1, indicating the probability that the point is an outlier. The process description function
Figure SMS_142
,
Figure SMS_144
, you can set the threshold yourself
Figure SMS_146
,Right now
Figure SMS_147
When
Figure SMS_148
is an outlier, remove it, and the set of all outliers is recorded as
Figure SMS_149
; For noise point determination, the number of channels in the regression layer is set to 3, indicating the inverse offset vector to be applied to the point, and the process description function
Figure SMS_140
,
Figure SMS_143
, the corrected point is
Figure SMS_145
.

步骤三,配准模块Step 3: Registration module

由于每次扫描得到的点云所处的坐标系并不相同,不能简单地将点云进行相加,而是要经过一定的旋转平移操作,将不同点云置于同一坐标系下,才可以进行不同点云的拼接,这个过程称之为点云配准。Since the coordinate systems of the point clouds obtained from each scan are not the same, the point clouds cannot be simply added together. Instead, they must go through certain rotation and translation operations to place different point clouds in the same coordinate system before they can be spliced. This process is called point cloud registration.

本模块的初始输入为上一模块处理好的经过去噪的多帧点云数据,为方便介绍这里假设为两帧点云数据,这里将它们分为源点云和目标点云,本模块的任务是将两帧点云数据对齐到同一坐标系下,使得它们能够在后续的处理中被有效地融合。为了实现这个目标,我们需要计算一个变换关系,将源点云对齐至目标点云所处的坐标系下。The initial input of this module is the denoised multi-frame point cloud data processed by the previous module. For the convenience of introduction, it is assumed that there are two frames of point cloud data. Here they are divided into source point cloud and target point cloud. The task of this module is to align the two frames of point cloud data to the same coordinate system so that they can be effectively fused in subsequent processing. To achieve this goal, we need to calculate a transformation relationship to align the source point cloud to the coordinate system of the target point cloud.

为了提取点云的全局特征,我们使用多层感知机MLP方法。首先,我们分别对源点云和目标点云进行特征提取,得到两个全局特征。然后,我们将这两个特征拼接起来得到两个点云的合并特征。这个拼接操作是为了让源点云和目标点云产生关联,以便在后续的处理中能够更好地进行匹配和对齐。In order to extract the global features of the point cloud, we use the multi-layer perceptron MLP method. First, we extract features from the source point cloud and the target point cloud respectively to obtain two global features. Then, we concatenate these two features to obtain the merged features of the two point clouds. This concatenation operation is to associate the source point cloud and the target point cloud so that they can be better matched and aligned in subsequent processing.

接下来,我们将合并特征作为输入,通过五个全连接层来计算点云之间的变换关系。最后的输出是一个七维的向量,其中前三位是平移向量,后四位是单位四元数。单位四元数可以转换为旋转矩阵,因此我们可以通过这个向量来表示源点云相对于目标点云的平移和旋转变换。将源点云进行平移和旋转操作,就可以将其对齐至目标点云所处的坐标系下。Next, we use the merged features as input and calculate the transformation relationship between point clouds through five fully connected layers. The final output is a seven-dimensional vector, where the first three are the translation vector and the last four are the unit quaternion. The unit quaternion can be converted to a rotation matrix, so we can use this vector to represent the translation and rotation transformation of the source point cloud relative to the target point cloud. By translating and rotating the source point cloud, it can be aligned to the coordinate system of the target point cloud.

最后,将对齐后的源点云和目标点云进行融合,得到一个完整的点云图像,在后续的处理中,我们可以使用这个点云图像来进行三维重建任务。Finally, the aligned source point cloud and target point cloud are fused to obtain a complete point cloud image. In subsequent processing, we can use this point cloud image to perform 3D reconstruction tasks.

本模块的功能是根据源点云和目标点云,计算一个变换关系,将源点云对齐 至目标点云所处的坐标系下,然后将两幅点云相加得到一个合并点云(对于多幅点云即两两相加后的结果再相加)。在具体的配准任务中,待配准的两幅点云必须包含重叠的部分,否则无法进行配准。所以在选取待配准点云时,通常会选择扫描站点比较靠近的两幅点云进行配准,这样两幅点云重叠部分多,易找到点之间的匹配关系。具体过程如下:The function of this module is to calculate a transformation relationship based on the source point cloud and the target point cloud, align the source point cloud to the coordinate system of the target point cloud, and then add the two point clouds to obtain a merged point cloud (for multiple point clouds, the results of adding two by two are added together). In the specific registration task, the two point clouds to be registered must contain overlapping parts, otherwise the registration cannot be performed. Therefore, when selecting the point clouds to be registered, two point clouds with relatively close scanning sites are usually selected for registration, so that the two point clouds have more overlapping parts and it is easy to find the matching relationship between the points. The specific process is as follows:

(1)特征提取。针对源点云和目标点云分别进行,使用多层感知机方法和最大池化层提取点云全局特征:

Figure SMS_150
Figure SMS_151
。(1) Feature extraction. This is done for the source point cloud and the target point cloud respectively, using the multi-layer perceptron method and the maximum pooling layer to extract the global features of the point cloud:
Figure SMS_150
,
Figure SMS_151
.

(2)拼接。将两个全局特征拼接得到两个点云的合并特征,作为后面五个全连接层的输入,进行拼接操作是为了让源点云和目标点云产生关联,这样回归函数才是综合考虑二者的结构信息预测出变换:

Figure SMS_152
+
Figure SMS_153
。(2) Splicing. The two global features are spliced to obtain the merged features of the two point clouds, which are used as the input of the following five fully connected layers. The purpose of the splicing operation is to associate the source point cloud with the target point cloud, so that the regression function can comprehensively consider the structural information of the two and predict the transformation:
Figure SMS_152
+
Figure SMS_153
.

(3)全连接层。将拼接得到的合并特征输入全连接层,最后的输出是一个七维的向量,该向量的前三位是平移向量,后四位是单位四元数,单位四元数可以转换为旋转矩阵。源点云经过平移和旋转与目标点云处于同一坐标系下。(3) Fully connected layer. The combined features obtained by concatenation are input into the fully connected layer. The final output is a seven-dimensional vector. The first three digits of the vector are the translation vector, and the last four digits are the unit quaternion. The unit quaternion can be converted into a rotation matrix. After translation and rotation, the source point cloud is in the same coordinate system as the target point cloud.

多幅点云配准也可以看成两两配准,先把1号和2号配准为一幅,再去和3号配准,最终多幅点云配准后的图像合成完整的点云图像。The registration of multiple point clouds can also be viewed as pairwise registration. First, register point clouds 1 and 2 into one image, and then register them with point cloud 3. Finally, the images after multiple point cloud registrations are synthesized into a complete point cloud image.

步骤四,表面重构模块Step 4: Surface reconstruction module

点云表面重构是从非结构化的点云中重构出曲面或者多边形网格,使其尽可能地拟合于原本的物体表面,经过表面重构得到的曲面模型或多边形网格模型,再经过一定的优化和渲染,就可以得到逼近于实物的三维模型。本模块的初始输入为上一模块拼接好的一帧完整点云数据,其功能是使用设计的神经网络分辨出点云的表面形状点,形状外点和形状内点,而后对表面形状点上的每一帧点云进行体素表面重构,得到点云数据的三角网格模型,形状外点和形状内点不参与重构。Point cloud surface reconstruction is to reconstruct a surface or polygonal mesh from an unstructured point cloud so that it fits the original object surface as closely as possible. The surface model or polygonal mesh model obtained by surface reconstruction, after certain optimization and rendering, can obtain a three-dimensional model close to the real object. The initial input of this module is a frame of complete point cloud data spliced by the previous module. Its function is to use the designed neural network to distinguish the surface shape points, shape external points and shape internal points of the point cloud, and then perform voxel surface reconstruction on each frame of point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data. The shape external points and shape internal points do not participate in the reconstruction.

如图3所示,本发明引入了占据概率

Figure SMS_154
的概念,给定一组经过配准的3D点云作为输入,本模块的目标是构建一个隐式函数
Figure SMS_155
,指示在任意给定查询点
Figure SMS_156
处的占用概率
Figure SMS_157
,使用包含在整个空间中的点云并带有标签的数据来使用神经网络学习这个函数,标签值为0表示这个点在物体形状的外部,即形状外点,可以把它当作背景或者噪声,标签值为1表示这个点在物体形状的内部,即形状内点,不参与物体的表面重构,这样大大提升了表面重构的效率。而后,可以将形状的表面提取为占用水平为0.5的隐式函数
Figure SMS_158
的等值面,占用水平0.5的最可能是表面,构成的等值面即都是0.5的点构成的表面。As shown in FIG3 , the present invention introduces the occupation probability
Figure SMS_154
Given a set of registered 3D point clouds as input, the goal of this module is to construct an implicit function
Figure SMS_155
, indicating that at any given query point
Figure SMS_156
Occupancy probability
Figure SMS_157
, using the point cloud contained in the entire space and labeled data to use the neural network to learn this function. The label value of 0 means that the point is outside the shape of the object, that is, the shape outside point, which can be regarded as background or noise. The label value of 1 means that the point is inside the shape of the object, that is, the shape inside point, and does not participate in the surface reconstruction of the object. This greatly improves the efficiency of surface reconstruction. Then, the surface of the shape can be extracted as an implicit function with an occupancy level of 0.5
Figure SMS_158
The isosurface of the occupancy level 0.5 is most likely the surface, and the isosurface is composed of points with a value of 0.5.

设计的神经网络学习过程如下:The designed neural network learning process is as follows:

(1)潜在向量编码:使用点卷积方法首先为每个输入点

Figure SMS_159
产生一个潜在向量
Figure SMS_160
,编码器
Figure SMS_161
可以由任何点云卷积网络实现,只需要改变最后一层的通道数来控制向量
Figure SMS_162
的维度,其中
Figure SMS_163
为经过配准后的点云中的点,n表示输入点的个数。(1) Latent vector encoding: Use the point convolution method to first encode each input point
Figure SMS_159
Generate a latent vector
Figure SMS_160
, encoder
Figure SMS_161
It can be implemented by any point cloud convolutional network, just need to change the number of channels in the last layer to control the vector
Figure SMS_162
The dimensions of
Figure SMS_163
is the point in the point cloud after registration, and n represents the number of input points.

(2)相对潜在向量编码。给定一个任意查询点

Figure SMS_165
,查询点
Figure SMS_167
来自输入点,构造一组点集邻居
Figure SMS_169
,它包括离
Figure SMS_170
最近的k个点,这里的k可根据实际情况自行设置,设点集邻居中的点为
Figure SMS_172
。而后用查询点
Figure SMS_174
相对于
Figure SMS_176
的局部坐标
Figure SMS_164
,对每个点集邻居
Figure SMS_166
中的点
Figure SMS_168
的潜在向量
Figure SMS_171
进行增强,这些增强的潜在向量经过
Figure SMS_173
处理得到相对潜在向量
Figure SMS_175
,其中
Figure SMS_177
为级联操作。其中查询点只是说明从这个输入点开始查询,最终需要对每一个输入点都完成一次查询。相对潜在向量的引入,使得查询点的特征与其邻居点集内的点产生联系,使得神经网络对于物体的局部形状特征的把握更加精确,对于复杂形状的生成,同样可以利用相对潜在向量将其结构为局部形状的组合,这样做大大提升了模型重构的可扩展性。(2) Relative latent vector encoding. Given an arbitrary query point
Figure SMS_165
, query point
Figure SMS_167
From the input point, construct a set of point set neighbors
Figure SMS_169
, which includes
Figure SMS_170
The nearest k points, where k can be set according to the actual situation, and the points in the point set neighbors are
Figure SMS_172
Then use the query point
Figure SMS_174
Relative to
Figure SMS_176
The local coordinates of
Figure SMS_164
, for each point set neighbor
Figure SMS_166
Points in
Figure SMS_168
The latent vector
Figure SMS_171
These enhanced latent vectors are enhanced by
Figure SMS_173
Processing to obtain the relative potential vector
Figure SMS_175
,in
Figure SMS_177
It is a cascade operation. The query point only indicates that the query starts from this input point, and eventually a query needs to be completed for each input point. The introduction of relative latent vectors connects the features of the query point with the points in its neighbor point set, making the neural network more accurate in grasping the local shape features of the object. For the generation of complex shapes, relative latent vectors can also be used to structure them into a combination of local shapes, which greatly improves the scalability of model reconstruction.

(3)特征加权。相对潜在向量

Figure SMS_194
的范数与输入点
Figure SMS_195
对于确定查询点
Figure SMS_196
的占用的重要性相关,由此可以用它来推断相对潜变量
Figure SMS_197
的重要性权重。由此引入了一个注意力机制,相对潜在向量
Figure SMS_198
经过由权重向量
Figure SMS_199
参数化的线性层,产生相对权重
Figure SMS_200
Figure SMS_179
表示点乘操作,这些权重通过softmax函数在
Figure SMS_181
内进行归一化,得到各自的插值权重
Figure SMS_182
,这些插值权重总和为1。实际上特征加权是通过学习
Figure SMS_185
个独立的线性层,由
Figure SMS_187
个相应的权重向量
Figure SMS_188
参数化,产生
Figure SMS_190
个相对权重
Figure SMS_193
,最后通过softmax将
Figure SMS_178
变为权重
Figure SMS_180
,并平均为
Figure SMS_183
。查询点
Figure SMS_184
处的特征向量
Figure SMS_186
由相邻点
Figure SMS_189
的相对潜在向量
Figure SMS_191
加权求和得到:
Figure SMS_192
。(3) Feature weighting. Relative latent vector
Figure SMS_194
The norm and input point
Figure SMS_195
To determine the query point
Figure SMS_196
The importance of occupancy is related to the relative latent variable
Figure SMS_197
The importance weight of . This introduces an attention mechanism, relative to the potential vector
Figure SMS_198
Through the weight vector
Figure SMS_199
Parameterized linear layer, generating relative weights
Figure SMS_200
,
Figure SMS_179
Represents the dot product operation, these weights are passed through the softmax function in
Figure SMS_181
Normalize them to get their respective interpolation weights
Figure SMS_182
, the sum of these interpolation weights is 1. In fact, feature weighting is learned by
Figure SMS_185
independent linear layers, consisting of
Figure SMS_187
The corresponding weight vector
Figure SMS_188
Parameterization, generating
Figure SMS_190
Relative weight
Figure SMS_193
, and finally through softmax
Figure SMS_178
Becomes weight
Figure SMS_180
, and the average is
Figure SMS_183
Query point
Figure SMS_184
The eigenvector at
Figure SMS_186
By adjacent points
Figure SMS_189
The relative potential vector
Figure SMS_191
The weighted summation yields:
Figure SMS_192
.

(4)解码为占据概率。线性层

Figure SMS_201
将特征向量
Figure SMS_202
解码为占据得分
Figure SMS_203
,这是一个维度为2的全连接层输出,然后通过softmax转换为占用概率
Figure SMS_204
,标签值为0表示这个点在物体形状的外部,可以把它当作背景或者噪声,标签值为1表示这个点在物体形状的内部,不参与物体的表面重构。(4) Decoded into occupancy probability. Linear layer
Figure SMS_201
The feature vector
Figure SMS_202
Decoded as occupancy score
Figure SMS_203
, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax
Figure SMS_204
, a label value of 0 means that the point is outside the shape of the object and can be regarded as background or noise. A label value of 1 means that the point is inside the shape of the object and does not participate in the surface reconstruction of the object.

(5)体素表面重构。得到各个点的占用概率

Figure SMS_205
后,把点云当作一个体素格子进行重构,首先初始化一个空三角形网格,然后执行以下操作:(5) Voxel surface reconstruction. Obtain the occupancy probability of each point
Figure SMS_205
Finally, reconstruct the point cloud as a voxel grid. First, initialize an empty triangle mesh and then do the following:

1、获取当前体素在(x, y, z)处的立方体cube,包含8个顶点。1. Get the cube of the current voxel at (x, y, z), which contains 8 vertices.

2、获取当前立方体cube的边界集合,该边界为所有在占据概率在[0.5-tau,0.5+tau]区间内的点的集合。2. Get the boundary set of the current cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau].

3、对于集合中的每个点,执行以下操作:3. For each point in the set, do the following:

i.选取顶点v1和v2,其中p1和p2分别为顶点v1和v2的占据概率,p1<0.5, p2>0.5。i. Select vertices v1 and v2, where p1 and p2 are the occupancy probabilities of vertices v1 and v2 respectively, p1<0.5, p2>0.5.

ii. 通过插值函数插值(v1, v2, p1, p2)获得一个新的占据概率等于0.5的插值点vertex。ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5.

iii. 将插值点vertex添加到三角形网格M中。iii. Add the interpolation point vertex to the triangle mesh M.

4、构建当前立方体cube的三角形面列表faces该列表包含当前cube中占据概率为0.5的点和刚生成的vertex点。4. Construct the triangle face list faces of the current cube. This list contains the points with an occupation probability of 0.5 in the current cube and the vertex points that have just been generated.

5、对于每个三角形面,将其添加到三角形网格M中,当所有三角形面都已经添加到三角形网格中,算法结束。5. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends.

伪代码如下:The pseudo code is as follows:

输入:3D体素格子V(尺寸为nx, ny, nz),占用概率阈值tauInput: 3D voxel grid V (size nx, ny, nz), occupancy probability threshold tau

输出:三角形网格MOutput: triangular mesh M

M = 空网格M = Empty grid

for x in range(nx-1):for x in range(nx-1):

for y in range(ny-1):for y in range(ny-1):

for z in range(nz-1):for z in range(nz-1):

cube = 获取立方体(x, y, z, V)cube = Get Cube(x, y, z, V)

set = 获取立方体集合(cube, tau)set = Get cube set (cube, tau)

for point in set:for point in set:

if p1<0.5 and p2>0.5if p1<0.5 and p2>0.5

vertex = 插值(v1, v2, p1, p2)vertex = interpolate(v1, v2, p1, p2)

M.添加顶点(vertex)M. Adding vertices

faces = 构建三角形面列表(cube, set)faces = build triangle face list (cube, set)

for face in faces:for face in faces:

M.添加三角形(face)M. Add triangle (face)

当所有三角形面都已经添加到三角形网格中,算法结束When all triangle faces have been added to the triangle mesh, the algorithm ends.

返回M其中,获取立方体函数“获取立方体(x, y, z, V) ”根据体素坐标返回包含八个顶点的立方体,获取立方体边界函数“获取立方体集合(cube, tau) ”返回一个包含所有在占据概率[0.5-tau,0.5+tau]区间内的点的集合,“插值 (v1, v2, p1, p2) ”返回一个占据概率等于0.5的插值点,“构建三角形函数构建三角形(cube, set) ”返回一个包含所有在占据概率等于0.5点的三角形面的列表。Returns M, where the cube get function "Get Cube (x, y, z, V)" returns a cube with eight vertices according to the voxel coordinates, the cube boundary get function "Get Cube Set (cube, tau)" returns a set of all points in the occupation probability interval [0.5-tau,0.5+tau], "Interpolation (v1, v2, p1, p2)" returns an interpolation point with an occupation probability equal to 0.5, and the "Build Triangle Function Build Triangle (cube, set)" returns a list of all triangular faces with an occupation probability equal to 0.5.

本发明还提供基于深度学习的点云数据三维重建系统,包括如下模块:The present invention also provides a point cloud data 3D reconstruction system based on deep learning, comprising the following modules:

点云获取模块,针对被扫描物体,利用三维激光扫描仪从多个不同的角度进行多次扫描,得到对于同一个待扫描物体的多幅点云数据;The point cloud acquisition module uses a 3D laser scanner to perform multiple scans from multiple different angles on the object to be scanned, thereby obtaining multiple point cloud data for the same object to be scanned;

预处理模块,用于对采集的多幅点云数据进行预处理,包括去除离散点和将噪声点恢复到正确的位置上;The preprocessing module is used to preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position;

配准模块,对于任意两幅点云数据,根据源点云和目标点云计算一个变换关系,将源点云对齐至目标点云所处的坐标系下,从而实现点云的配准,最终得到一个完整的点云图像;The registration module calculates a transformation relationship between the source point cloud and the target point cloud for any two point cloud data, aligns the source point cloud to the coordinate system of the target point cloud, thereby realizing point cloud registration and finally obtaining a complete point cloud image;

重构模块,用于对非结构化的点云图像进行表面重构,得到点云数据的三角网格模型,再经过优化和渲染,得到逼近于实物的三维模型。The reconstruction module is used to reconstruct the surface of unstructured point cloud images to obtain a triangular mesh model of the point cloud data, and then obtain a three-dimensional model that is close to the real object through optimization and rendering.

各模块的具体实现方式与各步骤相应,本发明不予撰述。The specific implementation method of each module corresponds to each step and is not described in detail in the present invention.

本文中所描述的具体实施例仅仅是对本发明精神作举例说明。本发明所属技术领域的技术人员可以对所描述的具体实施例做各种各样的修改或补充或采用类似的方式替代,但并不会偏离本发明的精神或者超越所附权利要求书所定义的范围。The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims (10)

1.基于深度学习的点云数据三维重建方法,其特征在于,包括如下步骤:1. A method for three-dimensional reconstruction of point cloud data based on deep learning, characterized in that it comprises the following steps: 步骤1,针对被扫描物体,利用三维激光扫描仪从多个不同的角度进行多次扫描,得到对于同一个待扫描物体的多幅点云数据;Step 1: Scan the object to be scanned multiple times from multiple different angles using a three-dimensional laser scanner to obtain multiple point cloud data for the same object to be scanned; 步骤2,对采集的多幅点云数据进行预处理,包括去除离散点和将噪声点恢复到正确的位置上;Step 2: preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position; 步骤3,对于任意两幅点云数据,根据源点云和目标点云计算一个变换关系,将源点云对齐至目标点云所处的坐标系下,从而实现点云的配准,最终得到一个完整的点云图像;Step 3: For any two point cloud data, a transformation relationship is calculated based on the source point cloud and the target point cloud, and the source point cloud is aligned to the coordinate system of the target point cloud, so as to realize the registration of the point cloud and finally obtain a complete point cloud image; 步骤4,对非结构化的点云图像进行表面重构,得到点云数据的三角网格模型,再经过优化和渲染,得到逼近于实物的三维模型。Step 4: reconstruct the surface of the unstructured point cloud image to obtain a triangular mesh model of the point cloud data, and then optimize and render it to obtain a three-dimensional model that is close to the real object. 2.如权利要求1所述的基于深度学习的点云数据三维重建方法,其特征在于:步骤2中,首先进行局部划分,将每幅点云数据分成多个局部patch,每个局部patch以一个中心点为中心,包含距离中心r范围内的所有点;2. The method for 3D reconstruction of point cloud data based on deep learning as claimed in claim 1, characterized in that: in step 2, firstly, local division is performed to divide each point cloud data into a plurality of local patches, each local patch is centered on a central point and includes all points within a range r from the center; 其次,将每个局部patch输入到四元空间转移网络,学习到四维向量,通过这个四维向量,将原始点云迁移到一个新的位姿状态,然后通过一个全连接层改变位姿状态的维度,使用多层感知机网络提取逐个点云的特征,并对每个点云的特征向量做逆变换,还原到原本的位姿;在获得处理完的逐点特征向量后,对一个patch中的每个点的特征向量进行融合得到一个局部patch的特征向量;Secondly, each local patch is input into the quaternion space transfer network to learn a four-dimensional vector. Through this four-dimensional vector, the original point cloud is transferred to a new posture state, and then the dimension of the posture state is changed through a fully connected layer. The multi-layer perceptron network is used to extract the features of each point cloud, and the feature vector of each point cloud is inversely transformed to restore it to the original posture. After obtaining the processed point-by-point feature vector, the feature vector of each point in a patch is fused to obtain the feature vector of a local patch. 最后对融合的特征向量使用若干全连接层进行回归处理,获得最终结果。Finally, the fused feature vector is regressed using several fully connected layers to obtain the final result. 3.如权利要求2所述的基于深度学习的点云数据三维重建方法,其特征在于:通过改变最后一个全连接层的通道数量实现离散点的去除和噪声点的恢复,对离群点的判定:最后一个全连接层的通道数量设定为1,过程描述函数
Figure QLYQS_2
Figure QLYQS_4
Figure QLYQS_5
表示该点为离群点的概率,自行设置阈值
Figure QLYQS_7
,即
Figure QLYQS_9
时,判定
Figure QLYQS_12
为离群点,将其去除,所有离群点的集合记为
Figure QLYQS_13
;对噪声点判定:最后一个全连接层的通道数量设定为3,过程描述函数
Figure QLYQS_1
Figure QLYQS_3
Figure QLYQS_6
表示学习到的每个点的偏移向量,修正后的点即为
Figure QLYQS_8
Figure QLYQS_10
表示原始的噪声点云,
Figure QLYQS_11
表示原始的噪声点云中的一个点。
3. The method for 3D reconstruction of point cloud data based on deep learning as claimed in claim 2, characterized in that: the number of channels in the last fully connected layer is changed to achieve the removal of discrete points and the recovery of noise points, and the determination of outliers: the number of channels in the last fully connected layer is set to 1, and the process description function
Figure QLYQS_2
,
Figure QLYQS_4
,
Figure QLYQS_5
Indicates the probability that the point is an outlier, and sets the threshold by yourself
Figure QLYQS_7
,Right now
Figure QLYQS_9
When
Figure QLYQS_12
is an outlier, remove it, and the set of all outliers is recorded as
Figure QLYQS_13
; For noise point determination: the number of channels in the last fully connected layer is set to 3, and the process description function
Figure QLYQS_1
,
Figure QLYQS_3
,
Figure QLYQS_6
Represents the offset vector of each learned point, and the corrected point is
Figure QLYQS_8
,
Figure QLYQS_10
represents the original noisy point cloud,
Figure QLYQS_11
Represents a point in the original noisy point cloud.
4.如权利要求1所述的基于深度学习的点云数据三维重建方法,其特征在于:步骤3中任意两幅点云数据进行配准的具体实现方式如下;4. The method for 3D reconstruction of point cloud data based on deep learning according to claim 1, characterized in that: the specific implementation method of registering any two pieces of point cloud data in step 3 is as follows; 首先,分别对源点云和目标点云进行特征提取,得到两个全局特征,然后,将这两个全局特征进行拼接得到两个点云的合并特征,所述拼接操作是为了让源点云和目标点云产生关联,以便在后续的处理中能够更好地进行匹配和对齐;First, feature extraction is performed on the source point cloud and the target point cloud respectively to obtain two global features. Then, the two global features are spliced to obtain the merged features of the two point clouds. The splicing operation is to associate the source point cloud and the target point cloud so that they can be better matched and aligned in subsequent processing. 接下来,将合并特征作为输入,通过五个全连接层来计算点云之间的变换关系,最后的输出是一个七维的向量,其中前三位是平移向量,后四位是单位四元数,将单位四元数转换为旋转矩阵,根据输出的平移向量和旋转矩阵将源点云进行平移和旋转操作,即将源点云对齐至目标点云所处的坐标系下;Next, the merged features are used as input, and the transformation relationship between point clouds is calculated through five fully connected layers. The final output is a seven-dimensional vector, of which the first three are translation vectors and the last four are unit quaternions. The unit quaternion is converted into a rotation matrix. The source point cloud is translated and rotated according to the output translation vector and rotation matrix, that is, the source point cloud is aligned to the coordinate system of the target point cloud. 最后,将对齐后的源点云和目标点云进行融合。Finally, the aligned source point cloud and target point cloud are fused. 5.如权利要求1所述的基于深度学习的点云数据三维重建方法,其特征在于:步骤4中,通过神经网络分辨出点云的表面形状点,形状外点和形状内点,而后对表面形状点上的每一帧点云进行体素表面重构,得到点云数据的三角网格模型,形状外点和形状内点不参与重构;表面重构的具体实现方式如下;5. The method for 3D reconstruction of point cloud data based on deep learning as claimed in claim 1, characterized in that: in step 4, the surface shape points, shape external points and shape internal points of the point cloud are distinguished by a neural network, and then voxel surface reconstruction is performed on each frame of point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data, and the shape external points and shape internal points do not participate in the reconstruction; the specific implementation method of the surface reconstruction is as follows; 首先初始化一个空三角形网格,然后执行以下操作:First initialize an empty triangle mesh, then do the following: 41,获取当前体素在(x, y, z)处的立方体cube,包含8个顶点;41. Get the cube of the current voxel at (x, y, z), which contains 8 vertices; 42,获取当前立方体cube的边界集合,该边界集合为所有占据概率在[0.5-tau,0.5+tau]区间内的点的集合,tau为设置的阈值;42. Get the boundary set of the current cube cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau], where tau is the set threshold; 43,对于集合中的每个点,执行以下操作:43. For each point in the set, do the following: i.选取顶点v1和v2,其中其中p1和p2分别为顶点v1和v2的占据概率,p1<0.5, p2>0.5;i. Select vertices v1 and v2, where p1 and p2 are the occupancy probabilities of vertices v1 and v2, respectively, p1<0.5, p2>0.5; ii.通过插值函数插值(v1, v2, p1, p2)获得一个新的占据概率等于0.5的插值点vertex;ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5; iii.将插值点vertex添加到三角形网格M中;iii. Add the interpolation point vertex to the triangle mesh M; 44,构建当前立方体cube的三角形面列表,该列表包含当前cube中占据概率为0.5的点和刚生成的vertex点;44. Build a triangular face list of the current cube, which contains points with an occupation probability of 0.5 in the current cube and the newly generated vertex points; 45,对于每个三角形面,将其添加到三角形网格M中,当所有三角形面都已经添加到三角形网格中,算法结束。45. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends. 6.如权利要求5所述的基于深度学习的点云数据三维重建方法,其特征在于:神经网络学习过程如下;6. The method for three-dimensional reconstruction of point cloud data based on deep learning as claimed in claim 5, characterized in that: the neural network learning process is as follows; (1)潜在向量编码:使用点卷积方法首先为每个输入点
Figure QLYQS_14
产生一个潜在向量
Figure QLYQS_15
,编码器
Figure QLYQS_16
由点云卷积网络实现,只需要改变最后一层的通道数来控制向量
Figure QLYQS_17
的维度,其中
Figure QLYQS_18
为经过配准后的点云中的点,n表示输入点的个数;
(1) Latent vector encoding: Use the point convolution method to first encode each input point
Figure QLYQS_14
Generate a latent vector
Figure QLYQS_15
, encoder
Figure QLYQS_16
Implemented by the point cloud convolutional network, only the number of channels in the last layer needs to be changed to control the vector
Figure QLYQS_17
The dimensions of
Figure QLYQS_18
is the point in the registered point cloud, and n represents the number of input points;
(2)相对潜在向量编码:给定一个任意查询点
Figure QLYQS_19
,查询点
Figure QLYQS_21
来自输入点,构造一组点集邻居
Figure QLYQS_24
,它包括离
Figure QLYQS_26
最近的k个点,设点集邻居中的点为
Figure QLYQS_28
,而后用查询点
Figure QLYQS_29
相对于
Figure QLYQS_31
的局部坐标
Figure QLYQS_20
,对每个点集邻居
Figure QLYQS_22
中的点
Figure QLYQS_23
的潜在向量
Figure QLYQS_25
进行增强,这些增强的潜在向量经过
Figure QLYQS_27
处理得到相对潜在向量
Figure QLYQS_30
,其中
Figure QLYQS_32
为级联操作;通过以上操作对每一个输入点都完成一次查询;
(2) Relative latent vector encoding: Given an arbitrary query point
Figure QLYQS_19
, query point
Figure QLYQS_21
From the input point, construct a set of point set neighbors
Figure QLYQS_24
, which includes
Figure QLYQS_26
The nearest k points, let the points in the point set neighbors be
Figure QLYQS_28
, and then use the query point
Figure QLYQS_29
Relative to
Figure QLYQS_31
The local coordinates of
Figure QLYQS_20
, for each point set neighbor
Figure QLYQS_22
Points in
Figure QLYQS_23
The latent vector
Figure QLYQS_25
These enhanced latent vectors are enhanced by
Figure QLYQS_27
Processing to obtain the relative potential vector
Figure QLYQS_30
,in
Figure QLYQS_32
It is a cascade operation; through the above operation, a query is completed for each input point;
(3)特征加权:通过学习
Figure QLYQS_34
个独立的线性层,由
Figure QLYQS_36
个相应的权重向量
Figure QLYQS_38
参数化,产生
Figure QLYQS_39
个相对权重
Figure QLYQS_41
Figure QLYQS_43
表示点乘操作,权重总和为1;最后通过softmax将相对权重
Figure QLYQS_45
变为权重
Figure QLYQS_33
,并平均为
Figure QLYQS_35
,查询点
Figure QLYQS_37
处的特征向量
Figure QLYQS_40
由相邻点
Figure QLYQS_42
的相对潜在向量
Figure QLYQS_44
加权求和得到:
Figure QLYQS_46
(3) Feature weighting: through learning
Figure QLYQS_34
independent linear layers, consisting of
Figure QLYQS_36
The corresponding weight vector
Figure QLYQS_38
Parameterization, generating
Figure QLYQS_39
Relative weight
Figure QLYQS_41
,
Figure QLYQS_43
represents the dot multiplication operation, the sum of the weights is 1; finally, the relative weights are converted to
Figure QLYQS_45
Becomes weight
Figure QLYQS_33
, and the average is
Figure QLYQS_35
, query point
Figure QLYQS_37
The eigenvector at
Figure QLYQS_40
By adjacent points
Figure QLYQS_42
The relative potential vector
Figure QLYQS_44
The weighted summation yields:
Figure QLYQS_46
;
(4)解码为占据概率:线性层
Figure QLYQS_47
将特征向量
Figure QLYQS_48
解码为占据得分
Figure QLYQS_49
,这是一个维度为2的全连接层输出,然后通过softmax转换为占用概率
Figure QLYQS_50
(4) Decoding into occupancy probability: linear layer
Figure QLYQS_47
The feature vector
Figure QLYQS_48
Decoded as occupancy score
Figure QLYQS_49
, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax
Figure QLYQS_50
;
(5)体素表面重构:得到各个点的占用概率
Figure QLYQS_51
后,将占用概率
Figure QLYQS_52
为0.5
Figure QLYQS_53
tau的点云当作一个体素格子进行重构。
(5) Voxel surface reconstruction: obtain the occupancy probability of each point
Figure QLYQS_51
After that, the occupation probability
Figure QLYQS_52
0.5
Figure QLYQS_53
The point cloud of tau is reconstructed as a voxel grid.
7.如权利要求1所述的基于深度学习的点云数据三维重建方法,其特征在于:步骤1中采用的三维激光扫描仪包括激光脉冲扫描仪和相位测距扫描仪。7. The method for three-dimensional reconstruction of point cloud data based on deep learning as described in claim 1 is characterized in that the three-dimensional laser scanner used in step 1 includes a laser pulse scanner and a phase ranging scanner. 8.基于深度学习的点云数据三维重建系统,其特征在于,包括如下模块:8. A point cloud data 3D reconstruction system based on deep learning, characterized by comprising the following modules: 点云获取模块,针对被扫描物体,利用三维激光扫描仪从多个不同的角度进行多次扫描,得到对于同一个待扫描物体的多幅点云数据;The point cloud acquisition module uses a 3D laser scanner to perform multiple scans from multiple different angles on the object to be scanned, thereby obtaining multiple point cloud data for the same object to be scanned; 预处理模块,用于对采集的多幅点云数据进行预处理,包括去除离散点和将噪声点恢复到正确的位置上;The preprocessing module is used to preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position; 配准模块,对于任意两幅点云数据,根据源点云和目标点云计算一个变换关系,将源点云对齐至目标点云所处的坐标系下,从而实现点云的配准,最终得到一个完整的点云图像;The registration module calculates a transformation relationship between the source point cloud and the target point cloud for any two point cloud data, aligns the source point cloud to the coordinate system of the target point cloud, thereby realizing point cloud registration and finally obtaining a complete point cloud image; 重构模块,用于对非结构化的点云图像进行表面重构,得到点云数据的三角网格模型,再经过优化和渲染,得到逼近于实物的三维模型。The reconstruction module is used to reconstruct the surface of unstructured point cloud images to obtain a triangular mesh model of the point cloud data, and then obtain a three-dimensional model that is close to the real object through optimization and rendering. 9.如权利要求7所述基于深度学习的点云数据三维重建系统,其特征在于:预处理模块中,首先进行局部划分,将每幅点云数据分成多个局部patch,每个局部patch以一个中心点为中心,包含距离中心r范围内的所有点;9. The point cloud data 3D reconstruction system based on deep learning as claimed in claim 7, characterized in that: in the preprocessing module, local division is first performed to divide each point cloud data into multiple local patches, each local patch is centered on a central point and includes all points within a range r from the center; 其次,将每个局部patch输入到四元空间转移网络,学习到四维向量,通过这个四维向量,将原始点云迁移到一个新的位姿状态,然后通过一个全连接层改变位姿状态的维度,使用多层感知机网络提取逐个点云的特征,并对每个点云的特征向量做逆变换,还原到原本的位姿;在获得处理完的逐点特征向量后,对一个patch中的每个点的特征向量进行融合得到一个局部patch的特征向量;Secondly, each local patch is input into the quaternion space transfer network to learn a four-dimensional vector. Through this four-dimensional vector, the original point cloud is transferred to a new posture state, and then the dimension of the posture state is changed through a fully connected layer. The multi-layer perceptron network is used to extract the features of each point cloud, and the feature vector of each point cloud is inversely transformed to restore it to the original posture. After obtaining the processed point-by-point feature vector, the feature vector of each point in a patch is fused to obtain the feature vector of a local patch. 最后对融合的特征向量使用若干全连接层进行回归处理,获得最终结果,具体实现方式为:Finally, the fused feature vector is regressed using several fully connected layers to obtain the final result. The specific implementation method is as follows: 通过改变最后一个全连接层的通道数量实现离散点的去除和噪声点的恢复,对离群点的判定:最后一个全连接层的通道数量设定为1,过程描述函数
Figure QLYQS_55
Figure QLYQS_57
Figure QLYQS_58
表示该点为离群点的概率,自行设置阈值
Figure QLYQS_60
,即
Figure QLYQS_62
时,判定
Figure QLYQS_63
为离群点,将其去除,所有离群点的集合记为
Figure QLYQS_65
;对噪声点判定:最后一个全连接层的通道数量设定为3,过程描述函数
Figure QLYQS_54
Figure QLYQS_56
Figure QLYQS_59
表示学习到的每个点的偏移向量,修正后的点即为
Figure QLYQS_61
Figure QLYQS_64
表示原始的噪声点云,
Figure QLYQS_66
表示原始的噪声点云中的一个点。
By changing the number of channels in the last fully connected layer, the discrete points can be removed and the noise points can be restored. For the determination of outliers, the number of channels in the last fully connected layer is set to 1. The process description function
Figure QLYQS_55
,
Figure QLYQS_57
,
Figure QLYQS_58
Indicates the probability that the point is an outlier, and sets the threshold by yourself
Figure QLYQS_60
,Right now
Figure QLYQS_62
When
Figure QLYQS_63
is an outlier, remove it, and the set of all outliers is recorded as
Figure QLYQS_65
; For noise point determination: the number of channels in the last fully connected layer is set to 3, and the process description function
Figure QLYQS_54
,
Figure QLYQS_56
,
Figure QLYQS_59
Represents the offset vector of each learned point, and the corrected point is
Figure QLYQS_61
,
Figure QLYQS_64
represents the original noisy point cloud,
Figure QLYQS_66
Represents a point in the original noisy point cloud.
10.如权利要求7所述基于深度学习的点云数据三维重建系统,其特征在于:重构模块中,通过神经网络分辨出点云的表面形状点,形状外点和形状内点,而后对表面形状点上的每一帧点云进行体素表面重构,得到点云数据的三角网格模型,形状外点和形状内点不参与重构;表面重构的具体实现方式如下;10. The point cloud data 3D reconstruction system based on deep learning as claimed in claim 7, characterized in that: in the reconstruction module, the surface shape points, shape external points and shape internal points of the point cloud are distinguished by a neural network, and then the voxel surface is reconstructed for each frame of the point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data, and the shape external points and shape internal points do not participate in the reconstruction; the specific implementation method of the surface reconstruction is as follows; 首先初始化一个空三角形网格,然后执行以下操作:First initialize an empty triangle mesh, then do the following: 41,获取当前体素在(x, y, z)处的立方体cube,包含8个顶点;41. Get the cube of the current voxel at (x, y, z), which contains 8 vertices; 42,获取当前立方体cube的边界集合,该边界集合为所有占据概率在[0.5-tau,0.5+tau]区间内的点的集合,tau为设置的阈值;42. Get the boundary set of the current cube cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau], where tau is the set threshold; 43,对于集合中的每个点,执行以下操作:43. For each point in the set, do the following: i.选取顶点v1和v2,其中p1<0.5, p2>0.5;i. Select vertices v1 and v2, where p1<0.5, p2>0.5; ii.通过插值函数插值(v1, v2, p1, p2)获得一个新的占据概率等于0.5的插值点vertex;ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5; iii.将插值点vertex添加到三角形网格M中;iii. Add the interpolation point vertex to the triangle mesh M; 44,构建当前立方体cube的三角形面列表,该列表包含当前cube中占据概率为0.5的点和刚生成的vertex点;44. Build a triangular face list of the current cube, which contains points with an occupation probability of 0.5 in the current cube and the newly generated vertex points; 45,对于每个三角形面,将其添加到三角形网格M中,当所有三角形面都已经添加到三角形网格中,算法结束;45. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends. 其中神经网络学习过程如下;The neural network learning process is as follows: (1)潜在向量编码:使用点卷积方法首先为每个输入点
Figure QLYQS_67
产生一个潜在向量
Figure QLYQS_68
,编码器
Figure QLYQS_69
由点云卷积网络实现,只需要改变最后一层的通道数来控制向量
Figure QLYQS_70
的维度,其中
Figure QLYQS_71
为经过配准后的点云中的点,n表示输入点的个数;
(1) Latent vector encoding: Use the point convolution method to first encode each input point
Figure QLYQS_67
Generate a latent vector
Figure QLYQS_68
, encoder
Figure QLYQS_69
Implemented by the point cloud convolutional network, only the number of channels in the last layer needs to be changed to control the vector
Figure QLYQS_70
The dimensions of
Figure QLYQS_71
is the point in the registered point cloud, and n represents the number of input points;
(2)相对潜在向量编码:给定一个任意查询点
Figure QLYQS_73
,查询点
Figure QLYQS_75
来自输入点,构造一组点集邻居
Figure QLYQS_77
,它包括离
Figure QLYQS_79
最近的k个点,设点集邻居中的点为
Figure QLYQS_81
,而后用查询点
Figure QLYQS_82
相对于
Figure QLYQS_85
的局部坐标
Figure QLYQS_72
,对每个点集邻居
Figure QLYQS_74
中的点
Figure QLYQS_76
的潜在向量
Figure QLYQS_78
进行增强,这些增强的潜在向量经过
Figure QLYQS_80
处理得到相对潜在向量
Figure QLYQS_83
,其中
Figure QLYQS_84
为级联操作;通过以上操作对每一个输入点都完成一次查询;
(2) Relative latent vector encoding: Given an arbitrary query point
Figure QLYQS_73
, query point
Figure QLYQS_75
From the input point, construct a set of point set neighbors
Figure QLYQS_77
, which includes
Figure QLYQS_79
The nearest k points, let the points in the point set neighbors be
Figure QLYQS_81
, and then use the query point
Figure QLYQS_82
Relative to
Figure QLYQS_85
The local coordinates of
Figure QLYQS_72
, for each point set neighbor
Figure QLYQS_74
Points in
Figure QLYQS_76
The latent vector
Figure QLYQS_78
These enhanced latent vectors are enhanced by
Figure QLYQS_80
Processing to obtain the relative potential vector
Figure QLYQS_83
,in
Figure QLYQS_84
It is a cascade operation; through the above operation, a query is completed for each input point;
(3)特征加权:通过学习
Figure QLYQS_86
个独立的线性层,由
Figure QLYQS_89
个相应的权重向量
Figure QLYQS_91
参数化,产生
Figure QLYQS_93
个相对权重
Figure QLYQS_96
Figure QLYQS_98
表示点乘操作,权重总和为1;最后通过softmax将相对权重
Figure QLYQS_99
变为权重
Figure QLYQS_87
,并平均为
Figure QLYQS_88
,查询点
Figure QLYQS_90
处的特征向量
Figure QLYQS_92
由相邻点
Figure QLYQS_94
的相对潜在向量
Figure QLYQS_95
加权求和得到:
Figure QLYQS_97
(3) Feature weighting: through learning
Figure QLYQS_86
independent linear layers, consisting of
Figure QLYQS_89
The corresponding weight vector
Figure QLYQS_91
Parameterization, generating
Figure QLYQS_93
Relative weight
Figure QLYQS_96
,
Figure QLYQS_98
represents the dot multiplication operation, the sum of the weights is 1; finally, the relative weights are converted to
Figure QLYQS_99
Becomes weight
Figure QLYQS_87
, and the average is
Figure QLYQS_88
, query point
Figure QLYQS_90
The eigenvector at
Figure QLYQS_92
By adjacent points
Figure QLYQS_94
The relative potential vector
Figure QLYQS_95
The weighted summation yields:
Figure QLYQS_97
;
(4)解码为占据概率:线性层
Figure QLYQS_100
将特征向量
Figure QLYQS_101
解码为占据得分
Figure QLYQS_102
,这是一个维度为2的全连接层输出,然后通过softmax转换为占用概率
Figure QLYQS_103
(4) Decoding into occupancy probability: linear layer
Figure QLYQS_100
The feature vector
Figure QLYQS_101
Decoded as occupancy score
Figure QLYQS_102
, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax
Figure QLYQS_103
;
(5)体素表面重构:得到各个点的占用概率
Figure QLYQS_104
后,将占用概率
Figure QLYQS_105
为0.5
Figure QLYQS_106
tau的点云当作一个体素格子进行重构。
(5) Voxel surface reconstruction: obtain the occupancy probability of each point
Figure QLYQS_104
After that, the occupation probability
Figure QLYQS_105
0.5
Figure QLYQS_106
The point cloud of tau is reconstructed as a voxel grid.
CN202310483537.7A 2023-05-04 2023-05-04 Method and system for 3D reconstruction of point cloud data based on deep learning Active CN116258835B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310483537.7A CN116258835B (en) 2023-05-04 2023-05-04 Method and system for 3D reconstruction of point cloud data based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310483537.7A CN116258835B (en) 2023-05-04 2023-05-04 Method and system for 3D reconstruction of point cloud data based on deep learning

Publications (2)

Publication Number Publication Date
CN116258835A true CN116258835A (en) 2023-06-13
CN116258835B CN116258835B (en) 2023-07-28

Family

ID=86679625

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310483537.7A Active CN116258835B (en) 2023-05-04 2023-05-04 Method and system for 3D reconstruction of point cloud data based on deep learning

Country Status (1)

Country Link
CN (1) CN116258835B (en)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116781837A (en) * 2023-08-25 2023-09-19 中南大学 An automated laser three-dimensional scanning system
CN116817771A (en) * 2023-08-28 2023-09-29 南京航空航天大学 Aerospace part coating thickness measurement method based on cylindrical voxel characteristics
CN117078912A (en) * 2023-07-10 2023-11-17 中国地质大学(武汉) Laser point cloud and image alignment method and device based on deep learning
CN117292067A (en) * 2023-11-24 2023-12-26 中影年年(北京)文化传媒有限公司 Virtual 3D model method and system based on scanning real object acquisition
CN117473105A (en) * 2023-12-28 2024-01-30 浪潮电子信息产业股份有限公司 Three-dimensional content generation method based on multi-mode pre-training model and related components
CN117690095A (en) * 2024-02-03 2024-03-12 成都坤舆空间科技有限公司 Intelligent community management system based on three-dimensional scene
CN117710977A (en) * 2024-02-02 2024-03-15 西南石油大学 Dam BIM three-dimensional model semantic quick extraction method and system based on point cloud data
CN117953167A (en) * 2024-03-27 2024-04-30 贵州道坦坦科技股份有限公司 Expressway auxiliary facility modeling method and system based on point cloud data
CN118212368A (en) * 2024-05-22 2024-06-18 堆知(北京)科技集团有限公司 Method and system for reconstructing three-dimensional image based on incomplete contour point cloud data
CN118570395A (en) * 2024-07-29 2024-08-30 深圳大学 Building group structured reconstruction method, apparatus, device, medium and program product
CN118967972A (en) * 2024-07-24 2024-11-15 山东省果树研究所 Method and system for establishing 3D model of chestnut germplasm

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011085433A1 (en) * 2010-01-14 2011-07-21 The University Of Sydney Acceptation/rejection of a classification of an object or terrain feature
CN105574929A (en) * 2015-12-15 2016-05-11 电子科技大学 Single vegetation three-dimensional modeling method based on ground LiDAR point cloud data
CN109118582A (en) * 2018-09-19 2019-01-01 东北大学 A kind of commodity three-dimensional reconstruction system and method for reconstructing
WO2020146547A1 (en) * 2019-01-08 2020-07-16 Apple Inc. Auxiliary information signaling and reference management for projection-based point cloud compression
CN111833449A (en) * 2020-06-30 2020-10-27 南京航空航天大学 Three-dimensional reconstruction of subway tunnel interior environment and intelligent identification method of disease
CN111899328A (en) * 2020-07-10 2020-11-06 西北工业大学 Point cloud three-dimensional reconstruction method based on RGB data and generation countermeasure network
CN112967219A (en) * 2021-03-17 2021-06-15 复旦大学附属华山医院 Two-stage dental point cloud completion method and system based on deep learning network
CN113358061A (en) * 2021-05-31 2021-09-07 东南大学 Single stripe three-dimensional point cloud measuring method for end-to-end calibration of deep learning network
WO2021175050A1 (en) * 2020-03-04 2021-09-10 华为技术有限公司 Three-dimensional reconstruction method and three-dimensional reconstruction device
WO2021179593A1 (en) * 2020-03-10 2021-09-16 山东大学 Deep learning-based three-dimensional pipeline reconstruction method, system, medium, and apparatus

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011085433A1 (en) * 2010-01-14 2011-07-21 The University Of Sydney Acceptation/rejection of a classification of an object or terrain feature
CN105574929A (en) * 2015-12-15 2016-05-11 电子科技大学 Single vegetation three-dimensional modeling method based on ground LiDAR point cloud data
CN109118582A (en) * 2018-09-19 2019-01-01 东北大学 A kind of commodity three-dimensional reconstruction system and method for reconstructing
WO2020146547A1 (en) * 2019-01-08 2020-07-16 Apple Inc. Auxiliary information signaling and reference management for projection-based point cloud compression
WO2021175050A1 (en) * 2020-03-04 2021-09-10 华为技术有限公司 Three-dimensional reconstruction method and three-dimensional reconstruction device
WO2021179593A1 (en) * 2020-03-10 2021-09-16 山东大学 Deep learning-based three-dimensional pipeline reconstruction method, system, medium, and apparatus
CN111833449A (en) * 2020-06-30 2020-10-27 南京航空航天大学 Three-dimensional reconstruction of subway tunnel interior environment and intelligent identification method of disease
CN111899328A (en) * 2020-07-10 2020-11-06 西北工业大学 Point cloud three-dimensional reconstruction method based on RGB data and generation countermeasure network
CN112967219A (en) * 2021-03-17 2021-06-15 复旦大学附属华山医院 Two-stage dental point cloud completion method and system based on deep learning network
CN113358061A (en) * 2021-05-31 2021-09-07 东南大学 Single stripe three-dimensional point cloud measuring method for end-to-end calibration of deep learning network

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
陈辉;马世伟;ANDREAS NUECHTER;: "基于激光扫描和SFM的非同步点云三维重构方法", 仪器仪表学报, no. 05, pages 1148 - 1157 *
黄明伟;方莉娜;唐丽玉;王思洁;: "改进泊松算法的图像三维重建点云模型网格化", 测绘科学, no. 04, pages 23 - 28 *

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117078912A (en) * 2023-07-10 2023-11-17 中国地质大学(武汉) Laser point cloud and image alignment method and device based on deep learning
CN116781837B (en) * 2023-08-25 2023-11-14 中南大学 An automated laser three-dimensional scanning system
CN116781837A (en) * 2023-08-25 2023-09-19 中南大学 An automated laser three-dimensional scanning system
CN116817771A (en) * 2023-08-28 2023-09-29 南京航空航天大学 Aerospace part coating thickness measurement method based on cylindrical voxel characteristics
CN116817771B (en) * 2023-08-28 2023-11-17 南京航空航天大学 Aerospace part coating thickness measurement method based on cylindrical voxel characteristics
CN117292067A (en) * 2023-11-24 2023-12-26 中影年年(北京)文化传媒有限公司 Virtual 3D model method and system based on scanning real object acquisition
CN117292067B (en) * 2023-11-24 2024-03-05 中影年年(北京)科技有限公司 Virtual 3D model method and system based on scanning real object acquisition
CN117473105A (en) * 2023-12-28 2024-01-30 浪潮电子信息产业股份有限公司 Three-dimensional content generation method based on multi-mode pre-training model and related components
CN117473105B (en) * 2023-12-28 2024-04-05 浪潮电子信息产业股份有限公司 Three-dimensional content generation method based on multi-mode pre-training model and related components
CN117710977B (en) * 2024-02-02 2024-04-26 西南石油大学 Dam BIM three-dimensional model semantic quick extraction method and system based on point cloud data
CN117710977A (en) * 2024-02-02 2024-03-15 西南石油大学 Dam BIM three-dimensional model semantic quick extraction method and system based on point cloud data
CN117690095A (en) * 2024-02-03 2024-03-12 成都坤舆空间科技有限公司 Intelligent community management system based on three-dimensional scene
CN117690095B (en) * 2024-02-03 2024-05-03 成都坤舆空间科技有限公司 Intelligent community management system based on three-dimensional scene
CN117953167A (en) * 2024-03-27 2024-04-30 贵州道坦坦科技股份有限公司 Expressway auxiliary facility modeling method and system based on point cloud data
CN117953167B (en) * 2024-03-27 2024-05-28 贵州道坦坦科技股份有限公司 Expressway auxiliary facility modeling method and system based on point cloud data
CN118212368A (en) * 2024-05-22 2024-06-18 堆知(北京)科技集团有限公司 Method and system for reconstructing three-dimensional image based on incomplete contour point cloud data
CN118212368B (en) * 2024-05-22 2024-08-13 堆知(北京)科技集团有限公司 Method and system for reconstructing three-dimensional image based on incomplete contour point cloud data
CN118967972A (en) * 2024-07-24 2024-11-15 山东省果树研究所 Method and system for establishing 3D model of chestnut germplasm
CN118570395A (en) * 2024-07-29 2024-08-30 深圳大学 Building group structured reconstruction method, apparatus, device, medium and program product

Also Published As

Publication number Publication date
CN116258835B (en) 2023-07-28

Similar Documents

Publication Publication Date Title
CN116258835B (en) Method and system for 3D reconstruction of point cloud data based on deep learning
Zorzi et al. Polyworld: Polygonal building extraction with graph neural networks in satellite images
Meuleman et al. Progressively optimized local radiance fields for robust view synthesis
Chen et al. Point-based multi-view stereo network
Li et al. Beyond grids: Learning graph representations for visual recognition
CN111627065B (en) Visual positioning method and device and storage medium
Hu et al. Structure‐aware 3D reconstruction for cable‐stayed bridges: A learning‐based method
CN108734728A (en) A kind of extraterrestrial target three-dimensional reconstruction method based on high-resolution sequence image
GB2573170A (en) 3D Skeleton reconstruction from images using matching 2D skeletons
CN114782634B (en) Method and system for reconstructing clothed human body from monocular image based on surface implicit function
Dipanda et al. 3-D shape reconstruction in an active stereo vision system using genetic algorithms
CN116682021A (en) A Method for Extracting Building Vector Outline Data from High Resolution Remote Sensing Image
Chen et al. Research on 3D reconstruction based on multiple views
Correia et al. 3D reconstruction of human bodies from single-view and multi-view images: A systematic review
Li et al. Unsupervised category-specific partial point set registration via joint shape completion and registration
Zhang et al. ElasticMVS: Learning elastic part representation for self-supervised multi-view stereopsis
Wang et al. RSBuilding: Towards General Remote Sensing Image Building Extraction and Change Detection with Foundation Model
CN113723294B (en) Data processing method and device and object recognition method and device
Hu et al. Learning structural graph layouts and 3D shapes for long span bridges 3D reconstruction
CN118314006A (en) Semantic-combined indoor plane layout reconstruction method and device and electronic equipment
Niu et al. Overview of image-based 3D reconstruction technology
Lei et al. Geometric feature learning for 3D meshes
CN112837420B (en) Shape complement method and system for terracotta soldiers and horses point cloud based on multi-scale and folding structure
CN115471651A (en) 4D target segmentation method based on point cloud space-time memory network
Yang et al. Learning to reconstruct and understand indoor scenes from sparse views

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant