JP7171087B2 - A mesh denoising method based on graph convolutional networks - Google Patents

Description

The present invention belongs to the field of computer graphics, and more particularly to mesh denoising methods based on graph convolutional networks.

Since vertex positions or surface normals are inherently 3D signals, the task of denoising on a mesh surface is similar to 2D images. Therefore, mesh denoising techniques have been heavily inspired by denoising techniques in images, and currently various low-pass and feature-preserving filters are introduced to perform mesh denoising. Here, bilateral filters are one of the most widely applied filters, but a general drawback of filter-based methods is that if features (especially weak features) are severely corrupted by noise, they (especially weak features) will be is difficult to recover in any way. Another method is based on optimized mesh denoising method. However, it applies only hypothetical meshes and fails to summarize noise modes well for meshes with different geometric features.

In contrast, learning-based methods do not give specific hypotheses to the underlying features or noise modes and are successfully applied to image denoising. However, unlike images, 3D meshes are generally irregular, so image-based convolution operations cannot be applied directly. To solve this problem, we present a new method to input irregular mesh block data directly into the graph convolutional network. The pattern convolution operation provides a graphical representation for the local surface geometry, and our network is better able to capture the intrinsic geometric features of the noisy source model than other conventional methods. can.

A graph convolutional network (GCN) is introduced to process the Euclidean structure. Early operations of GCN require a static graph structure and thus cannot be extended to meshes with varying topologies. According to the latest research in dynamic graph convolution, variable edges can be represented better. Our method utilizes the static graph structure in the block and the dynamic graph structure of the convolutional term structure to effectively learn the geometric features in the block. Some others are mesh development convolution operations, which are mainly used for understanding whole objects or large scenes and require very deep network structures. The denoising operation focuses more on local blocks and introduces convolutions in the dual space of the mesh planes.

The present invention provides a mesh denoising method based on GCN networks, and uses rotation-invariant graph representations in the dual space of mesh faces to achieve efficient feature learning through graph convolution. At the same time, the static and dynamic graph convolution operations in our architecture are serially connected to learn effective explicit structural features and potential implicit features between adjacent nodes.

The present invention is realized by the following technical solutions.

A mesh denoising method based on graph convolutional networks, comprising the following steps 1 and 2:
In step 1, generate a local block for each face in the noise mesh and employ a tensor voting algorithm to rotationally align the local block;
In step 2, the local blocks aligned in step 1 are transformed into a graph and shown, input to a trained graph convolutional neural network, noise-free normal directions are predicted, and the predicted normal directions are update the vertices of the mesh model based on to obtain the denoised model, wherein the convolutional neural network structure is L _e -layer EdgeConv, L _d -layer dynamic EdgeConv and L ₁ -layer fully connected (FC) layer Configured.

に対して、その領域の面積の一定の割合に応じてバウンディングスフィアを画定し、バウンディングスフィア内の全てのパッチを該ブロック

中の面とし、
サブステップ１．２では、該ブロックにおける全ての面

に対する投票テンソル

を定義し、かつ特徴値及び単位特徴ベクトルを取得し、
サブステップ１．３では、サブステップ１．２で得られた特徴ベクトルに基づいて一つの回転行列Ｒ_ｉを構築し、かつ

中の各小面の重心及び法線をＲ_ｉ ^－１に乗算して新たなブロックデータ

を生成する。 Further, said step 1 is performed by the following substeps 1.1 to 1.3,
In substep 1.1 the selected patch

, define a bounding sphere according to a certain percentage of the area of that region, and all patches within the bounding sphere are added to the block

as the inside surface,
In sub-step 1.2 all faces in the block

vote tensor for

and obtain feature values and unit feature vectors,
in substep 1.3, constructing one rotation matrix R _i based on the feature vectors obtained in substep 1.2; and

Multiply R _i ⁻¹ by the centroid and normal of each facet in the new block data

to generate

Further, step 2 is performed by the following substeps 2.1 to 2.4,
In sub-step 2.1, iterating the static edge convolution process with static EdgeConv on the input image to obtain adjacent surface features,
in substep 2.2, iterating the dynamic edge convolution process with dynamic EdgeConv on the result obtained in substep 2.1 to obtain the closest feature face in the feature space;
In sub-step 2.3, after graph convolution, connect the learned features together and aggregate the features via a fully connected layer,
In sub-step 2.4, symmetry pooling selects the most dominant features to finally predict the normal direction.

Further, a training dataset for said graph convolutional neural network is constructed in the following manner:
A tensor voting algorithm is applied to each face of the noiseless model in the dataset to obtain three feature values λ ₁ , λ ₂ , λ ₃ , and based on the feature values all model faces in all data are The block data is divided into two sets of "face" and "edge", and the block data is collected respectively to construct the training data set.

Furthermore, when training multiple cascaded graph convolutional neural networks to predict noise-free normal directions, training the preceding graph convolutional neural networks until the loss of the cascaded networks no longer drops. The predicted normal direction is used to construct the denoised mesh model and generate new data to train the next stage graph convolutional neural network.

Furthermore, in the final-stage graph convolutional neural network, iterative optimization is performed using the normal vector of the bilateral filter pair on the mesh, the vertices are updated at each iteration, and finally the mesh model after noise removal is obtained. .

The salient contributions of the present invention are as follows.

The present invention provides a GCN-Denoiser, a feature-reserving network denoising method based on graph convolutional networks (GCN). Unlike conventional artificial structural feature learning or voxel-based feature learning-based learning-based mesh noise reduction methods, the present invention explores the structure of the triangular mesh itself and introduces, as shown, Next, we introduce a graph convolution operation in the dual space of triangles. The present invention thus presents a graphical representation format that can naturally capture geometric features, while at the same time being lightweight for both the training and estimation stages. In order to facilitate effective feature learning, such networks utilize static and dynamic edge convolutions simultaneously, thereby extracting from implicit connections between explicit mesh structures and unconnected neighbors. Information can be learned. To better estimate the unknown noise function, the present invention introduces a cascaded optimization example of multiple GCNs to estimate the noiseless normal direction of the surface step by step. The present invention achieves best results on multiple noise data sets, including CAD models with well-defined features and original scanned models with true noise captured from different devices. The method of the present invention achieves a good balance between effectiveness and efficiency while achieving the currently most favorable results.

1 is a process schematic diagram of network noise cancellation of the present invention; FIG. 1 is the GCN network structure of the present invention; is the network noise elimination effect diagram of the present invention;

Since noise is a complex composition of mesh model surfaces, we generally estimate it using local methods. The target of the present invention is to take each patch in the dual domain of mesh triangles, block it with noise within a certain range, predict the original noiseless normal direction of the patch, and reproduce the noiseless model. Specifically, it includes the following steps.
Step 1: Generate a local block for each face in the noise mesh and employ a tensor voting algorithm to rotationally align the local block.
Step 2: The local blocks aligned in step 1 are transformed as shown in the figure, input to a trained graph convolutional neural network to predict noise-free normal directions, and further based on the predicted normal directions update the vertices of the mesh model to get the denoised model. Here, the convolutional neural network structure is composed of an L _e layer EdgeConv, an L _d layer dynamic EdgeConv and an L ₁ fully connected (FC) layer.

FIG. 1 illustrates the denoising flow of multiple graph convolutional neural network cascades in the present invention. The method of the present invention is further described below with reference to specific examples.

For the noise mesh, first define the input triangular mesh as M={V, F}, where V={v _k } ₁ N ^v sets all vertices and F={f _i } ₁ ^Nf is Set all faces. _Nv and _Nf are the number of vertices and the number of faces, respectively. For each face f _i in F, generate its local block data p _i . Define the set of all blocks in M as P={p _i } ₁ ^Nf . Similarly, the normal direction of the face f _i is denoted by n _i , its centroid by c _i and its area by a _i .

Here, block p _i refers to all facets (including f _i ) within a sphere of radius r located on the centroid c _i of facet f _i , i.e. p _i should satisfy:

This is because blocks with similar properties but in different locations create trouble for neural networks and make it difficult to learn spatial transformations for deep learning methods. To solve this problem, the present invention adopts tensor voting theory to unambiguously align surfaces to one public coordinate system, i.e., define all surface voting tensors in the surface, and define feature values and units Get the feature vector, specifically:

First transform pi to the origin [ _0,0,0 ] and then scale it to the unit bounding box. The definition for face f _i of one voting tensor T _i is:

where μ _j =(a _j /a _m )exp(−||c _j −c _i ||/σ), σ is a parameter, set to ⅓ in this embodiment, where a _m is is the maximum triangle area at p _i and n _j ' is the voting normal vector of f _j . n _j '=2(n _j ·w _j )w _j −n _j , where w _j =normalize{[(c _j −c _i )×n _j ]×(c _j −c _i )}. Since T _i is a semi-regular matrix, it can be expressed by its spectral decomposition as follows.

where λ ₁ ≧λ ₂ ≧λ ₃ are its feature values and e ₁ , e ₂ and e ₃ are the corresponding unit feature vectors, which constitute a set of orthogonal groups.

を生成する。 Next, construct one rotation matrix R _i =[e ₁ ,e ₂ ,e ₃ ] and multiply R _i ⁻¹ by the centroid and normal direction of each facet at p _i to obtain the new block data

to generate

内の各面ｆに図面上のノードｑ_ｉ∈Ｑ、及び一つの辺ｅ＝（ｑ_ｉ、ｑ_ｊ）∈Ｅを作成して対応する面ｆ_ｉ及びｆ_ｊが隣接する。Φはノード特徴を表し、一組のノード属性を含む。各のφ_ｉ∈Φ、対応面

．

及び

は整列後の平面ｆｉの重心及び法線方向を指す。ｄ_ｉはｆ_ｉの１環近傍における隣接する面の数であり、境界面を区別することに役立つ。 We then establish a diagram structure for each generated block to adapt it to our later pattern convolution network. Establish an undirected graph G = (Q, E, Φ), where block

For each face f in , create a node q _i εQ on the drawing and one edge e=(q _i , q _j )εE to adjoin the corresponding faces f _i and f _j . Φ represents a node feature and contains a set of node attributes. For each φ _i ∈ Φ, the corresponding surface

.

as well as

refers to the centroid and normal direction of the plane fi after alignment. d _i is the number of adjacent faces in one ring neighborhood of f _i and serves to distinguish the boundary faces.

It has been described that the GCN network of the present invention is composed of multiple convolutional layers, as shown in FIG. （Ｍａｒｔｉｎ Ｓｉｍｏｎｏｖｓｋｙ ａｎｄ Ｎｉｋｏｓ Ｋｏｍｏｄａｋｉｓ． ２０１７． Ｄｙｎａｍｉｃ ｅｄｇｅ－ｃｏｎｄｉｔｉｏｎｅｄ ｆｉｌｔｅｒｓ ｉｎ ｃｏｎｖｏｌｕｔｉｏｎａｌ ｎｅｕｒａｌ ｎｅｔｗｏｒｋｓ ｏｎ ｇｒａｐｈｓ． Ｉｎ ２０１７ ＩＥＥＥ Ｃｏｎｆｅｒｅｎｃｅ ｏｎ Ｃｏｍｐｕｔｅｒ Ｖｉｓｉｏｎ ａｎｄ Ｐａｔｔｅｒｎ Ｒｅｃｏｇｎｉｔｉｏｎ （ＣＶＰＲ）． ２９－３８． ）各層において、従来の畳み込みネットワークと類似し, the GCN of the present invention collectively updates the features of each node's neighbors, also called a convolution operation. Each figure has a certain degree of connectivity in the plane, but their structure is very different in blocks. Therefore, we use an ECC (Edge-Conditioned Convolution) policy to handle different structures in the convolution process. G _l = (Q _l , E _l , Φ _l ) is the first layer in the graph convolution and φ ⁱ is the feature vector of the _{i th} node in G _l . Update the node features in the following manner.

where Ψ is the feature set and h _Θ ^l = Linear _Θ ^l (F _l ⁱ , F _l ^j - F _l ⁱ ). There is a similar Linear _Θ at the base layer of each graph reel in the network.

Since the geometry-to-connectivity mapping is not a one-to-one function, some information may be lost in the convolution process using only the original geometry structure. To enrich the receiving area of graph nodes, the present invention also allows connecting non-adjacent graph nodes in the convolution process. Such graphics transforms are called dynamic Edge Conv. For this scheme, each node's neighbors are dynamically computed by KNN (where K=8 in the implementation of this embodiment) based on the node's Euclidean distance.
The network architecture of the present invention consists of a static Edge Conv on the L _e layer, a dynamic Edge Conv on the L _d layer and a fully connected (FC) layer on the L ₁ layer. After each layer of graph convolution, the learned features are connected together for pooling. In this example, both the average pool and the maximum pool are used as symmetry functions, and they can select the most important features. Finally, the FC layer regresses the 3D vector, ie the normal direction we predict. Except for the final FC layer, each layer in our architecture has a batch processing normalization and activation function LeakyReLU.

A preferred solution is to stepwise regress the noise-free normal direction with cascaded GCNs (GCN ₁ , . . . , GCN _X ). All GCNs in the cascade optimization have the same architecture, but different numbers of static Edge Conv, dynamic Edge Conv and FC layers. In this example, we use Le=3, Ld=3, and Ll=4 for the first GCN, and Le=2, Ld=2, and Ll=3 for the remaining GCNs. do.

には、該頂点の領域はその近傍頂点を含み、該頂点の領域はその隣接頂点の関連付けパッチを含む。

It is necessary to update the vertex position after each normal prediction update, and the vertex update in the method of the present invention can be defined as the following equation, where:

, the region of the vertex contains its neighboring vertices, and the region of the vertex contains the associated patches of its neighboring vertices.

k denotes the number of vertex update iterations, the superscript g denotes the target normal direction, in the previous x−1 stage GCN update, the predicted value of the network output, and in the final stage update, the post-optimization Indicates normal direction. eij indicates an edge between two vertices in the patch.

である。ここで、

は面ｆ_ｉの真の雑音メッシュの法線方向であり、Ｒは上記対応する回転行列である。 In the offline training step, the output of GCN _x is used to denoise the noisy meshes of the training set and then new data is generated from these updated meshes to train the next GCN _x+1 . Cascaded GCN can be stopped if the error on the validation set of the network does not decrease. The loss function is the MSE between the network output and the reference value, i.e. R ⁻¹

is. here,

is the normal direction of the true noise mesh of face f _i and R is the corresponding rotation matrix above.

A preferred solution is to generate and train different levels and types of noise for each 3D model. A tensor voting algorithm is applied to each face of the noiseless model in the dataset to obtain three feature values λ1, λ2, λ3. For each model, each face is divided into four sets, {fi|λi 2<0.01Λλi 3<0.001} are planes and {fi|λi 2>0.01Λλi 3<0.1} are edges , {fi|λi 3>0.1} are corner surfaces and the rest are transition surfaces. Side surfaces and corner surfaces are fewer than the other two types, and are further divided into two sets. Characteristic surfaces composed of planes and transition surfaces are characteristic surfaces composed of edge surfaces and corner surfaces. Blocks are uniformly collected as training data in these two sets. This policy applies to training all GCNs.

As a preferred solution, a bilateral filter (Youyi Zheng, Hongbo Fu, Oscar Kin-Chung Au , and Chiew-Lan Tai. 2011. Bilal normal filtering for mutenoising. IEEE Transions on Visualization and Computer Graphics 17, 10 (2011), 1521-1530.) can be applied to fine-tune the normal direction of the GCN prediction. can.

は反復最適化ｍ＋１回の法線方向であり、

はｍ回目の反復により得られた法線方向に基づいて頂点を更新した後のパッチに算出された法線方向である。Ω_ｉは隣接ｆ_ｉの集合であり、Ｗ_ｓとＷ_ｒはそれぞれσｓとσｒコアのガウス関数である。 This manipulation iteration can be repeated m times, but only applies to the normal output in the last cascaded GCN. here

is the normal direction of iterative optimization m+1 times, and

is the normal direction computed for the patch after updating the vertices based on the normal direction obtained by the m-th iteration. Ω _i is the set of neighbors f _i and W _s and W _r are Gaussian functions of σs and σr cores respectively.

Figures 1 and 3 show the denoising results of the original scanned model and the CAD model containing the distinct features, respectively, of the actual noise captured by the slave device of the method, the leftmost being the input noise model and the middle Result, rightmost is the original noiseless true value. As can be seen, the method of the present invention has a good denoising effect and can achieve a good balance between effectiveness and efficiency.

Claims (6)

A mesh denoising method based on graph convolutional networks, comprising the following steps 1 and 2:
In step 1, generate a local block for each face in the noise mesh and employ a tensor voting algorithm to rotationally align the local block;
In step 2, the local blocks aligned in step 1 are transformed into a graph and shown, input to a trained graph convolutional neural network, noise-free normal directions are predicted, and the predicted normal directions are update the vertices of the mesh model to obtain the denoised model, wherein the structure of the graph convolutional neural network is L _e -layer EdgeConv, L _d -layer dynamic EdgeConv and L ₁ -layer fully connected (FC) A method characterized by comprising layers. said step 1 being performed by the following substeps 1.1 to 1.3;
In substep 1.1 the selected patch

, define a bounding sphere according to a constant fraction of the area of the region, and delineate all patches within the bounding sphere into the local block

as the inside surface,
In substep 1.2 all faces in the local block

vote tensor for

and obtain feature values and unit feature vectors,
in substep 1.3, constructing one rotation matrix R _i based on the feature vectors obtained in substep 1.2; and

Multiply R _i ⁻¹ by the centroid and normal of each facet in the new block data

The graph convolutional network-based mesh denoising method of claim 1, wherein generating . said step 2 is performed by the following substeps 2.1 to 2.4,
In sub-step 2.1, iterating the static edge convolution process with static EdgeConv on the input image to obtain adjacent surface features,
in substep 2.2, iterating the dynamic edge convolution process with dynamic EdgeConv on the result obtained in substep 2.1 to obtain the closest feature face in the feature space;
In sub-step 2.3, after graph convolution, connect the learned features together and aggregate the features via a fully connected layer,
The method of mesh denoising based on graph convolutional networks according to claim 1, characterized in that sub-step 2.4 selects the most dominant features by symmetry pooling to finally predict the normal direction. A training data set for said graph convolutional neural network is constructed in the following manner,
A tensor voting algorithm is applied to each face of the noiseless model in the dataset to obtain three feature values λ ₁ , λ ₂ , λ ₃ , and based on the feature values all model faces in all data are The mesh denoising method based on graph convolutional network according to claim 1, characterized in that it divides into two sets of "faces" and "edges" and collects block data respectively to build a training data set. When training multiple cascaded graph convolutional neural networks to predict noise-free normal directions and training, until the loss of the cascaded networks no longer drops, the previous graph convolutional neural network predicts The graph convolution according to claim 1, wherein the normal direction is used to build a mesh model after noise removal and generate new data to train a graph convolution neural network in the next stage. A network-based mesh denoising method. In the final-stage graph convolutional neural network, iterative optimization is performed using the normal vector of the bilateral filter pair on the mesh, the vertices are updated at each iteration, and finally the mesh model after denoising is obtained. Graph convolutional network-based mesh denoising method according to claim 1.

Images