WO2022183852A1

WO2022183852A1 - Method for segmenting dental three-dimensional digital model

Info

Publication number: WO2022183852A1
Application number: PCT/CN2022/072239
Authority: WO
Inventors: 沈恺迪
Original assignee: 杭州朝厚信息科技有限公司
Priority date: 2021-03-01
Filing date: 2022-01-17
Publication date: 2022-09-09
Also published as: CN114998577A

Abstract

A method for segmenting a dental three-dimensional digital model, which is executed by a computer. The method comprises: acquiring a first dental three-dimensional digital model; generating a graph on the basis of the first dental three-dimensional digital model, wherein the graph comprises nodes, initial features of the nodes, and adjacent points, wherein the nodes are center points of patches of the first dental three-dimensional digital model; generating a coarse prediction result and an offset vector by using a trained graph neural network and on the basis of the graph, wherein the graph neural network comprises a feature extraction sub-network, a coarse prediction sub-network and an offset sub-network, the feature extraction sub-network generates a node feature matrix on the basis of the graph, the coarse prediction sub-network generates the coarse prediction result on the basis of the node feature matrix, and the offset sub-network generates the offset vector on the basis of the node feature matrix; on the basis of the offset vector, performing a clustering operation on nodes, belonging to a tooth, in the coarse prediction result; and performing weighted calculation on the basis of the coarse prediction result and a clustering result, so as to obtain a first segmentation result.

Description

Segmentation method of 3D digital model of teeth and jaws

technical field

The present application generally relates to a method for segmenting a three-dimensional digital model of a tooth and jaw, and more particularly, to a method for segmenting a three-dimensional digital model of the tooth and jaw using an artificial neural network.

Background technique

Today, dental treatment is increasingly relying on computer technology, and in many cases it is necessary to segment the scanned three-dimensional digital model of the jaw including the dentition and at least part of the gingiva, and segment the crown of each tooth, including The division between crown and gingiva and between adjacent crowns.

Due to the low efficiency of manually segmenting the 3D digital model of the teeth and jaws through the computer user interface, there have been many methods for automatically segmenting the 3D digital models of the teeth and jaws by computer. However, these methods cannot accurately and quickly segment teeth.

Therefore, it is necessary to provide a new segmentation method of the three-dimensional digital model of the teeth and jaws.

SUMMARY OF THE INVENTION

An aspect of the present application provides a computer-implemented method for segmenting a three-dimensional digital model of a tooth, including: acquiring a three-dimensional digital model of a first tooth; generating a graph based on the three-dimensional digital model of the first tooth, which includes nodes, nodes initial features and adjacent points, wherein the node is the center point of the face of the first three-dimensional digital model; using a trained graph neural network, based on the graph, to generate a rough prediction result and an offset vector, the The graph neural network includes a feature extraction sub-network, a coarse prediction sub-network and an offset sub-network. The feature extraction sub-network generates a node feature matrix based on the graph, and the coarse prediction sub-network generates the coarse prediction based on the node feature matrix. A prediction result, the offset sub-network generates the offset vector based on the node feature matrix; based on the offset vector, a clustering operation is performed on the nodes belonging to the teeth in the rough prediction result; and based on the rough prediction result The prediction result and the clustering result are weighted to obtain the first segmentation result.

In some embodiments, the node initial features include node coordinates, patch normals, and vectors from nodes to vertices of the patch.

In some embodiments, the adjacent points are nodes adjacent to each of the nodes calculated using the k-nearest neighbor algorithm.

In some embodiments, the feature extraction sub-network is a dynamic graph convolutional neural network.

In some embodiments, the coarse prediction sub-network is a convolution-based neural network.

In some embodiments, the coarse prediction sub-network employs an EdgeConv convolution operation.

In some embodiments, the offset sub-network is a recurrent neural network based on shared fully connected layers.

In some embodiments, the coarse prediction sub-network generates the coarse prediction result based on the node feature matrix and the offset vector generated by the offset sub-network.

In some embodiments, the clustering operation employs a density clustering based algorithm.

In some embodiments, the method for segmenting the three-dimensional digital model of teeth and jaws may further include: performing weighted calculation based on the rough prediction result and the clustering result to obtain a second segmentation result; and constructing a second segmentation result using the second segmentation result Markov random field, and use the graph cut algorithm to obtain the first segmentation result.

In some embodiments, the method for segmenting a three-dimensional digital model of a tooth and jaw may further include: acquiring a three-dimensional digital model of a second tooth and jaw; and simplifying the three-dimensional digital model of the second tooth and jaw to obtain the first tooth and jaw a three-dimensional digital model; and mapping the first segmentation result back to the second jaw three-dimensional digital model to obtain a third segmentation result.

In some embodiments, the segmentation method of the three-dimensional digital model of the teeth and jaws may further include: using a fuzzy clustering algorithm and a shortest path algorithm to optimize and smooth the third segmentation result.

In some embodiments, the fuzzy clustering algorithm considers patch area.

Description of drawings

The above and other features of the present application will be further described below with reference to the accompanying drawings and the detailed description. It should be understood that these drawings only illustrate several exemplary embodiments according to the present application, and therefore should not be construed as limiting the scope of protection of the present application. Unless otherwise indicated, the drawings are not necessarily to scale and like numerals refer to like parts therein.

1 is a schematic flowchart of a method for segmenting a three-dimensional digital model of a tooth and jaw in an embodiment of the present application;

FIG. 2 schematically shows the structure of a graph neural network in an embodiment of the present application;

FIG. 3 schematically shows the structure of a feature extraction sub-network in an embodiment of the present application;

Figure 4A shows the node distribution before clustering in an example of the present application; and

Figure 4B shows the distribution of the nodes shown in Figure 4A after clustering.

Detailed ways

In the following detailed description, reference is made to the accompanying drawings which form a part of this specification. The schematic embodiments mentioned in the description and drawings are for illustrative purposes only, and are not intended to limit the protection scope of the present application. Under the guidance of the present application, those skilled in the art can understand that many other embodiments may be employed and various changes may be made to the described embodiments without departing from the spirit and scope of the present application. It should be understood that the various aspects of the application described and illustrated herein may be arranged, substituted, combined, separated and designed in many different configurations, all of which are within the scope of the application.

One aspect of the present application provides a computer-implemented segmentation method of a three-dimensional digital model of a tooth and jaw,

Please refer to FIG. 1 , which is a schematic flowchart of a method 100 for segmenting a three-dimensional digital model of a tooth and jaw in an embodiment of the present application.

At 101, a three-dimensional digital model of the jaw is acquired.

In some embodiments, a patient's jaw can be scanned directly to obtain a three-dimensional digital model of the jaw. In yet other embodiments, a physical model of the patient's jaw, such as a plaster cast, may be scanned to obtain a three-dimensional digital model of the jaw. In yet other embodiments, a bite model of a patient's jaw may be scanned to obtain a three-dimensional digital model of the jaw.

In one embodiment, a three-dimensional digital model of a tooth and jaw may be constructed based on a triangular mesh, and the following takes such a three-dimensional digital model of the tooth and jaw as an example for description.

In 103, the three-dimensional digital model of the dental jaw is simplified.

In one embodiment, the three-dimensional digital model of the teeth and jaws obtained in 101 may be simplified to reduce the memory usage of subsequent calculations.

In one embodiment, an algorithm based on Quadric Error Metrics can be used to simplify the three-dimensional digital model of the teeth and jaws.

In one embodiment, the number N of face pieces of the simplified three-dimensional digital model of the posterior teeth and jaws can be preset, for example, N=10000 can be preset. It can be understood that the number N of the simplified back sheets may not be fixed in advance, for example, it may be simplified according to a preset density or ratio of the sheets.

After the simplified operation, a simplified three-dimensional digital model of the jaw is obtained. The simplified operation is a known technology in the industry, and will not be described in detail here.

At 105, a map is generated based on the face patch of the simplified three-dimensional digital model of the jaw.

In one embodiment, a graph may be generated based on the facets of the simplified three-dimensional digital model of the teeth as input to a Graph Neural Network. In one embodiment, the graph includes nodes, node initial features, and edges.

The center point of each patch is used as a node, and the coordinates of each node are the three-dimensional center coordinates of the corresponding patch. The node set can be expressed as P={c ₁ ,...,c _N }∈R ^N×3 , where , N represents the number of faces in the simplified three-dimensional digital model of the teeth and jaw, and c ₁ represents the three-dimensional center coordinates of the face with index 1 (ie, the node coordinates corresponding to the face).

The initial features of a node can include the center coordinates of the patch (3-dimensional vector), the normal direction (3-dimensional vector), and the vector (9-dimensional vector) from the center of the patch to each vertex of the patch (9-dimensional vector), that is, the initial feature of each node is 15-dimensional vector. Therefore, the initial features of the node set can be expressed as X∈R ^N×15 .

In one embodiment, a k-nearest neighbor algorithm (k-Nearest Neighbor) may be used to calculate k adjacent nodes for each node to form k edges. In one embodiment, the edges of a set of nodes can be represented by an N*k adjacency matrix, which can store the indices of each node's neighbors.

At 107, using the trained graph neural network, based on the graph, coarse prediction results and offset vectors are generated.

Referring to FIG. 2 , a graph neural network 200 in an embodiment of the present application is schematically shown, which includes a feature extraction sub-network 201 , a coarse prediction sub-network 203 and an offset sub-network 205 .

In one embodiment, the feature extraction sub-network 201 can use a modified dynamic graph convolutional neural network (Dynamic Graph CNN, DGCNN for short), for example, can use Yue Wang et al. in Acm Transactions On Graphics (tog) 38.5 (2019) : The DGCNN network structure disclosed in "Dynamic Graph CNN for Learning on Point Clouds" published by 1-12.

In one embodiment, the feature extraction sub-network 201 takes the node initial feature X and the adjacency matrix as input, and outputs a node feature matrix of N*1216.

Please refer to FIG. 3, which schematically shows the structure of the feature extraction sub-network 201 in an embodiment of the present application, which includes three EdgeConv modules 2011-2015, a shared fully connected layer, an Instance Normalization layer (not shown in the figure), Leaky ReLU activation function, concatenate operation, and global average pooling layer. Among them, each EdgeConv module receives the same adjacency matrix.

In one embodiment, the offset sub-network 205 is a regression network based on a shared fully connected layer, which includes an Instance Normalization layer and a Leaky ReLU activation function. The offset sub-network 205 predicts the normalized offset vector O={o ₁ ,...,o _N }∈R ^N× for each node based on the node feature matrix output by the feature extraction sub-network 201 to the center of the corresponding tooth ³ . The normalized offset vector is multiplied by a constant δ (in one embodiment, δ=6) to obtain the offset vector, and the node coordinate P is added to the offset vector to obtain the offset node coordinate Q={q _i |q _i =c _i +δ×o _i , i=1,...,N}∈R ^N×3 .

In one embodiment, the offset sub-network 205 may also calculate an adjacency matrix based on the offset node set, and output it to the coarse prediction sub-network 203 .

Please refer to FIG. 4A , which shows the original distribution of nodes in an example, and please refer to FIG. 4B , which shows the shifted node distribution in an example. It can be seen that the offset nodes are more concentrated in the center of the teeth and are more compact. On the one hand, it is easy to cluster, and on the other hand, the coarse prediction sub-network can better predict the classification.

The coarse prediction sub-network 203 is used to predict the probability distribution of 17 classes (including 16 teeth and gums) of nodes. It is a convolution-based network. In one embodiment, the convolution operation can use EdgeConv, KPConv, PointConv or X-Conv etc.

In one embodiment, the coarse prediction sub-network 203 includes a shared fully connected layer, a Leaky ReLU activation function, an Instance Normalization layer, and an EdgeConv module, where the EdgeConv module receives an adjacency matrix of shifted nodes. The coarse prediction sub-network 203 takes the node feature matrix output by the feature extraction network and the adjacency matrix of the offset node as input, and predicts a 17-class probability distribution for each face, indicating that the face belongs to the gingival and the left and right sides of the single jaw respectively. Probability of 16 teeth.

In one embodiment, the graph neural network 200 may be trained with annotated dental and jaw triangular mesh data.

In one embodiment, the training of the graph neural network 200 may employ a loss function expressed by the following equation (1):

where L _sem is the cross-entropy loss function that supervises the 17-class probability distribution,

is the mean rating error loss function for the supervised offset vector,

where o is the normalized offset vector output by the network,

is the true value of the offset vector of the node.

At 109, based on the offset vector, the patches that are coarsely predicted to belong to teeth are clustered.

In one embodiment, if a patch (ie, a node) has a rough predicted classification result of gingiva, its offset vector is reset to zero.

In one embodiment, based on the offset vector predicted by the offset sub-network 205, a density-based clustering algorithm (DBSCAN) can be used to cluster the rough predicted tooth patches, and use principal component analysis and k-means clustering The algorithm optimizes the clustering results to divide the patches into different clusters, and finally classifies these clusters to obtain preliminary segmentation results. The specific operations are as follows.

First, according to the 17-class probability distribution output by the coarse prediction sub-network 203, all nodes (patches) T classified as teeth are extracted.

Next, the offset node coordinates Q _T of T are clustered into m clusters using DBSCAN (parameters are ε=1.05, MinPts=30), and denoted as G={g ₁ ,...,g _m }.

For each g _i , if the number of patches is less than 60, the cluster is discarded and regarded as gingiva (because the number of patches is too small, it is most likely to be a bubble).

For each _gi (i=1,...,m), use principal component analysis to calculate the long axis and calculate the projection length of _gi on the long axis. If the projection length is greater than τ, use k-means clustering to g _i is divided into two clusters, which can separate the two teeth that are wrongly clustered into one. In one embodiment, for anterior teeth, τ=6.5 can be set, and for posterior teeth, τ=10 can be set.

For each _gi , the 17-class probability distributions of all nodes in _gi are averaged, and the tooth class with the highest probability is assigned to _gi . For the previously separated cluster, find the unassigned tooth category to assign to the cluster, if there is no unassigned tooth then do not assign.

At this point, the clustering operation is completed.

In 111, a preliminary segmentation result is weighted based on the coarse prediction result and the clustering result.

In one embodiment, a more accurate preliminary segmentation result may be obtained by weighted calculation based on the rough prediction result and the clustering result according to the following equation (3).

in,

Represents the probability that the patch i output by the coarse prediction sub-network 203 belongs to the category j ( _j =0, 1, . Equation (4) expresses,

In 113, a Markov random field is constructed using the preliminary segmentation result, and the classification result of each patch is obtained by using the graph cut algorithm.

Since some patches are wrongly segmented in the preliminary segmentation results, the preliminary segmentation results can be optimized. Based on the preliminary segmentation results (probability distribution), a Markov random field is constructed, and the graph cut algorithm is used to obtain the category of each facet, which is the final segmentation result of the simplified three-dimensional digital model of the teeth. In one embodiment, the "3d Tooth" published by X. Xu, C. Liu and Y. Zheng in IEEE Transactions on Visualization and Computer Graphics, Vol. 25, No. 7, pp. 2336-2348, 2018. The method disclosed in Segmentation and Labeling Using Deep Convolutional Neural Networks constructs a Markov random field.

In 115, the final segmentation result of the simplified three-dimensional digital model of the teeth is mapped back to the original three-dimensional digital model of the teeth.

In one embodiment, the k-nearest neighbor algorithm (k=1) may be used to map the final segmentation result of the simplified three-dimensional digital model of the tooth back to the original three-dimensional digital model of the tooth.

In 117, the tooth edges of the segmentation result of the original three-dimensional digital model of teeth are optimized and smoothed.

In one embodiment, the fuzzy clustering algorithm and the shortest path algorithm may be used to optimize and smooth each tooth boundary of the segmentation result of the original three-dimensional digital model of the teeth.

In one embodiment, the "3d Tooth" published by X. Xu, C. Liu and Y. Zheng in IEEE Transactions on Visualization and Computer Graphics, Vol. 25, No. 7, pp. 2336-2348, 2018. Fuzzy clustering and shortest path algorithms disclosed in Segmentation and Labeling Using Deep Convolutional Neural Networks.

In one embodiment, the capacity function of the fuzzy clustering algorithm in the above paper can be improved. For patch i, improve the capacity function of the subsequent patch i to its adjacent patch j as follows:

Among them, the definitions of C(i,j) and x are the same as in the original paper, C(i,j) is the flow capacity value between face i and face j, and x is the distance from the center of face i to the current tooth boundary The shortest geodesic distance, σ=0.05, a _i and a _j are the areas of patches i and j respectively, γ=50. The improved fuzzy clustering takes the patch area into account and increases the segmentation probability at the tooth boundary where the patch area is small.

Although various aspects and embodiments of the present application are disclosed herein, other aspects and embodiments of the present application will also be apparent to those skilled in the art in light of the present application. The various aspects and embodiments disclosed herein are for purposes of illustration only and not limitation. The scope and spirit of this application are to be determined only by the appended claims.

Likewise, the various diagrams may illustrate exemplary architectural or other configurations of the disclosed methods and systems, which may be helpful in understanding the features and functionality that may be included in the disclosed methods and systems. What is claimed is not limited to the exemplary architectures or configurations shown, and the desired features may be implemented in various alternative architectures and configurations. Additionally, with respect to the flowcharts, functional descriptions, and method claims, the order of blocks presented herein should not be limited to various embodiments that are implemented in the same order to perform the functions, unless the context clearly dictates otherwise. .

Unless expressly stated otherwise, the terms and phrases used herein, and variations thereof, are to be construed as open-ended rather than restrictive. In some instances, the appearance of expanding words and phrases such as "one or more," "at least," "but not limited to," or other similar expressions should not be construed as in instances where such expanding words may not be present Intent or need to indicate a narrowed situation.

Claims

A computer-implemented segmentation method of a three-dimensional digital model of teeth and jaws, comprising:

Obtain the 3D digital model of the first tooth and jaw;

generating a graph based on the first three-dimensional digital model of the teeth, which includes nodes, initial features of nodes, and adjacent points, wherein the nodes are the center points of the facets of the first three-dimensional digital model;

Using a trained graph neural network, based on the graph, to generate a coarse prediction result and an offset vector, the graph neural network including a feature extraction sub-network, a coarse prediction sub-network and an offset sub-network, the feature extraction sub-network based on The graph generates a node feature matrix, the coarse prediction sub-network generates the coarse prediction result based on the node feature matrix, and the offset sub-network generates the offset vector based on the node feature matrix;

Based on the offset vector, a clustering operation is performed on the nodes belonging to the teeth in the rough prediction result; and a weighted calculation is performed based on the rough prediction result and the clustering result to obtain a first segmentation result.
The method for segmenting a three-dimensional digital model of a tooth and jaw executed by a computer according to claim 1, wherein the initial features of the nodes include the coordinates of the nodes, the normal direction of the face, and the vector from the node to each vertex of the face.
The computer-implemented method for segmenting a three-dimensional digital model of teeth and jaws according to claim 1, wherein the adjacent points are the adjacent nodes calculated for each of the nodes by using the k-nearest neighbor algorithm.
The computer-implemented method for segmenting a three-dimensional digital model of teeth and jaws according to claim 1, wherein the feature extraction sub-network is a dynamic graph convolutional neural network.
The computer-implemented method for segmenting a three-dimensional digital model of teeth and jaws according to claim 1, wherein the coarse prediction sub-network is a convolution-based neural network.
The computer-implemented segmentation method of the three-dimensional digital model of teeth and jaws according to claim 5, wherein the coarse prediction sub-network adopts EdgeConv convolution operation.
The computer-implemented segmentation method of the three-dimensional digital model of teeth and jaws according to claim 1, wherein the offset sub-network is a recurrent neural network based on a shared fully-connected layer.
The computer-implemented method for segmenting a three-dimensional digital model of teeth and jaws according to claim 1, wherein the coarse prediction sub-network generates the rough forecast results.
The computer-implemented method for segmenting a three-dimensional digital model of teeth and jaws according to claim 1, wherein the clustering operation adopts an algorithm based on density clustering.
The segmentation method of the three-dimensional digital model of teeth and jaws as claimed in claim 1, characterized in that, it further comprises:

A second segmentation result is obtained by weighted calculation based on the rough prediction result and the clustering result; and a Markov random field is constructed by using the second segmentation result, and the first segmentation result is obtained by using a graph cut algorithm.
The segmentation method of the three-dimensional digital model of teeth and jaws according to claim 1, characterized in that, it further comprises:

Obtain the 3D digital model of the second jaw;

Simplifying the three-dimensional digital model of the second jaw to obtain the three-dimensional digital model of the first jaw; and mapping the first segmentation result back to the three-dimensional digital model of the second jaw to obtain a third segmentation result.
The method for segmenting a three-dimensional digital model of a tooth and jaw according to claim 11, further comprising: using a fuzzy clustering algorithm and a shortest path algorithm to optimize and smooth the third segmentation result.
The segmentation method of the three-dimensional digital model of teeth and jaws according to claim 12, wherein the fuzzy clustering algorithm considers the area of the face.