WO2019214339A1 - 牙齿三维数字模型的局部坐标系设定方法 - Google Patents

牙齿三维数字模型的局部坐标系设定方法 Download PDF

Info

Publication number
WO2019214339A1
WO2019214339A1 PCT/CN2019/077805 CN2019077805W WO2019214339A1 WO 2019214339 A1 WO2019214339 A1 WO 2019214339A1 CN 2019077805 W CN2019077805 W CN 2019077805W WO 2019214339 A1 WO2019214339 A1 WO 2019214339A1
Authority
WO
WIPO (PCT)
Prior art keywords
coordinate system
axis
local coordinate
dimensional digital
digital model
Prior art date
Application number
PCT/CN2019/077805
Other languages
English (en)
French (fr)
Inventor
冯洋
刘晓林
周博文
Original Assignee
无锡时代天使医疗器械科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 无锡时代天使医疗器械科技有限公司 filed Critical 无锡时代天使医疗器械科技有限公司
Priority to US16/967,671 priority Critical patent/US11694397B2/en
Publication of WO2019214339A1 publication Critical patent/WO2019214339A1/zh

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61C—DENTISTRY; APPARATUS OR METHODS FOR ORAL OR DENTAL HYGIENE
    • A61C7/00—Orthodontics, i.e. obtaining or maintaining the desired position of teeth, e.g. by straightening, evening, regulating, separating, or by correcting malocclusions
    • A61C7/002—Orthodontic computer assisted systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/005—Tree description, e.g. octree, quadtree
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/50—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for simulation or modelling of medical disorders
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61C—DENTISTRY; APPARATUS OR METHODS FOR ORAL OR DENTAL HYGIENE
    • A61C7/00—Orthodontics, i.e. obtaining or maintaining the desired position of teeth, e.g. by straightening, evening, regulating, separating, or by correcting malocclusions
    • A61C7/002—Orthodontic computer assisted systems
    • A61C2007/004—Automatic construction of a set of axes for a tooth or a plurality of teeth
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS

Definitions

  • the present application generally relates to a local coordinate system setting method for a three-dimensional digital model of a tooth.
  • a three-dimensional digital model of the teeth is commonly used.
  • a world coordinate system can be established and a local coordinate system can be established for each tooth.
  • the world coordinate system and the local coordinate system of the tooth can be combined to indicate the orientation of the tooth.
  • the setting of the local coordinate system is critical.
  • the local coordinate system is usually set manually, but there are several shortcomings in manually setting the local coordinate system.
  • each technician may have different understandings of the local coordinate system, so it is difficult to ensure the local coordinate system.
  • the consistency is determined.
  • it takes a lot of time and manpower to manually set the local coordinate system for each tooth.
  • the optimization of the local coordinate system is equivalent to re-manual setting, therefore, manually setting the local Coordinate system time and labor costs are higher.
  • An aspect of the present application provides a method for setting a local coordinate system of a three-dimensional digital model of a tooth performed by a computer, comprising: acquiring a first three-dimensional digital model, which is a three-dimensional digital model representing a first tooth based on a world coordinate system; Using the first artificial neural network, a local coordinate system is set based on the first three-dimensional digital model, wherein the first artificial neural network is a trained artificial neural network with deep learning capability.
  • the first artificial neural network can be a multilayer perceptron.
  • the method for setting a local coordinate system of the computer-based three-dimensional digital model may further include: obtaining, by using the first artificial neural network, a first prediction based on the first three-dimensional digital model a vector corresponding to a first coordinate axis of the local coordinate system, the first coordinate axis being one of a y-axis and a z-axis of the local coordinate system, the y-axis and the z-axis of the local coordinate system being divided The other one outside the first coordinate axis is a second coordinate axis; using a principal component analysis method, determining an x-axis of the local coordinate system based on the first three-dimensional digital model; based on the determined x-axis and a prediction vector, determining the second coordinate axis; and determining the first coordinate axis based on the determined x-axis and the second coordinate axis.
  • the principal component analysis method can be a normal-based principal component analysis method.
  • the method for setting a local coordinate system of the computer-based three-dimensional digital model may further include: obtaining, by using the second artificial neural network, a second prediction vector based on the first three-dimensional digital model, Wherein the second artificial neural network is a trained artificial neural network with deep learning capability for predicting an x-axis of a local coordinate system, the second prediction vector corresponding to an x-axis of the local coordinate system; Using the principal component analysis method, generating three feature vectors based on the first three-dimensional digital model; and selecting one of the three feature vectors that has the smallest angle with the straight line of the second prediction vector, and according to the The second prediction vector gives the selected symbol of the selected feature vector the correct sign as the x-axis of the local coordinate system.
  • the second coordinate axis is determined based on the determined x-axis and the first prediction vector using a cross multiplication; and using a cross multiplication, based on the determined x-axis and the second coordinate axis, The first coordinate axis is determined.
  • the first coordinate axis of the local coordinate system can be the z-axis.
  • the method for setting a local coordinate system of the computer-based three-dimensional digital model may further include: simplifying the first three-dimensional digital model such that the number of vertices is equal to a predetermined N, and obtaining the first a digital data set using the first artificial neural network to obtain the first prediction vector based on the first digital data set, wherein the N is a natural number.
  • the method for setting a local coordinate system of the computer-based three-dimensional digital model may further include: centralizing the simplified data set to obtain a second digital data set, An artificial neural network, based on the second digital data set, obtaining the first prediction vector, and using the principal component analysis method, determining an x-axis of the local coordinate system based on the second digital data set.
  • the method for setting a local coordinate system of the computer-based three-dimensional digital model may further include: normalizing the second digital data set to obtain a third digital data set, and utilizing The first artificial neural network obtains the first prediction vector based on the third digital data set.
  • the output layer of the first artificial neural network includes an EuclideanLoss cost function for training the various layer parameters by backpropagation.
  • the method for setting a local coordinate system of the computer-based dental three-dimensional digital model may further include: selecting the first one from the plurality of artificial neural networks according to the type of the first tooth An artificial neural network, wherein the plurality of artificial neural networks are trained artificial neural networks with deep learning capabilities for respectively setting local coordinate systems for different types of teeth.
  • the first artificial neural network can be one of: a multilayer perceptron, an octree-based convolutional neural network, a convolutional neural network, a recurrent neural network, reinforcement learning, and a generation confrontation network.
  • the first artificial neural network is trained by manually calibrating a plurality of three-dimensional digital models of teeth in a local coordinate system, wherein the plurality of three-dimensional digital models of teeth are of the same type as the first teeth 3D digital model of the teeth.
  • FIG. 1 is a schematic flowchart of a method for setting a local coordinate system of a three-dimensional digital model of a tooth implemented by a computer according to an embodiment of the present application;
  • FIG. 2 is a schematic diagram showing the structure of a multilayer perceptron artificial neural network in one embodiment of the present application
  • Figure 3 schematically shows the quadrant distribution of the teeth
  • FIG. 4 is a schematic flowchart of a method for setting a local coordinate system of a computer-implemented three-dimensional digital model of teeth in an embodiment of the present application
  • Figure 5 schematically illustrates the structure of an O-CNN network in one embodiment of the present application.
  • the inventor of the present application developed a computer-based local coordinate system setting method for a three-dimensional digital model of teeth based on deep learning, using a trained artificial neural network with deep learning ability to set teeth.
  • the local coordinate system of the 3D digital model was developed.
  • FIG. 1 is a schematic flowchart of a method 100 for setting a local coordinate system of a three-dimensional digital model of a tooth performed by a computer in an embodiment of the present application.
  • the method 100 of setting a local coordinate system of a three-dimensional digital model of a tooth performed by a computer is based on a Multi-Layer Perceptron (MLP).
  • MLP Multi-Layer Perceptron
  • an artificial neural network may be established for each type of tooth, for example, an artificial neural network is established for each of the 1st to 7th teeth, respectively for corresponding Types of teeth set the local coordinate system.
  • a first three-dimensional digital model is acquired.
  • the first three-dimensional digital model is a three-dimensional digital model representing the first tooth based on the world coordinate system, and the coordinate values of the vertices thereof are coordinate values in the world coordinate system.
  • the world coordinate system can be set manually.
  • the vector perpendicular to the occlusal plane can be used as the Z axis of the world coordinate system, and the cusp of the two 6th teeth can be connected as the direction of the X coordinate of the world coordinate system, and then the Y axis can be determined based on the two coordinate axes.
  • the patient's jaw can be directly scanned to obtain a three-dimensional digital model representing the patient's dentition.
  • a solid model of the patient's jaw such as a plaster cast, can be scanned to obtain a three-dimensional digital model representing the patient's dentition.
  • the impression of the patient's jaw can be scanned to obtain a three-dimensional digital model representing the patient's dentition.
  • a three-dimensional digital model representing each tooth is obtained by dividing a three-dimensional digital model representing the patient's dentition.
  • a three-dimensional digital model of the tooth can be constructed based on a triangular mesh, and such a three-dimensional digital model is exemplified below. It can be understood that the three-dimensional digital model of the teeth can also be constructed based on other types of meshes, for example, a quadrilateral mesh, a pentagonal mesh, a hexagonal mesh, etc., which will not be described again here.
  • the first three-dimensional digital model is simplified to obtain a first digital data set.
  • the number of vertices/facets of the unprocessed original tooth three-dimensional digital model may vary, which is not conducive to training the artificial neural network, and is not conducive to setting the local coordinate system with the trained artificial neural network. Therefore, before the 3D digital model is input into the artificial neural network, it can be simplified first, so that the number of vertices/patch is equal to the predetermined value N.
  • the predetermined value can be set to 2048 or 1024, that is, the number of vertices of the simplified three-dimensional digital model is 2048 or 1024. It can be understood that the predetermined value can also be set to other numbers according to specific situations and needs.
  • the first digital data set represents a simplified three-dimensional digital model.
  • the first three-dimensional digital model may be simplified using a Quadric Error Metrics with a quadratic error as a metric.
  • the Q matrix can be calculated according to equation (1):
  • the merge error can be calculated for each pair of adjacent vertex combinations, then the vertex combination that selects the smallest error is iterated to shrink, and the errors for all associated edges are updated. Based on the Q matrix calculation, a simplified set of vertices can be obtained.
  • the number of vertices of the simplified three-dimensional digital model does not necessarily equal the predetermined value, and at this time, the number of vertices of the first digital data set may be equal to the predetermined value by adding or deleting vertices.
  • the first three-dimensional digital model can be simplified using any other suitable algorithm in addition to the edge contraction algorithm with quadratic error as a measure.
  • the first digital data set is centered to obtain a second digital data set.
  • the coordinates of the vertices in the first digital data set may be averaged to obtain the coordinates of the center point, and then the coordinate of the center point is subtracted from the coordinate of each vertex to obtain a second digital data set.
  • the second digital data set is normalized to obtain a third digital data set.
  • the second digital data set can be separately normalized in each dimension to obtain a third digital data set.
  • normalization on the X-axis of the world coordinate system can be performed according to the following equation (6):
  • m-data(X) represents the normalized data on the X-axis of the world coordinate system
  • min(data(X)) represents the minimum X coordinate value of the vertex
  • max(data(X)) represents the maximum X of the vertex Coordinate value
  • the normalized data can be saved in the hdf5 format (Hierarchical Data Format version 5.0), that is, the third digital data set is in the hdf5 format.
  • hdf5 format Hierarchical Data Format version 5.0
  • the normalization of the Y-axis and Z-axis components of the world coordinate system can also be performed by using min(data(X)) and max(data(X)), that is, performing homogeneity normalization. It is proved that the effect is better, and will not be described here.
  • homomorphic normalization can also use min(data(Y)) and max(data(Y)), or min(data(Z)) and max(data(Z). ).
  • a first prediction vector corresponding to the z-axis of the local coordinate system is obtained.
  • a Multi-Layer Perceptron may be employed to predict the first prediction vector based on the third digital data set.
  • the structure of the multilayer perceptron artificial neural network 200 in one embodiment of the present application is schematically illustrated.
  • multi-layer perceptron 200 includes an input layer 201, six fully connected layers 203, 205, 207, 211, 215, and 217, two Dropout layers 209 and 213, and an output layer 219.
  • the ReLink activation function is included in the fully connected layers 203, 205, and 207 to achieve non-linearity and avoid gradient dispersion, providing inclusiveness to the depth and breadth of the model.
  • the all-connected layer enhances the model's ability to fit through linear and nonlinear changes while preserving all input information, extracting effective features.
  • Dropout layers 209 and 213 can enhance model generalization and reduce overfitting.
  • the multilayer perceptron can be trained with a three-dimensional digital model of the tooth that manually calibrates the local coordinate system.
  • a multi-layer perceptron artificial neural network can be established for different teeth and separately trained to improve the accuracy of the prediction.
  • the label item (ie, the direction of the artificial neural network learning) can be set to the z-axis of the local coordinate system during training. It can be understood that the label item can also be set to the y-axis of the local coordinate system.
  • the output layer 219 includes an EuclideanLoss cost function, which represents the root mean square error of the prediction result and the labeling result, thereby training the parameters of each layer of the MLP through Back Propagation (BP).
  • the mathematical expression of the EuclideanLoss cost function is:
  • N represents the amount of data used for training (ie, the number of three-dimensional digital models of the type of teeth for manual calibration of the local coordinate system for training)
  • label i indicates the label item of the i-th training data
  • predicition i indicates the artificial nerve
  • the network predicts the terms based on the i-th training data.
  • the backpropagation algorithm is a learning algorithm suitable for multi-layer neural networks under supervised conditions. It is based on the gradient descent method.
  • the backpropagation algorithm is iteratively iteratively looped through two links (excitation propagation and weight update) until the network's response to the input reaches a predetermined target range.
  • the learning process of the backpropagation algorithm consists of a forward propagation process and a back propagation process. In the forward propagation process, the input information is processed through the hidden layer through the input layer and processed layer by layer and transmitted to the output layer. If the expected output value is not obtained in the output layer, the cost function is used as the objective function, and the backward propagation is performed.
  • the partial derivative of the objective function to the weight of each neuron is obtained layer by layer, and the ladder of the objective function is added to the weight vector. Quantity, as the basis for modifying the weight.
  • the learning of the network is completed during the weight modification process. When the error reaches the desired value, the network learning ends.
  • the first prediction vector of the MLP output may be three components on the X, Y, and Z axes of the world coordinate system.
  • the label item is the z-axis of the local coordinate system
  • the first prediction vector is the z-axis of the local coordinate system predicted by the MLP.
  • the training process can be implemented in ubuntu system, python combined with caffe, and the training parameters and their settings are as follows:
  • Stepsize 20000 (or 40000)
  • the learning rate base_lr and stepsize can be adjusted according to the training feedback to get a better model.
  • the x-axis of the local coordinate system is determined based on the second digital data set using a normal-based principal component analysis algorithm.
  • the principal component analysis algorithm based on normal direction is insensitive to point sequence change, translation, rotation and other transformations, and can strongly describe the representation of a patch data in a three-dimensional coordinate system, which reduces the trouble in the matching problem.
  • the inventors of the present application have found through a large number of experiments that the NPCA algorithm is used to analyze the tooth data, and the obtained feature vector has a very high degree of coincidence with the x-axis direction of the local coordinate system. In the case of about 99%, both of them The angle is less than 1 degree. Therefore, one of the three feature vectors obtained by processing the second digital data set by using the NPCA algorithm can be selected according to the x-axis spatial condition of the local coordinate system, and the correct symbol is given as the local coordinate system. x axis.
  • the following describes how to determine the x-axis of the local coordinate system by using the NPCA algorithm.
  • the coordinate axes of the world coordinate system are represented by X, Y, and Z, and the coordinate axes of the local coordinate system are represented by x, y, and z.
  • the x-axis of the local coordinate system can be directly determined using the NPCA algorithm.
  • E represents the sum of the surface areas of all triangular faces.
  • the vector z' corresponding to the z-axis of the local coordinate system having the smallest angle can be selected from the feature vectors.
  • the angle between the feature vector and the -Z vector is calculated; for the third or fourth quadrant teeth, the angle with the Z-axis vector is calculated.
  • This view is the projection of the upper and lower jaw teeth from the back to the front.
  • the first quadrant 1 and the second quadrant 2 are the upper jaw teeth
  • the third quadrant 3 and the fourth quadrant 4 are the lower jaw teeth.
  • the first digit of each tooth number is the number of the quadrant
  • the second digit is the number of the tooth.
  • the number 1 on the left side of the upper jaw is numbered 11, and so on.
  • one of the remaining two feature vectors is selected as x', and the corresponding symbol is given as the x-axis of the local coordinate system.
  • which of the remaining two feature vectors is x' can be determined based on the following method.
  • a pseudo coordinate system x 1 y 1 z 1 can be constructed using x', y', z'.
  • the right-hand rule which are [x', y'] respectively (the first vector of the remaining two feature vectors is x 1 and the second vector is y 1 ) , [y', -x'] (the first of the remaining two feature vectors is y 1 , the second vector is -x 1 ), [-x', -y'] (the remaining two feature vectors will be The first vector in the vector is -x 1 , the second vector is -y 1 ) and [-y',x'] (the first vector of the remaining two feature vectors is taken as -y 1 and the second vector is taken as x 1 ).
  • the following rules can be used according to the positional relationship with the X and Y axes of the world coordinate system and the shape difference of the teeth themselves
  • Rule 2 For the teeth of the first and second quadrants, it can be specified that the component of x' on the Y-axis of the world coordinate system is less than zero, and the component of y' on the X-axis of the world coordinate system is less than zero; for the third and fourth quadrants The teeth can specify that the component of x' on the Y-axis of the world coordinate system is less than zero, and the component of y' on the X-axis of the world coordinate system is greater than zero.
  • Rule 3 For the teeth of the first and fourth quadrants, it can be specified that when x' component on the X-axis of the world coordinate system is less than zero, the component of x' on the Y-axis of the world coordinate system is less than about -sin (pi/6).
  • the value is an empirical value obtained by the inventor of the present application based on a large number of experiments; when the component of x' on the X-axis of the world coordinate system is greater than zero, the component of x' on the Y-axis of the world coordinate system is smaller than About -sin(pi/3), where the value is an empirical value obtained by the inventors of the present application based on a large number of experiments.
  • the component of x' on the Y-axis of the world coordinate system is less than about -sin(pi/3), when x 'When the component on the X-axis of the world coordinate system is greater than zero, the component of x' on the Y-axis of the world coordinate system is less than about -sin(pi/6).
  • the midpoint of the maximum and minimum values in the z' direction is the dividing point, and the upper part of the crown is retained.
  • the NPCA analysis is performed again on the upper crown, in which case the vector probability of the first principal component is substantially parallel to the line of the local coordinate system x-axis. Therefore, the vector having the smallest angle between the feature vector of the complete crown NPCA and the line of the first principal component vector of the upper crown NPCA can be taken as x'. Then, the above four cases can be reduced to two. Then, according to Rule 1 and Rule 2, the direction of the x-axis of the local coordinate system is finally determined.
  • x' and y' can be distinguished based on the absolute value of the component on the X-axis of the world coordinate system.
  • the absolute value is x'
  • the absolute value is y'.
  • the judgment method can replace the rule. two.
  • intersection sets A and B are obtained, respectively.
  • [x', y'] is more suitable if there are many cases where the intersection set A is concave
  • [y', -x'] is more suitable if the intersection set B is concave.
  • the component of y' on the Y-axis of the world coordinate system is greater than -sqrt(3)/2, wherein the value is an empirical value obtained by the inventors of the present application based on a large number of experiments;
  • the component of y' on the Y-axis of the world coordinate system is smaller than sqrt(3)/2.
  • the combination of the local coordinate system y axis and the world coordinate system X axis can be selected to be the smallest.
  • the NPCA and MLP can also be combined to determine the x-axis of the local coordinate system.
  • 28 MLP networks can be established and trained for each of the teeth for the x-axis of the local coordinate system with reference to the above method.
  • the third digital data set is input to the corresponding MLP network to obtain a second prediction vector, that is, the x-axis of the local coordinate system predicted by the MLP network based on the third digital data set.
  • the second digital data set is processed by NPCA, and three eigenvectors x', y', and z' respectively correspond to the x, y, and z axes of the local coordinate system, but at this time, the three eigenvectors cannot be distinguished.
  • the smallest angle between the three feature vectors and the line of the Z coordinate of the world coordinate system is selected as z'.
  • the second prediction vector is compared with the line where the remaining two feature vectors are located, and the feature vector with the smallest angle is selected, and the correct symbol is given as the x-axis of the local coordinate system. For example, when the angle between the selected feature vector and the second prediction vector is less than 90 degrees, the "+" sign is given to indicate that the direction is not changed; when the angle between the selected feature vector and the second prediction vector is greater than 90 Degree is given its "-" sign, indicating that its direction is reversed.
  • the y-axis and the z-axis of the local coordinate system are determined based on the determined local coordinate system x-axis and the first prediction vector.
  • the determined local coordinate system x-axis can be cross-multiplied with the first prediction vector to obtain a local coordinate system y-axis. Then, the x-axis and the y-axis of the determined local coordinate system are multiplied to obtain the corrected local coordinate system z-axis.
  • the coordinate center of the local coordinate system may be the center of the cavity line. At this point, the local coordinate system is set.
  • FIG. 4 is a schematic flowchart of a method 400 for setting a local coordinate system of a computer-implemented three-dimensional digital model of a tooth in an embodiment of the present application.
  • the method 400 for setting the local coordinate system of the computer-implemented three-dimensional digital model of the tooth is to use an Octree-Based Convolutional Neural Networks (O-CNN).
  • O-CNN Octree-Based Convolutional Neural Networks
  • a first three-dimensional digital model is acquired.
  • the first three-dimensional digital model is a three-dimensional digital model representing the first tooth based on the world coordinate system, that is, the coordinate values of the vertices thereof are coordinate values in the world coordinate system.
  • the first three-dimensional digital model is refined to obtain a second three-dimensional digital model.
  • the first three-dimensional model may be refined and upsampled to obtain a second three-dimensional digital model.
  • the specific operation is as follows.
  • the first three-dimensional digital model may be upsampled 4 times to obtain a second three-dimensional digital model.
  • the second three-dimensional digital model is subjected to octree voxelization to obtain a volume data set.
  • the second three-dimensional digital model can be placed in a unit bounding box (all length, width, and height are 1), and then the unit bounding box is recursively divided into eight sub-bounding boxes in a breadth-first order. The recursion is repeated until the preset octree depth d is satisfied.
  • some of the information needed in the O-CNN transformation can be gathered after voxelization.
  • the shuffle key defines the relative position of the current child node under the parent node, which is a convolution calculation service.
  • the O-label defines who the parent node of the current child node is, and the first few non-empty child nodes under the parent node. This process serves the pooling operation.
  • a third prediction vector and a fourth prediction vector corresponding to two coordinate axes of the corresponding local coordinate system are predicted based on the volume data set.
  • the O-CNN network when training the O-CNN network, can be separately established and trained for the three coordinate axes of the local coordinate system, and according to the results in the training process, the two O- predictions with the most accurate prediction are selected.
  • the CNN network (corresponding to two of the coordinate axes of the local coordinate system, respectively) is used to set the local coordinate system.
  • the refined three-dimensional digital model for training can be rotated around the three axes of the local coordinate system in steps of 15 degrees [- 45,45] degrees, the data can therefore be increased by 7 3 times.
  • O-CNN Octree-Based Convolutional Neural Networks for 3D Shape Analysis
  • FIG. 5 the structure of an O-CNN network 500 in one embodiment of the present application is schematically illustrated.
  • the O-CNN network 500 includes an input layer 501, O-CNN[d] 503, 505, and 507, Dropout layers 509 and 513, fully connected layers 511 and 515, and an output layer 517.
  • each of O-CNN[d] 503, 505, and 507 includes a convolution layer, a batch normalization layer, an activation function ReLU, and a pooling layer.
  • the convolutional layer computes local features by convolving data in a small range.
  • the pooling layer achieves invariance such as rotational translation by upsampling.
  • the batch normalization layer solves the problem that the output distribution of each layer of data is not uniform to a certain extent, and has the advantages of being able to use a higher learning rate, avoiding over-fitting, and reasonably avoiding gradient saturation.
  • the input data value is the average normal of all vertices in the bounding box where each leaf node is located. If the leaf node is empty, set the normal to 0.
  • three coordinate axes of the local coordinate system are determined based on the third prediction vector and the fourth prediction vector.
  • one of the third prediction vector and the fourth prediction vector may be determined as the first coordinate axis of the local coordinate system based on the magnitude of the deviation predicted by the O-CNN network. Then, the third prediction vector and the fourth prediction vector are cross-multiplied to obtain a second coordinate axis of the local coordinate system. Finally, the first coordinate axis and the second coordinate axis are multiplied to obtain a third coordinate axis of the local coordinate system.
  • the third prediction vector corresponds to the x-axis of the local coordinate system
  • the fourth prediction vector corresponds to the z-axis of the local coordinate system
  • the third The prediction vector is taken as the x-axis of the local coordinate system.
  • the third prediction vector and the fourth prediction vector are cross-multiplied to obtain the y-axis of the local coordinate system.
  • the x-axis and the y-axis are multiplied to obtain the z-axis of the local coordinate system.
  • CNN Convolutional Neural Networks
  • RNN Recurrent Neural Networks
  • RL Reinforcement Learning
  • GANs Generative Adversarial Networks
  • various diagrams may illustrate exemplary architectures or other configurations of the disclosed methods and systems that facilitate understanding of features and functionality that may be included in the disclosed methods and systems.
  • the claimed content is not limited to the exemplary architecture or configuration shown, and the desired features can be implemented in various alternative architectures and configurations.
  • the order of the blocks presented herein should not be limited to the various embodiments that are implemented in the same order to perform the described functions, unless clearly indicated in the context .

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • Data Mining & Analysis (AREA)
  • Public Health (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Epidemiology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Dentistry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Veterinary Medicine (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Pathology (AREA)
  • Primary Health Care (AREA)
  • Image Analysis (AREA)

Abstract

一种计算机执行的牙齿三维数字模型的局部坐标系的设定方法，包括：获取第一三维数字模型，它是基于世界坐标系表示第一牙齿的三维数字模型；以及利用第一人工神经网络，基于所述第一三维数字模型，为其设定局部坐标系，其中，所述第一人工神经网络是经训练的具有深度学习能力的人工神经网络。

Description

牙齿三维数字模型的局部坐标系设定方法 技术领域
本申请总体上涉及牙齿三维数字模型的局部坐标系设定方法。
背景技术
随着计算机科学的不断发展，牙科专业人员越来越多地借助计算机技术来提高牙科诊疗的效率。
在借助计算机的牙科诊疗中，常用到牙齿的三维数字模型。为了便于处理和计算，可以建立一个世界坐标系并且为每一颗牙齿建立一个局部坐标系，对于每一颗牙齿，可以结合所述世界坐标系以及该牙齿的局部坐标系来表示该牙齿的方位。在一些借助计算机的牙科诊疗中，例如，借助计算机的牙科正畸方案制定，局部坐标系的设定非常关键。
当前，局部坐标系通常是人工设定的，但人工设定局部坐标系存在以下几点不足之处：第一，每个技术人员对局部坐标系的认识可能不同，故难以保证局部坐标系设定的一致性；第二，分别为每一颗牙齿人工设定局部坐标系会花费大量时间和人力，另外，对局部坐标系的优化相当于重新进行一次人工设定，因此，人工设定局部坐标系时间和人力成本较高。
鉴于以上，有必要提供一种新的局部坐标系设定方法。
发明内容
本申请的一方面提供了一种计算机执行的牙齿三维数字模型的局部坐标系的设定方法，包括：获取第一三维数字模型，它是基于世界坐标系表示第一牙齿的三维数字模型；以及利用第一人工神经网络，基于所述第一三维数字模型，为 其设定局部坐标系，其中，所述第一人工神经网络是经训练的具有深度学习能力的人工神经网络。
在一些实施方式中，所述第一人工神经网络可以是多层感知器。
在一些实施方式中，所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法还可以包括：利用所述第一人工神经网络，基于所述第一三维数字模型，获得第一预测向量，其与所述局部坐标系的第一坐标轴相对应，该第一坐标轴为所述局部坐标系的y轴和z轴之一，所述局部坐标系的y轴和z轴除所述第一坐标轴外的另一个为第二坐标轴；利用主成分分析法，基于所述第一三维数字模型，确定所述局部坐标系的x轴；基于所述已确定的x轴以及第一预测向量，确定所述第二坐标轴；以及基于所述已确定的x轴以及第二坐标轴，确定所述第一坐标轴。
在一些实施方式中，所述主成分分析法可以是基于法向的主成分分析法。
在一些实施方式中，所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法还可以包括：利用第二人工神经网络，基于所述第一三维数字模型，获得第二预测向量，其中，所述第二人工神经网络是经训练的具有深度学习能力的人工神经网络，用于预测局部坐标系的x轴，所述第二预测向量与所述局部坐标系的x轴相对应；利用所述主成分分析法，基于所述第一三维数字模型产生三个特征向量；以及从所述三个特征向量中选取与所述第二预测向量所在直线夹角最小的一个，并根据所述第二预测向量赋予所述被选中的特征向量正确的符号，作为所述局部坐标系的x轴。
在一些实施方式中，利用叉乘，基于所述已确定的x轴与第一预测向量确定所述第二坐标轴；以及利用叉乘，基于所述已确定的x轴以及第二坐标轴，确定所述第一坐标轴。
在一些实施方式中，所述局部坐标系的第一坐标轴可以是z轴。
在一些实施方式中，所述的基于计算机的牙齿三维数字模型的局部坐标系的 设定方法还可以包括：对所述第一三维数字模型进行简化，使其顶点数等于预定的N，获得第一数字数据集，利用所述第一人工神经网络，基于该第一数字数据集，获得所述第一预测向量，其中，所述N是自然数。
在一些实施方式中，所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法还可以包括：对所述简化后数据集进行中心化，获得第二数字数据集，利用所述第一人工神经网络，基于该第二数字数据集，获得所述第一预测向量，利用所述主成分分析法，基于所述第二数字数据集，确定所述局部坐标系的x轴。
在一些实施方式中，所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法还可以包括：对所述第二数字数据集进行归一化处理，获得第三数字数据集，利用所述第一人工神经网络，基于该第三数字数据集，获得所述第一预测向量。
在一些实施方式中，所述第一人工神经网络的输出层包括EuclideanLoss代价函数，用于通过反向传播训练各层参数。
在一些实施方式中，所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法还可以包括：根据所述第一牙齿的类型，从多个人工神经网络中选定所述第一人工神经网络，其中，所述多个人工神经网络是经训练的具有深度学习能力的人工神经网络，分别用于为不同类型的牙齿设定局部坐标系。
在一些实施方式中，所述第一人工神经网络可以是以下之一：多层感知器、基于八叉树的卷积神经网络、卷积神经网络、递归神经网络、强化学习以及生成对抗网络。
在一些实施方式中，所述第一人工神经网络是以手工标定局部坐标系的多个牙齿三维数字模型进行训练，其中，所述多个牙齿三维数字模型均是与所述第一牙齿相同类型的牙齿的三维数字模型。
附图说明
以下将结合附图及其详细描述对本申请的上述及其他特征作进一步说明。应当理解的是，这些附图仅示出了根据本申请的若干示例性的实施方式，因此不应被视为是对本申请保护范围的限制。除非特别指出，附图不必是成比例的，并且其中类似的标号表示类似的部件。
图1为本申请一个实施例中计算机实施的牙齿三维数字模型的局部坐标系的设定方法的示意性流程图；
图2示意性地展示了本申请一个实施例中多层感知器人工神经网络的结构；
图3示意性地展示了牙齿的象限分配；
图4为本申请一个实施例中的计算机实施的牙齿三维数字模型的局部坐标系的设定方法的示意性流程图；
图5示意性地展示了本申请一个实施例中的O-CNN网络的结构。
具体实施方式
以下的详细描述引用了构成本说明书一部分的附图。说明书和附图所提及的示意性实施方式仅仅是出于说明性之目的，并非意图限制本申请的保护范围。在本申请的启示下，本领域技术人员能够理解，可以采用许多其他实施方式，并且可以对所描述实施方式做出各种改变，而不背离本申请的主旨和保护范围。应当理解的是，在此说明并图示的本申请的各个方面可以按照很多不同的配置来布置、替换、组合、分离和设计，这些不同配置都在本申请的保护范围之内。
经过大量的研发工作，本申请的发明人开发出了一种计算机执行的基于深度学习的牙齿三维数字模型的局部坐标系设定方法，利用经训练的具有深度学习能力的人工神经网络设定牙齿三维数字模型的局部坐标系。
请参图1，为本申请一个实施例中的计算机执行的牙齿三维数字模型的局部坐标系的设定方法100的示意性流程图。
在一个实施例中，计算机执行的牙齿三维数字模型的局部坐标系的设定方法100是基于多层感知器(Multi-Layer Perceptron，简称MLP)。
在一个实施例中，为了提高人工神经网络的预测精度，可以为每一种类型的牙齿分别建立一个人工神经网络，例如，为1至7号牙分别建立一个人工神经网络，分别用于为相应类型的牙齿设定局部坐标系。
在101中，获取第一三维数字模型。
第一三维数字模型是基于世界坐标系的表示第一牙齿的三维数字模型，其各顶点的坐标值是所述世界坐标系下的坐标值。
在一个实施例中，世界坐标系可以手工设定。例如，可以将垂直于咬合面的向量作为世界坐标系的Z轴，将两颗6号牙的牙尖连线作为世界坐标系X轴的方向，然后基于该两坐标轴确定Y轴。
获取患者牙齿的数字三维模型的方法有多种。在一个实施例中，可以直接扫描患者的牙颌(上颌和/或下颌)，获得表示患者牙列的三维数字模型。在又一实施例中，可以扫描患者牙颌的实体模型，例如石膏模型，获得表示患者牙列的三维数字模型。在又一实施例中，可以扫描患者牙颌的印模，获得表示患者牙列的三维数字模型。将表示患者牙列的三维数字模型进行分割即可获得表示各牙齿的三维数字模型。
在一个实施例中，可以基于三角网格构建牙齿的三维数字模型，下面以此类三维数字模型为例进行说明。可以理解，还可以基于其他类型的网格构建牙齿的三维数字模型，例如，四边形网格、五边形网格、六边形网格等，此处不再进行一一说明。
在103中，对第一三维数字模型进行简化获得第一数字数据集。
未经处理的原始牙齿三维数字模型的顶点/面片数量可能各不相同，不利于对人工神经网络进行训练，也不利于用经训练的人工神经网络来设定其局部坐标系。因此，在把三维数字模型输入人工神经网络之前，可以先对其进行简化处理，使其顶点/面片数量等于预定的值N。在一个实施例中，可以把该预定的值设为2048或1024，即经简化处理后的三维数字模型的顶点数量为2048或1024。可以理解，该预定的值也可以根据具体情况和需求设置为其他数字。此处，第一数字数据集表示经简化后的三维数字模型。
在一个实施例中，可以利用以二次误差作为度量代价的边收缩算法(Quadric Error Metrics)对第一三维数字模型进行简化处理。
首先，对于第一三维数字模型的每个顶点，可以根据方程式(1)计算Q矩阵：
Q＝∑ p∈planes(v)K p                                   方程式(1)
其中，planes(v)表示原始顶点(未经简化的第一三维数字模型的顶点)相关平面的集合，K p由以下方程式(2)表达，
Figure PCTCN2019077805-appb-000001
其中，p由以下方程式(3)表达，
p＝[a b c d] T                                   方程式(3)
其中，p代表以下方程式(4)的平面方程的系数，
ax+by+cz+d＝0                                 方程式(4)
其中，a、b以及c满足以下条件，
a 2+b 2+c 2＝1                                     方程式(5)
在一个实施例中，可以针对每一对相邻的顶点组合计算合并误差，然后迭代选取最小误差的顶点组合进行收缩，并更新所有相关的边的误差。基于Q矩阵的计算，可以获得简化后的顶点集合。
经过简化后的三维数字模型的顶点数量不一定等于所述预定的值，此时，可以通过添加或删减顶点使得第一数字数据集的顶点数量等于所述预定的值。
在本申请的启发下，可以理解，除了以二次误差作为度量代价的边收缩算法之外，可以采用任何其他适用的算法对第一三维数字模型进行简化。
在105中，对第一数字数据集进行中心化处理获得第二数字数据集。
在一个实施例中，可以对第一数字数据集中的各顶点坐标求均值，获得中心点坐标，然后把各顶点坐标减去该中心点坐标得到第二数字数据集。
在107中，对第二数字数据集进行归一化处理获得第三数字数据集。
在一个实施例中，可以对第二数字数据集在各维度分别进行归一化，获得第三数字数据集。例如，在世界坐标系X轴上的归一化可以根据下列方程式(6)进行：
Figure PCTCN2019077805-appb-000002
其中，m-data(X)代表在世界坐标系X轴上归一化后的数据；min(data(X))代表顶点的最小X坐标值；max(data(X))代表顶点的最大X坐标值。
在一个实施例中，可以将归一化后的数据保存为hdf5格式(Hierarchical Data Format version 5.0)，即第三数字数据集为hdf5格式。
在一个实施例中，对于世界坐标系Y轴和Z轴分量的归一化也可以采用min(data(X))和max(data(X))，即进行各项同性归一化，经实验证明其效果较佳，此处不再赘述。在本申请的启发下，可以理解，各项同性归一化也可以采用min(data(Y))和max(data(Y))，或min(data(Z))和max(data(Z))。
在109中，利用经训练的具有深度学习能力的人工神经网络，基于第三数字数据集，得到对应局部坐标系z轴的第一预测向量。
在一个实施例中，可以采用多层感知器(Multi-Layer Perceptron，简称MLP)，基于第三数字数据集，预测得到第一预测向量。
请参图2，示意性地展示了本申请一个实施例中多层感知器人工神经网络200的结构。
在一个实施例中，多层感知器200包括输入层201，6个全连接层203、205、207、211、215以及217，2个Dropout层209和213，以及输出层219。
全连接层203、205以及207中包括ReLU激活函数，以实现非线性及避免梯度弥散，对模型的深度和广度提供了包容性。全连接层在保留所有输入信息的基础上，通过线性与非线性变化提升模型拟合能力，提取有效特征。
Dropout层209和213能够加强模型泛化性，减少过拟合产生。
【多层感知器的训练】
在一个实施例中，可以用手工标定局部坐标系的牙齿三维数字模型来训练多层感知器。
在一个实施例中，可以为不同的牙齿分别建立多层感知器人工神经网络，并分别进行训练，以提高预测的准确性。
在一个实施例中，在训练时，label项(即人工神经网络学习的方向)可以设为局部坐标系的z轴。可以理解，label项也可以设为局部坐标系的y轴。
输出层219中包括EuclideanLoss代价函数，表示预测结果与标记结果的均方根误差，以此通过反向传播(Back Propagation，简称BP)来训练MLP各层参数。EuclideanLoss代价函数的数学式表达为：
Figure PCTCN2019077805-appb-000003
其中，N代表训练采用的数据量(即用于训练的人工标定了局部坐标系的该类牙齿的三维数字模型的数量)，label i表示第i个训练数据的label项，predicition i表示人工神经网络基于第i个训练数据预测得到的项。
反向传播算法是在有监督的情况下，适合多层神经网络的一种学习算法。它建立在梯度下降法的基础上。反向传播算法主要由两个环节(激励传播和权重更新)反复循环迭代，直到网络对输入的响应达到预定的目标范围为止。反向传播算法的学习过程由正向传播过程和反向传播过程组成。在正向传播过程中，输入信息通过输入层经隐含层，逐层处理并传向输出层。如果在输出层得不到期望的输出值，以代价函数作为目标函数，转入反向传播，逐层求出目标函数对各神经元权值的偏导数，构成目标函数对权值向量的梯量，作为修改权值的依据。网络的学习在权值修改过程中完成。误差达到所期望值时，网络学习结束。
MLP输出的第一预测向量可以是在世界坐标系X、Y、Z轴上的三个分量。当label项为局部坐标系的z轴时，那么该第预测一向量就是MLP预测得到的局部坐标系的z轴。
在一个实施例中，训练过程可以在ubuntu系统下，python结合caffe来实现，训练参数及其设定值如下：
max_iter:40000
base_lr:0.001(或0.01)
Lr_policy:step
Stepsize＝20000(或40000)
Gamma:0.1
Momentum:0.9
Momentum2:0.99
Weight_decay＝0.005
Solver:SGDSolver
Weight_filler:Xavier
其中学习率base_lr及stepsize可根据训练反馈情况进行调整，以得到较佳模型。
在111中，利用基于法向的主成分分析算法，基于第二数字数据集，确定局部坐标系的x轴。
请参由Papadakis P、Pratikakis I、Perantonis S等在Pattern Recognition,2007,40(9):2437-2452上发表的“Efficient 3D Shape Matching and Retrieval Using a Concrete Radialized Spherical Projection Representation”，披露了一种基于法向的三维数据的主成分分析法(Normals Principal Component Analysis，简称NPCA)。
基于法向的主成分分析算法对于点序变化、平移、旋转等变换不敏感，能够强有力地描述一个面片数据在三维坐标系下的表示，使得在匹配问题上减少了许多麻烦。
本申请的发明人通过大量的实验发现，利用NPCA算法对牙齿数据进行分析，得到的某一特征向量与局部坐标系的x轴方向吻合度极高，在99％左右的情况下，两者的夹角小于1度。因此，可以从利用NPCA算法对第二数字数据集进行处理而获得的三个特征向量中符合局部坐标系x轴空间条件的那个特征向量挑选出，赋予其正确的符号后，作为局部坐标系的x轴。
下面对如何利用NPCA算法确定局部坐标系x轴的过程进行详细描述，其中，以X、Y、Z表示世界坐标系的坐标轴，以x、y、z表示局部坐标系的坐标轴。
在一个实施例中，可以利用NPCA算法直接确定局部坐标系的x轴。
【利用NPCA直接确定局部坐标系的x轴】
首先，计算每一个三角面片的法向量n i和表面积E i。
接着，根据以下方程式(7)计算协方差矩阵，
Figure PCTCN2019077805-appb-000004
其中，E表示所有三角面片的表面积总和。
然后，基于计算得到的协方差矩阵计算得到三个特征向量x’、y’以及z’，分别与局部坐标系的x、y及z轴相对应，在赋予x’正确的符号后，作为局部坐标系的x轴。
此时，还无法确定计算得到的三个特征向量中的哪一个与局部坐标系的x轴相对应。在一个实施例中，可以基于以下方法来确定。
首先，可以基于特征向量与世界坐标系Z轴所在直线的夹角，从特征向量中选出夹角最小的作为与局部坐标系z轴相对应的向量z’。对于第一或第二象限牙齿，计算特征向量与-Z向量的夹角；对于第三或第四象限牙齿，计算与Z轴向量的夹角。
请参图3，示意性地展示了牙齿的象限分配。
该视图是上、下颌牙齿沿从后往前方向的投影。其中，第一象限1和第二象限2中的为上颌牙齿，第三象限3和第四象限4中的为下颌牙齿。各牙齿的编号中第一位是象限的编号，第二位是牙齿的编号。例如，上颌左侧1号牙的编号为11，以此类推。
然后，从剩余的两个特征向量中选择其一为x’，再赋予其相应的符号，作为局部坐标系的x轴。在一个实施例中，可以基于以下方法来确定剩余的两个特征向量中哪一个是x’。
在一个实施例中，可以利用x’、y’、z’构建一个伪坐标系x 1y 1z 1。在考虑方向变化的情况下，有4种符合右手定则的情况，分别为[x’,y’](将剩余两个特征 向量中的第一向量作为x 1，第二向量作为y 1)、[y’,-x’](将剩余两个特征向量中的第一向量作为y 1，第二向量作为-x 1)、[-x’,-y’](将剩余两个特征向量中的第一向量作为-x 1，第二向量作为-y 1)以及[-y’,x’](将剩余两个特征向量中的第一向量作为-y 1，第二向量作为x 1)。对于1～6号牙，可以根据与世界坐标系X、Y轴的位置关系以及牙齿本身的形状差异，基于以下规则进行处理。
规则一：对于第一、第四象限的牙齿，可以规定x’在世界坐标系X轴上的分量小于零；对于第二、第三象限的牙齿，可以规定x’在世界坐标系X轴上的分量大于零。
规则二：对于第一、第二象限的牙齿，可以规定x’在世界坐标系Y轴上的分量小于零，y’在世界坐标系X轴上的分量小于零；对于第三、第四象限的牙齿，可以规定x’在世界坐标系Y轴上的分量小于零，y’在世界坐标系X轴上的分量大于零。
规则三：对于第一、第四象限的牙齿，可以规定当x’在世界坐标系X轴上的分量小于零时，x’在世界坐标系Y轴上的分量小于约-sin(pi/6)，其中，该值是本申请的发明人基于大量实验而得出的经验值；当x’在世界坐标系X轴上的分量大于零时，x’在世界坐标系Y轴上的分量小于约-sin(pi/3)，其中，该值是本申请的发明人基于大量实验而得出的经验值。对于第二、第三象限牙齿，可以规定当x’在世界坐标系X轴上的分量小于零时，x’在世界坐标系Y轴上的分量小于约-sin(pi/3)，当x’在世界坐标系X轴上的分量大于零时，x’在世界坐标系Y轴上的分量小于约-sin(pi/6)。
由于形态的特殊性，对于1～3号牙齿，可以z’方向上最大最小值中点为分割点，保留牙冠上部。然后，对上部牙冠再次进行NPCA分析，在这种情况下，第一主成分的向量大概率与局部坐标系x轴所在直线基本平行。因此，可以将完整牙冠NPCA的特征向量中与上部牙冠NPCA的第一主成分向量所在直线夹角最小的向量作为x’。那么，上述的4种情况就可以被简化为2种。然后，再依据规则一和规则二来最终确定局部坐标系x轴的方向。
对于中切牙，可以基于在世界坐标系X轴上的分量的绝对值大小来区分x’和y’，绝对值大的为x’，绝对值小的为y’，该判断方法可以替代规则二。
对于4～6号牙，在经过基于规则三的判断之后，有可能出现多种情况均符合要求，例如，[x’,y’]、[y’,-x’]。
医学规则规定局部坐标系的x轴应与中央沟平行。那么可以基于以下规则进行进一步判断。
分别用10组垂直于局部坐标系x轴的平面与牙齿模型求交，得到不同x值下的10组交线。在这两种情况下(即[x’,y’]和[y’,-x’])，分别获得交线集合A和B。若交线集合A凹的情况较多，则[x’,y’]更加合适；若交线集合B凹的情况较多，则[y’,-x’]更加合适。为了判断交线是否为凹，可以用z＝a(a为10个在曲线z值范围内的等间距的值)的直线与交线求交，若交点数大于2，则判断该曲线为凹。比较A、B情况下，基于交线为凹的数量，最终确定局部坐标系的x轴。
对于7号牙，可以根据以下规则四进行判断。对于第一、第三象限的牙齿，y’在世界坐标系Y轴上的分量大于-sqrt(3)/2，其中，该值是本申请的发明人基于大量实验而得出的经验值；对于第二、第四象限的牙齿，y’在世界坐标系Y轴上的分量小于sqrt(3)/2。
若经过规则四和规则二后，依旧有一个以上的情况满足规则，那么可以选择局部坐标系y轴与世界坐标系X轴夹角最小的组合。
若基于以上依旧无法判断对应关系，那么随机输出结果，这就是1％误差的来源。
在又一实施例中，还可以结合NPCA和MLP来确定局部坐标系的x轴。
【结合NPCA和MLP确定局部坐标系的x轴】
在一个实施例中，可以参照以上的方法，针对局部坐标系的x轴，建立并训练28个MLP网络，分别对应每一颗牙齿。
然后，将第三数字数据集输入对应的MLP网络，获得第二预测向量，即该MLP网络基于第三数字数据集预测得到的局部坐标系的x轴。
接着，以NPCA处理第二数字数据集，得到三个特征向量x’、y’、z’分别与局部坐标系的x、y、z轴相对应，但此时还无法区分三个特征向量与局部坐标系的x、y、z轴的对应关系。
对于不同象限的牙齿，挑选出三个特征向量中与世界坐标系的Z轴所在直线夹角最小的作为z’。
将第二预测向量与剩下两个特征向量所在直线进行比较，选取夹角最小的那个特征向量，并赋予其正确的符号，作为局部坐标系的x轴。例如，当被选中的特征向量与第二预测向量的夹角小于90度，则赋予其“+”号，表示不改变其方向；当被选中的特征向量与第二预测向量的夹角大于90度，则赋予其“-”号，表示反转其方向。
在113中，基于已确定的局部坐标系x轴以及第一预测向量，确定局部坐标系的y轴和z轴。
在一个实施例中，可以将已确定的局部坐标系x轴与第一预测向量进行叉乘，得到局部坐标系y轴。然后，再将已确定的局部坐标系的x轴和y轴进行叉乘，得到修正后的局部坐标系z轴。
在一个实施例中，局部坐标系的坐标中心可以是牙洞线中心。至此，局部坐标系设定完毕。
请参图4，为本申请一个实施例中的计算机实施的牙齿三维数字模型的局部坐标系的设定方法400的示意性流程图。
计算机实施的牙齿三维数字模型的局部坐标系的设定方法400是利用基于八叉树的卷积神经网络(Octree-Based Convolutional Neural Networks，简称O-CNN)。
在401中，获取第一三维数字模型。
第一三维数字模型是基于世界坐标系的表示第一牙齿的三维数字模型，也就是说，其各顶点的坐标值是所述世界坐标系下的坐标值。
在403中，对第一三维数字模型进行细化得到第二三维数字模型。
在一个实施例中，为后续体素化后的数据增加更多信息，可以对第一三维模型进行细化上采样，得到第二三维数字模型。具体操作如下。
首先，计算三角面片f i三边的中点a i、b i、c i。
然后，连接a ib i、a ic i、b ic i，此时，原三角面片f i被分割为4个小三角面片。
分割完所有三角面片后，还可以迭代上述操作，直至达到经验要求。在一个实施例中，可以对第一三维数字模型进行4次上采样，得到第二三维数字模型。
在405中，对第二三维数字模型进行八叉树体素化得到体数据集。
在一个实施例中，可以把第二三维数字模型放入一个单位包围盒内(长宽高均为1)，然后，以广度优先的顺序将该单位包围盒递归地分割为8个子包围盒，重复递归直到满足预先设定的八叉树深度d。
在一个实施例中，在体素化后可以搜集一些O-CNN变换中需要的信息，如shuffle key和O-label。
其中，shuffle key定义了父节点下的当前子节点的相对位置，其为卷积计算服务。O-label定义了当前子节点的父节点是谁，以及是父节点下的第几个非空子节点，该过程为pooling操作服务。
在407中，利用经训练的O-CNN网络，基于体数据集，预测得到对应局部坐标系其中两个坐标轴的第三预测向量和第四预测向量。
在一个实施例中，在训练O-CNN网络时，可以针对局部坐标系的三个坐标轴分别建立O-CNN网络并进行训练，根据训练过程中的结果，选择预测最准确 的两个O-CNN网络(分别对应局部坐标系的其中两个坐标轴)来设定局部坐标系。
【O-CNN网络的训练】
在训练O-CNN网络时，为了提高O-CNN网络的泛化能力，可以将用于训练的细化后的三维数字模型以15度为步长分别绕局部坐标系的三个轴旋转[-45,45]度，数据因此可增广7 3倍。
请参由Wang P S、Liu Y、Guo Y X等在Acm Transactions on Graphics,2017,36(4):72上发表的“O-CNN:Octree-Based Convolutional Neural Networks for 3D Shape Analysis”，披露了一种在以体素化3D数据为输入的卷积神经网络，该网络为3D模型的分类分割问题提供了非常好的解决方案。针对局部坐标系坐标轴的预测，可以将O-CNN的分类层改成以EuclideanLoss为代价函数的回归层。
请参图5，示意性地展示了本申请一个实施例中的O-CNN网络500的结构。
O-CNN网络500包括输入层501、O-CNN[d]503、505及507、Dropout层509和513、全连接层511和515以及输出层517。
其中，O-CNN[d]503、505及507的每一个均包括卷积层(convolution)、批量正规化层(batch normalization)、激活函数ReLU以及池化层(pooling)。卷积层通过对小范围域内的数据通过卷积的方式计算局部特征。池化层通过上采样实现部分的旋转平移等不变性。批量正规化层在一定程度上解决了数据的每层输出分布不统一的问题，具有能够使用更高的学习率，避免过拟合，合理避免梯度饱和等优点。
在一个实施例中，输入的数据值为每一个叶子节点所在包围盒中所有顶点的平均法向。若叶子节点为空，则将法向设定为0。
在409中，基于第三预测向量和第四预测向量确定局部坐标系的三个坐标轴。
在一个实施例中，可以根据O-CNN网络预测的偏差大小决定将第三预测向 量和第四预测向量之一作为局部坐标系的第一坐标轴。然后，将第三预测向量和第四预测向量进行叉乘得到局部坐标系的第二坐标轴。最后将第一坐标轴和第二坐标轴进行叉乘得到局部坐标系的第三坐标轴。
例如，若第三预测向量对应局部坐标系的x轴，第四预测向量对应局部坐标系的z轴，若该O-CNN网络对局部坐标系的x轴的预测更加准确，那么，将第三预测向量作为局部坐标系的x轴。然后，将第三预测向量和第四预测向量进行叉乘得到局部坐标系的y轴。最后，将x轴和y轴进行叉乘得到局部坐标系的z轴。
在本申请的启发下，可以理解，除了以上具有深度学习能力的人工神经网络，还可以采用其他适用的人工神经网络设定局部坐标系的坐标轴，例如，卷积神经网络(Convolutional Neural Networks，简称CNN)、递归神经网络(Recurrent Neural Networks，简称RNN)、强化学习(Reinforcement Learning，简称RL)以及生成对抗网络(Generative Adversarial Networks，简称GANs)。
尽管在此公开了本申请的多个方面和实施例，但在本申请的启发下，本申请的其他方面和实施例对于本领域技术人员而言也是显而易见的。在此公开的各个方面和实施例仅用于说明目的，而非限制目的。本申请的保护范围和主旨仅通过后附的权利要求书来确定。
同样，各个图表可以示出所公开的方法和系统的示例性架构或其他配置，其有助于理解可包含在所公开的方法和系统中的特征和功能。要求保护的内容并不限于所示的示例性架构或配置，而所希望的特征可以用各种替代架构和配置来实现。除此之外，对于流程图、功能性描述和方法权利要求，这里所给出的方框顺序不应限于以同样的顺序实施以执行所述功能的各种实施例，除非在上下文中明确指出。
除非另外明确指出，本文中所使用的术语和短语及其变体均应解释为开放式的，而不是限制性的。在一些实例中，诸如“一个或多个”、“至少”、“但不限于”这样的扩展性词汇和短语或者其他类似用语的出现不应理解为在可能没有这种 扩展性用语的示例中意图或者需要表示缩窄的情况。

Claims (14)

  1. 一种计算机执行的牙齿三维数字模型的局部坐标系的设定方法，包括：
    获取第一三维数字模型，它是基于世界坐标系表示第一牙齿的三维数字模型；以及
    利用第一人工神经网络，基于所述第一三维数字模型，为其设定局部坐标系，其中，所述第一人工神经网络是经训练的具有深度学习能力的人工神经网络。
  2. 如权利要求1所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，所述第一人工神经网络是多层感知器。
  3. 如权利要求2所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，它还包括：
    利用所述第一人工神经网络，基于所述第一三维数字模型，获得第一预测向量，其与所述局部坐标系的第一坐标轴相对应，该第一坐标轴为所述局部坐标系的y轴和z轴之一，所述局部坐标系的y轴和z轴除所述第一坐标轴外的另一个为第二坐标轴；
    利用主成分分析法，基于所述第一三维数字模型，确定所述局部坐标系的x轴；
    基于所述已确定的x轴以及第一预测向量，确定所述第二坐标轴；以及
    基于所述已确定的x轴以及第二坐标轴，确定所述第一坐标轴。
  4. 如权利要求3所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，所述主成分分析法是基于法向的主成分分析法。
  5. 如权利要求3所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，它还包括：
    利用第二人工神经网络，基于所述第一三维数字模型，获得第二预测向量，其中，所述第二人工神经网络是经训练的具有深度学习能力的人工神经网络，用于预测局部坐标系的x轴，所述第二预测向量与所述局部坐标系的 x轴相对应；
    利用所述主成分分析法，基于所述第一三维数字模型产生三个特征向量；以及
    从所述三个特征向量中选取与所述第二预测向量所在直线夹角最小的一个，并根据所述第二预测向量赋予所述被选中的特征向量正确的符号，作为所述局部坐标系的x轴。
  6. 如权利要求3所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，
    利用叉乘，基于所述已确定的x轴与第一预测向量确定所述第二坐标轴；以及
    利用叉乘，基于所述已确定的x轴以及第二坐标轴，确定所述第一坐标轴。
  7. 如权利要求3所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，所述局部坐标系的第一坐标轴是z轴。
  8. 如权利要求3所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，它还包括：对所述第一三维数字模型进行简化，使其顶点数等于预定的N，获得第一数字数据集，利用所述第一人工神经网络，基于该第一数字数据集，获得所述第一预测向量，其中，所述N是自然数。
  9. 如权利要求8所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，它还包括：对所述简化后数据集进行中心化，获得第二数字数据集，利用所述第一人工神经网络，基于该第二数字数据集，获得所述第一预测向量，利用所述主成分分析法，基于所述第二数字数据集，确定所述局部坐标系的x轴。
  10. 如权利要求9所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，它还包括：对所述第二数字数据集进行归一化处理，获得第三数字数据集，利用所述第一人工神经网络，基于该第三数字数据集，获得所 述第一预测向量。
  11. 如权利要求2所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，所述第一人工神经网络的输出层包括EuclideanLoss代价函数，用于通过反向传播训练各层参数。
  12. 如权利要求1所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，它还包括：根据所述第一牙齿的类型，从多个人工神经网络中选定所述第一人工神经网络，其中，所述多个人工神经网络是经训练的具有深度学习能力的人工神经网络，分别用于为不同类型的牙齿设定局部坐标系。
  13. 如权利要求1所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，所述第一人工神经网络是以下之一：多层感知器、基于八叉树的卷积神经网络、卷积神经网络、递归神经网络、强化学习以及生成对抗网络。
  14. 如权利要求1所述的基于计算机的牙齿三维数字模型的局部坐标系的设定方法，其特征在于，所述第一人工神经网络是以手工标定局部坐标系的多个牙齿三维数字模型进行训练，其中，所述多个牙齿三维数字模型均是与所述第一牙齿相同类型的牙齿的三维数字模型。
PCT/CN2019/077805 2018-05-09 2019-03-12 牙齿三维数字模型的局部坐标系设定方法 WO2019214339A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/967,671 US11694397B2 (en) 2018-05-09 2019-03-12 Method for setting a local coordinate system of a tooth 3D digital model

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810445610.0 2018-05-09
CN201810445610.0A CN110473283B (zh) 2018-05-09 2018-05-09 牙齿三维数字模型的局部坐标系设定方法

Publications (1)

Publication Number Publication Date
WO2019214339A1 true WO2019214339A1 (zh) 2019-11-14

Family

ID=68466876

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/077805 WO2019214339A1 (zh) 2018-05-09 2019-03-12 牙齿三维数字模型的局部坐标系设定方法

Country Status (3)

Country Link
US (1) US11694397B2 (zh)
CN (1) CN110473283B (zh)
WO (1) WO2019214339A1 (zh)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019002631A1 (en) 2017-06-30 2019-01-03 Promaton Holding B.V. 3D CLASSIFICATION AND MODELING OF 3D DENTO-MAXILLO-FACIAL STRUCTURES THROUGH DEEP LEARNING PROCESSES
EP3462373A1 (en) 2017-10-02 2019-04-03 Promaton Holding B.V. Automated classification and taxonomy of 3d teeth data using deep learning methods
EP3503038A1 (en) * 2017-12-22 2019-06-26 Promaton Holding B.V. Automated 3d root shape prediction using deep learning methods
EP3561778A1 (en) 2018-04-26 2019-10-30 Promaton Holding B.V. Automated correction of metal affected voxel representations of x-ray data using deep learning techniques
TWI712396B (zh) * 2020-01-16 2020-12-11 中國醫藥大學 口腔缺陷模型之修補方法及口腔缺陷模型之修補系統
CN113822305A (zh) * 2020-06-19 2021-12-21 杭州朝厚信息科技有限公司 产生表示牙齿正畸治疗的目标牙齿布局的数字数据集的方法
CN112790879B (zh) * 2020-12-30 2022-08-30 正雅齿科科技(上海)有限公司 一种牙齿模型的牙轴坐标系构建方法及系统
CN113421333B (zh) * 2021-07-01 2023-10-24 正雅齿科科技(上海)有限公司 牙齿局部坐标系确定方法和系统、设备及计算机存储介质
CN113570709B (zh) * 2021-08-05 2024-05-10 杭州隐捷适生物科技有限公司 一种基于八叉树结构的3d牙齿分割方法
WO2023242767A1 (en) * 2022-06-16 2023-12-21 3M Innovative Properties Company Coordinate system prediction in digital dentistry and digital orthodontics, and the validation of that prediction
EP4307229A1 (en) * 2022-07-15 2024-01-17 3Shape A/S Method and system for tooth pose estimation
CN117351293B (zh) * 2023-12-04 2024-02-06 天津医科大学口腔医院 一种联合学习的牙周病图像分类方法及设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008096017A2 (en) * 2007-02-09 2008-08-14 Universiteit Antwerpen 3-d orthognathic surgery simulation
CN105931291A (zh) * 2016-05-16 2016-09-07 杭州美齐科技有限公司 一种数字化完整牙颌建模方法
CN107260335A (zh) * 2017-06-26 2017-10-20 达理 一种基于人工智能的牙列畸形自动化分类和设计方法
CN107301673A (zh) * 2017-05-17 2017-10-27 杭州美齐科技有限公司 一种利用标准模型库构造牙齿的局部坐标系的算法
CN107374758A (zh) * 2017-07-14 2017-11-24 西安科技大学 基于拟合优化的自动排牙方法和装置

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102073749B (zh) * 2009-11-04 2015-12-16 常州高新技术产业开发区三维工业技术研究所有限公司 设计中间全颌牙齿数字模型的方法
CN202154758U (zh) * 2011-07-26 2012-03-07 北京大学口腔医学院 一种数字化口腔修复体制作设备
US9632981B2 (en) * 2012-07-12 2017-04-25 Vital Connect, Inc. Calibration of a chest-mounted wireless sensor device for posture and activity detection
KR101865655B1 (ko) * 2014-06-26 2018-06-11 한국과학기술원 증강현실 상호 작용 서비스 제공 장치 및 방법
US20160019688A1 (en) * 2014-07-18 2016-01-21 University Of Georgia Research Foundation, Inc. Method and system of estimating produce characteristics
CN105380723B (zh) * 2015-12-23 2017-03-29 南京航空航天大学 制作全口义齿的排牙方法
EP3151164A3 (en) * 2016-12-26 2017-04-12 Argosai Teknoloji Anonim Sirketi A method for foreign object debris detection
EP3612132B1 (en) * 2017-04-21 2023-08-23 Martz, Andrew, S. Method and system for the fabrication of dental appliances
CN115688877A (zh) * 2017-06-06 2023-02-03 格兰菲智能科技有限公司 一种用于对待量化数据进行定点化处理的方法及计算装置
CN108010036B (zh) * 2017-11-21 2020-01-21 江南大学 一种基于rgb-d相机的物体对称轴检测方法
US10643384B2 (en) * 2018-02-08 2020-05-05 Google Llc Machine learning-based geometric mesh simplification

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008096017A2 (en) * 2007-02-09 2008-08-14 Universiteit Antwerpen 3-d orthognathic surgery simulation
CN105931291A (zh) * 2016-05-16 2016-09-07 杭州美齐科技有限公司 一种数字化完整牙颌建模方法
CN107301673A (zh) * 2017-05-17 2017-10-27 杭州美齐科技有限公司 一种利用标准模型库构造牙齿的局部坐标系的算法
CN107260335A (zh) * 2017-06-26 2017-10-20 达理 一种基于人工智能的牙列畸形自动化分类和设计方法
CN107374758A (zh) * 2017-07-14 2017-11-24 西安科技大学 基于拟合优化的自动排牙方法和装置

Also Published As

Publication number Publication date
CN110473283A (zh) 2019-11-19
US20210217233A1 (en) 2021-07-15
CN110473283B (zh) 2024-01-23
US11694397B2 (en) 2023-07-04

Similar Documents

Publication Publication Date Title
WO2019214339A1 (zh) 牙齿三维数字模型的局部坐标系设定方法
US8639477B2 (en) Reconstruction of non-visible part of tooth
WO2018218988A1 (zh) 牙颌三维数字模型的分割方法
Singh et al. Topological methods for the analysis of high dimensional data sets and 3d object recognition.
CN113454637A (zh) 使用深度学习对非欧式3d数据集的自动语义分割
EP2598034B1 (en) Adaptive visualization for direct physician use
CN111968146B (zh) 三维牙颌网格模型分割方法
JP2023512083A (ja) ニューラルネットワークを用いた歯群のセグメンテーション
Zheng et al. TeethGNN: semantic 3D teeth segmentation with graph neural networks
WO2022183852A1 (zh) 牙颌三维数字模型的分割方法
WO2020181974A1 (zh) 基于人工神经网络的去除牙齿三维数字模型的表面气泡的方法
WO2023242757A1 (en) Geometry generation for dental restoration appliances, and the validation of that geometry
WO2024127316A1 (en) Autoencoders for the processing of 3d representations in digital oral care
WO2024127311A1 (en) Machine learning models for dental restoration design generation
WO2023242768A1 (en) Defect detection, mesh cleanup, and mesh cleanup validation in digital dentistry
CN115830287A (zh) 基于激光口扫与cbct重建的牙齿点云融合方法、设备及介质
CN115761125A (zh) 基于点云注意力和齿间碰撞损失的齿科数字化正畸方法
CN115146333A (zh) 用于生成机器学习模型的3D CAD对象的UV-net表示的技术
Unal et al. Generating shapes by analogies: An application to hearing aid design
EP4307229A1 (en) Method and system for tooth pose estimation
RU2717911C1 (ru) Способ обучения сверточной нейронной сети осуществлять разметки телерентгенограмм в прямой и боковой проекциях
WO2024082284A1 (zh) 基于网格特征深度学习的口腔正畸自动排牙方法和系统
WO2023242767A1 (en) Coordinate system prediction in digital dentistry and digital orthodontics, and the validation of that prediction
TW202416910A (zh) 整合式牙齒復原設計方法及系統
WO2023242776A1 (en) Bracket and attachment placement in digital orthodontics, and the validation of those placements

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19800307

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 20.04.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19800307

Country of ref document: EP

Kind code of ref document: A1