KR102651466B1 - Device and method for 3d teeth reconstruction from panoramic radiographs using neural implicit functions - Google Patents

Abstract

An apparatus and method for three-dimensional tooth reconstruction from panoramic X-ray (PX) images using a nested function neural network are disclosed. The three-dimensional tooth reconstruction method is performed by a computing device including a processor, and includes the steps of acquiring a PX image of a subject, dividing each tooth of the PX image, Generating a condition vector, and predicting a 3D image from the condition vector using a pre-trained neural implicit function-based reconstruction model, wherein the condition vector is It is generated by adding the encoding value of the tooth patch corresponding to the segmented tooth and the embedding of the tooth number.

Description

Apparatus and method for three-dimensional tooth reconstruction from panoramic radiographs using nested function neural network {DEVICE AND METHOD FOR 3D TEETH RECONSTRUCTION FROM PANORAMIC RADIOGRAPHS USING NEURAL IMPLICIT FUNCTIONS}

The present invention relates to a three-dimensional tooth reconstruction algorithm, and more specifically, to an apparatus and method for reconstructing single teeth in three dimensions using a neural implicit function.

Deep neural network-based 3D reconstruction technology from 2D images is a field in which research is being actively conducted based on various data and structures. Representative technologies using deep neural networks include technologies using convolutional neural networks, generative adversarial networks, and recurrent neural networks. Methods previously proposed using these technologies include papers such as UNet, GAN, 3D-R2N2, and Occupancy Network.

Due to the nature of the imaging method, panoramic radiography creates a two-dimensional image that spreads the dental structure, including the upper and lower jaw, horizontally. However, in the case of diagnosis and treatment using 2D images, there is a limitation in that there is no information about actual 3D teeth. Therefore, the topic of 3D tooth reconstruction is an important field in more precise dental medical diagnosis and applications. However, due to the nature of 3D data, there are problems such as limitations in computing resources and low tooth prediction accuracy. In addition, in the case of the existing voxel format generation method, the computational complexity is high because 3D teeth are learned as input and output at once, and accuracy problems arise depending on the input and output resolution.

In addition, the existing voxel-based method of generating results has memory limitations and difficulties in predicting detailed tooth shape. Anatomically, there are up to 32 teeth in humans, and individual teeth have different shapes depending on the tooth number. High resolution, that is, a large number of voxels, is required to represent detailed tooth shapes of different numbers. However, in the case of voxel-based methods, as the resolution increases, the size of the model also becomes very large, so there are memory limitations and post-processing work, such as smoothing the surface using a threshold, is required. In addition, 3D results use more computing resources than general images, making 3D reconstruction one of the difficult research areas in the medical field. Therefore, in order to perform 3D tooth reconstruction from panoramic radiographs, a new structural technique that can solve the problems mentioned above is needed.

Republic of Korea Patent No. 1739091 (announced on May 24, 2017) Republic of Korea Patent Publication No. 2022-0154471 (published on November 22, 2022) Republic of Korea Patent Publication No. 2017-0025241 (published on March 8, 2017) Republic of Korea Patent No. 2117290 (announced on June 1, 2020)

The technical problem to be achieved by the present invention is to provide an apparatus and method for three-dimensional tooth reconstruction from panoramic radiographs using a nested function neural network.

The 3D tooth reconstruction method according to an embodiment of the present invention is a 3D tooth reconstruction method from a panoramic X-ray (PX) image, which is performed by a computing device including a processor, and the PX image of the subject is Obtaining, segmenting each tooth of the PX image, generating a condition vector, and using a reconstruction model based on a pre-trained implicit function neural network, from the condition vector It includes predicting a 3D image, and the condition vector is generated by adding the encoding value of the tooth patch corresponding to the segmented tooth and the embedding of the tooth number.

According to an embodiment of the present invention, the apparatus and method for 3D tooth reconstruction from panoramic radiographs using a nested function neural network can perform accurate 3D individual tooth creation from panoramic radiographs, thereby improving dental medical diagnosis and It can be used as a three-dimensional auxiliary visualization material for treatment.

Additionally, 3D reconstructed tooth data can be used to develop medical simulations and medical learning tools.

In order to more fully understand the drawings cited in the detailed description of the present invention, a detailed description of each drawing is provided.
Figure 1 shows examples of CBCT and panoramic radiographs.
Figure 2 is an overall schematic diagram of the technique proposed in the present invention.
Figure 3 shows the 3D reconstruction prediction results according to the present invention.
Figure 4 is a flowchart for explaining a 3D tooth reconstruction method according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining the embodiments according to the concept of the present invention. They may be implemented in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention can make various changes and have various forms, the embodiments will be illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all changes, equivalents, or substitutes included in the spirit and technical scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component and similarly a second component The component may also be named a first component.

When a component is said to be “connected” or “connected” to another component, it is understood that it may be directly connected to or connected to that other component, but that other components may also exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "immediately between" or "neighboring" and "directly adjacent to" should be interpreted similarly.

The terms used in this specification are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in this specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings attached to this specification. However, the scope of the patent application is not limited or limited by these examples. The same reference numerals in each drawing indicate the same members.

Panoramic radiography (panoramic X-ray, or PX) is mainly used in the fields of dental examination and diagnosis. PX generates 2D images from panoramic scanning, while Cone-Beam Computed Tomography (CBCT) provides 3D information about dental, oral, and maxillofacial structures. Despite providing more comprehensive information than PX, CBCT is expensive and involves greater radiation exposure to patients. Therefore, 3D tooth reconstruction from PX is of great significance. In this regard, Figure 1 shows examples of CBCT and panoramic radiographs.

The technique proposed in the present invention consists of 2D teeth segmentation and 3D teeth reconstruction. 2D tooth segmentation performs segmentation of a plurality of teeth (eg, 32 teeth) from the PX using a predetermined segmentation technique (eg, UNet++ model). The UNet++ model, an exemplary tooth segmentation model, is described in the paper by Zhou et al. (Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., Liang, J.: Unet++: A nested u-net architecture for medical image segmentation. In: Deep Learning It was proposed in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 3-11. Springer (2018).

Individually segmented teeth and tooth numbers (tooth classes) are input into the 3D tooth reconstruction model for reconstruction. The reconstruction process predicts a 3D representation of the tooth based on a neural implicit function. The overall outline of the proposed technique is shown in Figure 2.

Referring to FIG. 2, (a) the PX image is divided into a plurality of teeth (eg, 32 teeth) using a tooth segmentation model (eg, UNet++ model). At this time, each tooth may be given a unique tooth number (which may also be named a unique number or identification number) depending on the position, shape, role, etc. of the tooth. For each tooth, a segmented patch (which may mean a segmented image corresponding to an individual tooth) is created by cropping the input PX with the predicted segmentation mask. The segmented patches are encoded through an image encoder. The tooth number is encoded by an embedding layer to generate a number embedding (class embedding). The segmented patches and number embeddings are summed (combined) to create a condition vector.

(b) The reconstruction process consists of N (N is any natural number) ResBlocks that calculate the occupancy features of points sampled from 3D space. Condition vectors from PX are processed by the Conditional eXcitation module (CX module) merged within ResBlocks.

(c) Convolutional layer (Conv) with CX sub-module consists of batch normalization, conditional activation (CX), activation function (e.g., ReLU), and convolutional layer (conv1D). FC (Fully Connected layer) with CX is similar to ResBlock with CX. However, there is a difference in that the Conv1D layer has been replaced with a fully connected layer.

(d) CX injects condition information into the reconstruction network using excitation values. CX uses a trainable weight matrix to encode a condition vector into an excitation vector through a gating function. Input features are scaled through component-wise multiplication using activation vectors.

The above-described reconstruction model may refer to a model learned using training data. The reconstruction model can be created by repeatedly performing a learning process to output 3D teeth corresponding to the segmented tooth patches and updating the weights. At this time, the 3D tooth is a 3D tooth image corresponding to the segmented tooth and may mean a CBCT image.

2D Teeth Segmentation

The teeth in the input PX (PX image) are divided into a plurality of teeth (eg, 32 teeth). The tooth number for each tooth may be according to traditional numbering for teeth. Therefore, the tooth number can be understood as including information about the position, shape, mechanics, etc. of the tooth.

The input of the model is a PX image of H×W size. The split output has dimensions of C×H×W. Here, the channel dimension C=33 may represent the number of tooth numbers. One of the 33 numbers may represent a number for the background, and 32 may represent a number for the teeth. Therefore, the H×W output from each channel is a divided output for each tooth number. The segmented output is used to generate tooth patches for reconstruction. For segmentation, a pre-trained tooth segmentation model (eg, UNet++) can be used. UNet++ has strengths in the field of medical image segmentation. However, the present invention does not limit the adoption of other partitioning techniques, and various partitioning techniques such as UNet may be used depending on the embodiment.

3D Teeth Reconstruction

Neural Implicit Representation. Typical representations of 3D shapes include point-based, voxel-based, or mesh-based techniques. These techniques explicitly represent 3D shapes through sets of discrete points, vertices, and faces. Recently, implicit representation methods based on continuous functions that define the boundaries of 3D shapes have become known. Occupancy Networks use neural networks to approximate an implicit function for an object's occupancy (Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger , A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4460-4470 (2019)). The term "occupancy" refers to whether a point in space lies inside or outside the boundary of an object. The occupancy function maps 3D points to 0 or 1. At this time, 0 indicates occupancy (or non-occupancy), and 1 indicates non-occupancy (or non-occupancy). o Let _A be the occupancy function for object A. o _A is the same as Equation 1.

[Equation 1]

o _A is, It can be estimated solely from the set of observations for object A, denoted by . Examples of observations are projected images or point cloud data from an object. The purpose of the present invention is It is to estimate the occupancy function in the state. especially, A function that estimates the occupancy probability of a point in 3D space based on (occupancy probability function) I want to find . function is expressed as Equation 2.

[Equation 2]

In the present invention, the segmented tooth patch and the corresponding tooth number are used as observations (represented by the condition vector c). In particular, the input to the function is a set of T randomly sampled positions (points) within a predefined space (e.g., a unit cube), and the function outputs the occupancy probability of the input. Therefore, the function is It is given as follows.

The model for is shown in (b) of Figure 2. The sampled positions are projected into high-dimensional, for example, 128-dimensional feature vectors, using 1D convolution. Next, the features are processed by a series of ResNet blocks (which may also be named convolutional layers, convolutional layers with CX, convolutional blocks, etc.) followed by fully connected layers (FC layers). do. The condition vector c is used for each block through conditional eXcitation (CX). Additionally, depending on the embodiment, the input of the first convolutional layer may be added to the output of the last convolutional layer and then input to the fully connected layer.

Tooth number-specific conditional features (Class-specific Conditional Features). A distinguishing feature of the tooth reconstruction task is that teeth with the same tooth number share features such as surface and root shapes. Therefore, the present invention proposes using tooth number information in combination with segmented tooth patches from PX. The tooth number is processed by a learnable embedding layer that outputs a tooth number embedding vector.

Next, a square patch of teeth is created using the segmented output as follows. A binary mask of the segmented teeth is created by applying thresholding to the segmentation output. A tooth patch is created by cutting out a tooth region from the input PX. That is, a binary mask is applied to the input PX (bitwise AND) to obtain the patch. The segmented tooth patch is encoded with a predetermined model, such as a pre-trained encoding model, such as the ResNet18 model (He, K., Zhang, of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)). The ResNet18 model, an example encoding model, outputs a patch embedding vector. The patch and tooth number embeddings are added to create a condition vector for the reconstruction model. This process is shown in Figure 2(a).

Conditional eXcitation (CX). To efficiently inject 2D observations into a reconstruction network, the Squeeze-and-Excitation Network (SENet) (Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Inspired by the conference on computer vision and pattern recognition. pp. 7132-7141 (2018), we propose conditional activation (CX). In SENet, excitation refers to scaling input features according to importance. In the present invention, the concept of “activity” is extended to incorporating conditional features into the network. First of all, the condition vector is encoded into the activation vector e. Next, the activation vector is applied to the input features by scaling the feature elements by e. The CX process is expressed by Equation 3 and Equation 4.

[Equation 3]

[Equation 4]

In the above equation, c is a condition vector, σ is a gate function, W is a learnable weight matrix, α is a hyperparameter for the excitation result, and F _ext is an activation function ( excitation function). A sigmoid function can be used as σ, and component-wise multiplication for activation is am. The CX module is shown in Figure 2(d). CX differs from SENet in that CX derives activations from condition vectors, whereas SENet derives activations from input features. The proposed technique is Occupancy Networks using Conditional Batch Normalization (CBN) (Dumoulin, V., Belghazi, I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., Courville, A.: Adversarially learned inference. arXiv preprint arXiv:1606.00704 (2016), De Vries, H., Strub, F., Mary, J., Larochelle, H., Pietquin, O., Courville, AC: Modulating early visual processing by language. Advances in Neural Information Processing Systems 30 (2017)).

Figure 3 shows the 3D reconstruction prediction results according to the present invention. Specifically, Figure 3 (a) shows a PX image, (b) shows the prediction result using the reconstruction model proposed in the present invention, and (c) shows the correct answer image. Referring to Figure 3, it can be seen that 3D reconstruction is performed even on the root portion of the tooth, which is difficult to predict from the PX image.

Figure 4 is a flowchart for explaining a 3D tooth reconstruction method according to an embodiment of the present invention. Hereinafter, in explaining the 3D tooth reconstruction method, detailed description of content that overlaps with the previous description will be omitted.

The 3D tooth reconstruction method may be performed by a computing device including at least a processor and/or memory. That is, at least some of the steps of the 3D tooth reconstruction method may be understood as operations of a processor included in the computing device. The computing device may include a personal computer (PC), a server, a laptop computer, a tablet PC, etc. Depending on the embodiment, the computing device may include a PX device (PX imaging device, PX imaging device, etc.) that generates a PX image. It can also mean itself. The computing device may be named a 3D tooth reconstruction device, etc.

First, a panoramic radiography (PX) image of the subject's or patient's teeth is acquired (S110). PX images, which are two-dimensional (2D) images, can be received through a wired or wireless communication network. PX images may be received from a PX imaging device. Depending on the embodiment, the PX image may be received from a storage device such as a USB memory device through a predetermined input/output interface, or may be stored in advance in the storage of the computing device. Additionally, when the computing device refers to the PX imaging device itself, the PX image may be obtained by taking a PX image of the subject's or patient's teeth.

Each tooth is segmented from the PX image (S120). Teeth may be segmented using any tooth segmentation technique, such as UNet++. At this time, each divided tooth may be assigned a unique tooth number.

Next, a condition vector, which is the input of the reconstruction model, is created (S130). The condition vector is created by combining (summing) the tooth patch embedding and the tooth number embedding. Tooth patch embedding refers to a patch for each segmented tooth, encoded (or embedded) through a predetermined encoding technique, and tooth number embedding may refer to embedding of the tooth number for the corresponding tooth.

Three-dimensional (3D) tooth reconstruction is performed using a reconstruction model based on a nested function neural network (S140). The reconstruction model may receive a condition vector for each tooth and generate a 3D tooth image corresponding to the condition vector.

Specifically, a multidimensional feature vector is generated from a plurality of points sampled within a predefined space. The generated feature vector may be processed by at least one convolutional layer, and then a three-dimensional tooth image (which may mean whether each point is occupied) may be derived by at least one fully connected layer.

The device described above may be implemented as a hardware component, a software component, and/or a set of hardware components and software components. For example, devices and components described in the embodiments include, for example, a processor, a controller, an Arithmetic Logic Unit (ALU), a Digital Signal Processor, a microcomputer, a Field Programmable Array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a Programmable Logic Unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device may include multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes specially configured hardware devices to store and execute program instructions, such as magneto-optical media, ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely illustrative, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the attached registration claims.

Claims (5)

In a three-dimensional tooth reconstruction method from a panoramic X-ray (PX) image, performed by a computing device including a processor,
Obtaining a PX image of a subject;
Segmenting each tooth of the PX image;
generating a condition vector; and
Predicting a 3D image from the condition vector using a pre-trained neural implicit function-based reconstruction model,
The condition vector is generated by adding the encoding value of the tooth patch corresponding to the segmented tooth and the embedding of the tooth number,
The step of predicting the 3D image involves generating a feature vector with increased dimension based on a plurality of points randomly sampled within a predetermined space.
3D tooth reconstruction method. According to paragraph 1,
The dividing step involves dividing each tooth using UNet++.
3D tooth reconstruction method. delete According to paragraph 1,
The step of predicting the 3D image involves predicting whether each of the plurality of points is occupied by a 3D shape by inputting the feature vector into at least one convolutional layer followed by a fully connected layer. doing,
3D tooth reconstruction method. According to paragraph 4,
The at least one convolutional layer further includes a conditional activation layer,
The conditional activation layer performs inter-element multiplication between the input feature vector and the excitation vector and outputs it,
The activation vector (e) is calculated by the equation,
The above equation is ego,
The c is the condition vector, W is a weight matrix, σ is a gate function, and α is a hyperparameter with a predetermined value,
3D tooth reconstruction method.

Images