TWI807809B - System, computer program and computer-readable medium for assisting in the recognition of periodontitis and dental caries by using the convolutional neural network of deep learning - Google Patents

Abstract

The present invention relates to a system, a computer program, and a computer-readable medium for assisting in the recognition of periodontitis and dental caries by using a convolutional neural network of deep learning. A plurality of single-tooth images retrieved from periapical x-ray images are stored in a database, in which the plurality of single-tooth images are classified into normal-tooth images, periodontal disease images, dental caries images, and the composite of periodontal disease and dental caries images by normal-tooth label, a periodontal disease label, and a dental caries label. A computer is installed with an image processing module, a convolutional neural network with an attention model of deep learning. The computer performs resizing and dada augmentation on the plurality of single-tooth images, uses the image processing module to enhance features of the plurality of single-tooth images, and then inputs the processed images to the convolutional neural network of deep learning for training; moreover, the attention model is utilized to enhance the weight distribution so that a multi-label classification model for assisting in recognition of periodontitis and dental caries is obtained. The multi-label classification model serves to recognize a to-be-detected single-tooth image retrieved from a periapical x-ray image and to classify it as the normal-tooth image, the periodontal disease image, the dental caries image, or the composite of periodontal disease and dental caries image, thereby assisting in determing the health status of teeth.

Description

Using deep learning convolutional neural network to assist in the identification of periodontal disease and dental caries systems, computer programs and computer readable media

The present invention relates to a system, a computer program, and a computer-readable medium for assisting the identification of periodontal disease and dental caries using deep learning convolutional neural networks, and in particular refers to the invention of combining multiple single tooth X-ray images obtained from apical X-ray images in a data database with an image processing module, deep learning convolutional neural networks, and an attention model to establish a multi-label classification model for assisting identification of periodontal disease and dental caries, so as to assist in the detection of dental health status.

Dental health is an important cornerstone of maintaining overall health. The two most common diseases are periodontal disease and dental caries. Among them, the radiation level of X-ray images is relatively low and can be obtained immediately. Therefore, digital X-ray images are often used in hospitals and dental clinics to assist diagnosis and treatment.

In recent years, artificial intelligence has developed very rapidly, among which Convolutional Neural Network (CNN) is one of the most important models in image classification tasks. With the recent rapid evolution of deep learning, many classic CNN network architectures have emerged, such as AlexNet, GoogLeNet, VGG, ResNet and other neural networks, which have been successfully applied in the field of computer vision. Among them, there are also many application cases and research literature in medical treatment, such as the application of deep learning in the auxiliary identification of dental X-ray images.

For example, the detection of periodontal disease is documented in the following literature: "Diagnosis and Prediction of Periodontally Compromised Teeth Using a Deep Learning-based Convolutional Neural Network Algorithm" proposed by J.H.Lee et al. in 2018, uses X-ray images of root tips to detect premolars and molars. The method is to use 16 of the pre-trained VGGNet 19 framework After the feature map is obtained by the convolutional layer, it is classified through 3 layers of fully connected layers, including healthy teeth, teeth with moderate periodontal damage and teeth with severe periodontal damage. Among them, the obtained root tip X-ray images are incremented through rotation, translation and shearing to increase the amount of training.

The document "An Interpretable Computer-aided Diagnosis Method for Periodontitis from Panoramic Radiographs" proposed by H.Li et al. in 2021 uses full-mouth X-ray images to classify periodontal disease, including four categories: no periodontal disease, mild periodontal disease, moderate periodontitis, and severe periodontitis. The method is to use the model based on Mask Region-Based Convolutional Neural Networks (Mask Region-Based Convolutional Neural Networks) to cut and number the teeth, and calculate the characteristics of alveolar bone loss. The machine learning method of eXtreme Gradient Boosting (XGBoost) is used to predict the whole mouth X-ray image.

The above literature discloses that root tip or full-mouth X-ray images can be trained with convolutional neural network or machine learning to identify periodontal disease with artificial intelligence.

Testing for dental caries, such as the following documents: J.H.Lee et al. "Detection and Diagnosis of Dental Caries USing a Deep Learning-Based CONVORAL Network Algorithm", The base X -ray image obtained was detected to the front molars and molars, and the pre -trained Googlenet was transferred to the transfer learning. Incremental translation, zooming, shearing and horizontal flipping randomly increases the amount of data by 10 times to make predictions more accurate.

The document "Dental Caries Diagnosis in Digital Radiographs Using Back-propagation Neural Network" proposed by V.Geetha et al. in 2020 is to classify dental caries from apical X-ray images, use Laplacian Filter and adaptive threshold (Adaptive Thresholding) image processing for feature extraction, and then use neural network for classification.

The above-mentioned literature discloses that root tip X-ray images can be processed by using convolutional neural network or neural network after image processing, so as to identify dental caries with artificial intelligence.

For image processing of root tip X-ray images, the document "Automatic Image Processing Based Dental Image Analysis Using Automatic Gaussian Fitting Energy and Level Sets" proposed by P. Pandey et al. Image processing before cutting to facilitate subsequent image cutting and analysis.

Refer to Chinese Patent No. CN202110360226.2 "Method for Recognition of Caries Depth in Panoramic Films Based on Deep Learning and Attention Mechanism", which discloses the use of a deep learning model including attention mechanism to realize caries classification in different stages of oral panoramas.

Different from the above-mentioned previous proposals, the present invention proposes a system that utilizes deep learning convolutional neural networks to simultaneously assist in the identification of periodontal disease and dental caries.

The system of the present invention includes: a data database, which stores a plurality of single tooth X-ray images obtained from apical X-ray images, and uses three labels of normal, periodontal disease and dental caries to classify the above-mentioned single tooth Tooth X-ray images are classified into four types: normal tooth images, periodontal disease tooth images, carious tooth images, and periodontal disease/caries composite tooth images. A computer is connected to the data database, and the computer is installed with an image processing module, a deep learning convolutional neural network, and an attention model.

The computer obtains the X-ray image of the single tooth, first adjusts the size and increment, and then processes the X-ray image of the single tooth through the image processing module to enhance the characteristics of the X-ray image of the single tooth, and then inputs the convolutional neural network for training, and uses the attention model to strengthen the weight distribution to obtain a multi-label classification model for periodontal disease and dental caries assisted identification. Input an X-ray image of an apex to the computer, select the X-ray image of the root apex and resize it to obtain an X-ray image of a single tooth to be detected, process the X-ray image of the single tooth to be detected through the image processing module, and then use the multi-label classification model for the auxiliary identification of periodontal disease and dental caries to identify the X-ray image of the single tooth to be detected as any one of the normal tooth image, the periodontal disease tooth image, the carious tooth image or the periodontal disease/caries composite tooth image.

Further, the convolutional neural network is a residual network model.

Further, the computer adjusts the image size of the above-mentioned single tooth X-ray image to 100×100 and performs incremental processing, so that the above-mentioned single tooth X-ray image of the periodontal disease tooth image, carious tooth image and periodontal disease/carious composite tooth image are consistent in the number of images input to the convolutional neural network. The incremental processing is, for example, any or a combination of the above-mentioned single tooth X-ray image rotation 90 degrees, 180 degrees, 270 degrees, horizontal flip or vertical flip. Furthermore, the number of normal tooth images of the above-mentioned single tooth X-ray image input to the convolutional neural network is twice that of the above-mentioned periodontal disease tooth images, dental caries tooth images, and periodontal disease/caries composite tooth images.

Further, the image processing module includes contrast-limited adaptive histogram averaging and a bilateral filter; the above-mentioned single tooth X-ray image has three image processing results: the contrast-limited adaptive histogram The result of image averaging processing, the result of the bilateral filter processing, and the result of image processing first performed by the contrast-limited adaptive histogram averaging to enhance contrast, and then image processing is performed again by the bilateral filter to reduce noise and preserve the enhanced image edge.

The present invention further provides a computer program for being installed in the aforementioned system using deep learning convolutional neural network to assist in identifying periodontal disease and dental caries.

The present invention further provides a computer-readable medium storing the aforementioned computer program.

According to the above-mentioned technical features, the following effects can be achieved:

1. The multi-label classification model for periodontal disease and dental caries assisted identification established by the system of the present invention resizes and incrementally processes the single tooth X-ray image before performing image training, so that the above-mentioned periodontal disease tooth image, dental caries tooth image and periodontal disease/dental caries composite tooth image can be consistent in the number of images input to the convolutional neural network, avoiding the inconsistency of the number of images that leads to the decline in the predictive ability of deep learning.

2. The multi-label classification model for periodontal disease and dental caries assisted identification established by the system of the present invention combines image processing modules, deep learning convolutional neural networks and attention models during image training. The image processing module uses contrast-limited adaptive histogram averaging for image processing to enhance local contrast, uses a bilateral filter for image processing to preserve image edges, and respectively obtains three processed images from the contrast-limited adaptive histogram averaging, the bilateral filter, and sequentially combining the contrast-limited adaptive histogram averaging and the bilateral filter for training; the attention model can strengthen the weight distribution of image features to enhance the learning performance of the convolutional neural network. When actually performing artificial intelligence detection, it can effectively improve the prediction performance of the detection results such as sensitivity, specificity, accuracy and precision.

3. The present invention uses a single multi-label classification model for periodontal disease and dental caries assisted identification to detect periodontal disease and dental caries at the same time. It does not need to establish independent models for periodontal disease and dental caries, which can reduce the burden on software and hardware.

4. Using the deep learning convolutional neural network of the present invention to assist in the identification of periodontal disease and dental caries, the system can conduct preliminary screening of the patient's teeth, allowing people in areas with insufficient medical resources to spend a lot of time and money going to and from hospitals and clinics, and can also regularly check their teeth to maintain oral health. In addition, the screening results can assist medical decision-making units to investigate medical needs, plan medical manpower, and reduce the gap between urban and rural areas.

1: Data database

2: computer

21: Image processing module

211:Contrast-Limited Adaptive Histogram Averaging

212: bilateral filter

22: Convolutional Neural Networks

23: Attention Model

24: Multi-label classification model for auxiliary identification of periodontal disease and dental caries

A: X-ray image of a single tooth

A1: normal tooth image

A2: Periodontal Disease Teeth Imaging

A3: Tooth image with caries

A4: Periodontal disease/caries compound tooth image

B: root tip X-ray image

B1: X-ray image of a single tooth to be detected

[The first figure] is a schematic diagram of the system architecture of the embodiment of the present invention.

[The second figure] is a flowchart of an embodiment of the present invention.

[The third figure] is a schematic diagram of the model building process of the embodiment of the present invention.

[Third Figure A] is a specific architecture diagram of image processing and deep learning in the embodiment of the present invention.

[Figure 4] is a comparison of X-ray images of a single tooth after contrast-limited adaptive histogram averaging, bilateral filtering, and contrast-limiting adaptive histogram averaging and bilateral filtering in an embodiment of the present invention.

[Figure 5] is the confusion matrix for detecting normal teeth, periodontal disease, and dental caries in the embodiment of the present invention.

[Figure 6] is the ROC curve for detecting normal teeth, periodontal disease and dental caries in the embodiment of the present invention.

[Figure 7] is the PR curve for detecting normal teeth, periodontal disease and dental caries in the embodiment of the present invention.

Based on the above technical features, the main functions of the system, computer program and computer-readable media of the present invention for assisting identification of periodontal disease and dental caries using deep learning convolutional neural network will be clearly presented in the following embodiments.

As shown in the first and second figures, the system of this embodiment includes: a data database 1 storing a plurality of single tooth X-ray images A, using three labels of normal, periodontal disease and caries to classify the above single tooth X-ray images A into a normal tooth image A1, a periodontal disease tooth image A2, a carious tooth image A3 and a periodontal disease/caries composite tooth image A4. A computer 2 is connected to the data database 1, and the computer is equipped with an image processing module 21, a deep learning convolutional neural network 22 and an attention model 23. The computer 2 adjusts the size and increment of the single tooth X-ray image A, and performs image processing through the image processing module 21 to enhance the characteristics of the single tooth X-ray image A, and then inputs the convolutional neural network 22 for training, and uses the attention model 23 to strengthen weight distribution, and obtains a multi-label classification model 24 for periodontal disease and dental caries auxiliary identification. of classification.

Referring to Figures 1 to 3 A, in this embodiment, 1139 root-apical X-ray images are used, and each root-apical X-ray image is framed after being diagnosed and marked by a dentist to obtain the X-ray image A of a single tooth. During the test, the computer 2 adjusted the image size of the above-mentioned single tooth X-ray image A to 100×100, and then performed incremental processing, such as rotating the above-mentioned single tooth X-ray image A by 90 degrees, 180 degrees, 270 degrees, horizontal flip or vertical flip, and randomly selected the incremental image to solve the problem of uneven data distribution. In this embodiment, the above-mentioned periodontal disease tooth image A2, caries tooth image The number of images input to the convolutional neural network 22 of A3 and periodontal disease/caries composite tooth image A4 can be consistent, and the number of the above-mentioned normal tooth image A1 input to the convolutional neural network 22 is twice that of the above-mentioned periodontal disease tooth image A2, caries tooth image A3 and periodontal disease/caries composite tooth image A4, thereby avoiding the inconsistency of the number of images and causing the decline in the prediction ability of deep learning. Specifically, in this embodiment, the convolutional neural network 22 is input with a total of 8000 training sets, a total of 2000 verification sets, and a total of 1000 test sets; in the training set, there are 3200 normal tooth images A1, 1600 each of the periodontal disease tooth image A2, carious tooth image A3, and periodontal disease/carious composite tooth image A4; There are 400 images of image A3 and composite tooth image A4 with periodontal disease/dental caries; in the test set, there are 400 images of normal teeth mentioned above A1, and 200 images of teeth with periodontal disease A2, dental caries A3 and composite teeth with periodontal disease/caries A4. After the computer 2 obtains the X-ray image A of the single tooth, it processes the X-ray image A of the single tooth through the image processing module 21 to enhance the features of the X-ray image A of the single tooth. Specifically, the image processing module 21 includes a contrast-limited adaptive histogram averaging 211 and a bilateral filter 212; the above-mentioned single tooth X-ray image A is respectively processed by the contrast-limited adaptive histogram averaging 211, the bilateral filter 212, and sequentially combined with the contrast-limited adaptive histogram averaging 211 and the bilateral filter 212 to obtain three processed images.

The above single tooth X-ray image A is input to the convolutional neural network 22 for training after the three kinds of image processing of the above image processing module 21, and the attention model 23 is used to improve the convolutional neural network 22 to strengthen the weight distribution, so as to obtain a multi-label classification model 24 for the auxiliary identification of periodontal disease and dental caries. The convolutional neural network 22 is enhanced to focus on important pixel values to improve prediction performance.

When actually used for detection, input a root tip X-ray image B of the patient to the computer 2, obtain an X-ray image B1 of a single tooth to be detected by frame selection of the root tip X-ray image B and adjust the image size, perform image processing on the X-ray image B1 of the single tooth to be detected through the image processing module 21, and then use the multi-label classification model 24 for periodontal disease and dental caries to identify the label of the X-ray image B1 of the single tooth to be detected. Any one of the normal tooth image A1, the periodontal disease tooth image A2, the carious tooth image A3, or the periodontal disease/carious composite tooth image A4 can be used for preliminary screening of the patient's teeth, to assist dentists in identifying the patient's dental health status, and to simultaneously identify the composite disease of periodontal disease and dental caries.

Referring to the first and fourth figures, the single tooth X-ray image A processed by the contrast-limited adaptive histogram averaging 211 can effectively enhance the local contrast compared with the original single tooth X-ray image A. The X-ray image A of a single tooth processed by the bilateral filter 212 can effectively reduce the image noise, blur the areas irrelevant to the symptoms, and retain the information used to identify the characteristics of periodontal disease and dental caries, such as alveolar bone loss or erosion and loss of teeth. This embodiment combines the above-mentioned contrast-limited adaptive histogram averaging 211 and bilateral filter 212 to enhance the contrast of the single tooth X-ray image A to make the outline of the tooth clearer and reduce the detail noise while maintaining the original outline of the image.

Refer to the fifth to seventh figures, calculate the confusion matrix (Confusion Matrix), and evaluate it with accuracy (Accuracy), sensitivity (Sensitivity, Recall), specificity (Specificity), precision (Precision), F ₁ score (F ₁ Score), receiver operating characteristic curve (Receiver Operating Characteristic Curve, ROC Curve) and Precision-Recall (PR) curve The performance of the multi-label classification model24 for the aided identification of periodontal disease and dental caries.

See Table 1 below. The average sensitivity (Sen.), specificity (Spe.), accuracy (Acc.), precision (Pre.), F ₁ score, Area Under the ROCCurve (AUROC) and Area Under the PR Curve (AUPR) for normal teeth, periodontal disease, and dental caries are 82.50%, 79.89%, 80.93%, 73.22%, and 77.0%, respectively. 58%, 89.10% and 84.81%. In terms of predicting periodontal disease, the sensitivity, specificity, accuracy, AUROC and AUPR reached 80.00%, 80.33%, 80.20%, 88.00% and 83.38%, respectively. In terms of caries prediction, the sensitivity, specificity, accuracy, AUROC and AUPR reached 82.00%, 78.83%, 80.10%, 88.88% and 86.36%, respectively.

Refer to Table 2 to Table 4 below. After the single tooth X-ray image A is processed by ResNet, Image Processing (IP) combined with ResNet, ResNet combined with Attention Model (ATT), Image Processing (IP) combined with ResNet and Attention Model (ATT), the confusion matrix of normal teeth, periodontal disease and caries is calculated first, and the normal teeth detection, periodontal disease detection and caries detection are calculated through the confusion matrix. The sensitivity (Sen.), specificity (Spe.), accuracy (Acc.), precision (Pre.), F ₁ score, AUROC and AUPR. Table 5 below presents the average performance of the three.

Table 5 (average)

According to the above Table 3, in the detection of periodontal disease, image processing or attention model can improve the performance of residual network prediction. Combining the two can further improve the performance, the sensitivity is increased from 75.00% to 80.00%, the specificity is increased from 79.83% to 80.33%, the accuracy is increased from 77.90% to 80.20%, the accuracy is increased from 71.26% to 73.06%, and the _F1 score is increased from 73.08%. to 76.37%, AUROC increased from 84.61% to 88.00%, and AUPR increased from 80.56% to 83.38%.

According to the above table 4, in the detection of dental caries, image processing or attention model can improve the performance of residual network prediction. Combining the two can further improve the performance, the sensitivity is increased from 77.00% to 82.00%, the specificity is increased from 70.50% to 78.83%, the accuracy is increased from 73.10% to 80.10%, the accuracy is increased from 63.51% to 72.09%, and the _F1 score is increased from 69.61%. 76.73%, AUROC increased from 80.22% to 88.88%, and AUPR increased from 75.57% to 86.36%.

According to the above table 5, the average performance of the three can be improved by adding image processing or attention model. Combining the two can further improve the performance, the sensitivity is increased from 79.08% to 82.50%, the specificity is increased from 76.67% to 79.89%, the accuracy is increased from 77.63% to 80.93%, the accuracy is increased from 69.47% to 73.22%, and the F ₁ score is increased from 73.91% to 77.5% 8%, AUROC increased from 84.58% to 89.10%, and AUPR increased from 79.15% to 84.81%.

According to the above Tables 2 to 5, image processing can improve the prediction performance of periodontal disease and dental caries, attention model can improve the prediction performance of normal, periodontal disease and dental caries, and the combination of the two can further improve the performance.

Based on the description of the above-mentioned embodiments, the operation, use and the effects of the present invention can be fully understood, but the above-mentioned embodiments are only preferred embodiments of the present invention, and the scope of the present invention can not be limited with this, that is, simple equivalent changes and modifications made according to the scope of the patent application for the present invention and the contents of the description of the invention are all within the scope of the present invention.

1: Data database

2: computer

21: Image processing module

211:Contrast-Limited Adaptive Histogram Averaging

212: bilateral filter

22: Convolutional Neural Networks

23: Attention Model

24: Multi-label classification model for auxiliary identification of periodontal disease and dental caries

A: X-ray image of a single tooth

A1: normal tooth image

A2: Periodontal Disease Teeth Imaging

A3: Tooth image with caries

A4: Periodontal disease/caries compound tooth image

B: root tip X-ray image

B1: X-ray image of a single tooth to be detected

Claims (5)

A system for identifying periodontal disease and dental caries using deep learning convolutional neural networks, including: a data database that stores multiple X-ray images of single teeth obtained from apical X-ray images, using three labels of normal, periodontal disease, and caries to classify the X-ray images of single teeth into four types: normal tooth images, periodontal disease tooth images, carious tooth images, and periodontal disease/dental caries composite tooth images; a computer, which is connected to the data database. Neural network and an attention model; the computer obtains the above-mentioned single tooth X-ray image, first adjusts the size and increment, so that the periodontal disease tooth image, dental caries tooth image and periodontal disease/caries composite tooth image of the above-mentioned single tooth X-ray image are consistent in the number of images input to the convolutional neural network, and the number of normal tooth images of the above-mentioned single tooth X-ray image input to the convolutional neural network is 2 times that of the above-mentioned periodontal disease tooth image, dental caries tooth image, and periodontal disease/caries composite tooth image. There are 8000 training sets, 2000 verification sets and 1000 test sets in the network. In the training set, there are 3200 images of normal teeth, 1600 images of teeth with periodontal disease, 1600 images of teeth with caries, and 1600 images of composite teeth with periodontal disease/caries. 400 images, 200 images of periodontal disease teeth, carious teeth and periodontal disease/caries composite teeth; then image processing of the above single tooth X-ray images through the image processing module to strengthen the characteristics of the above single tooth X-ray images, and then input the convolutional neural network for training, and use the attention model to strengthen the weight distribution, and obtain a multi-label classification model for periodontal disease and dental caries auxiliary identification. X-ray images of teeth are respectively averaged by the contrast-limited adaptive histogram, the bilateral filter, and sequentially combined with the contrast-limited adaptive histogram After image processing by averaging and the bilateral filter to obtain three types of processed images, input the convolutional neural network for training, and sequentially combine the contrast-limited adaptive histogram averaging with the bilateral filter to obtain a single tooth X-ray image. An X-ray image of a tooth; image processing is performed on the X-ray image of the single tooth to be detected through the image processing module, and then the multi-label classification model of the aided identification of periodontal disease and caries is used to identify the X-ray image of the single tooth to be detected into any one of the normal tooth image, the periodontal disease tooth image, the caries tooth image or the periodontal disease/caries composite tooth image. The system for assisting identification of periodontal disease and dental caries using a deep learning convolutional neural network as described in Claim 1, wherein the convolutional neural network is a residual network model. The system for identifying periodontal disease and dental caries using deep learning convolutional neural network as described in claim 1, wherein the incremental processing includes any or a combination of rotating the X-ray image of the single tooth by 90 degrees, 180 degrees, 270 degrees, horizontal flip or vertical flip. A computer program for being installed in the system for assisting the identification of periodontal disease and dental caries using a deep learning convolutional neural network as described in any one of claim 1 to claim 3. A computer-readable medium storing the computer program as described in Claim 4.