CN113160151A

CN113160151A - Panoramic film dental caries depth identification method based on deep learning and attention mechanism

Info

Publication number: CN113160151A
Application number: CN202110360226.2A
Authority: CN
Inventors: 叶冠琛; 曹政; 朱海华; 朱赴东; 陈谦明
Original assignee: Zhejiang University ZJU
Current assignee: Zhejiang University ZJU
Priority date: 2021-04-02
Filing date: 2021-04-02
Publication date: 2021-07-23
Anticipated expiration: 2041-04-02
Also published as: CN113160151B

Abstract

The invention discloses a panoramic film decayed tooth depth identification method based on a depth learning and attention mechanism, which realizes the grading of different decayed tooth lesion periods in an oral panoramic film by using a depth learning model containing the attention mechanism. The entire caries grading system consists of three modules: the system comprises a segmentation network module, an attention extraction module and a classification module. The automatic identification of the decayed tooth area in the panoramic picture of the oral cavity and the derivation of the results of different decayed tooth pathological changes are realized, which is helpful for assisting accurate grading, provides basis for preventing the occurrence and development of decayed tooth and has important clinical and social significance for the maintenance of oral health.

Description

Panoramic film dental caries depth identification method based on deep learning and attention mechanism

Technical Field

The invention belongs to the field of medical treatment, and relates to a panoramic film decayed tooth depth identification method based on a deep learning and attention mechanism.

Background

Dental caries is a progressive lesion of hard tooth tissue caused by the combined action of various factors in the oral cavity, is a main common disease of the oral cavity and is one of the most common diseases of human beings. Clinically, the color, shape and quality of the decayed tooth can be changed, the change of color and shape is the result of quality change, the lesion enters the dentin from the enamel along with the development of the course of disease, the tissue is continuously destroyed and disintegrated to gradually form a decayed cavity, and the decayed tooth is divided into three stages of light, medium and deep decayed tooth according to the extent of decayed tooth. Due to the increasing demand for oral health by the general public at present, more and more people go to hospitals or clinics for oral consultation or treatment. However, because of more patients and limited diagnosis and treatment time, sometimes doctors have to pay attention to the symptomatic teeth in priority, but neglect other dental caries with lower potential degree, further development of the dental caries is avoided, and treatment difficulty and treatment cost are improved. The panoramic film is the most common auxiliary examination means in clinic oral cavity, comprises all teeth in the mouth, and has low cost and small radiation quantity. With the improvement of modern medical level, how to realize automatic detection and classification based on the oral panoramic by using electronic information technology becomes a popular research topic.

As an important component of artificial intelligence, the deep learning technology has good effect in the field of auxiliary medical diagnosis. Deep learning can achieve better generalization performance by learning features using more data volume than traditional methods. The deep neural network in deep learning can automatically extract features in the network without artificial feature selection, and the extracted features can be classified according to the full connection layer of the network, so that feature extraction and classification are combined together, and a result superior to that obtained by a traditional method is obtained.

However, most of the existing deep learning methods use images enhanced by simple data to identify decayed teeth. Firstly, an input image in training data is transmitted into a neural network consisting of a series of convolution structures and full-connection structures, then trained neural network parameters are stored, and finally the trained neural network is used for predicting test data to obtain a prediction result of dental caries classification. Due to the black box characteristic of the neural network, the deep learning method may lose some small characteristic information, only highlights main characteristics, and cannot accurately identify some decayed tooth areas with small expression range in the aspect of caries classification.

Therefore, the development of an improved deep learning algorithm for accurately performing the automatic identification method of the decayed tooth is a problem which needs to be solved in the field.

Disclosure of Invention

The invention aims to provide a panoramic caries depth identification method based on deep learning and attention mechanism on the basis of grading caries degree by using the existing deep learning method, and provides an innovative algorithm for assisting caries classification by using the attention mechanism and feature fusion. The evaluation index of the grading system is improved while the running speed and the generalization performance are ensured.

The invention uses the oral panoramic picture as data input to enter the segmentation network, calculates the corresponding attention characteristic in each layer of characteristic output of the segmentation network, and finally fuses the attention characteristic and the original convolution characteristic, so that the network attention is focused on partial fine characteristic areas which are beneficial to classification in the segmentation process, thereby improving the accurate segmentation of the dental caries lesion in the fine area, and then uses the classification network to extract and classify the characteristics of the segmentation network, thereby effectively improving the accuracy of dental caries classification. The entire caries grading system consists of three modules: the system comprises a segmentation network module, an attention extraction module and a classification module.

In order to achieve the purpose, the invention provides the following technical steps:

1. data preprocessing: carrying out data marking and standardized preprocessing by using the oral panoramic film in the database to prepare a training set; specifically, (1) collect clear oral panoramic picture of picture from the radio examination database, regulate luminance and contrast in unison; (2) the carious part of the carious tooth in the oral cavity panoramic film is artificially marked.

2. Dividing the decayed tooth area: in the segmentation network module, a U-Net segmentation network is used for segmenting normal teeth and decayed tooth parts in the panoramic picture of the oral cavity, separating an oral cavity background area and a target area and extracting corresponding image blocks; the image digital sampling is carried out on the collected oral panoramic picture, and then the segmentation and extraction of the background and the target area are realized through a segmentation network module.

3. Attention calculation: calculating an attention feature map captured by a neural network corresponding to the caries position in a corresponding U-Net segmentation network by using an attention extraction module; specifically, the method comprises the following steps: (1) taking a feature graph generated by a segmentation network as input, and obtaining a feature graph with a weight value through an attention extraction module, namely the feature graph representing attention in deep learning; (2) the label of the original image corresponding to the feature map can obtain a loss value through the loss function, and the loss value is used as a part of the total loss value.

4. Grading the degree of caries: in a classification module, combining the extracted image blocks output by the object segmentation network and the feature map output by the attention module thereof, and grading the lesion degree of the decayed tooth by using a ResNet classification network;

specifically, the method comprises the following steps: (1) extracting the characteristics of the middle layers of the segmentation network module and the attention extraction module to be used as input data of the modules, and obtaining the attention weight of the input data under the action of a convolutional neural network; (2) respectively applying different attention to the two input data to obtain attention characteristics; (3) and fusing the two attention feature graphs to obtain a new feature graph, and simply classifying the newly generated feature graph to obtain a final prediction result.

The attention extraction module takes a feature map generated by segmenting the network as input, and after the network is classified through the feature map, a probability vector of 3 categories is obtained, and the probability vector and a label of an original image corresponding to the feature map can obtain a loss value through a loss function, wherein the loss value is used as a part of a total loss value. The original image classification network module takes an original image as input, and after passing through the original image classification network, a probability vector of 3 categories is obtained, and the probability vector and a label of the original image can obtain a loss value through a loss function, wherein the loss value is used as a part of the same total loss value.

And the classification module in the fourth step extracts the characteristics of the middle layers of the segmentation network module and the attention extraction module as input data of the module, obtains the attention weight of the input data through the function of a convolutional neural network, then obtains an attention characteristic diagram by respectively applying different attention to the two input data, finally fuses the two attention characteristic diagrams to obtain a new characteristic diagram, and then simply classifies the newly generated characteristic diagram to obtain a final prediction result.

The invention has the advantages that:

1. different from the traditional method of directly predicting the decayed tooth grading by simply using deep learning, the invention introduces an attention mechanism on the basis of a simple deep learning prediction system, combines with a segmentation network to segment the decayed tooth area, and then uses a classification network to grade the decayed tooth area, so that the stability of the system is better.

2. Due to the adoption of the attention mechanism, the system can well identify and grade the decayed tooth area with smaller characteristics and difficult grading, so that the performance of the system is greatly improved as a whole.

3. The invention does not need manual participation in the using process, and belongs to a full-automatic grading system based on an attention mechanism and feature fusion.

Drawings

Fig. 1 is an overall work flow diagram.

Fig. 2 is a diagram of a split network module.

Fig. 3 is a diagram of an attention-extracting module.

Fig. 4 is a block diagram of a classification network.

Detailed Description

The whole process of the panoramic film caries depth identification method based on the depth learning and attention mechanism will be described with reference to the drawings and the embodiment.

In the whole method, the decayed tooth is graded and decomposed into three mutually related modules, and finally, a decayed tooth grading system based on attention mechanism is realized.

Example 1

Referring to fig. 1, the workflow of the entire recognition method is shown: first, the original panoramic image of the oral cavity is converted into 572 × 572 in a unified manner, and the converted image is loaded to the segmentation network module as input data. And inputting the feature map output by the segmentation network module into an attention extraction module to obtain a corresponding attention feature map. And finally, inputting the characteristic diagram output by the segmentation network module and the characteristic diagram output by the attention mechanism module into a classification network module to obtain a final caries classification result.

The operation method of each module in the present invention is described with reference to fig. 2 to 4.

Fig. 2 illustrates a segmented net module, where the input of the net is a picture of 572 x 572, and the left side of the net is a series of downsampling operations consisting of convolution and max pooling, each operation consisting of 4 blocks, each block using 3 effective convolutions and 1 max pooling downsampling, and the number of feature maps after each downsampling is multiplied by 2. A characteristic map with a dimension of 32 × 32 was finally obtained. The right part of the network also consists of 4 blocks, the size of the feature map is multiplied by 2 by deconvolution before each block starts, meanwhile, the number of the blocks is halved, and then the feature maps are merged with the feature map of the left symmetrical compression path, and finally, the feature map with the size of 388 × 388 is obtained, namely, the result of the network segmentation.

Figure 3 illustrates an attention mechanism module. The input feature graph is respectively subjected to maximum pooling and average pooling, then 7x7 convolution operation of the image is carried out by using the convolutional layer, the convolutional layer outputs corresponding features to be loud, then a weight graph Ms is obtained through a sigmoid activation function, and then the weight graph Ms and the input feature graph are subjected to weighted average operation to obtain a final feature graph. The weight graph output by the module reflects the importance of different positions of the input feature graph, and places with large weights indicate that the places are more important, and the network puts more attention on the places, namely the application of the attention mechanism in the system.

The classification network module is presented in fig. 4, and in consideration of the balance between the system speed and the accuracy, a residual network of 41 layers is used in the present invention. Phase 1, phase 2, …, and phase 5 are 5 phases of the Resnet network. 2X and 4X are module repetition times. In Conv (64,256, k is (1,1), s is 1, and p is 0), k is the convolution kernel size, s is the sliding step, p is the fill pixel, 256 is the convolution kernel channel number, 64 is the channel number output by the previous convolution layer, and the rest of the convolution layers are similar. The main network is formed by stacking residual error learning modules, the head end and the tail end of each residual error learning module are 1 multiplied by 1 convolution kernel, and the middle of each residual error learning module is 3 multiplied by 3 convolution kernel. In the first learning module of each stage of the network, except the series connection of 3 convolutional layers, the input and the output are connected through a convolutional layer bypass to increase the number of channels of the input characteristic diagram, so that the input characteristic diagram is convenient to fuse with the output characteristic diagram, and the number of channels of the input characteristic diagram and the output characteristic diagram of the subsequent residual learning module is consistent, so that the addition operation can be directly carried out without increasing the dimension through the convolutional layers. The structure can effectively reduce characteristic loss and improve the training effect of the model.

Claims

1. The panoramic film decayed tooth depth identification method based on the deep learning and attention mechanism is characterized by comprising the following steps of:

(1) data preprocessing: carrying out data marking and standardized preprocessing by using the oral panoramic film in the database to prepare a training set;

(2) dividing the decayed tooth area: using a U-Net object segmentation network to segment normal teeth and decayed tooth parts in the oral panoramic image, separating an oral background area and a target area, and extracting corresponding image blocks;

(3) attention calculation: calculating an attention feature map captured by a neural network corresponding to the caries position in a corresponding U-Net segmentation network by using an attention extraction module;

(4) grading the degree of caries: and (3) grading the lesion degree of the decayed tooth by using a ResNet classification network in combination with the extracted image blocks output by the object segmentation network and the feature map output by the attention extraction module.

2. The method according to claim 1, characterized in that said step (1) comprises in particular:

(a) collecting clear oral panoramic pictures from a radiology examination database, and uniformly adjusting brightness and contrast;

(b) the carious part of the carious tooth in the oral cavity panoramic film is artificially marked.

3. The method according to claim 1, wherein the step (2) comprises in particular: the collected oral panoramic picture is subjected to image digital sampling, and then the segmentation and extraction of the background and the target area are realized through a segmentation network module.

4. The method according to claim 1, wherein the step (3) comprises in particular:

(a) the feature graph generated by the segmentation network is used as input, and a feature graph with weight, namely the feature graph representing attention in deep learning, is obtained through the attention extraction module.

(b) The label of the original image corresponding to the feature map can obtain a loss value through the loss function, and the loss value is used as a part of the total loss value.

5. The method according to claim 1, characterized in that said step (4) comprises in particular:

(a) extracting the characteristics of the middle layers of the segmentation network module and the attention extraction module to be used as input data of the modules, and obtaining the attention weight of the input data under the action of a convolutional neural network;

(b) respectively applying different attention to the two input data to obtain attention characteristics;

(c) and fusing the two attention feature graphs to obtain a new feature graph, and simply classifying the newly generated feature graph to obtain a final prediction result.