CN114998300A - Corneal ulcer classification method based on multi-scale information fusion network - Google Patents
Corneal ulcer classification method based on multi-scale information fusion network Download PDFInfo
- Publication number
- CN114998300A CN114998300A CN202210743828.0A CN202210743828A CN114998300A CN 114998300 A CN114998300 A CN 114998300A CN 202210743828 A CN202210743828 A CN 202210743828A CN 114998300 A CN114998300 A CN 114998300A
- Authority
- CN
- China
- Prior art keywords
- scale information
- information fusion
- corneal ulcer
- network model
- fusion network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 71
- 201000007717 corneal ulcer Diseases 0.000 title claims abstract description 68
- 206010064996 Ulcerative keratitis Diseases 0.000 title claims abstract description 60
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000009499 grossing Methods 0.000 claims abstract description 29
- 238000012757 fluorescence staining Methods 0.000 claims abstract description 25
- 238000012360 testing method Methods 0.000 claims abstract description 14
- 238000012549 training Methods 0.000 claims abstract description 13
- 238000007781 pre-processing Methods 0.000 claims abstract description 10
- 238000003062 neural network model Methods 0.000 claims abstract description 9
- 238000009826 distribution Methods 0.000 claims description 17
- 238000011176 pooling Methods 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 14
- 238000006243 chemical reaction Methods 0.000 claims description 13
- 230000003321 amplification Effects 0.000 claims description 8
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 8
- 238000010606 normalization Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 7
- 238000011156 evaluation Methods 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 4
- 238000013434 data augmentation Methods 0.000 claims description 2
- 238000013461 design Methods 0.000 claims description 2
- 239000000654 additive Substances 0.000 claims 1
- 230000000996 additive effect Effects 0.000 claims 1
- 238000011160 research Methods 0.000 abstract description 5
- 238000001514 detection method Methods 0.000 abstract description 4
- 238000012216 screening Methods 0.000 abstract description 4
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 abstract description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000013527 convolutional neural network Methods 0.000 description 11
- 239000007850 fluorescent dye Substances 0.000 description 10
- 230000001575 pathological effect Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000003745 diagnosis Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000007306 turnover Effects 0.000 description 4
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 201000004569 Blindness Diseases 0.000 description 2
- 208000025865 Ulcer Diseases 0.000 description 2
- 230000006378 damage Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000002207 retinal effect Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 231100000397 ulcer Toxicity 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 238000002679 ablation Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 210000004087 cornea Anatomy 0.000 description 1
- 208000021921 corneal disease Diseases 0.000 description 1
- 210000003683 corneal stroma Anatomy 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 210000005081 epithelial layer Anatomy 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 208000030533 eye disease Diseases 0.000 description 1
- GNBHRKFJIUUOQI-UHFFFAOYSA-N fluorescein Chemical compound O1C(=O)C2=CC=CC=C2C21C1=CC=C(O)C=C1OC1=CC(O)=CC=C21 GNBHRKFJIUUOQI-UHFFFAOYSA-N 0.000 description 1
- 208000015181 infectious disease Diseases 0.000 description 1
- 230000002458 infectious effect Effects 0.000 description 1
- 230000002757 inflammatory effect Effects 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30041—Eye; Retina; Ophthalmic
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a corneal ulcer classification method based on a multi-scale information fusion network, which belongs to the field of slit lamp image classification and comprises the steps of preprocessing images in an image data set, constructing a multi-scale information fusion network model, adding a multi-scale information fusion device in the network model, introducing a label smoothing strategy, training and testing the multi-scale information fusion network model, finally classifying two-dimensional slit lamp fluorescence staining target images by using the multi-scale information fusion network model, combining the image preprocessing, the building, the training and the testing of a deep neural network model, enabling the follow-up research on corneal ulcer diseases to be greatly assisted, making better judgment on the two-dimensional slit lamp fluorescence staining images, being beneficial to the classification and the detection of the two-dimensional slit lamp fluorescence staining images and improving the screening efficiency of the two-dimensional slit lamp fluorescence staining images, improving the accuracy of classifying different types of corneal ulcers.
Description
Technical Field
The invention belongs to the field of slit lamp image classification, and particularly relates to a corneal ulcer classification method based on a multi-scale information fusion network.
Background
Corneal ulcer is a serious blinding eye disease, is particularly prominent in developing countries, is one of the main causes of corneal blindness, is an inflammatory or more serious infectious corneal disease, comprises damage of an epithelial layer and involvement of corneal stroma, can bring huge pain to patients, can bring irreversible injury or blindness to human eyes due to untimely treatment or improper treatment, and is an effective way for reducing the blinding rate of corneal ulcer.
Due to the fluorescent characteristic and high visibility of fluorescein under low concentration, the fluorescent staining is the most widely applied method in optometry and ophthalmic diagnosis, and the fluorescent staining is commonly used for observing the integrity of the ocular surface, particularly the integrity of the cornea, so that the fluorescent staining technology becomes a common tool for assisting ophthalmologists in diagnosing corneal ulcer, great convenience is provided for diagnosis and treatment of corneal ulcer, the slit lamp microscope is combined with the fluorescent staining technology to carry out ocular surface examination on high risk groups, and the fluorescent staining image is used for identifying corneal ulcer to play an important role in formulating a treatment scheme.
The current corneal ulcer prevention and treatment mode is that an ophthalmologist classifies (TG) standards according to the general mode and type of the ulcer, the general mode of the ulcer is based on the shape and distribution characteristics of the corneal ulcer and can be divided into three types of point corneal ulcer, point and sheet mixed corneal ulcer and sheet corneal ulcer, and the TG classification method classifies (0-4) types of the corneal ulcer according to the specific type of the corneal ulcer.
However, there is a great challenge in achieving accurate classification of corneal ulcer, mainly because due to differences in subjective experience and professional knowledge of ophthalmologists, doctors have certain differences in identification of corneal ulcer based on fluorescence staining images, which easily causes some severe patients with corneal ulcer to fail to be treated in time, some patients may relapse after treatment, and differences in treatment time and treatment mode selection of the doctors on the relapsed corneal ulcer may cause uncertainty of treatment, meanwhile, corneal ulcer has complex pathological features and is very susceptible to noise interference, different types of corneal ulcers have similarities in pathological forms and distribution, so that the similarity of fluorescence staining images of different types of corneal ulcer is also very high, and finally, the accuracy in classification of different types of corneal ulcer is greatly reduced.
Disclosure of Invention
Problems to be solved
The invention provides a corneal ulcer classification method based on a multi-scale information fusion network, aiming at the problems that the identification of corneal ulcers by existing doctors based on a fluorescence staining image has certain difference, different types of corneal ulcers have similarity in pathological forms and distribution, the fluorescence staining image similarity is very high, and the accuracy of corneal ulcer classification is greatly reduced.
Technical scheme
In order to solve the above problems, the present invention adopts the following technical solutions.
A corneal ulcer classification method based on a multi-scale information fusion network comprises the following steps:
step 1: acquiring a two-dimensional slit lamp fluorescence staining image data set, and preprocessing the image;
step 2: constructing a multi-scale information fusion network model;
and 3, step 3: training and testing the multi-scale information fusion network model;
and 4, step 4: and classifying the two-dimensional slit lamp fluorescence staining target images by using a multi-scale information fusion network model.
Preferably, the preprocessing of the image in step 1 is to perform downsampling on all images in the data set by using a bilinear interpolation method, and then perform normalization processing while performing an online data amplification operation on the image data.
Further, the online data augmentation operation includes random 30 degree rotation, random horizontal flipping, and random vertical flipping of the image.
Preferably, the multi-scale information fusion network model constructed in step 2 is to set up a backbone network, design and add a multi-scale information fusion device on the basis of the backbone network, and finally introduce a label smoothing strategy to optimize the network model.
Furthermore, the multi-scale information fusion network model takes a two-dimensional convolution neural network model as a main network, the main network is provided with a plurality of two-dimensional convolution kernels, four dense connection layers and three conversion layers, the size of the first two-dimensional convolution kernel is 7 × 7, the step length is 2, the sizes of the other two-dimensional convolution kernels are 3 × 3 and 1 × 1 respectively, and the step length of the convolution kernel is 1 or 2.
Furthermore, the multi-scale information fusion device is composed of two self-adaptive average pooling layers, two full-connection layers and an adder, wherein the two self-adaptive average pooling layers are respectively connected with a second conversion layer and a fourth dense connection layer in a backbone network, output characteristics of the second conversion layer and the fourth dense connection layer respectively compress spatial dimensions through the two self-adaptive average pooling layers, obtained results are respectively converted into predicted probability distribution with result dimensions of N1 through the two full-connection layers, N represents the category number of corneal ulcer, and the two predicted probability distributions are finally subjected to addition fusion operation through the adder to obtain fused N1 predicted probability distribution.
Further, the tag smoothing strategy is implemented by adding noise to the tag, and the tag smoothing strategy is formulated as follows:
wherein t and t' respectively represent one-hot labels of the label smooth label, epsilon is a random number between 0.1 and 0.2, I is a matrix with the same dimension as t, the element values of I are all 1, and N is the total number of categories.
Further, the label smoothing strategy adopts a cross entropy loss function, and the formula is as follows:
where m is the number of samples per small batch, K is the total number of input image class labels, x i Representing an input corneal ulcer image, t i Is a class label of the input image, f (-) is an indication function, if t i Equal to k, it is 1, otherwise it is 0.
Preferably, the training of the multi-scale information fusion network model in step 3 is to minimize a cost function through an optimizer Adam, the basic learning rate and the weight attenuation are both set to 0.0001, the batch size is set to 16, and the iteration number is set to 50.
Preferably, three classification evaluation indexes are set for testing the multi-scale information fusion network model in the step 3, where the three classification evaluation indexes are accuracy, a weighted average F1 score and a weighted average area under a curve, and a definition formula of the accuracy and the weighted average F1 score is as follows:
wherein Accuracy is Accuracy, W _ F1-score is F1 score of weighted average, TP, FP, TN and FN are true positive, false positive, true negative and false negative respectively, and W _ P and W _ R represent Accuracy of weighted average and recall of weighted average respectively.
A corneal ulcer classification method based on a multi-scale information fusion network comprises the steps of preprocessing images in an image data set, constructing a multi-scale information fusion network model, adding a multi-scale information fusion device in the network model, introducing a label smoothing strategy, training and testing the multi-scale information fusion network model, classifying two-dimensional slit lamp fluorescence staining target images by using the multi-scale information fusion network model, combining image preprocessing, building, training and testing of a deep neural network model, enabling follow-up researches on corneal ulcer diseases, such as lesion region segmentation, automatic diagnosis and research and the like to be greatly assisted, making better judgment on the two-dimensional slit lamp fluorescence staining images, facilitating classification and detection of the two-dimensional slit lamp fluorescence staining images, and improving screening efficiency of the two-dimensional slit lamp fluorescence staining images, improving the accuracy of classifying different types of corneal ulcers.
Advantageous effects
Compared with the prior art, the invention has the beneficial effects that:
(1) the acquired two-dimensional slit lamp fluorescent staining image data set is preprocessed, a down-sampling method and normalization processing of bilinear interpolation are carried out on the two-dimensional slit lamp fluorescent staining image data set, online data amplification is carried out, the down-sampling method can prevent processing memory overflow, the calculation cost of a learning task is reduced, the difference degree can be improved through the normalization processing, and the diversity and the anti-interference performance of data can be improved through the online data amplification;
(2) the network model constructed by the method is based on the improvement of a convolutional neural network, a multi-scale information fusion device is designed on the basis of the improvement, shallow local information and deep global information can be combined, the loss of low-resolution characteristic information related to categories in a shallow layer is avoided, the shallow information is fully utilized, and meanwhile, the depth prediction information is fused to enhance the robustness of a prediction result;
(3) according to the invention, a label smoothing strategy is introduced in the optimization process of the network model, the category weight corresponding to the real label and the risk of overfitting can be reduced by increasing noise, and the problems of inter-category similarity and intra-category diversity can be alleviated by adopting a cross entropy loss function based on the label smoothing strategy.
Drawings
In order to more clearly illustrate the embodiments or exemplary technical solutions of the present application, the drawings needed to be used in the embodiments or exemplary descriptions will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application and therefore should not be considered as limiting the scope, and it is also possible for those skilled in the art to obtain other drawings according to the drawings without inventive efforts.
FIG. 1 is a schematic representation of the steps of the present invention;
FIG. 2 is a schematic flow chart of the present invention;
FIG. 3 is a schematic diagram of a translation layer structure in the network model of the present invention;
FIG. 4 is a schematic diagram of a dense module structure in the network model of the present invention;
FIG. 5 is a diagram illustrating the classification result of corneal ulcer in the general mode of the present invention;
FIG. 6 is a diagram illustrating the classification of corneal ulcers according to a specific embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, it is obvious that the described embodiments are a part of the embodiments of the present application, but not all of the embodiments, and generally, components of the embodiments of the present application described and illustrated in the drawings herein can be arranged and designed in various different configurations.
Therefore, the following detailed description of the embodiments of the present application, as presented in the figures, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application, and all other embodiments that can be derived by one of ordinary skill in the art based on the embodiments in the present application without making creative efforts fall within the scope of the claimed application.
Example 1
As shown in fig. 1, a corneal ulcer classification method based on a multi-scale information fusion network includes the following specific processes:
the method comprises the steps of obtaining a two-dimensional slit lamp fluorescence staining image data set, preprocessing the image, down-sampling all images in the data set by using a bilinear interpolation method, then normalizing, and simultaneously carrying out online data amplification operation on the image data, wherein the online data amplification operation comprises random 30-degree rotation, random horizontal turnover and random vertical turnover on the image.
A multi-scale information fusion network model is constructed, a backbone network is firstly established, then a multi-scale information fusion device is designed and added on the basis of the backbone network, and finally a label smoothing strategy is introduced to optimize the network model.
The multi-scale information fusion network model takes a two-dimensional convolution neural network model as a backbone network, the backbone network is provided with a plurality of two-dimensional convolution kernels, four dense connection layers and three conversion layers, the size of the first two-dimensional convolution kernel is 7 × 7, the step length is 2, the sizes of other two-dimensional convolution kernels are 3 × 3 and 1 × 1 respectively, and the step length of the convolution kernels is 1 or 2;
the multi-scale information fusion device comprises two self-adaptive average pooling layers, two full-connection layers and an adder, wherein the two self-adaptive average pooling layers are respectively connected with a second conversion layer and a fourth dense connection layer in a main network, the output characteristics of the second conversion layer and the fourth dense connection layer are respectively compressed through the two self-adaptive average pooling layers, the obtained results are respectively converted into prediction probability distribution with result dimensionality of N1 through the two full-connection layers, wherein N represents the category number of corneal ulcer, and the two prediction probability distributions are finally subjected to addition fusion operation through the adder to obtain fused N1 prediction probability distribution;
the label smoothing strategy is realized by adding noise to the label, and the formula of the label smoothing strategy is as follows:
wherein t and t' respectively represent one-hot labels of the label smooth label, epsilon is a random number between 0.1 and 0.2, I is a matrix with the same dimension as t, the element values of I are all 1, and N is the total number of categories;
the label smoothing strategy adopts a cross entropy loss function, and the formula is as follows:
wherein m is eachNumber of samples of small lot, K is total number of input image class labels, x i Representing an input corneal ulcer image, t i Is a class label of the input image, f (-) is an indication function, if t i Equal to k, it is 1, otherwise it is 0.
Training and testing a multi-scale information fusion network model, wherein the training is to minimize a cost function through an optimizer Adam, the basic learning rate and the weight attenuation are both set to be 0.0001, the batch size is set to be 16, and the iteration number is set to be 50; the test needs to set three classification evaluation indexes, namely accuracy, a weighted average F1 score and a weighted average area under a curve, wherein the accuracy and the weighted average F1 score are defined by the following formulas:
wherein Accuracy is Accuracy, W _ F1-score is F1 score of weighted average, TP, FP, TN and FN are true positive, false positive, true negative and false negative respectively, and W _ P and W _ R represent Accuracy of weighted average and recall of weighted average respectively.
And finally, classifying the two-dimensional slit lamp fluorescence staining target images by using a multi-scale information fusion network model.
According to the description, in the embodiment, the images in the image data set are preprocessed to construct the multi-scale information fusion network model, the multi-scale information fusion device is added in the network model, the label smoothing strategy is introduced to train and test the multi-scale information fusion network model, and finally the multi-scale information fusion network model is used for classifying the two-dimensional slit lamp fluorescence staining target images, so that the two-dimensional slit lamp fluorescence staining images can be judged more optimally, the classification and the detection of the two-dimensional retinal fundus color photographic images are facilitated, the screening efficiency of the two-dimensional retinal fundus color photographic images is improved, and the subsequent research on the corneal ulcer diseases, such as pathological area segmentation, automatic diagnosis research and the like, is greatly facilitated by combining the image preprocessing and the building, training and testing of the deep neural network model.
Example 2
Firstly, a two-dimensional slit lamp fluorescence staining image data set is obtained, images are preprocessed, the size of an original two-dimensional slit lamp fluorescence staining image in the image data set is 2592 multiplied by 1728, the general mode of corneal ulcer can be divided into point corneal ulcer, point and sheet mixed corneal ulcer and sheet corneal ulcer, the specific mode of corneal ulcer can be divided into 0 to 4 type corneal ulcer, in order to reduce the calculation cost of a learning task, all two-dimensional slit lamp images are downsampled by utilizing bilinear interpolation, the image size is made to be 320 multiplied by 320, then normalization processing is carried out to improve the difference, in order to prevent overfitting of the model and enhance the generalization capability of the model, and meanwhile, online data amplification operation is carried out on the image data, wherein the online data amplification operation comprises random 30-degree rotation, random horizontal turnover and random vertical turnover of the image.
The construction of the multi-scale information fusion network model needs to use a two-dimensional convolution neural network model (DenseNet121) as a backbone network, and aiming at the complex pathological characteristics of different corneal ulcers, a multi-scale information fusion device is designed and added on the basis of the two-dimensional convolution neural network model (DenseNet121), and considering that the pathological forms and the distribution of different types of corneal ulcers have similarity, a label smoothing strategy is introduced in the network optimization process.
The original two-dimensional convolutional neural network model (Densenet121) is composed of a conversion layer and dense connection layers and comprises a two-dimensional convolutional layer, a two-dimensional maximum pooling layer, a two-dimensional batch normalization layer, an average pooling layer, a global average pooling layer, a full connection layer and a softmax output layer, wherein the dense connection layers are composed of dense modules, and 4 dense connection layers are respectively composed of 6, 12, 24 and 16 dense modules.
The main network of the multi-scale information fusion network model is provided with a plurality of two-dimensional convolution kernels, four dense connection layers and three conversion layers, the size of the first two-dimensional convolution kernel is 7 x 7, the step size is 2, the sizes of other two-dimensional convolution kernels are 3 x 3 and 1 x 1 respectively, the step size of the convolution kernel is 1 or 2, the sizes of the kernels of the maximum pooling layer and the average pooling layer are 3 x 3, and the step size is 2, so that the purpose is to not merge depth information early, and the parameter amount of the network can be reduced and the robustness of the network can be enhanced.
The multi-scale information fusion device is composed of two self-adaptive average pooling layers, two full-connection layers and an adder, wherein the two self-adaptive average pooling layers are respectively connected with a second conversion layer and a fourth dense-connection layer in a backbone network, output characteristics of the second conversion layer and the fourth dense-connection layer are respectively compressed through the two self-adaptive average pooling layers, the obtained result dimensions are 256 × 1 and 1024 × 1, the numerical distribution conditions of corresponding 256 and 1024 characteristic graphs are shown, a calculation result is respectively converted into a prediction probability distribution with a result dimension of N × 1 through the two full-connection layers, wherein N represents the classification number of corneal ulcer, and the two prediction probability distributions are finally subjected to addition fusion operation through the adder to obtain the fused N × 1 prediction probability distribution.
The multi-scale information fusion device can combine the shallow local information with the deep global information, on one hand, the shallow information is fully utilized, the loss of low-resolution characteristic information related to categories in the shallow layer is avoided, and on the other hand, the robustness of a prediction result can be improved by combining the shallow edge information with the deep semantic information.
The label smoothing strategy is implemented by adding noise to the label, assuming D e (x) i ,y i ) (i ═ 1,2, …, M) is a sorted dataset of M samples, where x is i And y i Representing the input image and the corresponding class label, respectively, a standard multi-classification problem is to predict the input image x i Probability of belonging to class k (y) i K) when class k is encoded as a vector t (0,0, …,0,1,0, …,0) by one-hot tags, where the value of the kth position is 1 and the rest are all 0, this form of tag encoding encourages the model to learn towards the direction of maximum difference between the correct and wrong tags, meaning that only the loss of the correct tag position is calculated during the model's optimization, however, when one-hot tags are encoded as vectors t (0,0, …,0,1,0, …,0)When the inter-class similarity and intra-class difference are relatively large, it may result in a network overfitting.
In order to solve the above problems, the present invention introduces label smoothing, which is a regularization strategy, and mainly reduces the weight of the real label category by adding noise, and slightly increases the penalty of the wrong label category in the model training, and the formula of the label smoothing strategy is as follows:
wherein t and t' respectively represent one-hot labels of the label smooth label, epsilon is a random number between 0.1 and 0.2, I is a matrix with the same dimension as t, the element values of I are all 1, and N is the total number of categories;
the label smoothing strategy adopts a cross entropy loss function, and the formula is as follows:
where m is the number of samples per small batch, K is the total number of input image class labels, x i Representing an input corneal ulcer image, t i And k a class label of the input image, f (-) is an indication function, if t i Equal to k, it is 1, otherwise it is 0.
712 two-dimensional slit-lamp fluorescence staining images are used as a data set, 5-fold cross validation is adopted on the whole data set to evaluate the performance of the invention, in order to reduce the calculation cost of a learning task, all the two-dimensional slit-lamp fluorescence staining images are down-sampled to 320 x 320 by using a bilinear interpolation method, and then normalization processing is carried out to improve the difference.
To prevent overfitting of the model and enhance the generalization ability of the model, the data is augmented online during training to increase the diversity of the data, including random rotation by 30 degrees, random horizontal flipping, and random vertical flipping.
Training and testing of the model are completed based on an integrated environment of a Pytorch and an NVIDIA GTX Titan X GPU with 12GB storage space, the model is trained by minimizing a loss function through a back propagation algorithm, a cost function is minimized by using an optimizer Adam, and both a basic learning rate and weight attenuation are set to be 0.0001. The batch size was set to 16 and the number of iterations was set to 50.
In order to quantitatively evaluate the performance of the invention, the test needs to set three classification evaluation indexes, namely accuracy, a weighted average F1 score and a weighted average area under a curve, wherein the accuracy and the weighted average F1 score are defined by the following formulas:
wherein Accuracy is Accuracy, W _ F1-score is F1 score of weighted average, TP, FP, TN and FN are true positive, false positive, true negative and false negative respectively, and W _ P and W _ R represent Accuracy of weighted average and recall of weighted average respectively.
To demonstrate the effectiveness of the multi-scale fuser and label smoothing strategy, a series of ablation experiments were performed. The results of the experiment are shown in fig. 5 and 6:
"backbone network" represents the original two-dimensional convolutional neural network model (DenseNet 121);
"backbone network + multi-scale information fusion device" means adding multi-scale information fusion device in original two-dimensional convolution neural network model (DenseNet 121);
"backbone network + tag smoothing" means that a tag smoothing strategy is used in the original two-dimensional convolutional neural network model (DenseNet 121);
"LmNet" means a method published on journal name "IEEEACCESS" (DOI) of 10.1109/ACCESS.2021.3093308 on 7/3/2021;
the "proposed method" means the method proposed in the present invention, i.e., "backbone network + multi-scale information fusion device + label smoothing".
For the classification of corneal ulcer in general mode, it can be seen from fig. 5 that the classification accuracy of the original two-dimensional convolutional neural network model (DenseNet121) is 84.39%, the classification accuracy of LmNet is 85.52%, and the improved classification Accuracy (ACC) of the present invention can reach 87.07%. The weighted average F1 score (W _ F1) and the weighted average area under the curve (W _ AUC) of the present invention were 86.82% and 92.20%, respectively, which were 3.05% and 1.63% higher than the original densnet 121, respectively.
From fig. 5, it can be seen that the multi-scale information fusion device and the label smoothing strategy designed in the present invention can effectively improve the classification accuracy of the original two-dimensional convolutional neural network model (DenseNet121), and the classification accuracy is higher than that of LmNet.
For the corneal ulcer classification of a specific pattern, as can be seen from fig. 6, the classification accuracy of the original two-dimensional convolutional neural network model (DenseNet121) is 81.45%, the classification accuracy of LmNet is 82.42%, the improved classification Accuracy (ACC) of the present invention can reach 83.84%, and the accuracy is improved by 2.93% and 1.72% respectively compared with the original two-dimensional convolutional neural network model (DenseNet121) and LmNet. The weighted average F1 score (W _ F1) and the weighted average area under the curve (W _ AUC) of the present invention were 80.52% and 91.11%, respectively, which were 4.19% and 0.62% higher than the original densnet 121, respectively.
From fig. 6, it can be seen that the multi-scale information fusion device and the label smoothing strategy designed in the present invention can effectively improve the classification accuracy of the original two-dimensional convolutional neural network model (DenseNet 121).
The invention provides a multi-scale information fusion device and a quoted label smoothing strategy, so that the accuracy of classification and identification of corneal ulcer is ensured.
Thus, a corneal ulcer classification method for slit-lamp fluorescence staining images has been implemented and validated. The performance of the method in an experiment is superior to that of an original two-dimensional convolutional neural network model (DenseNet121), the method can make better judgment on a two-dimensional slit lamp fluorescent staining image, on the other hand, a multi-scale information fusion device designed in the method is not complex and can be embedded into any other convolutional neural network, so that the characteristic extraction capability of the network model is stronger, the overall performance of the network model is improved, the classification and detection of the two-dimensional slit lamp fluorescent staining image are facilitated, and the screening efficiency of the two-dimensional slit lamp fluorescent staining image is greatly improved.
The above examples are merely representative of preferred embodiments of the present invention, and the description thereof is more specific and detailed, but not to be construed as limiting the scope of the present invention. It should be noted that various changes, modifications and substitutions may be made by those skilled in the art without departing from the spirit of the invention, and all are intended to be included within the scope of the invention.
Claims (10)
1. A corneal ulcer classification method based on a multi-scale information fusion network is characterized by comprising the following steps:
step 1: acquiring a two-dimensional slit lamp fluorescence staining image data set, and preprocessing the image;
step 2: constructing a multi-scale information fusion network model;
and step 3: training and testing the multi-scale information fusion network model;
and 4, step 4: and classifying the two-dimensional slit lamp fluorescence staining target images by using a multi-scale information fusion network model.
2. The corneal ulcer classification method based on the multi-scale information fusion network as claimed in claim 1, wherein: the preprocessing of the image in the step 1 is to perform downsampling on all images in the data set by using a bilinear interpolation method, then perform normalization processing, and perform online data amplification operation on the image data.
3. The corneal ulcer classification method based on the multi-scale information fusion network as claimed in claim 2, wherein: the online data augmentation operation includes random 30 degree rotation, random horizontal flipping, and random vertical flipping of the image.
4. The corneal ulcer classification method based on the multi-scale information fusion network as claimed in claim 1, wherein: the multi-scale information fusion network model constructed in the step 2 is to set up a backbone network, design and add a multi-scale information fusion device on the basis of the backbone network, and finally introduce a label smoothing strategy to optimize the network model.
5. The corneal ulcer classification method based on the multi-scale information fusion network as claimed in claim 4, wherein: the multi-scale information fusion network model takes a two-dimensional convolution neural network model as a backbone network, the backbone network is provided with a plurality of two-dimensional convolution kernels, four dense connection layers and three conversion layers, the size of the first two-dimensional convolution kernel is 7 x 7, the step length is 2, the sizes of other two-dimensional convolution kernels are 3 x 3 and 1 x 1 respectively, and the step length of the convolution kernel is one of 1 and 2.
6. The corneal ulcer classification method based on the multi-scale information fusion network as claimed in claim 5, wherein: the multi-scale information fusion device is composed of two self-adaptive average pooling layers, two full-connection layers and an adder, wherein the two self-adaptive average pooling layers are respectively connected with a second conversion layer and a fourth dense-connection layer in a backbone network, output characteristics of the second conversion layer and the fourth dense-connection layer are respectively compressed through the two self-adaptive average pooling layers, obtained results are respectively converted into predicted probability distribution with result dimensionality of N1 through the two full-connection layers, N represents the category number of corneal ulcer, and the two predicted probability distributions are finally subjected to additive fusion through the adder to obtain fused N1 predicted probability distribution.
7. The corneal ulcer classification method based on the multi-scale information fusion network as claimed in claim 6, wherein: the label smoothing strategy is realized by adding noise to the label, and the formula of the label smoothing strategy is as follows:
wherein t and t' respectively represent one-hot labels of the label smooth label, epsilon is a random number between 0.1 and 0.2, I is a matrix with the same dimension as t, the element values of I are all 1, and N is the total number of categories.
8. The corneal ulcer classification method based on the multi-scale information fusion network as claimed in claim 7, wherein: the label smoothing strategy adopts a cross entropy loss function, and the formula is as follows:
where m is the number of samples per small batch, K is the total number of input image class labels, x i Representing the input corneal ulcer image, t i Is a class label of the input image, f (-) is an indication function, if t i Equal to k, it is 1, otherwise it is 0.
9. The corneal ulcer classification method based on the multi-scale information fusion network as claimed in claim 1, wherein: in the step 3, the multi-scale information fusion network model is trained by minimizing a cost function through an optimizer, the basic learning rate and the weight attenuation are both set to be 0.0001, the batch size is set to be 16, and the iteration number is set to be 50.
10. The corneal ulcer classification method based on the multi-scale information fusion network as claimed in claim 1, wherein: in the step 3, three classification evaluation indexes, namely, an accuracy, a weighted average F1 score and a weighted average area under a curve, need to be set for testing the multi-scale information fusion network model, where the accuracy and the weighted average F1 score are defined by the following formulas:
wherein Accuracy is Accuracy, W _ F1-score is F1 score of weighted average, TP, FP, TN and FN are true positive, false positive, true negative and false negative respectively, and W _ P and W _ R represent Accuracy of weighted average and recall of weighted average respectively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210743828.0A CN114998300A (en) | 2022-06-27 | 2022-06-27 | Corneal ulcer classification method based on multi-scale information fusion network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210743828.0A CN114998300A (en) | 2022-06-27 | 2022-06-27 | Corneal ulcer classification method based on multi-scale information fusion network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114998300A true CN114998300A (en) | 2022-09-02 |
Family
ID=83037458
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210743828.0A Pending CN114998300A (en) | 2022-06-27 | 2022-06-27 | Corneal ulcer classification method based on multi-scale information fusion network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114998300A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117877103A (en) * | 2024-03-13 | 2024-04-12 | 宁波市眼科医院 | Intelligent keratitis screening method based on deep meta learning |
-
2022
- 2022-06-27 CN CN202210743828.0A patent/CN114998300A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117877103A (en) * | 2024-03-13 | 2024-04-12 | 宁波市眼科医院 | Intelligent keratitis screening method based on deep meta learning |
| CN117877103B (en) * | 2024-03-13 | 2024-05-24 | 宁波市眼科医院 | Intelligent keratitis screening method based on deep meta learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | Fully automated detection of retinal disorders by image-based deep learning | |
| Yan et al. | Joint segment-level and pixel-wise losses for deep learning based retinal vessel segmentation | |
| Luo et al. | Ophthalmic disease detection via deep learning with a novel mixture loss function | |
| CN111259982A (en) | Premature infant retina image classification method and device based on attention mechanism | |
| Uysal et al. | Computer-aided retinal vessel segmentation in retinal images: convolutional neural networks | |
| CN111046835A (en) | Eyeground illumination multiple disease detection system based on regional feature set neural network | |
| Da Rocha et al. | Diabetic retinopathy classification using VGG16 neural network | |
| CN112308830A (en) | Attention mechanism and deep supervision strategy-based automatic division identification method for retinopathy of prematurity | |
| Islam et al. | Transfer learning based diabetic retinopathy detection with a novel preprocessed layer | |
| CN113610118B (en) | Glaucoma diagnosis method, device, equipment and method based on multitasking course learning | |
| CN117058676B (en) | Blood vessel segmentation method, device and system based on fundus examination image | |
| Qin et al. | A review of retinal vessel segmentation for fundus image analysis | |
| CN114998300A (en) | Corneal ulcer classification method based on multi-scale information fusion network | |
| Tian et al. | Learning discriminative representations for fine-grained diabetic retinopathy grading | |
| Galveia et al. | Computer aided diagnosis in ophthalmology: Deep learning applications | |
| Tuncer et al. | Hybrid CNN Based Computer-Aided Diagnosis System for Choroidal Neovascularization, Diabetic Macular Edema, Drusen Disease Detection from OCT Images. | |
| Burlina et al. | Artificial intelligence and deep learning in retinal image analysis | |
| Saranya et al. | Detection of exudates from retinal images for non-proliferative diabetic retinopathy detection using deep learning model | |
| Chakraborty et al. | Early Prediction of Cataract using Convolutional Neural Network | |
| Ahmed et al. | An effective deep learning network for detecting and classifying glaucomatous eye. | |
| Pavani et al. | Robust semantic segmentation of retinal fluids from SD-OCT images using FAM-U-Net | |
| Sharma et al. | Blood vessel segmentation using SegNet | |
| Hatode et al. | Evolution and Testimony of Deep Learning Algorithm for Diabetic Retinopathy Detection | |
| CN114926460B (en) | Training method of fundus image classification model, and fundus image classification method and system | |
| CN114913592B (en) | Fundus image classification method based on convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |