CN116310525A - Pathological image classification method based on contrast representation distillation and output distillation - Google Patents
Pathological image classification method based on contrast representation distillation and output distillation Download PDFInfo
- Publication number
- CN116310525A CN116310525A CN202310194883.3A CN202310194883A CN116310525A CN 116310525 A CN116310525 A CN 116310525A CN 202310194883 A CN202310194883 A CN 202310194883A CN 116310525 A CN116310525 A CN 116310525A
- Authority
- CN
- China
- Prior art keywords
- distillation
- image
- classification
- pathological
- loss
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000001575 pathological effect Effects 0.000 title claims abstract description 101
- 238000004821 distillation Methods 0.000 title claims abstract description 65
- 238000000034 method Methods 0.000 title claims abstract description 46
- 206010058467 Lung neoplasm malignant Diseases 0.000 claims abstract description 24
- 201000005202 lung cancer Diseases 0.000 claims abstract description 24
- 208000020816 lung neoplasm Diseases 0.000 claims abstract description 24
- 230000007170 pathology Effects 0.000 claims description 28
- 238000000605 extraction Methods 0.000 claims description 24
- 230000006870 function Effects 0.000 claims description 21
- 238000012549 training Methods 0.000 claims description 18
- 238000002372 labelling Methods 0.000 claims description 10
- 230000003902 lesion Effects 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 7
- 230000004913 activation Effects 0.000 claims description 6
- 239000013598 vector Substances 0.000 claims description 6
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 239000003086 colorant Substances 0.000 claims description 3
- 238000002790 cross-validation Methods 0.000 claims description 3
- 201000010099 disease Diseases 0.000 claims description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 230000008685 targeting Effects 0.000 claims description 3
- 238000012706 support-vector machine Methods 0.000 claims description 2
- 238000010276 construction Methods 0.000 claims 2
- 238000005457 optimization Methods 0.000 abstract description 2
- 239000000284 extract Substances 0.000 abstract 1
- 238000003745 diagnosis Methods 0.000 description 14
- 201000011510 cancer Diseases 0.000 description 7
- 206010028980 Neoplasm Diseases 0.000 description 6
- 210000004072 lung Anatomy 0.000 description 5
- 208000010507 Adenocarcinoma of Lung Diseases 0.000 description 3
- 201000005249 lung adenocarcinoma Diseases 0.000 description 3
- 206010041823 squamous cell carcinoma Diseases 0.000 description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000000052 comparative effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000000452 restraining effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000002512 chemotherapy Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002651 drug therapy Methods 0.000 description 1
- 238000013399 early diagnosis Methods 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010827 pathological analysis Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000004393 prognosis Methods 0.000 description 1
- 238000001959 radiotherapy Methods 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 230000004083 survival effect Effects 0.000 description 1
- 238000002626 targeted therapy Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a pathological image classification method based on contrast representation distillation and output distillation. The weak supervision model is used as a classification network, the data dependence of the model is greatly reduced, the comparison is adopted to represent distillation, so that the weak supervision model extracts more obvious and more distinguishable deep features, and the output distillation is adopted to guide the optimization direction of the weak supervision model through the high-precision and high-accuracy full supervision model. The output distillation and the comparison show that the distillation can greatly improve the prediction capability of different lung cancer subtypes while the weak supervision model keeps high recognition capability of normal tissue samples. In addition, through the design of the depth gating attention module, accurate prediction of normal tissue samples and lung cancer subtypes is realized.
Description
Technical Field
The invention relates to the technical field of medical image processing and the field of artificial intelligence, in particular to a lung cancer pathological image classification method based on contrast expression distillation and output distillation.
Background
According to the data of the occurrence and death of Chinese cancer in 2020, the new cases and the death cases of the lung cancer respectively account for 17.9% and 23.8% of all cancer cases. Lung cancer is malignant tumor with highest incidence and mortality rate in China, and seriously threatens the life health and safety of people. If the patient is in early stage of lung cancer at the time of diagnosis, methods such as targeted therapy, drug therapy, surgery and the like can be adopted, and the medical cost of the patient in early stage of lung cancer is usually less than 10 ten thousand yuan. If the patient has progressed to the middle and late stage when lung cancer is diagnosed, only radiotherapy or chemotherapy with great harm to the body can be adopted, the medical cost is usually 10 to 30 ten thousand yuan, and the average cost of people is 2.6 times that of early patients. Therefore, early diagnosis of lung cancer is an important approach to increase patient survival, improve prognosis, and reduce medical costs.
The pathological diagnosis is a gold standard for lung cancer diagnosis, a doctor acquires tissues of a patient through a lung fine needle puncture, a surgical operation and the like, then makes a full-view digital pathological image, and then searches a focus area in the pathological image through continuously adjusting the view scale to finish diagnosis. However, the manual diagnosis is easily affected by subjective factors, fatigue, diagnosis experience and other external factors, and misdiagnosis or missed diagnosis is easily caused, so that the treatment of the patient is delayed. Fortunately, with the development of artificial intelligence technology and the improvement of computer power, deep learning technology is widely applied to the medical field, and provides a new idea for lung cancer diagnosis and subtype classification. The convolutional neural network can intelligently learn and analyze the pathological images to provide objective and scientific diagnosis suggestions for pathologists, and greatly improve the diagnosis efficiency and accuracy of the pathologists.
Currently, the mainstream full-field digital pathological image classification method based on deep learning is divided into a pathological image classification method based on full-supervised learning and a pathological image classification method based on weak-supervised learning. The classification method based on full-supervised learning has higher classification precision on cancer areas, but has poorer identification capability on normal tissue samples, and a large number of pathological images carrying accurate focus marks are required to train a network, so that data is extremely difficult to obtain and the collection cost is high. The classification method based on weak supervision learning only needs an image-level label, can be easily obtained from an open source database TCGA, and a large sample enables the weak supervision network to have stronger recognition capability on normal tissue samples, but the accuracy of the model on classifying the cancer subtypes is lower due to the fact that the focus area cannot be effectively obtained and the characteristic information related to subtype diagnosis is extracted, so that the auxiliary diagnosis effect cannot be achieved.
Therefore, there is a need for a full-field digital pathology image classification method that has high classification accuracy for both normal tissue samples and different cancer subtypes, and that is simple to collect data sets.
Disclosure of Invention
The invention aims to solve the problems that the existing lung cancer pathological image classification method based on full-supervision learning is poor in normal tissue sample recognition capability and the lung cancer subtype classification method based on weak-supervision learning is low in accuracy, and provides a pathological image classification method based on comparison expression distillation and output distillation, wherein the advantages of a full-supervision network are transferred to a weak-supervision network through comparison expression distillation and output distillation, so that the accurate classification of pathological images is realized.
In order to solve the technical problems, the invention adopts the following technical scheme:
a pathological image classification method based on contrast representation distillation and output distillation specifically comprises the following steps:
step S1: the method for constructing the data set of the full-supervision teacher classification network specifically comprises the following steps:
step S1.1: collecting full-field digital pathology images of normal tissue and different lung cancer subtypes;
step S1.2: performing color standardization on the collected full-field digital pathology image to eliminate color difference of the pathology image;
step S1.3: labeling focus areas of the lung cancer full-view digital pathological images, and labeling all tissue pathological areas of a normal tissue sample;
step S1.4: generating a mask with the same size and position according to the marking information, and cutting the image into a plurality of small image blocks with the size of n multiplied by n by using a sliding window, wherein n represents the length and the width of the small image blocks;
step S1.5: comparing the small image block obtained by cutting with the generated mask, if the focus area in the image block is more than 50%, storing the image block, otherwise discarding the image block. Marking a small image block cut out of a pathological image carrying focus marks as m t ；
Step S2: the method for constructing the weakly supervised student classification network data set specifically comprises the following steps:
step S2.1: collecting full-field digital pathology images of normal tissue and different lung cancer subtypes;
step S2.2: performing color standardization on the collected full-field digital pathology image to eliminate color difference of the pathology image;
step S2.3: automatically dividing the standardized pathological image, filtering blank background and natural holes, cutting into n×n small image blocks, wherein n represents the length and width of the small image blocks, and marking the small image blocks cut from pathological image without focus mark as m s ；
Step S3: constructing a double distilled pathology image classification network, the double distilled pathology image classification network comprising:
step S3.1: full supervision feature extraction module F T For extracting pathological image m carrying focus labeling t Deep characteristic information h of (2) t And distilled until the pathological image m is marked with the lesion which is not carried s The network is classified for the input weakly supervised students as follows:
h t ＝F T (m t )
step S3.2: full supervision classification module C T The method is used for judging the disease type of the input image block;
step S3.3: weak supervision feature extraction module F S For extracting pathological image blocks m without focus labeling s Deep characteristic information h of (2) s ；
Step S3.4: depth-gated attention module A S For assigning to each image block an attention score representing the degree of importance of the image block to the classification of the network;
step S3.5: auxiliary clustering module C L The feature space is used for restraining the input image blocks, so that the positive features and the negative features in each category are linearly separable;
step S3.6: weak supervision classification module C S The method comprises the steps of making a final category prediction for a pathological image;
step S4: pathological image m carrying focus mark t As input, back propagation using a random gradient descent algorithm, targeting minimization of cross entropy loss functions to train and optimize parameters of a fully supervised teacher classification network, including a fully supervised feature extraction module F T And a full supervision classification module C T The trained fully supervised teacher classification network is used as an auxiliary reference to optimize the training of the weakly supervised student classification network;
step S5: marking pathological image block m with non-carried focus s For input, training and optimizing parameters of a weakly supervised student classification network, and specifically comprises the following steps:
step S5.1: randomly initializing depth-gated attention module A S Auxiliary clustering module C L And weak supervision classification module C S Parameters of (2);
step S5.2: pathological image block m without focus mark s Sending the feature to a weak supervision feature extraction module F S Obtaining deep features h of each image block s The formula is as follows:
h s ＝F S (m s )
step S5.3: combined weak supervision characteristic extraction module F S Output characteristics h s And full supervision feature extraction module F T Output characteristics h t The calculated contrast represents the distillation loss L F ；
Step S5.4: deep layer feature h generated in step S5.2 s Into depth-gated attention module A S In (2), a concentration score a representing the importance of each image block to network classification is obtained k,n The formula is as follows:
wherein n represents the number of categories of the classification task; n is the total number of image blocks cut out of one pathological image; p (P) a,n Is a full connection layer belonging to the nth class; g a 、H a 、J a A linear layer sharing weights for all categories; tan h and sigm represent tan h and sigmoid activation functions, respectively;
step S5.5: normalizing the attention score generated in the step S5.4, and combining the normalized attention score of each image block with the full-supervision classification module C T Output classification result, calculate output distillation loss L O ；
Step S5.6: deep layer feature h generated in step S5.2 s Sending into an auxiliary clustering module C L In the method, attention scores output by a depth gating attention module are used as pseudo-label supervision clusters, and auxiliary cluster loss L is calculated C ；
Step S5.7: taking the attention score as a corresponding weight, calculating the weighted sum of all feature vectors, and aggregating to generate the feature h of the pathological image level W The formula is as follows:
step S5.8: feature h of pathological image level W Sending the classified data to a weak supervision classification module C S Obtaining category prediction of the full-view digital pathological image, and calculating the category loss L of pathological image level features by combining with a real label W ；
Step S5.9: calculate the total loss function L total The total loss function includes the classification loss L of the pathological image level features W Auxiliary clustering loss L C Loss of output distillation L O Comparison shows distillation loss L F The specific formula is as follows:
L total ＝λ W L W +λ C L C +λ O L O +λ F L F
wherein each λ is the weight of the corresponding loss function;
step S5.10: to minimize the total loss function L total Training a weak supervision student classification network for a target, stopping training when the loss of continuous k rounds of iteration is not reduced any more, and determining an optimal model by using Monte Carlo ten-fold cross validation to obtain a trained weak supervision student classification network;
step S6: and (3) extracting the normalized attention score generated in the step S5.7, generating small image blocks mapped by different RGB colors, covering the small image blocks on the original pathological image with set transparency, and obtaining a focus detection heat map after blurring and smoothing operation.
Further, the double-distillation pathological image classification network comprises a full supervision feature extraction module F T Full supervision classification module C T Weak supervision feature extraction module F S Depth-gated attention module A S Auxiliary clustering module C L Weak supervision classification module C S 。
Further, the weak supervision feature extraction module F S For a pre-trained network, as a preference, resNet-50, resNet-18, resNet-101, resNet-152 pre-trained in the ImageNet natural image dataset may be employed;
further, not all image blocks need to be sent to the auxiliary clustering module C L Only m image blocks with the highest attention score and the lowest attention score in each category are sent to an auxiliary clustering module, and 8 is taken as a preferable m;
further, the output distillation loss L O And the comparison shows the distillation loss L F Not all of the time is involved in the training,preferably, both types of distillation losses are added only after the 40 th iteration;
further, the auxiliary clustering loss L C Using a smooth support vector machine penalty as the preference; classification loss L of the pathological image-level features W Using cross entropy loss as a preference; said output distillation loss L O And the comparison shows the distillation loss L F Using KL divergence or cross entropy loss as the preference;
further, the final full-field digital pathological image classification network is composed of a weak supervision feature extraction module F S Depth-gated attention module A S Weak supervision classification module C S Constructing;
further, the full-field digital pathology image is a medical full-field digital pathology image.
Compared with the prior art, the invention has the beneficial effects that:
(1) The invention designs a double-distillation pathological image classification network aiming at the problems that the existing classification method based on full-supervised learning has poor recognition capability on normal tissue samples, data integration is expensive and difficult to collect, and the classification method based on weak-supervised learning has low accuracy in classifying lung cancer subtypes and cannot achieve the effect of auxiliary diagnosis. The full-supervision classification network is used as a teacher model by combining a contrast representation learning method, and the weak supervision model is extracted into more obvious and more distinguishable deep features by carrying out contrast representation distillation on the full-supervision model. In addition, the output result of the full supervision network is used as a soft label to guide the optimization direction of the attention module in the weak supervision model so as to improve the detection precision of the model on the lesion area. The model realizes accurate prediction of normal tissue samples and lung cancer subtypes by outputting distillation and comparing the distillation.
(2) According to the invention, an auxiliary clustering module is introduced in the training process, so that the positive characteristic and the negative characteristic of a model for predicting a certain cancer subtype can be linearly separated as much as possible, the distinguishing degree of the characteristics is improved, and the accurate prediction of the lung cancer subtype is realized.
(3) The depth gating attention module designed by the invention firstly uses the tanh activation function to introduce preliminary positive and negative gradients, then combines the gating thought to introduce the sigmoid activation function in the form of tensor product as weighting, integrates different characteristics of the two activation functions, ensures that feature vectors containing key information have higher attention scores, and further strengthens the attention scores of the feature vectors to the model. In addition, the attention module converts the focus detection problem into a classification problem, generates a focus detection heat map with high resolution, and maintains the texture structure of the underlying cell tissue while revealing the canceration probability.
Drawings
FIG. 1 is a flow chart of a training of a weakly supervised student network based on a comparative representation distillation and output distillation pathology image classification method of the present invention;
FIG. 2 is a diagram showing a network structure of a pathological image classification method based on comparative representation distillation and output distillation according to the present invention;
FIG. 3 is a flow chart of the full field digital pathology image classification of the present invention.
Detailed Description
For a clearer understanding of the objects, aspects and advantages of the present application, reference is made to the following detailed description and examples, taken in conjunction with the accompanying drawings. It is obvious that the examples described are only illustrative of the invention and are not limiting to the embodiments of the invention, and that other variants or modifications can be made on the basis of the examples, which need not be and cannot be exhaustive of all the embodiments. All other embodiments, which come within the spirit and principles of the invention and are desired to be protected by the following claims, are to be construed as equivalents and alternatives falling within the spirit and scope of the invention.
The invention provides a pathological image resolution-reducing rapid diagnosis method based on contrast expression learning and condition countermeasure, which is further described in detail by taking full-field digital pathological image classification for lung cancer diagnosis (lung adenocarcinoma, lung squamous carcinoma and normal tissue sample) as an example and combining a specific implementation method.
The embodiment of the invention comprises the following steps:
training phase:
step S1: the method for constructing the data set of the full-supervision teacher classification network specifically comprises the following steps:
step S1.1: collecting 50 cases of pathological images of normal tissues, lung adenocarcinoma and lung squamous carcinoma from an open source database TCGA and TCIA respectively;
step S1.2: color standardization is carried out on the collected 150 full-view digital pathology images, and color differences of the pathology images are eliminated;
step S1.3: labeling focus areas of 100 lung cancer full-view digital pathological images, and labeling all tissue pathological areas of 50 normal tissue samples;
step S1.4: generating a mask with the same size and position according to the marking information, and cutting the image into a plurality of small image blocks with the size of 256 multiplied by 256 by using a sliding window;
step S1.5: comparing the small image block obtained by cutting with the generated mask, if the focus area in the image block is more than 50%, storing the image block, otherwise discarding the image block. Marking a small image block cut out of a pathological image carrying focus marks as m t ；
Step S2: the method for constructing the weakly supervised student classification network data set specifically comprises the following steps:
step S2.1: additionally collecting 1000 cases of normal tissue, lung adenocarcinoma and lung squamous carcinoma pathological images from the open source databases TCGA and TCIA, wherein all the images do not contain any pixel level, block level and ROI level labels;
step S2.2: color standardization is carried out on 3000 collected full-view digital pathology images, and color differences of the pathology images are eliminated;
step S2.3: automatically segmenting the standardized pathological image, filtering blank background and natural holes, cutting into a plurality of small image blocks with the size of 256 multiplied by 256, and marking the small image blocks cut out by the pathological image without focus marks as m s ；
Step S3: constructing a double distilled pathology image classification network, the double distilled pathology image classification network comprising:
step S3.1: full supervision feature extraction module F T Illustratively, resNet-18 is used for extracting pathological images m carrying lesion marking t Deep characteristic information h of (2) t And distilled until the pathological image m is marked with the lesion which is not carried s The network is classified for the input weakly supervised students as follows:
h t ＝F T (m t )
step S3.2: full supervision classification module C T Illustratively, the full-connection layer and the Softmax layer are used for judging the disease type of the input image block;
step S3.3: weak supervision feature extraction module F S Illustratively, resNet-50 pre-trained on ImageNet is used for extracting pathological image blocks m carrying no lesion marking s Deep characteristic information h of (2) s ；
Step S3.4: depth-gated attention module A S For assigning to each image block an attention score representing the degree of importance of the image block to the classification of the network;
step S3.5: auxiliary clustering module C L The feature space is used for restraining the input image blocks, so that the positive features and the negative features in each category are linearly separable;
step S3.6: weak supervision classification module C S Illustratively, it is composed of a fully connected layer and a Softmax layer for making a final class prediction for the pathology image;
step S4: pathological image m carrying focus mark t As input, back propagation using a random gradient descent algorithm, targeting minimization of cross entropy loss functions to train and optimize parameters of a fully supervised teacher classification network, including a fully supervised feature extraction module F T And a full supervision classification module C T In this embodiment, adam is used as an optimizer, the learning rate is set to 0.0001, the maximum iteration number is 200, and the trained fully-supervised teacher classification network is used as an auxiliary reference to optimize training of the weakly-supervised student classification networkTraining;
step S5: marking pathological image block m with non-carried focus s For input, training and optimizing parameters of a weakly supervised student classification network, see fig. 1, specifically comprises the following steps:
step S5.1: randomly initializing depth-gated attention module A S Auxiliary clustering module C L And weak supervision classification module C S Parameters of (2);
step S5.2: pathological image block m without focus mark s Sending the feature to a weak supervision feature extraction module F S Obtaining deep features h of each image block s The formula is as follows:
h s ＝F S (m s )
step S5.3: combined weak supervision characteristic extraction module F S Output characteristics h s And full supervision feature extraction module F T Output characteristics h t The calculated contrast represents the distillation loss L F Illustratively, the comparison represents that distillation loss uses cross entropy loss;
step S5.4: deep layer feature h generated in step S5.2 s Into depth-gated attention module A S In (2), a concentration score a representing the importance of each image block to network classification is obtained k,n The formula is as follows:
wherein n represents the number of categories of the classification task; n is the total number of image blocks cut out of one pathological image; p (P) a,n Is a full connection layer belonging to the nth class; g a 、H a 、J a A linear layer sharing weights for all categories; tanh and sigm represent tanh and sigmoid activation functions, respectively.
Step S5.5: normalizing the attention score generated in the step S5.4, and combining the normalized attention score of each image block with the full-supervision classification module C T Output classification result, calculate output distillation loss L O Illustratively, the loss of distillation is outputUsing cross entropy loss;
step S5.6: deep layer feature h generated in step S5.2 s Sending into an auxiliary clustering module C L In the method, attention scores output by a depth gating attention module are used as pseudo-label supervision clusters, and auxiliary cluster loss L is calculated C Illustratively, the auxiliary cluster penalty uses a smooth vector machine penalty;
step S5.7: taking the attention score as a corresponding weight, calculating the weighted sum of all feature vectors, and aggregating to generate the feature h of the pathological image level W The formula is as follows:
step S5.8: feature h of pathological image level W Sending the classified data to a weak supervision classification module C S Obtaining category prediction of the full-view digital pathological image, and calculating the category loss L of pathological image level features by combining with a real label W Illustratively, the classification penalty of pathological image-level features uses cross entropy penalty;
step S5.9: calculate the total loss function L total The total loss function includes the classification loss L of the pathological image level features W Auxiliary clustering loss L C Loss of output distillation L O Comparison shows distillation loss L F The specific formula is as follows:
L total ＝λ W L W +λ C L C +λ O L O +λ F L F
wherein each lambda is the weight of the corresponding loss function, and lambda is set W ＝0.7，λ C ＝0.1，λ O ＝0.1，λ F ＝0.1。
Step S5.10: to minimize the total loss function L total Training a weak supervision student classification network for targets, using Adam as an optimizer, setting a learning rate to 0.0001, stopping training when the loss of continuous 30 rounds of iteration is not reduced any more, and determining an optimal model by using Monte Carlo ten-fold cross-validation to obtain trainingGood weakly supervised student classification network, see fig. 2;
step S6: and (3) extracting the normalized attention score generated in the step S5.7, generating small image blocks mapped by different RGB colors, covering the small image blocks on the original pathological image with the transparency of 0.4, and obtaining a focus detection heat map after blurring and smoothing operation.
The application phase, as shown in fig. 3:
step S1: acquiring a full-field digital pathological image of the lung of a patient;
step S2: preprocessing the full-view digital pathological image of the patient by using a method for preprocessing each full-view digital pathological image during training to obtain a plurality of small image blocks;
step S4: and sending all the small image blocks into a trained weak supervision student classification network to obtain a prediction result of the full-field digital pathological image.
Step S5: a lesion detection heat map is generated using the attention scores in the depth-gated attention module.
Claims (8)
1. The pathological image classification method based on contrast representation distillation and output distillation is characterized by comprising the following steps of:
step S1: constructing a full-supervision teacher classification network data set, and marking a small image block cut out from a pathological image carrying focus marks as m t ；
Step S2: constructing a weak supervision student classification network data set, and marking a small image block cut out from a pathological image without focus marks as m s ；
Step S3: constructing a double distilled pathology image classification network, the double distilled pathology image classification network comprising:
step S3.1: full supervision feature extraction module F T For extracting pathological image m carrying focus labeling t Deep characteristic information h of (2) t And distilled until the pathological image m is marked with the lesion which is not carried s The network is classified for the input weakly supervised students as follows:
h t ＝F T (m t )
step S3.2: full supervision classification module C T The method is used for judging the disease type of the input image block;
step S3.3: weak supervision feature extraction module F S For extracting pathological image blocks m without focus labeling s Deep characteristic information h of (2) s ；
Step S3.4: depth-gated attention module A S For marking pathological image blocks m for lesions not carried s Assigning an attention score representing the degree of importance to the classification of the network;
step S3.5: auxiliary clustering module C L For constraining pathological image blocks m without lesion marking s To make the positive and negative features in each category linearly separable;
step S3.6: weak supervision classification module C S For marking pathological image blocks m for lesions which are not carried s Making a final category prediction;
step S4: pathological image m carrying focus mark t As input, back propagation using a random gradient descent algorithm, targeting minimization of cross entropy loss functions to train and optimize parameters of a fully supervised teacher classification network, including a fully supervised feature extraction module F T And a full supervision classification module C T The trained fully supervised teacher classification network is used as an auxiliary reference to optimize the training of the weakly supervised student classification network;
step S5: marking pathological image block m with non-carried focus s For input, training and optimizing parameters of a weakly supervised student classification network, and specifically comprises the following steps:
step S5.1: randomly initializing depth-gated attention module A S Auxiliary clustering module C L And weak supervision classification module C S Parameters of (2);
step S5.2: pathological image block m without focus mark s Sending the feature to a weak supervision feature extraction module F S Obtaining deep features h of each image block s The formula is as follows:
h s ＝F S (m s )
step S5.3: combined weak supervision characteristic extraction module F S Output characteristics h s And full supervision feature extraction module F T Output characteristics h t The calculated contrast represents the distillation loss L F ；
Step S5.4: deep layer feature h generated in step S5.2 s Into depth-gated attention module A S In (2), a concentration score a representing the importance of each image block to network classification is obtained k,n The formula is as follows:
wherein n represents the number of categories of the classification task; n is the total number of image blocks cut out of one pathological image; p (P) a,n Is a full connection layer belonging to the nth class; g a 、H a 、J a A linear layer sharing weights for all categories; tan h and sigm represent tan h and sigmoid activation functions, respectively;
step S5.5: normalizing the attention score generated in the step S5.4, and combining the normalized attention score of each image block with the full-supervision classification module C T Output classification result, calculate output distillation loss L O ；
Step S5.6: deep layer feature h generated in step S5.2 s Sending into an auxiliary clustering module C L In the method, attention scores output by a depth gating attention module are used as pseudo-label supervision clusters, and auxiliary cluster loss L is calculated C ；
Step S5.7: taking the attention score as a corresponding weight, calculating the weighted sum of all feature vectors, and aggregating to generate the feature h of the pathological image level W The formula is as follows:
step S5.8: feature h of pathological image level W Sending the classified data to a weak supervision classification module C S Obtaining category prediction of the full-view digital pathological image, and calculating the category loss L of pathological image level features by combining with a real label W ；
Step S5.9: calculate the total loss function L total The total loss function includes the classification loss L of the pathological image level features W Auxiliary clustering loss L C Loss of output distillation L O Comparison shows distillation loss L F The specific formula is as follows:
L total ＝λ W L W +λ C L C +λ O L O +λ F L F
wherein each λ is the weight of the corresponding loss function;
step S5.10: to minimize the total loss function L total Training a weak supervision student classification network for a target, stopping training when the loss of continuous k rounds of iteration is not reduced any more, and determining an optimal model by using Monte Carlo ten-fold cross validation to obtain a trained weak supervision student classification network;
step S6: and (3) extracting the normalized attention score generated in the step S5.7, generating small image blocks mapped by different RGB colors, covering the small image blocks on the original pathological image with set transparency, and obtaining a focus detection heat map after blurring and smoothing operation.
2. A method of classifying pathological images based on contrast-indicative and output distillation according to claim 1, wherein: the method comprises the steps of constructing a full-supervision teacher classification network data set, and marking a small image block cut out from a pathological image carrying focus marks as m t The method comprises the steps of carrying out a first treatment on the surface of the The method specifically comprises the following steps:
step S1.1: collecting full-field digital pathology images of normal tissue and different lung cancer subtypes;
step S1.2: performing color standardization on the collected full-field digital pathology image to eliminate color difference of the pathology image;
step S1.3: labeling focus areas of the lung cancer full-view digital pathological images, and labeling all tissue pathological areas of a normal tissue sample;
step S1.4: generating a mask with the same size and position according to the marking information, and cutting the image into a plurality of small image blocks with the size of n multiplied by n by using a sliding window, wherein n represents the length and the width of the small image blocks;
step S1.5: comparing the small image block obtained by cutting with the generated mask, if the focus area in the image block is more than 50%, storing the image block, otherwise discarding; marking a small image block cut out of a pathological image carrying focus marks as m t 。
3. A method of classifying pathological images based on contrast-indicative and output distillation according to claim 1, wherein: the construction of the weak supervision student classification network data set marks a small image block cut out from a pathological image without focus marks as m s The method comprises the steps of carrying out a first treatment on the surface of the The data set construction method specifically comprises the following steps:
step S2.1: collecting full-field digital pathology images of normal tissue and different lung cancer subtypes;
step S2.2: performing color standardization on the collected full-field digital pathology image to eliminate color difference of the pathology image;
step S2.3: automatically dividing the standardized pathological image, filtering blank background and natural holes, cutting into n×n small image blocks, wherein n represents the length and width of the small image blocks, and marking the small image blocks cut from pathological image without focus mark as m s 。
4. A pathological image classification method based on contrast-based representation distillation and output distillation according to claim 1, wherein said weakly supervised feature extraction module F S For the pre-trained network, resNet-50, resNet-18, resNet-101, resNet-152 pre-trained in the ImageNet natural image dataset are employed.
5. A pair-based according to claim 1The pathological image classification method of the ratio representing distillation and output distillation is characterized by comprising the following steps of: not all image blocks need to be sent to the auxiliary clustering module C L And sending m image blocks with the highest attention score and the lowest attention score in each category into an auxiliary clustering module, wherein m is 8.
6. A pathological image classifying method based on contrast-based representation distillation and output distillation according to claim 1, wherein said output distillation loss L O And the comparison shows the distillation loss L F Rather than taking part in training entirely, the two types of distillation losses described above were added only after the 40 th iteration.
7. A pathological image classification method based on contrast-representation distillation and output distillation according to claim 1, wherein said auxiliary clustering loss L C Loss using a smooth support vector machine; classification loss L of the pathological image-level features W Using cross entropy loss; said output distillation loss L O And the comparison shows the distillation loss L F KL divergence or cross entropy loss is used.
8. The pathological image classification method based on contrast representation distillation and output distillation according to claim 1, wherein the trained full-field digital pathological image classification network is composed of a weak supervision feature extraction module F S Depth-gated attention module A S Weak supervision classification module C S The composition is formed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310194883.3A CN116310525A (en) | 2023-02-28 | 2023-02-28 | Pathological image classification method based on contrast representation distillation and output distillation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310194883.3A CN116310525A (en) | 2023-02-28 | 2023-02-28 | Pathological image classification method based on contrast representation distillation and output distillation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116310525A true CN116310525A (en) | 2023-06-23 |
Family
ID=86812524
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310194883.3A Pending CN116310525A (en) | 2023-02-28 | 2023-02-28 | Pathological image classification method based on contrast representation distillation and output distillation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116310525A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116994032A (en) * | 2023-06-28 | 2023-11-03 | 河北大学 | Rectal polyp multi-classification method based on deep learning |
-
2023
- 2023-02-28 CN CN202310194883.3A patent/CN116310525A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116994032A (en) * | 2023-06-28 | 2023-11-03 | 河北大学 | Rectal polyp multi-classification method based on deep learning |
| CN116994032B (en) * | 2023-06-28 | 2024-02-27 | 河北大学 | Rectal polyp multi-classification method based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Sharma et al. | Dermatologist-level classification of skin cancer using cascaded ensembling of convolutional neural network and handcrafted features based deep neural network | |
| Shah et al. | A robust approach for brain tumor detection in magnetic resonance images using finetuned efficientnet | |
| CN109493308B (en) | Medical image synthesis and classification method for generating confrontation network based on condition multi-discrimination | |
| CN108364006B (en) | Medical image classification device based on multi-mode deep learning and construction method thereof | |
| US20170249739A1 (en) | Computer analysis of mammograms | |
| CN108257135A (en) | The assistant diagnosis system of medical image features is understood based on deep learning method | |
| CN111767952B (en) | Interpretable lung nodule benign and malignant classification method | |
| Chen et al. | Pulmonary nodule detection on chest radiographs using balanced convolutional neural network and classic candidate detection | |
| Mahapatra et al. | A novel framework for retinal vessel segmentation using optimal improved frangi filter and adaptive weighted spatial FCM | |
| CN111430025B (en) | Disease diagnosis model training method based on medical image data augmentation | |
| CN107516312A (en) | A kind of Chinese medicine complexion automatic classification method using shallow-layer neutral net | |
| CN112785603B (en) | Brain tissue segmentation method based on Unet and superpixel | |
| JP7312510B1 (en) | Whole-slide pathological image classification system and construction method considering tumor microenvironment | |
| Ma et al. | A novel two-stage deep method for mitosis detection in breast cancer histology images | |
| Wahid et al. | Lung and colon cancer detection with convolutional neural networks on histopathological images | |
| CN116310525A (en) | Pathological image classification method based on contrast representation distillation and output distillation | |
| CN115471701A (en) | Lung adenocarcinoma histology subtype classification method based on deep learning and transfer learning | |
| CN114549452A (en) | New coronary pneumonia CT image analysis method based on semi-supervised deep learning | |
| Hu et al. | A Hybrid Convolutional Neural Network Model Based on Different Evolution for Medical Image Classification. | |
| AU2016201298A1 (en) | Computer analysis of mammograms | |
| CN117036288A (en) | Tumor subtype diagnosis method for full-slice pathological image | |
| Krishna et al. | Optimization empowered hierarchical residual VGGNet19 network for multi-class brain tumour classification | |
| Cao et al. | 3D convolutional neural networks fusion model for lung nodule detection onclinical CT scans | |
| CN116228759A (en) | Computer-aided diagnosis system and apparatus for renal cell carcinoma type | |
| Masood | Developing improved algorithms for detection and analysis of skin cancer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |