CN114972341A - WSI image classification method, system and medium based on Bayesian assisted learning - Google Patents

WSI image classification method, system and medium based on Bayesian assisted learning Download PDF

Info

Publication number
CN114972341A
CN114972341A CN202210895166.9A CN202210895166A CN114972341A CN 114972341 A CN114972341 A CN 114972341A CN 202210895166 A CN202210895166 A CN 202210895166A CN 114972341 A CN114972341 A CN 114972341A
Authority
CN
China
Prior art keywords
wsi
image
image block
classification
classification model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210895166.9A
Other languages
Chinese (zh)
Other versions
CN114972341B (en
Inventor
余晋刚
吴梓浩
吴锦全
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Fangxin Medical Technology Co ltd
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to CN202210895166.9A priority Critical patent/CN114972341B/en
Publication of CN114972341A publication Critical patent/CN114972341A/en
Application granted granted Critical
Publication of CN114972341B publication Critical patent/CN114972341B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/136—Segmentation; Edge detection involving thresholding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30024—Cell structures in vitro; Tissue sections in vitro

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Pure & Applied Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Computational Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Mathematical Analysis (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Algebra (AREA)
  • Probability & Statistics with Applications (AREA)
  • Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a WSI image classification method, a system and a medium based on Bayesian assisted learning, wherein the method comprises the following steps: acquiring a WSI image and performing threshold segmentation to obtain a pathological image block set; constructing a WSI classification model based on a Bayesian decision theory, wherein the WSI classification model comprises an image block feature extractor, an image block classifier and a feature fusion and classification module; initializing a WSI classification model, and iteratively training the WSI classification model on the pathological image block set by using an EM (effective ray spectroscopy) method to obtain a trained WSI classification model; and inputting the WSI image to be diagnosed into the trained WSI classification model, and outputting the class label of the WSI image to be diagnosed. The WSI classification model is constructed based on the Bayesian decision theory, the image block classifier is introduced to perform an auxiliary learning task, information in the WSI image is fully extracted, and meanwhile, the EM method is used for performing dynamic learning in the iterative training process, so that the WSI classification performance and the model robustness are improved.

Description

WSI image classification method, system and medium based on Bayesian assisted learning
Technical Field
The invention belongs to the technical field of digital pathological full-section image classification, and particularly relates to a WSI image classification method, a system and a medium based on Bayesian assisted learning.
Background
Pathological diagnosis is recognized as a gold standard for clinical cancer diagnosis, and has important significance for the establishment of a treatment scheme and prognosis of cancer patients. In the traditional method, a pathologist carries out pathological diagnosis by dragging and magnifying different positions of a pathological slide by means of a microscope; the diagnosis process is complicated and has strong subjectivity, and the consistency of the diagnosis results given by different doctors to difficult cases is poor. With the progress of the slide digital scanning technology, digital pathology full-section images (WSI) are easier to obtain, and the design of automated and intelligent computer-aided diagnosis methods for digital pathology images is also receiving wide attention. However, the digital pathological full-section image classification diagnosis still faces some technical difficulties, which restrict the application of the auxiliary diagnosis method: 1) the WSI has ultrahigh resolution, such as 100,000x100,000 pixels, and is difficult to directly process by adopting a common deep learning method, so the WSI is generally divided into a plurality of small image blocks (patch), and then classification diagnosis is carried out on the WSI based on the patch, but the WSI has ultrahigh resolution, so the WSI is divided into a great number of patches, and on average, each WSI has 8000 to 15000 patches; when network training is carried out, a large number of intermediate feature graphs need to be stored, so that memory overflow is easily caused, and therefore, under the condition that only WSI integral class labels are provided, all patches cannot be used for carrying out end-to-end training on the feature extractor and the fusion model; 2) the fine pixel level label of the WSI is difficult to obtain, and usually only an integral class label can be obtained, and the label information corresponding to each patch cannot be known; 3) the WSI image has abundant changes of cell morphology, structure and texture, has high heterogeneity, and is less in data volume compared with other medical image data, so that the phenomenon of overfitting of a deep learning model is easily caused.
Aiming at the problems, the CLAM method proposed by Lu.M.Y and the like extracts patch features by utilizing a ResNet-50 network pre-trained by an ImageNet data set, and then learns a fusion module based on an attention mechanism through the patch features to finally realize the classification of WSI; however, the process of training the feature extractor and the fusion model by the method is completely split, so that the classifier is not favorable for learning feature information which is beneficial to distinguishing different types of WSI, and the feature information cannot be sufficiently coupled to improve the classification performance of the WSI, so that the classification performance of the WSI is limited by the quality of the patch features finally. Selecting one with the highest prediction score from all the patches to represent the WSI image based on the maximum pooling multi-instance learning method, and training the whole network by using the patches, wherein the method can simultaneously train a feature extractor and a fusion model; however, training can be performed only through a small number of patches, information in a large number of WSI images is lost, so that the stability of the model is poor, noise is high, the overall performance depends on selection strategies such as clustering, and the learning of a high-quality feature extractor is not facilitated. Hou et al put forward a method for updating Patch classifier with EM algorithm in the document "Patch-based constrained neural network for whole slice image classification", supposing that all Patch class labels in WSI are consistent with WSI whole label, then training a Patch-level classifier, i.e. Patch feature extractor, generating probability graph of WSI through the classifier, and performing Gaussian smoothing processing on the probability graph, selecting higher confidence Patch from the probability graph according to threshold value, using the higher confidence Patch to retrain the classifier, iterating until the selected Patch is consistent, then using the trained classifier to retrain prediction for all patches in WSI, counting prediction class distribution as WSI whole feature, and using the whole feature training logic classifier to perform classification regression on WSI; however, the method assumes that it is unreasonable that all patch category labels in the WSI are consistent with the WSI overall label, and a background area is also included in the label, so that irrelevant noise is introduced, and the performance of the classifier is reduced; meanwhile, the process of training the feature extractor and the fusion model is completely split, which is not beneficial to learning feature information which is beneficial to distinguishing different types of WSI by the classifier, and enables the two structures to be mutually independent and not to be fully coupled so as to improve the classification performance of the WSI; finally, the adoption of the class distribution histogram of patch as the WSI feature is not reasonable, and is not enough to represent rich information in the WSI image.
Disclosure of Invention
The invention mainly aims to overcome the defects of the prior art and provide a WSI image classification method, a system and a medium based on Bayesian assisted learning, which solve the technical problem of poor classification performance caused by the fact that all patches can not be used for updating a feature extractor and a fusion model simultaneously, construct a WSI classification model based on Bayesian theorem, introduce image block classification as an assisted learning task training feature extractor, and fully utilize information in a WSI image to learn a high-quality feature extractor; and meanwhile, the EM method is used for dynamic learning in the iterative training process so as to improve the classification performance of the WSI and the robustness of the model.
In order to achieve the purpose, the invention adopts the following technical scheme:
the first purpose is to provide a WSI image classification method based on Bayesian assisted learning, which comprises the following steps:
acquiring a WSI image and a corresponding class label, converting the WSI image into an HSV space, extracting a foreground area according to saturation, and dividing the foreground area into a plurality of image blocks to obtain a pathological image block set;
constructing a WSI classification model based on a Bayesian decision theory, wherein the WSI classification model comprises an image block feature extractor, an image block classifier and a feature fusion and classification module; the feature fusion and classification module comprises an attention fusion module and a WSI classifier;
the image block feature extractor is used for extracting features of image blocks in the pathological image block set; the image block classifier is used for giving a pseudo label to the image block; the attention fusion module is used for acquiring attention weight corresponding to the image block characteristics; the WSI classifier is used for classifying the WSI images;
initializing a WSI classification model, iteratively training the WSI classification model on the pathological image block set by using an EM (effective ray spectroscopy) method, and optimizing model parameters until the model converges to obtain a trained WSI classification model;
and inputting the WSI image to be diagnosed of the cancer patient into the trained WSI classification model, outputting the classification result of the WSI image to be diagnosed and drawing a probability thermodynamic diagram.
As an optimal technical scheme, the WSI classification model is built based on a Bayesian decision theory and is built in a pathological image block set
Figure 89383DEST_PATH_IMAGE001
The training learning parameters areθ=(θ 1 ,θ 2 ,θ 3 ) WSI classification modelF θ Wherein
Figure 100002_DEST_PATH_IMAGE002
Is shown asiA sheet of the WSI image is printed,
Figure 100002_DEST_PATH_IMAGE004
is shown asiIn a WSI imagenThe number of the image blocks is one,N i is shown asiThe number of image blocks of a WSI image block,y i is shown asiA category label corresponding to the WSI image,θ 1 are parameters of the image block feature extractor,θ 2 for the parameters of the feature fusion and classification module,θ 3 for the parameters of the image block classifier, based on the input WSI imageXPredicting category labelsy= F θ (X) The method specifically comprises the following steps:
will train the learning parameters asθ=(θ 1 ,θ 2 ,θ 3 ) WSI classification modelF θ Defined as the maximum likelihood estimation problem, expressed as:
Figure 579139DEST_PATH_IMAGE005
wherein the content of the first and second substances,θ * representing theoretical parameters of the WSI classification modelDI represents the number of WSI images in the pathological image block set;
introducing hidden variables
Figure 100002_DEST_PATH_IMAGE006
A class label set for representing the image block is obtainedObjective function of WSI classification model:
Figure 60936DEST_PATH_IMAGE007
wherein the content of the first and second substances,
Figure 100002_DEST_PATH_IMAGE008
representing each image block
Figure 599233DEST_PATH_IMAGE004
The implicit true category label is that the user is,Crepresenting a set of category labels contained in the WSI image.
As a preferred technical scheme, the image block feature extractor adopts a ResNet-50 network as a backbone network;
the initialization WSI classification model specifically comprises the following steps:
pre-training an image block feature extractor on an ImageNet data set, and initializing parameters of the image block feature extractor;
the initialized image block feature extractor encodes an input image block into a 1024-dimensional feature vector;
and initializing the parameters of the image block classifier, the attention fusion module and the WSI classifier by adopting random initialization.
As a preferred technical scheme, before iteratively training a WSI classification model on a pathological image block set by using an EM method, an initialized image block feature extractor is used to extract initialized image block features of the pathological image block set, and then an attention fusion module and a WSI classifier are trained by using the initialized image block features and class labels corresponding to WSI images, and parameters of a feature fusion and classification module are updated;
and classifies the image blocks
Figure 100002_DEST_PATH_IMAGE010
Fraction value of corresponding category label of all image blocks belonging to WSI image in medium prediction
Figure 333141DEST_PATH_IMAGE011
The number of the given 1 s is 1,
Figure 478952DEST_PATH_IMAGE011
is shown asiIn a WSI imagenPrediction score values for image blocks.
As a preferred technical scheme, the step of iteratively training the WSI classification model by the EM method comprises E-step and M-step;
the E-step is based on the input WSI imageX i And corresponding category labely i And parameters of the WSI classification modelθ t() Assigning a pseudo label to each image block in the set of pathological images, whereinθ t() Denotes the firsttModel parameters of the round iteration;
defining an objective function based on a WSI classification model
Figure 100002_DEST_PATH_IMAGE012
And according to the equation
Figure 634995DEST_PATH_IMAGE013
And decomposing the objective function of the WSI classification model into:
Figure 100002_DEST_PATH_IMAGE014
Figure 649088DEST_PATH_IMAGE015
wherein the content of the first and second substances,
Figure 183974DEST_PATH_IMAGE017
the set of image block classes representing the prediction by the WSI classification model is equal to the a posteriori probability of the true label underlying the image block,J(θ,θ t() ) Indicating WSI Classification model usagetParameters of a wheel iteration modelθ t() Training to obtain model parameters ofθThe objective function of (1);
and further expanding to obtain an objective function expansion formula of the WSI classification model:
Figure 100002_DEST_PATH_IMAGE018
determining parameters of the M-step for updating the image block feature extractor according to an objective function expansion of a WSI classification modelθ 1 And parameters of image block classifierθ 3 Then according to the parameters of image block feature extractorθ 1 Updating parameters of a feature fusion and classification moduleθ 2 。
As a preferable technical scheme, the E-step is specifically as follows:
inputting the pathological image block data set into a WSI classification model, and using an image block feature extractor
Figure 424987DEST_PATH_IMAGE019
Extracting image block features:
Figure 100002_DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 474852DEST_PATH_IMAGE021
is shown asiIn a WSI imagenPage image block characteristics;
inputting the image data into an attention fusion module omega to obtain attention weight values of all image blocks in the WSI imagea n } i ：
Figure 100002_DEST_PATH_IMAGE022
Wherein the leafa n } i Is shown asiIn a WSI imagenAttention weight of the image blocks;
simultaneous input of image block features into image block classifier
Figure 682848DEST_PATH_IMAGE010
Fraction value of corresponding category label of all image blocks belonging to WSI image in medium prediction
Figure 541083DEST_PATH_IMAGE023
：
Figure 100002_DEST_PATH_IMAGE024
The attention weighted value of the same image block is readya n } i And score value
Figure 280893DEST_PATH_IMAGE023
Performing dot product to obtain integral fraction of pathological image block sets n } i ：
Figure 490157DEST_PATH_IMAGE025
Making an integral score of a pathological image block sets n } i Sorting in descending order, and assigning pseudo label to image block according to threshold lambdaz n } i ；
The M-step comprises M1-step and M2-step, and specifically comprises the following steps:
m1-step: pseudo label for using image blockz n } i Training image block feature extractor by full-supervision training method
Figure 846052DEST_PATH_IMAGE019
And an image block classifier
Figure 230897DEST_PATH_IMAGE010
；
M2-step: fixed image block feature extractor
Figure 482887DEST_PATH_IMAGE019
The parameters of the image block are extracted again, and the updated image block features are usedClass label retraining feature fusion and classification module corresponding to WSI image
Figure 100002_DEST_PATH_IMAGE026
The parameter (c) of (c).
As a preferred technical scheme, the loss function of the WSI classification model is constructed by adopting cross entropy, and comprises an L1 loss function and an L2 loss function;
the L1 loss function is used to train the updated WSI classifier, and is expressed as:
Figure 507344DEST_PATH_IMAGE027
wherein N represents the total number of classes predicted by the WSI classification model,
Figure 100002_DEST_PATH_IMAGE028
representing a WSI classification model predicting that a WSI image belongs to a categoryiThe score of (a) is calculated,y i e {0,1} represents a category label corresponding to the WSI image subjected to 0-1 encoding;
the L2 loss function is used to train the update image block classifier, represented as:
Figure 100002_DEST_PATH_IMAGE029
wherein N +1 represents the total number of classes predicted by the image block classifier,
Figure DEST_PATH_IMAGE030
representing image block classifier to predict image block to belong to WSI image real categoryiThe value of the fraction of (c) is,z i e {0,1} represents the image block pseudo label after 0-1 encoding.
As a preferred technical solution, the mapping probabilistic thermodynamic diagram is used for diagnosing cancer patients, and specifically includes:
inputting the WSI image to be diagnosed of the cancer patient into the trained WSI classification model, and outputting a classification result;
drawing the classification resultFractional values of all image blocks in the WSI image to be diagnosed predicted by the image block classifier
Figure 919127DEST_PATH_IMAGE023
Obtaining the probability values of the image blocks belonging to the various categories corresponding to the WSI image to be diagnosed through the normalization of the softmax function;
creating an all-0 matrix, and downsampling the length and the width of the matrix for a WSI image to be diagnosednDoubling;
performing coordinate conversion according to the position of the image block in the WSI image to be diagnosed, and filling the probability value of the image block into the corresponding position of the matrix to obtain a probability matrix;
and constructing a color mapping table corresponding to different probability values, mapping colors to corresponding positions of the WSI image to be diagnosed according to the probability values of the probability matrix, and obtaining a probability thermodynamic diagram.
The second purpose is to provide a WSI image classification system based on Bayesian assisted learning, which is applied to the WSI image classification method based on Bayesian assisted learning and comprises a data acquisition module, a model construction module, a model training module and a diagnosis module;
the data acquisition module is used for acquiring a WSI image and a corresponding class label, converting the WSI image into an HSV space, extracting a foreground area according to saturation and dividing the foreground area into a plurality of image blocks to obtain a pathological image block set;
the model construction module is used for constructing a WSI classification model based on a Bayesian decision theory, and the WSI classification model comprises an image block feature extractor, an image block classifier and a feature fusion and classification module; the feature fusion and classification module comprises an attention fusion module and a WSI classifier;
the image block feature extractor is used for extracting features of image blocks in the pathological image block set; the image block classifier is used for giving a pseudo label to the image block; the attention fusion module is used for acquiring attention weight corresponding to the image block characteristics; the WSI classifier is used for classifying the WSI images;
the model training module is used for initializing the WSI classification model, iteratively training the WSI classification model on the pathological image block set by using an EM (effective electromagnetic radiation) method, and optimizing model parameters until the model converges to obtain the trained WSI classification model;
the diagnosis module is used for inputting the WSI images to be diagnosed of the cancer patients into the trained WSI classification model, outputting the classification results of the WSI images to be diagnosed and drawing a probability thermodynamic diagram.
A third object is to provide a computer-readable storage medium, which stores a program that, when executed by a processor, implements the above-mentioned WSI image classification method based on bayesian-assisted learning.
Compared with the prior art, the invention has the following advantages and beneficial effects:
firstly, the invention trains the feature extractor in the WSI classification model by introducing an image block classification auxiliary task, so that rich feature expression with discrimination capability can be better learned from highly heterogeneous digital pathology full-section image data, and the accuracy of WSI image classification can be substantially improved;
secondly, the problem modeling and formula derivation are carried out by combining the Bayesian decision theory, and the method is different from other WSI classification deep learning methods.
Thirdly, a better image block classifier can be obtained according to the introduced auxiliary learning task, the WSI image can be conveniently scanned, a corresponding probability thermodynamic diagram is obtained, and a better visualization effect is achieved; and the visualization result can help a doctor to locate an interested region in the WSI image, thereby being beneficial to the diagnosis of cancer.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flowchart of a WSI image classification method based on Bayesian-assisted learning according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of iterative training of a WSI classification model by using an EM method according to an embodiment of the present invention;
FIG. 3 is a class activation diagram generated by the image block classifier for different iterations according to an embodiment of the present invention;
FIG. 4 is a probability thermodynamic diagram obtained by predicting all patches in a WSI image by an image block classifier under different iteration times in the embodiment of the present invention;
FIG. 5 is a diagram illustrating the structure of a WSI image classification system based on Bayesian-assisted learning according to an embodiment of the present invention;
fig. 6 is a block diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
Referring to fig. 1, in an embodiment of the present application, a WSI image classification method based on bayesian-assisted learning is provided, including the following steps:
s1, acquiring the WSI image and the corresponding class label, converting the WSI image into an HSV space, extracting a foreground region according to saturation, and dividing the foreground region into a plurality of image blocks to obtain a pathological image block set;
firstly, acquiring a digital pathological full-section WSI imageX i And its corresponding category labely i Then the WSI image is processedX i Converting the space into HSV space, extracting a foreground region according to the saturation, extracting an effective pathological tissue part from the foreground region, and removing most of the remaining white background region, so that a large amount of storage cost can be saved, and meanwhile, the noise of a data set is reduced; then, the tissue region of the WSI image is divided into 256 × 256 sized image blocks (patch) at 20 × magnification to obtain a pathological image block set, so that the subsequent processing can be performed using the deep neural network.
S2, constructing a WSI classification model based on a Bayesian decision theory, wherein the WSI classification model comprises an image block feature extractor, an image block classifier and a feature fusion and classification module; the feature fusion and classification module comprises an attention fusion module and a WSI classifier;
the image block feature extractor is used for extracting features of image blocks in the pathological image block set; the image block classifier is used for giving a false label to the image block; the attention fusion module is used for acquiring attention weight corresponding to the image block characteristics; the WSI classifier is used for classifying the WSI images.
Based on Bayesian decision theory, by using the pathological image block set
Figure 283112DEST_PATH_IMAGE001
The training learning parameters areθ=(θ 1 ,θ 2 ,θ 3 ) WSI classification modelF θ Wherein
Figure 971583DEST_PATH_IMAGE002
Is shown asiA sheet of the WSI image is printed,
Figure 421018DEST_PATH_IMAGE004
is shown asiIn a WSI imagenThe number of image blocks is one,N i is shown asiThe number of image blocks of a WSI image block,θ 1 are parameters of the image block feature extractor,θ 2 for the parameters of the feature fusion and classification module,θ 3 parameters of image block classifier are used to make WSI classification modelF θ Capable of being based on inputted WSI imageXPredicting category labelsy = F θ (X) The method specifically comprises the following steps:
will train the learning parameters asθ=(θ 1 ,θ 2 ,θ 3 ) WSI classification modelF θ Is defined as the Maximum Likelihood Estimation (MLE) problem, expressed as:
Figure 867785DEST_PATH_IMAGE005
wherein the content of the first and second substances,θ * theoretical parameters representing the WSI classification modelDI represents the number of WSI images in the pathological image block set;
the classification process of each WSI image is mutually independent, and then hidden variables are introduced into the maximum likelihood estimation function
Figure 820698DEST_PATH_IMAGE006
And representing a class label set of the image block to obtain an objective function of the WSI classification model:
Figure 821015DEST_PATH_IMAGE007
wherein the content of the first and second substances,
Figure 351222DEST_PATH_IMAGE008
representing each image block
Figure 727977DEST_PATH_IMAGE004
The implicit true category label is that the user is,Crepresenting a set of category labels contained in the WSI image.
Due to hidden variablesZ i Is unknown and therefore cannot directly optimize the objective function to obtain the normType parameters, therefore the EM method is used for iterative optimization until the model stops converging.
S3, initializing the WSI classification model, iteratively training the WSI classification model on the pathological image block set by using an EM (effective ray spectroscopy) method, and optimizing model parameters until the model converges to obtain the trained WSI classification model;
in the embodiment, the image block feature extractor adopts a ResNet-50 network as a backbone network;
in the stage of initializing the WSI classification model, pre-training an image block feature extractor on an ImageNet data set, and initializing parameters of the image block feature extractor; the initialized image block feature extractor can encode the input image block into a 1024-dimensional feature vector;
and initializing parameters of the image block classifier, the attention fusion module and the WSI classifier by adopting random initialization.
Considering that if the E-Step of the first iteration is directly performed by using the parameters obtained by random initialization, the pseudo label of each image block is predicted, the quality of the pseudo label is poor, and the performance of the next iteration process is possibly further influenced; and feature extractor
Figure 66554DEST_PATH_IMAGE019
And feature fusion and classification module
Figure 831248DEST_PATH_IMAGE026
The combination of the two is actually a multi-class attention multi-instance learning model, and since the attention multi-instance learning model is used in a WSI image classification task, all image blocks are usually encoded into feature vectors by using a pre-training network, and then a subsequent attention fusion module is trained, which is equivalent to freezing parameters of a preceding feature extractor, before performing E-step of a first iteration, initialized image block features of a pathological image block set are extracted by using the initialized image block feature extractor, and then the attention fusion module and the WSI classifier are trained by using the initialized image block features and class labels corresponding to the WSI images, which is equivalent to updating once first
Figure 583172DEST_PATH_IMAGE026
The parameters of (1); and classifies the image blocks
Figure 750236DEST_PATH_IMAGE010
Fraction value of corresponding category label of all image blocks belonging to WSI image in medium prediction
Figure 208899DEST_PATH_IMAGE023
The number of the given 1 s is given to the target,
Figure 675652DEST_PATH_IMAGE023
is shown asiIn a WSI imagenPrediction score values for image blocks.
As shown in fig. 2, the process of iteratively training the WSI classification model using the EM method is described in detail below:
the step of iteratively training the WSI classification model by the EM method comprises E-step and M-step;
where the E-step key is to estimate the posterior probability distribution
Figure 321397DEST_PATH_IMAGE031
I.e. from the input WSI imageX i And corresponding category labely i And parameters of the WSI classification modelθ t() Assigning a pseudo label to each image block in the set of pathological images, whereinθ t() Is shown astThe model parameters of the round iteration are specifically as follows:
inputting the pathological image block data set into a WSI classification model, and using an image block feature extractor
Figure 899009DEST_PATH_IMAGE019
Extracting image block features:
Figure 949529DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 321605DEST_PATH_IMAGE021
is shown asiIn a WSI imagenPage image block characteristics;
inputting the image data into an attention fusion module omega to obtain attention weight values of all image blocks in the WSI imagea n } i ：
Figure 595591DEST_PATH_IMAGE022
Wherein the leafa n } i Is shown asiIn a WSI imagenAttention weight of the image blocks;
simultaneous input of image block features into image block classifier
Figure 976894DEST_PATH_IMAGE010
Fraction value of corresponding category label of all image blocks belonging to WSI image in medium prediction
Figure 472466DEST_PATH_IMAGE023
：
Figure 156389DEST_PATH_IMAGE024
The attention weighted value of the same image block is readya n } i And score value
Figure 104622DEST_PATH_IMAGE023
Performing dot product to obtain integral fraction of pathological image block sets n } i ：
Figure 24036DEST_PATH_IMAGE025
Making an integral score of a pathological image block sets n } i Sorting in descending order, and assigning pseudo label to image block according to threshold lambdaz n } i (ii) a Image with score value higher than threshold value λ in the present embodimentThe method comprises the steps that a block is endowed with a class label corresponding to a WSI image, image blocks with the score values lower than a threshold lambda are regarded as irrelevant to classification, and the image blocks are classified into a background class;
in practical process, whole fraction of all image blocks in different WSI imagess n } i The value range is different, the threshold lambda aiming at different WSI images is difficult to confirm, and the threshold has high influence on the quality of the pseudo label, so the invention adopts an approximate method according to a great Chinese styles n } i Selecting a top-N image block with the highest score to be endowed with a WSI real category label, and dividing the top-N image block with the lowest score into a background category; since the threshold change only affects the category of the image blocks with the scores in the middle, and the average number of the image blocks in the WSI image is 8000 to 15000, the image blocks at the head end and the tail end cannot be affected; on the other hand, if all image blocks are used to train the feature extractor
Figure 783570DEST_PATH_IMAGE019
The introduction of noise is inevitable and this way reduces the number of image blocks used for training and thus improves their quality, so that
Figure 763027DEST_PATH_IMAGE019
Features that are beneficial for WSI classification are easier to learn.
M-step is to update the parameters of the WSI classification model so as to make the objective function of the WSI classification model reach the maximum;
defining an objective function based on a WSI classification model
Figure 870660DEST_PATH_IMAGE012
And according to the equation
Figure 328186DEST_PATH_IMAGE013
And decomposing the objective function of the WSI classification model into:
Figure 939296DEST_PATH_IMAGE014
Figure 89655DEST_PATH_IMAGE015
wherein the content of the first and second substances,
Figure 419005DEST_PATH_IMAGE017
the set of image block classes representing the prediction by the WSI classification model is equal to the a posteriori probability of the true label underlying the image block,J(θ,θ t() ) Indicating WSI Classification model usagetParameters of a wheel iteration modelθ t() Training to obtain model parameters ofθCan be understood as: from an input WSI imageX i And the parameters areθ t() The WSI classification model predicts that the WSI class label is equal to the corresponding class label of the WSI imagey i And according to the input WSI imageX i With the parameters ofθ t() The WSI classification model and the corresponding class label of the WSI imagey i The prediction image block class label is equal to the likelihood that the image block implies a true label. The maximum likelihood estimation is to find the optimal model parameters
Figure DEST_PATH_IMAGE032
So that the probability that the predicted WSI class label and image block class label are equal to the respective true labels is maximized, and thereforeJ(θ,θ t() ) Is also an objective function to be optimized;
and further expanding to obtain an objective function expansion formula of the WSI classification model:
Figure 883484DEST_PATH_IMAGE018
therefore, according to the expansion, M-step first optimizes the parameters of the first term, i.e. updates the parameters of the image block feature extractorθ 1 And parameters of image block classifierθ 3 Then the updated parameters are used againθ 1 To optimizeParameters of two termsθ 2 I.e. based on the parameters of the image block feature extractorθ 1 Updating parameters of a feature fusion and classification moduleθ 2 And completing the parameter updating of the whole WSI classification model, specifically comprising the following steps:
the M-step is divided into two steps according to the expansion, the first step M1-step: pseudo label for using image blockz n } i Training image block feature extractor by full-supervision training method
Figure 605891DEST_PATH_IMAGE019
And an image block classifier
Figure 192730DEST_PATH_IMAGE010
；
Second step M2-step: fixed image block feature extractor
Figure 9376DEST_PATH_IMAGE019
The pathological image blocks are encoded again, and the encoded image block features and the class labels corresponding to the WSI images are used for retraining the parameters of the attention fusion module and the WSI classifierθ 2 。
In iterative training, a loss function of the WSI classification model is constructed by adopting cross entropy, and the loss function comprises an L1 loss function and an L2 loss function;
the L1 loss function is used to train the updated WSI classifier, and is expressed as:
Figure 808705DEST_PATH_IMAGE027
wherein N represents the total number of categories predicted by the WSI classification model,
Figure 659986DEST_PATH_IMAGE028
representing a WSI classification model predicting that a WSI image belongs to a categoryiThe score of (a) is calculated,y i e {0,1} represents a category label corresponding to the WSI image subjected to 0-1 encoding;
the L2 loss function is used to train the update graph block classifier, represented as:
Figure 293093DEST_PATH_IMAGE029
wherein N +1 represents the total number of classes predicted by the image block classifier,
Figure 331456DEST_PATH_IMAGE030
representing image block classifier to predict image block to belong to WSI image real categoryiThe value of the fraction of (c) is,z i e {0,1} represents the image block pseudo label after 0-1 coding.
And S4, inputting the WSI image to be diagnosed of the cancer patient into the trained WSI classification model, outputting the classification result of the WSI image to be diagnosed and drawing a probability thermodynamic diagram.
The probabilistic thermodynamic diagram can clearly reflect which part of the image belongs to the category 1 and which part of the image belongs to the category 2 in the binary problem; if the multi-classification problem is analyzed, a probability thermodynamic diagram for performing two-classification on each class and other classes needs to be drawn separately; in this embodiment, two classification tasks on the Camelyon data set to determine whether a cancer patient has breast cancer metastasis are taken as examples, and specific steps for drawing a probability thermodynamic diagram are described as follows:
after a trained WSI classification model is obtained, the WSI image to be diagnosed is input, and the image block classifier is used for predicting the score values of all image blocks in the WSI image
Figure 200055DEST_PATH_IMAGE023
After the value range of the score value is changed into 0,1 after the normalization of the softmax function]Taking the normalized scores as probability values of the image blocks belonging to all the categories, wherein the sum of the scores of the image blocks belonging to all the categories is 1;
then, an all-0 matrix is created, the length and width of which are downsampled for the WSI image to be diagnosednObtaining the length and width of the matrix by down-sampling 64 times in the embodiment;
according to the position of an image block in the WSI image to be diagnosed, simple coordinate conversion is carried out, the probability value predicted by each image block is filled in the corresponding position of a matrix, and finally a probability matrix is obtained, wherein the values in the probability matrix are all decimals between 0 and 1, and the size of the probability matrix is 64 times of the size of the WSI image to be diagnosed;
and constructing a color mapping table corresponding to different probability values, mapping colors to corresponding positions of the WSI image to be diagnosed according to the probability values of the probability matrix, and obtaining a probability thermodynamic diagram.
In this embodiment, a color mapping table from blue to red is constructed, and the color mapping table corresponds to different probability values, where a probability value of 0 is represented by pure blue, a probability value of 1 is represented by pure red, and a probability value of 0 to 1 is mapped to a gradient color between pure blue and pure red; finally, a probability thermodynamic diagram equal to the size of the WSI image 1/64 to be diagnosed is obtained, the probability of breast cancer metastasis of each region in the WSI image is reflected, the probability that metastasis occurs is higher when the color is red, and the probability value that metastasis occurs is lower when the color is blue; the pathologist can directly and quickly lock the area with the breast cancer metastasis from the red areas and then carry out more careful examination, thereby accelerating the diagnosis process.
The existing WSI image weak supervision classification method does not fully learn the characteristics of the image, and more, the WSI classification accuracy is improved by learning a better fusion model, such as a multi-example learning model, and fusing information of different patches; however, as can be seen from the machine learning principle, only better features are learned to enable the system to generate better pattern discrimination capability. Therefore, the invention adopts a new idea, effectively processes image data with high heterogeneity by directly learning more discriminative image characteristics, and essentially improves the accuracy of WSI classification diagnosis.
The invention provides a WSI image classification method (BAL) based on Bayesian assisted learning, which is characterized in that image block classification is used as an auxiliary task, a WSI classification model is constructed by applying Bayesian decision theory, and model parameters are updated by using EM algorithm, so that end-to-end learning from a feature extractor to a feature fusion and classification module is realized. Compared with the prior art, the method has the advantages that the features of the image blocks are effectively and dynamically updated, and the features with rich judgment information are finally learned, so that the accuracy of WSI classification is improved; in addition, the invention provides a more reasonable strategy for labeling the false labels, and the local information of the image block classification and the global information required by the WSI classification are fully combined, so that the image block classification labels with high quality are screened out and used for training the image block classifier; meanwhile, the image block classifier obtained by the auxiliary task learning can be used for generating a probability thermodynamic diagram corresponding to the WSI, so that the accurate positioning of a cancerous region is realized, and the interpretability of the WSI auxiliary diagnosis system is greatly improved.
To verify the effectiveness of the method of the invention, experiments were performed on open datasets of three different cancer types, the three datasets being the Camelyon16 dataset, the TCGA-NSCLC dataset and the TCGA-RCC dataset, respectively;
the camellyon 16 data set is a public data set for pathological biopsy of the sentinel lymph node of the breast cancer, and comprises 270H & E staining WSIs in a training set and 129 WSI images in a testing set, and is used for diagnosing whether the sentinel lymph node of the breast cancer is transferred or not and performing a classification task; in the test, 15% of samples are randomly extracted from a training set to serve as a verification set, the rest 85% of data are used as training, and a test set divided by an original data set is reserved for testing.
The TCGA-NSCLC data set is a non-small cell lung cancer digital pathological full-section image data set, comprises 535 lung adenocarcinoma WSIs and 512 lung squamous carcinoma WSIs, and 1047 WSI images in total, and is used for typing two subtypes of non-small cell lung cancer and performing a classification task; in order to carry out more sufficient tests, the training/testing set is divided by common 5-fold cross validation, the proportion of adenocarcinoma/squamous carcinoma of each fold is ensured to be the same as that of the original data set, and in each division, 20% of samples are randomly selected from the training set to serve as a validation set for selecting models and hyper-parameters.
TCGA-RCC is a renal cell carcinoma digital pathology whole slice image dataset, comprises 297 renal papillary cell carcinoma WSI, 121 renal chromophobe cell carcinoma WSI and 519 renal clear cell carcinoma WSI, and 937 WSI images in total, and is used for typing three subtypes of renal cell carcinoma and performing three classification tasks; the test was also performed using a 5-fold cross-validation method, as described in the TCGA-NSCLC data set.
The test selects MAX-MIL, ABMIL, EM-CNN, CLAM and TransMIL methods in the prior art to carry out the test on three public data sets, and the results are shown in the following table:
Figure DEST_PATH_IMAGE033
from the above table, the final performance of the method of the present invention exceeds that of the existing method on three data sets, wherein BAL (t0) represents the model of the method of the present invention when just initialized, BAL (t3) represents the model of the method of the present invention after 3 rounds of EM iteration, it can be seen that the performance of the model just initialized is not as good as that of the current optimal methods such as CLAMs and transamls, but after 3 rounds of EM iteration, the performance of the model is greatly improved and finally exceeds that of other methods, which proves that the iterative updating of the feature extractor and the feature fusion and classification model by using the EM algorithm is effective and can help for WSI classification.
In addition, as shown in fig. 3, the given 6 image blocks are images containing breast cancer lymph node metastasis, it can be seen that the class activation map can better cover the tumor region in the patch, and as the EM iteration proceeds from left to right, the class activation map has better coverage degree in the tumor region, and the region outline is more obvious, so that it should be proved that the discrimination capability of the image block classifier in the EM iteration is improved, so that the network can learn better feature expression, and the image block classifier obtained by the auxiliary task can realize better tumor region localization capability, which is deficient in other WSI classification methods.
In addition, as shown in fig. 4, 3 WSIs are selected from the Camelyon16 data set of breast cancer lymph node metastasis, and probability thermodynamic diagrams of the WSIs drawn at different iteration times, so that as the iteration progresses, the probability thermodynamic diagrams can better cover the region of the WSI image where the breast cancer metastasis occurs, and help a doctor quickly locate a region of interest in the WSI image to assist the doctor in diagnosis.
It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present invention is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present invention.
Based on the same idea as the WSI image classification method based on Bayesian assisted learning in the embodiment, the invention also provides a WSI image classification system based on Bayesian assisted learning, and the system can be used for executing the WSI image classification method based on Bayesian assisted learning. For convenience of illustration, the structural diagram of the WSI image classification system based on bayesian-assisted learning in the embodiment of the present invention only shows the parts related to the embodiment of the present invention, and those skilled in the art will understand that the illustrated structure does not constitute a limitation to the apparatus, and may include more or less components than those illustrated, or combine some components, or arrange different components.
Referring to fig. 5, in another embodiment of the present application, a WSI image classification system based on bayesian-assisted learning is provided, the system includes a data acquisition module, a model construction module, a model training module, and a diagnosis module;
the data acquisition module is used for acquiring a WSI image and a corresponding class label, converting the WSI image into an HSV space, extracting a foreground region according to saturation and dividing the foreground region into a plurality of image blocks to obtain a pathological image block set;
the model construction module is used for constructing a WSI classification model based on a Bayesian decision theory, and comprises an image block feature extractor, an image block classifier and a feature fusion and classification module; the feature fusion and classification module comprises an attention fusion module and a WSI classifier;
the model training module is used for initializing the WSI classification model, iteratively training the WSI classification model on the pathological image block set by using an EM (effective electromagnetic radiation) method, and optimizing model parameters until the model converges to obtain the trained WSI classification model;
the diagnosis module is used for inputting the WSI images to be diagnosed of the cancer patients into the trained WSI classification model, outputting the classification results of the WSI images to be diagnosed and drawing a probability thermodynamic diagram.
It should be noted that, the WSI image classification system based on bayesian-aided learning of the present invention corresponds to the WSI image classification method based on bayesian-aided learning of the present invention one to one, and the technical features and the beneficial effects thereof described in the above embodiment of the WSI image classification method based on bayesian-aided learning are both applicable to the embodiment of the WSI image classification system based on bayesian-aided learning, and specific contents can be referred to the description in the embodiment of the method of the present invention, which is not repeated herein, and thus the present invention is declared.
In addition, in the implementation of the WSI image classification system based on bayesian-assisted learning in the foregoing embodiment, the logical division of each program module is only an example, and in practical applications, the foregoing function allocation may be performed by different program modules according to needs, for example, due to configuration requirements of corresponding hardware or convenience of implementation of software, that is, the internal structure of the WSI image classification system based on bayesian-assisted learning is divided into different program modules to perform all or part of the functions described above.
Referring to fig. 6, in an embodiment, a computer-readable storage medium is provided, in which a program is stored in a memory, and when the program in the memory is executed by a processor, the method for classifying a WSI image based on bayesian-assisted learning is implemented, and specifically, the method includes:
acquiring a WSI image and a corresponding class label, converting the WSI image into an HSV space, extracting a foreground area according to saturation, and dividing the foreground area into a plurality of image blocks to obtain a pathological image block set;
constructing a WSI classification model based on a Bayesian decision theory, wherein the WSI classification model comprises an image block feature extractor, an image block classifier and a feature fusion and classification module; the feature fusion and classification module comprises an attention fusion module and a WSI classifier;
the image block feature extractor is used for extracting features of image blocks in the pathological image block set; the image block classifier assigns a false label to the image block; the attention fusion module is used for acquiring attention weight corresponding to the image block characteristics; the WSI classifier is used for classifying the WSI images;
initializing a WSI classification model, iteratively training the WSI classification model on the pathological image block set by using an EM (effective ray spectroscopy) method, and optimizing model parameters until the model converges to obtain a trained WSI classification model;
and inputting the WSI image to be diagnosed of the cancer patient into the trained WSI classification model, outputting the classification result of the WSI image to be diagnosed and drawing a probability thermodynamic diagram.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.

Claims (10)

1. The WSI image classification method based on Bayesian assisted learning is characterized by comprising the following steps of:
acquiring a WSI image and a corresponding class label, converting the WSI image into an HSV space, extracting a foreground area according to saturation, and dividing the foreground area into a plurality of image blocks to obtain a pathological image block set;
constructing a WSI classification model based on a Bayesian decision theory, wherein the WSI classification model comprises an image block feature extractor, an image block classifier and a feature fusion and classification module; the feature fusion and classification module comprises an attention fusion module and a WSI classifier;
the image block feature extractor is used for extracting features of image blocks in the pathological image block set; the image block classifier is used for giving a false label to the image block; the attention fusion module is used for acquiring attention weight corresponding to the image block characteristics; the WSI classifier is used for classifying the WSI images;
initializing a WSI classification model, iteratively training the WSI classification model on the pathological image block set by using an EM (effective ray spectroscopy) method, and optimizing model parameters until the model converges to obtain a trained WSI classification model;
and inputting the WSI image to be diagnosed of the cancer patient into the trained WSI classification model, outputting the classification result of the WSI image to be diagnosed and drawing a probability thermodynamic diagram.
2. The WSI image classification method based on Bayesian assisted learning as per claim 1, wherein the WSI classification model is constructed based on Bayesian decision theory and is constructed by collecting pathological image blocks
Figure 32803DEST_PATH_IMAGE001
The training learning parameters areθ=(θ 1 ,θ 2 ,θ 3 ) WSI classification modelF θ Wherein
Figure DEST_PATH_IMAGE002
Is shown asiA sheet of the WSI image is printed,
Figure DEST_PATH_IMAGE004
is shown asiIn a WSI imagenThe number of image blocks is one,N i is shown asiThe number of image blocks of a WSI image block,y i is shown asiA category label corresponding to the WSI image,θ 1 are parameters of the image block feature extractor,θ 2 for the parameters of the feature fusion and classification module,θ 3 for the parameters of the image block classifier, based on the input WSI imageXPredicting category labelsy= F θ (X) The method specifically comprises the following steps:
will train the learning parameters asθ=(θ 1 ,θ 2 ,θ 3 ) WSI classification modelF θ Defined as the maximum likelihood estimation problem, expressed as:
Figure 74577DEST_PATH_IMAGE005
wherein the content of the first and second substances,θ * theoretical parameters representing the WSI classification modelDI represents the number of WSI images in the pathological image block set;
introducing hidden variables
Figure DEST_PATH_IMAGE006
And representing a class label set of the image block to obtain an objective function of the WSI classification model:
Figure 385473DEST_PATH_IMAGE007
wherein the content of the first and second substances,
Figure DEST_PATH_IMAGE008
representing each image block
Figure 568630DEST_PATH_IMAGE004
The implicit true category label is that the user is,Crepresenting a set of category labels contained in the WSI image.
3. The WSI image classification method based on Bayesian assisted learning of claim 2, wherein the image block feature extractor adopts a ResNet-50 network as a backbone network;
the initialization WSI classification model specifically comprises the following steps:
pre-training an image block feature extractor on an ImageNet data set, and initializing parameters of the image block feature extractor;
the initialized image block feature extractor encodes an input image block into a 1024-dimensional feature vector;
and initializing the parameters of the image block classifier, the attention fusion module and the WSI classifier by adopting random initialization.
4. The WSI image classification method based on Bayesian assisted learning according to claim 3, wherein before an EM method is used for iterative training of a WSI classification model on a pathological image block set, an initialized image block feature of the pathological image block set is extracted by using an initialized image block feature extractor, and then an attention fusion module and a WSI classifier are trained by using the initialized image block feature and a class label corresponding to the WSI image, so that parameters of a feature fusion and classification module are updated;
and classifies the image blocks
Figure DEST_PATH_IMAGE010
Fraction value of corresponding category label of all image blocks belonging to WSI image in medium prediction
Figure 304374DEST_PATH_IMAGE011
The number of the given 1 s is given to the target,
Figure 189154DEST_PATH_IMAGE011
is shown asiIn a WSI imagenPrediction score values for image blocks.
5. The WSI image classification method based on Bayesian assistant learning of claim 4, wherein the step of iteratively training WSI classification models by EM method comprises E-step and M-step;
the E-step is based on the input WSI imageX i And corresponding category labely i And parameters of the WSI classification modelθ t() Assigning a pseudo label to each image block in the set of pathological images, whereinθ t() Is shown astModel parameters of the round iteration;
defining an objective function based on a WSI classification model
Figure DEST_PATH_IMAGE012
And according to the equation
Figure 252925DEST_PATH_IMAGE013
And decomposing the objective function of the WSI classification model into:
Figure DEST_PATH_IMAGE014
Figure 982983DEST_PATH_IMAGE015
wherein the content of the first and second substances,
Figure 979758DEST_PATH_IMAGE017
the set of image block classes representing the prediction of the WSI classification model is equal to the a posteriori probability of the true label underlying the image block,J(θ,θ t() ) Indicating WSI Classification model usagetParameters of a wheel iteration modelθ t() Training to obtain model parameters ofθThe objective function of (1);
and further expanding to obtain an objective function expansion formula of the WSI classification model:
Figure DEST_PATH_IMAGE018
determining parameters of the M-step for updating the image block feature extractor according to an objective function expansion of a WSI classification modelθ 1 And parameters of image block classifierθ 3 Then according to the parameters of image block feature extractorθ 1 Updating parameters of a feature fusion and classification moduleθ 2 。
6. The WSI image classification method based on Bayesian assisted learning as recited in claim 5, wherein the E-step is specifically:
inputting the pathological image block data set into a WSI classification model, and using an image block feature extractor
Figure 769859DEST_PATH_IMAGE019
Extracting image block features:
Figure DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 448490DEST_PATH_IMAGE021
is shown asiIn a WSI imagenPage image block characteristics;
inputting the image data into an attention fusion module omega to obtain attention weight values of all image blocks in the WSI imagea n } i ：
Figure DEST_PATH_IMAGE022
Wherein the leafa n } i Is shown asiIn a WSI imagenAttention weight of the image blocks;
simultaneous input of image block features into image block classifier
Figure 982239DEST_PATH_IMAGE010
Fraction value of corresponding category label of all image blocks belonging to WSI image in medium prediction
Figure 567942DEST_PATH_IMAGE011
：
Figure 466627DEST_PATH_IMAGE023
The attention weighted value of the same image block is readya n } i And score value
Figure 36149DEST_PATH_IMAGE011
Performing dot product to obtain integral fraction of pathological image block sets n } i ：
Figure DEST_PATH_IMAGE024
Making an integral score of a pathological image block sets n } i Sorting in descending order, and assigning pseudo label to image block according to threshold lambdaz n } i ；
The M-step comprises M1-step and M2-step, and specifically comprises the following steps:
m1-step: pseudo label for using image blockz n } i Training image block feature extractor by full-supervision training method
Figure 967065DEST_PATH_IMAGE019
And an image block classifier
Figure 813798DEST_PATH_IMAGE010
；
M2-step: fixed image block feature extractor
Figure 742440DEST_PATH_IMAGE019
The parameters of the method are used for extracting the characteristics of the pathological image block again, and the updated image block characteristics and the class label corresponding to the WSI image are used for retraining the characteristic fusion and classification module
Figure 802187DEST_PATH_IMAGE025
The parameter (c) of (c).
7. The WSI image classification method based on Bayesian assistant learning of claim 6, wherein the loss function of the WSI classification model is constructed by adopting cross entropy, and comprises an L1 loss function and an L2 loss function;
the L1 loss function is used to train the updated WSI classifier, and is expressed as:
Figure DEST_PATH_IMAGE026
wherein N represents the total number of classes predicted by the WSI classification model,
Figure DEST_PATH_IMAGE027
representing a WSI classification model predicting that a WSI image belongs to a categoryiThe score of (a) is calculated,y i e {0,1} represents a category label corresponding to the WSI image subjected to 0-1 encoding;
the L2 loss function is used to train the update image block classifier, represented as:
Figure DEST_PATH_IMAGE028
wherein N +1 represents the total number of classes predicted by the image block classifier,
Figure DEST_PATH_IMAGE029
representing that an image block classifier predicts that an image block belongs to a WSI image true categoryiThe value of the fraction of (c) is,z i e {0,1} represents the image block pseudo label after 0-1 encoding.
8. The WSI classification diagnosis method based on Bayesian assisted learning according to claim 7, wherein the drawing probability thermodynamic diagram specifically comprises:
inputting the WSI image to be diagnosed of the cancer patient into the trained WSI classification model, and outputting a classification result;
the fraction values of all image blocks in the WSI image to be diagnosed, which are predicted by the image block classifier in the classification result
Figure 271215DEST_PATH_IMAGE011
Obtaining the probability values of the image blocks belonging to the various categories corresponding to the WSI image to be diagnosed through the normalization of the softmax function;
creating an all-0 matrix, wherein the length and the width of the matrix are down-sampled for the WSI image to be diagnosednDoubling;
performing coordinate conversion according to the position of the image block in the WSI image to be diagnosed, and filling the probability value of the image block into the corresponding position of the matrix to obtain a probability matrix;
and constructing a color mapping table corresponding to different probability values, mapping colors to corresponding positions of the WSI image to be diagnosed according to the probability values of the probability matrix, and obtaining a probability thermodynamic diagram.
9. The WSI image classification system based on Bayesian assisted learning is characterized by being applied to the WSI image classification method based on Bayesian assisted learning in any one of claims 1-8, and comprising a data acquisition module, a model construction module, a model training module and a diagnosis module;
the data acquisition module is used for acquiring a WSI image and a corresponding class label, converting the WSI image into an HSV space, extracting a foreground region according to saturation and dividing the foreground region into a plurality of image blocks to obtain a pathological image block set;
the model construction module is used for constructing a WSI classification model based on a Bayesian decision theory, and the WSI classification model comprises an image block feature extractor, an image block classifier and a feature fusion and classification module; the feature fusion and classification module comprises an attention fusion module and a WSI classifier;
the image block feature extractor is used for extracting features of image blocks in the pathological image block set; the image block classifier is used for giving a pseudo label to the image block; the attention fusion module is used for acquiring attention weight corresponding to the image block characteristics; the WSI classifier is used for classifying the WSI images;
the model training module is used for initializing the WSI classification model, iteratively training the WSI classification model on the pathological image block set by using an EM (effective electromagnetic radiation) method, and optimizing model parameters until the model converges to obtain the trained WSI classification model;
the diagnosis module is used for inputting the WSI images to be diagnosed of the cancer patients into the trained WSI classification model, outputting the classification results of the WSI images to be diagnosed and drawing a probability thermodynamic diagram.
10. A computer-readable storage medium storing a program, wherein the program, when executed by a processor, implements the bayesian-aided learning-based WSI image classification method of any of claims 1-8.
CN202210895166.9A 2022-07-28 2022-07-28 WSI image classification method, system and medium based on Bayesian assisted learning Active CN114972341B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210895166.9A CN114972341B (en) 2022-07-28 2022-07-28 WSI image classification method, system and medium based on Bayesian assisted learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210895166.9A CN114972341B (en) 2022-07-28 2022-07-28 WSI image classification method, system and medium based on Bayesian assisted learning

Publications (2)

Publication Number Publication Date
CN114972341A true CN114972341A (en) 2022-08-30
CN114972341B CN114972341B (en) 2022-12-16

Family

ID=82968570

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210895166.9A Active CN114972341B (en) 2022-07-28 2022-07-28 WSI image classification method, system and medium based on Bayesian assisted learning

Country Status (1)

Country Link
CN (1) CN114972341B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115797729A (en) * 2023-01-29 2023-03-14 有方（合肥）医疗科技有限公司 Model training method and device, and motion artifact identification and prompting method and device
CN116228759A (en) * 2023-05-08 2023-06-06 浙江大学滨江研究院 Computer-aided diagnosis system and apparatus for renal cell carcinoma type
CN116741347A (en) * 2023-05-12 2023-09-12 中山大学附属第一医院 Pathological image patches extraction and deep learning modeling method
CN117038023A (en) * 2023-10-08 2023-11-10 中国医学科学院北京协和医院 dMMR germ line mutation subtype classification method and system based on colorectal cancer pathological image
CN116741347B (en) * 2023-05-12 2024-06-04 中山大学附属第一医院 Pathological image patches extraction and deep learning modeling method

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200184643A1 (en) * 2018-12-11 2020-06-11 New York University Classification and mutation prediction from histopathology images using deep learning
CN111814342A (en) * 2020-07-16 2020-10-23 中国人民解放军空军工程大学 Complex equipment reliability hybrid model and construction method thereof
CN112435243A (en) * 2020-11-26 2021-03-02 山东第一医科大学附属省立医院(山东省立医院) Automatic analysis system and method for full-slice digital pathological image
CN113221978A (en) * 2021-04-27 2021-08-06 浙江师范大学 Colorectal cancer digital pathological image discrimination method and system based on weak supervised learning
CN114783604A (en) * 2022-04-21 2022-07-22 华南理工大学 Method, system and storage medium for predicting sentinel lymph node metastasis of breast cancer

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200184643A1 (en) * 2018-12-11 2020-06-11 New York University Classification and mutation prediction from histopathology images using deep learning
CN111814342A (en) * 2020-07-16 2020-10-23 中国人民解放军空军工程大学 Complex equipment reliability hybrid model and construction method thereof
CN112435243A (en) * 2020-11-26 2021-03-02 山东第一医科大学附属省立医院(山东省立医院) Automatic analysis system and method for full-slice digital pathological image
CN113221978A (en) * 2021-04-27 2021-08-06 浙江师范大学 Colorectal cancer digital pathological image discrimination method and system based on weak supervised learning
CN114783604A (en) * 2022-04-21 2022-07-22 华南理工大学 Method, system and storage medium for predicting sentinel lymph node metastasis of breast cancer

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
JIANJIN DENG ET AL.: "A simple graph-based semi-supervised learning approach for imbalanced classification", 《PATTERN RECOGNITION》 *
黄雷 等: "CT淋巴造影联合三维可视技术在乳腺癌淋巴回流路径研究中的应用", 《实用医学杂志》 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115797729A (en) * 2023-01-29 2023-03-14 有方（合肥）医疗科技有限公司 Model training method and device, and motion artifact identification and prompting method and device
CN115797729B (en) * 2023-01-29 2023-05-09 有方（合肥）医疗科技有限公司 Model training method and device, motion artifact identification and prompting method and device
CN116228759A (en) * 2023-05-08 2023-06-06 浙江大学滨江研究院 Computer-aided diagnosis system and apparatus for renal cell carcinoma type
CN116228759B (en) * 2023-05-08 2023-08-01 浙江大学滨江研究院 Computer-aided diagnosis system and apparatus for renal cell carcinoma type
CN116741347A (en) * 2023-05-12 2023-09-12 中山大学附属第一医院 Pathological image patches extraction and deep learning modeling method
CN116741347B (en) * 2023-05-12 2024-06-04 中山大学附属第一医院 Pathological image patches extraction and deep learning modeling method
CN117038023A (en) * 2023-10-08 2023-11-10 中国医学科学院北京协和医院 dMMR germ line mutation subtype classification method and system based on colorectal cancer pathological image

Also Published As

Publication number Publication date
CN114972341B (en) 2022-12-16

Similar Documents

Publication Publication Date Title
CN114972341B (en) WSI image classification method, system and medium based on Bayesian assisted learning
Saxena et al. Machine learning methods for computer-aided breast cancer diagnosis using histopathology: a narrative review
Man et al. Classification of breast cancer histopathological images using discriminative patches screened by generative adversarial networks
Beevi et al. A multi-classifier system for automatic mitosis detection in breast histopathology images using deep belief networks
Cao et al. A multi-kernel based framework for heterogeneous feature selection and over-sampling for computer-aided detection of pulmonary nodules
CN112101451A (en) Breast cancer histopathology type classification method based on generation of confrontation network screening image blocks
Balamurugan et al. Brain tumor segmentation and classification using hybrid deep CNN with LuNetClassifier
CN109214397A (en) The dividing method of Lung neoplasm in a kind of lung CT image
CN109671060B (en) Computer-aided breast mass detection method based on selective search and CNN
Liu et al. Fine-grained breast cancer classification with bilinear convolutional neural networks (BCNNs)
Jiang et al. Breast cancer detection and classification in mammogram using a three-stage deep learning framework based on PAA algorithm
CN114841947A (en) Method and device for multi-scale feature extraction and prognosis analysis of H & E staining pathological image tumor region
Damkliang et al. Tissue classification for colorectal cancer utilizing techniques of deep learning and machine learning
Morkūnas et al. Machine learning based classification of colorectal cancer tumour tissue in whole-slide images
CA3145370A1 (en) Systems and methods for mesothelioma feature detection and enhanced prognosis or response to treatment
Sarath et al. A two-stage multiple instance learning framework for the detection of breast cancer in mammograms
He et al. Multiphase level set model with local K-means energy for histology image segmentation
Mehak et al. Automated Grading of Breast Cancer Histopathology Images Using Multilayered Autoencoder.
CN116884597A (en) Pathological image breast cancer molecular typing method and system based on self-supervision pre-training and multi-example learning
Adhikary et al. Feature assisted cervical cancer screening through DIC cell images
Abbas et al. Multi-cell type and multi-level graph aggregation network for cancer grading in pathology images
Chugh et al. Morphological and otsu’s technique based mammography mass detection and deep neural network classifier based prediction
Das et al. Kernel-based Fisher discriminant analysis on the Riemannian manifold for nuclear atypia scoring of breast cancer
Kalsoom et al. An efficient liver tumor detection using machine learning
Guo et al. A novel 2D ground-glass opacity detection method through local-to-global multilevel thresholding for segmentation and minimum bayes risk learning for classification

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20231026

Address after: Room 219, Building G2, No. 31 Kefeng Road, Huangpu District, Guangzhou City, Guangdong Province, 510663

Patentee after: Guangzhou Fangxin Medical Technology Co.,Ltd.

Address before: 510640 No. five, 381 mountain road, Guangzhou, Guangdong, Tianhe District

Patentee before: SOUTH CHINA University OF TECHNOLOGY

TR01 Transfer of patent right