WO2022228349A1 - 基于弱监督学习的结直肠癌数字病理图像判别方法及系统 - Google Patents

基于弱监督学习的结直肠癌数字病理图像判别方法及系统 Download PDF

Info

Publication number
WO2022228349A1
WO2022228349A1 PCT/CN2022/088794 CN2022088794W WO2022228349A1 WO 2022228349 A1 WO2022228349 A1 WO 2022228349A1 CN 2022088794 W CN2022088794 W CN 2022088794W WO 2022228349 A1 WO2022228349 A1 WO 2022228349A1
Authority
WO
WIPO (PCT)
Prior art keywords
pathological image
module
digital pathological
supervised learning
weakly supervised
Prior art date
Application number
PCT/CN2022/088794
Other languages
English (en)
French (fr)
Inventor
朱信忠
徐慧英
李军
赵建民
Original Assignee
浙江师范大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浙江师范大学 filed Critical 浙江师范大学
Publication of WO2022228349A1 publication Critical patent/WO2022228349A1/zh
Priority to ZA2023/10619A priority Critical patent/ZA202310619B/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems

Definitions

  • the present application relates to the technical field of pathological images, in particular to a method and system for discriminating digital pathological images of colorectal cancer based on weakly supervised learning.
  • CRC Colorectal cancer
  • Tumors are usually diagnosed with suspicious lesions by imaging, and then further removed by needle biopsy or surgery. Tissue samples of 3 to 4 ⁇ m are taken from the patient’s body.
  • the pathology department uses a relatively complex staining and preparation process to make them that can be observed under a microscope. monolayer cell slice samples. An experienced pathologist can diagnose suspicious lesions by observing the morphological structure of cells under a high-power optical microscope. If it is a diseased tissue, it is necessary to further clarify the necessary information in tumor diagnosis, such as its degree of differentiation and its pathological type. These information will greatly affect the patient's treatment plan and prognosis. Therefore, pathological examination is the core key in tumor diagnosis. steps, whose diagnostic results are known as the gold standard for tumor diagnosis.
  • tumor tissue is composed of a complex microenvironment (Tumor Micro Environment, TME), and the tissue collected by pathological biopsy usually contains a variety of cells .
  • TME Tumor Micro Environment
  • Pathological images are an intuitive manifestation of cell morphology, so pathological images often contain more complex image information.
  • Digital pathology images are different from common medical images. Images under different microscope magnifications are usually saved in a file in the form of an image pyramid. Among them, the image under 20x magnification can clearly observe the tissue structure of cells, and can effectively complete the diagnosis, which is an ideal magnification. However, an image at an ideal magnification usually has hundreds of millions of pixels. Under the existing computer performance, common image processing algorithms are not suitable for such large-scale images, so it is necessary to effectively preprocess digital pathological images. Its special preprocessing method also makes the diagnosis problems based on digital pathological images different from conventional problems.
  • the purpose of this application is to provide a method and system for discriminating digital pathological images of colorectal cancer based on weakly supervised learning, aiming at the defects of the prior art.
  • Colorectal cancer digital pathology image discrimination system based on weakly supervised learning including:
  • an acquisition module for acquiring a digital pathological image data set of colorectal cancer
  • the preprocessing module is used to preprocess the data in the collected data set to obtain the preprocessed data
  • the first classification module is used to construct a sampling block discrimination model based on a weakly supervised learning algorithm, input the preprocessed data into the constructed sampling block discrimination model for processing, and obtain the classification results of all the pathological image blocks in the full slice sampling package;
  • the second classification module is used to construct a decision fusion model, and input the obtained classification results of the pathological image blocks into the decision fusion model for fusion processing to obtain the classification results of all-digital pathological images.
  • the preprocessing module includes:
  • a tissue localization module for localizing valid tissue in the input digital pathology image and generating a sampling data set
  • the normalization module is used to perform dye normalization processing on the generated sampling data set to obtain the preprocessed data.
  • tissue positioning module includes:
  • the background processing module is used to perform foreground and background segmentation on the thumbnail of the down-sampled image of the digital pathological image, and extract the effective image area in the thumbnail;
  • the positioning module is used to locate the effective tissue in the digital pathological image based on the result obtained by the background processing module;
  • the sampling module is used to traverse and sample the located area to generate a digital pathological sampling block that meets the needs of the algorithm.
  • the normalization module includes:
  • the conversion module is used to convert the RGB value of the input tissue slice image into the OD optical density value
  • the removal module is used to set the threshold ⁇ and remove the pixels whose optical density value is less than ⁇ ;
  • the first calculation module is used to calculate the SVD decomposition of the OD matrix
  • the projection module is used to project the data into the created plane and normalize it
  • the second calculation module is used to calculate the angle between each point and the first singular value, and find the extreme value of the angle
  • the output module is used to convert the extreme value into the OD domain and output the normalized image.
  • the construction of the sampling block discrimination model based on the weakly supervised learning algorithm in the first classification module specifically includes: optimizing the training process of the deep residual network through multi-instance learning in the weakly supervised learning algorithm to obtain the sampling block discrimination model.
  • the fusion processing in the second classification module is specifically: combining all the pathological image blocks according to the spatial relationship before sampling to obtain a probability map similar to the full digital pathological image, and introducing a full connection to the probability map.
  • the conditional random field is used to correct the classification probability map of the pathological image block, and the classification result of the all-digital pathological image is obtained.
  • a method for discriminating digital pathological images of colorectal cancer based on weakly supervised learning including:
  • step S2 includes:
  • the construction of the sampling block discrimination model based on the weakly supervised learning algorithm in the step S3 specifically includes: optimizing the training process of the deep residual network through multi-instance learning in the weakly supervised learning algorithm to obtain the sampling block discrimination model.
  • the fusion processing in the step S4 is specifically: combining the spatial relationships of all the pathological image blocks before sampling to obtain a probability map similar to the full digital pathological image, and introducing a fully connected conditional random field into the probability map. , the classification probability map of the pathological image block is modified, and the classification result of the all-digital pathological image is obtained.
  • the first stage uses a weakly supervised algorithm of multi-instance learning, and uses coarse-grained packet labeling to realize the sampling process.
  • the classification model on the instance obtains a classification performance similar to that of the doctor's labeling result; in the second stage, the classification results of the instances in the package are decided and fused, and the spatial relationship between the sampled instances is fully utilized.
  • the classification instance is subject to result correction. In the performance test and visualization results, it is proved that the algorithm has certain effectiveness.
  • Embodiment 1 is a structural diagram of a colorectal cancer digital pathological image discrimination system based on weakly supervised learning provided by Embodiment 1;
  • Fig. 2 is the whole pathological slice discrimination flow chart based on multi-instance learning provided by Embodiment 1;
  • FIG. 3 is a schematic diagram of the MICCAI2019 challenge data provided by Embodiment 1;
  • Embodiment 4 is a schematic diagram of effective tissue extraction from a digital pathological image provided by Embodiment 1;
  • Embodiment 5 is a schematic diagram of a patch generation method provided by Embodiment 1;
  • Example 6 is a schematic diagram of a pathological sample before and after staining normalization provided in Example 1;
  • FIG. 7 is a schematic structural diagram of a residual block provided in Embodiment 1;
  • Embodiment 8 is a schematic diagram of the Basic Block and Bottleneck Block provided by Embodiment 1;
  • Embodiment 9 is a schematic diagram of global average pooling provided by Embodiment 1.
  • FIG. 10 is a schematic diagram of a resnet network structure provided by Embodiment 1;
  • FIG. 11 is a schematic diagram of a training process flow of a digital pathological image block classification algorithm for multi-instance learning provided by Embodiment 1;
  • Embodiment 12 is a schematic diagram of a fully connected conditional random field probability correction provided by Embodiment 1;
  • FIG. 13 is a schematic diagram of a data tag provided by Embodiment 2.
  • Figure 15 is a microsatellite instability identification probability map provided by Embodiment 2;
  • FIG. 16 is a schematic diagram of classification ratios in different microsatellite typing samples provided in Embodiment 2;
  • FIG. 17 is a ROC curve diagram for identifying microsatellite instability provided in the second embodiment.
  • the present application provides a method and system for discriminating digital pathological images of colorectal cancer based on weakly supervised learning.
  • the colorectal cancer digital pathological image discrimination system based on weakly supervised learning provided in this embodiment, as shown in Figure 1-2, includes:
  • the acquisition module 11 is used for acquiring a digital pathological image data set of colorectal cancer
  • the preprocessing module 12 is used for preprocessing the data in the collected data set to obtain the preprocessed data
  • the first classification module 13 is used to construct a sampling block discrimination model based on a weakly supervised learning algorithm, and input the preprocessed data into the constructed sampling block discrimination model for processing to obtain the classification results of all pathological image blocks in the full slice sampling package ;
  • the second classification module 14 is used for constructing a decision fusion model, and inputting the obtained classification results of the pathological image blocks into the decision fusion model for fusion processing to obtain the classification results of all-digital pathological images.
  • the massive medical images in the open source database can be fully utilized to train an effective classification algorithm.
  • the trained model can also be used as a pre-trained model to apply to other pathological image problems through transfer learning.
  • Using the multi-instance learning method based on weakly supervised learning can effectively optimize the performance of the algorithm through a large number of coarse-grained labeled pathological image libraries, so as to obtain fine-grained labeling of sample data, and further improve the performance of the computer-aided discriminant model.
  • the goal of the first stage is to train a classifier of pathological image patches (Patch).
  • a WSI can generate thousands of pathological image blocks after the preprocessing process, and the patches generated by the same WSI form a Whole Slide Bag (WSB).
  • This data structure constitutes a multi-instance learning problem.
  • the WSI marked as normal by experts forms a WSB after preprocessing, and all patches in the package are normal samples; however, in the WSI containing lesions, there is no guarantee that every preprocessing A patch contains diseased tissue.
  • the purpose of multi-instance learning is to learn the discriminative model on the patch through only the data marked by WSI.
  • the goal of the second stage is to obtain the discriminative result of WSI through the fusion decision model.
  • a WSB contains multiple patches, which effectively integrates the discriminant results of the first-stage model on the WSB, and further obtains the discriminant results of WSI by integrating the discriminant results on the optimized patch.
  • a digital pathological image data set of colorectal cancer is acquired.
  • the data used in this example comes from the MICCAI2019 pathological challenge and TCGA database, which are used for the research of different problems. in:
  • MICCAI2019 data are digital pathology slides of colon cancer from 4 medical centers. A total of 410 negative data and 250 positive data were included, all of which were digital pathology images at 20x magnification. Among them, 250 positive data are labeled by the pathologist at the pixel level of the malignant lesion tissue in the image. Each data corresponds to a mask, which is a binary image. All the pixel values of the lesion area selected by the pathologist is 255, and the rest of the pixel values are zero, as shown in Figure 3. None of the 410 negative samples contained any malignant lesions. In this embodiment, the annotations of all pathologists are not used for the training of the algorithm, but are only used for evaluating the performance of the algorithm. Figure 3 is an example of the original image of positive data and sample annotation.
  • each of the above sets of data is randomly divided into 70% training set for algorithm training, 15% validation set for model screening, and 15% test set for performance testing.
  • data in the collected data set is preprocessed to obtain preprocessed data.
  • tissue localization module includes a tissue localization module and a normalization module.
  • the tissue localization module is used to locate the effective tissue in the input digital pathological image and generate a sampling data set; it specifically includes:
  • the background processing module is used to perform foreground and background segmentation on the thumbnail of the down-sampled image of the digital pathological image, and extract the effective image area in the thumbnail;
  • the positioning module is used to locate the effective tissue in the digital pathological image based on the result obtained by the background processing module;
  • the sampling module is used to traverse and sample the located area to generate a digital pathological sampling block that meets the needs of the algorithm.
  • the digital pathological image has hundreds of millions of pixels under the target magnification, it is necessary to traverse the pixels in the image in the process of image preprocessing, and the slice contains a large number of blank backgrounds, accounting for the entire pathological image. It ranges from 30% to 60%, and the process of traversing the invalid area takes up a lot of computing resources. Therefore, through effective target tissue positioning, some invalid samples and blank backgrounds can be removed, which greatly improves the efficiency of preprocessing.
  • openslide is a digital pathology image processing library integrated in python. Through openslide, pathological images under different microscopic magnifications can be accessed, and the target area under the specified magnification can be read by means of coordinates (x, y, w, h).
  • the effective image area in the thumbnail is extracted by performing foreground and background segmentation on the downsampled image.
  • the OTSU algorithm is an adaptive threshold algorithm based on inter-class variance. It assumes that an image consists of foreground and background, and selects a threshold through the maximum inter-class variance to distinguish the foreground and background as much as possible. Set a certain threshold T, then N 0 and N 1 are the front and rear background pixels, respectively, and the total number of S pixels, then:
  • the grayscale accumulation value is:
  • the between-class variance is:
  • the threshold that maximizes the inter-class variance g is the optimal threshold T.
  • the threshold T calculated by the OTSU algorithm can effectively divide the thumbnail into foreground and background, and generate a mask for the target area.
  • the tissue area can be quickly located, and at the same time, by calculating the area of the connected domain, the small damaged and stained areas in the slice can be removed.
  • the mask should be inflated to ensure that the small areas in the tissue section are not lost.
  • Figure 4 shows the effective tissue extraction process of digital pathological images.
  • the WSI needs to be sampled by the sliding window method to obtain a patch that satisfies the processing conditions of the subsequent algorithm.
  • the sliding window samples the WSI, it is possible to select whether to overlap according to the amount of data, that is, whether there is overlap of border pixels between adjacent patches.
  • the left side is the non-overlapping sampling method
  • the right side is the 50% pixel overlapping sampling method
  • the overlapping sampling method can generate more sampling data in one WSI.
  • the obtained sampled images it is also necessary to further effectively distinguish the foreground and background images.
  • Some of the sampled images contain a small proportion of cells and tissues and contain a large amount of white background, which cannot be used for model training. Only those containing valid foreground content should be selected, that is, the tissue present in the patch should be larger than a certain threshold.
  • the patch is selected by calculating the variance ⁇ between the pixels of the three channels of R, G, and B in the patch.
  • the parameter 18 obtained through repeated experimental experience can effectively distinguish the front and back backgrounds.
  • the patch is a background image that lacks effective organization and is not included in the training data set.
  • the target data can be processed more efficiently, and the full-section pathological image is sampled into a full-section sampling package composed of multiple patches.
  • the normalization module is used to perform dye normalization processing on the generated sampling data set to obtain the preprocessed data.
  • the tissue sections were stained by HE. Hematoxylin was used to stain the nucleus with blue-purple, and eosin was used to color the extracellular matrix red.
  • the staining process was affected by many factors and could not be consistent. Pathological images of different tissues and staining batches may have large color differences, which affect the performance of the algorithm to a certain extent. Therefore, it is necessary to normalize the staining of images for different slices.
  • the normalization process was performed on the pathological patches obtained after segmentation.
  • the staining normalization includes: input: image P 1 to be normalized, normalized template image P 2 ; output: normalized image P N .
  • the conversion module is used to convert the RGB value of the input tissue slice image into the OD optical density value
  • the removal module is used to set the threshold ⁇ and remove the pixels whose optical density value is less than ⁇ ;
  • the first calculation module is used to calculate the SVD decomposition of the OD matrix
  • the projection module is used to project the data into the created plane and normalize it
  • the second calculation module is used to calculate the angle between each point and the first singular value, and find the extreme value of the angle
  • the output module is used to convert the extreme value into the OD domain and output the normalized image.
  • a standard image is used as a staining template, and the remaining data are processed with the best staining vector, so that the color difference between samples is improved.
  • Fig. 6(a) is the sampling data of stained sections with large differences, and the result after normalization of staining is shown in Fig. 6(b).
  • a sampling block discrimination model is constructed based on the weakly supervised learning algorithm, and the preprocessed data is input into the constructed sampling block discrimination model for processing, and the classification results of all pathological image blocks in the full slice sampling package are obtained .
  • the construction of the sample block discrimination model based on the weakly supervised learning algorithm is as follows: the training process of the deep residual network is optimized through the multi-instance learning in the weakly supervised learning algorithm, and the sample block discrimination model is obtained.
  • MIL Multiple instance learning
  • Instance the labeling of the training set is usually in bags, and a bag contains several instances without specific labels ( Instance).
  • a bag is marked as positive if it contains at least one positive example, and is marked as negative if all instances in a bag are negative.
  • Algorithms hope to learn instance-based classifiers from packet-based data.
  • X ⁇ X 1 , X 2 , . . . X N ⁇ is a training set containing N packets.
  • X and the hidden variable H are produced by the following generative model:
  • the hidden variable H ⁇ H 1 ,H 2 ,...,H N ⁇ , where H i,j is a decision variable indicating whether x i,j is the label y i of the bag X i . Further assuming that all x i,j in each bag are independent, then the above formula can be further expressed as:
  • D is the data set that has a decision-making effect on the classification of the package. Assuming that all non-decision-making data obey a uniform distribution, the above formula can be simplified to:
  • the above formula can be described as a discriminant model.
  • the model is a convolutional neural network.
  • the multi-instance learning algorithm optimizes the algorithm training process based on the existing discriminant model.
  • a convolutional neural network is used as the discriminative model to be optimized.
  • Convolutional neural networks have been mainly used for image classification algorithms since their inception. After years of continuous optimization by researchers, great breakthroughs have been made in their network structure and performance. In the process of development, researchers try to improve the performance of the algorithm by continuously deepening the depth of the neural network to obtain more abstract feature extraction, and to obtain better feature learning with more parameters.
  • the common "convolution-activation-pooling" neural network structure has the problem of performance degradation as the depth deepens.
  • the degradation of the neural network may be due to the fact that the "convolution-activation-pooling" structure cannot effectively represent the identity transformation.
  • the shallow network model has obtained the optimal feature representation, more hidden layers cause the network to Performance deteriorates.
  • the residual block model in Resnet is the key to solving this problem, and its structure is shown in Figure 7 as a schematic diagram of the residual block structure.
  • the residual unit of Resnet consists of two "convolution-activation" layers and shortcut connections.
  • the input is the output of the previous residual unit, and the output is the feature map after the convolution operation and the feature map after the direct shortcut connection without the convolution operation. If the input of the residual unit is x and the output is y, the relationship between the input and output can be expressed as:
  • the mapping relationship is the residual between the input and the output.
  • all the network parameters can be set to zero to represent the identity transformation.
  • the residual module can be divided into Basic Block and Bottleneck Block, as shown in Figure 8.
  • bottlesneck Block In Basic Block, all convolution kernels are 3 ⁇ 3 convolution kernels, and the feature dimension is always kept unchanged in the residual block.
  • Bottleneck Block the number of channels is compressed and expanded in the residual block by introducing a 1 ⁇ 1 convolution in the first and last layers.
  • both Basic Block and Bottleneck Block implement feature downsampling through convolution with a stride of 2.
  • Bottleneck Block by introducing 1 ⁇ 1 convolution and introducing fewer parameters to increase the number of feature mappings in the residual block, it is often used in resnet structures with many layers.
  • GAP global average pooling
  • the multi-dimensional features are usually rearranged to form a one-dimensional column vector to further complete the target classification task.
  • the size of the feature map is a three-dimensional tensor with dimensions of 7 ⁇ 7 ⁇ 2048. If such redundant features are directly expanded one-dimensionally, the fully connected layer needs a corresponding number of nodes, and each node needs a corresponding number of nodes. Having the same number of parameters will largely lead to problems such as overfitting of the network and oversized models.
  • the global average pooling method calculates the average value of all the feature values in each channel of the output feature layer. Taking the 5*5 feature as an example, the global average pooling method reduces the original feature amount by 25 times. In resnet, global average pooling obtains a 1 ⁇ 2048-dimensional one-dimensional vector by weighting and averaging each 7 ⁇ 7 feature of the convolutional output layer, which effectively reduces network parameters and improves operational efficiency.
  • Regularly combining and stacking residual blocks can form resnet network models of different depths.
  • the Basic Block residual block structure is used in the shallow resnet of 18 and 34 layers. Resnet with more than 50 layers introduces the residual block structure of Bottleneck Block.
  • the output of the last layer of the shallow resnet convolutional layer is a 512-dimensional vector, while the output of the last layer of the deep resnet is a 2048-dimensional vector.
  • the resnet network structure with different depths is shown in Figure 10.
  • the multi-instance model training process is specifically:
  • resnet Since resnet has strong feature representation ability and has achieved good performance in multiple machine vision tasks, resnet is used as the discriminative model in the multi-instance learning process.
  • the multi-instance learning algorithm uses the iterative EM process to make the sample data used for training each time the model has the maximum expectation.
  • the iterative process of the algorithm is as follows:
  • the image is preprocessed by a preprocessing scheme in a sliding window mode, and the data is saved in a package-instance structure.
  • the process of sample traversal is a necessary step before each round of neural network training.
  • the model saved in the last training result all samples in the training dataset are traversed, and the classification results and classification probabilities of all patches are saved in WSB units.
  • the batch size of network input is 512 images.
  • the basis of packet-instance-based data screening is the probability value calculated by the neural network on the instance. During each iteration, the instances in the bag are sorted by their probability on the bag class.
  • the principle of filtering data is to select data with relatively high probability values inferred by the model. There are also different filtering methods according to probability. In this experiment, the samples were screened in the following ways, as shown in Table 1.
  • the data sets obtained through the data filtering step are all data pairs containing labels.
  • the continuous iterative optimization of the neural network relies on the loss between the classification results of the computing network on the batch training data and the real labels of the samples, and the neural network model is continuously optimized through the back-propagation algorithm and optimizer.
  • the essence of the cross-entropy function is the application of the cross-entropy in the information theory to the classification problem, which can better characterize the performance of the model in the classification problem, the gap between the model prediction result and the real label and the calculated value of the cross-entropy. proportional relationship between them.
  • the cross-entropy function can also be derived from the maximum likelihood function under the condition of Bernoulli distribution, and the process of minimizing the cross-entropy function is the maximization of the log-likelihood function. Compared with the mean square loss function, the cross-entropy function has the characteristics that when the error is large, the weight update is faster; when the error is small, the weight update range is small.
  • the cross-entropy loss function for a single sample, where y is the true label, For the model prediction result, its loss L is calculated as follows:
  • Learning rate decay is also used during neural network training to optimize the network training process.
  • the initial learning rate of the network is 0.01, and the learning rate is decayed by a factor of 0.1 after every 10 epochs.
  • the entire training process uses two GPUs to train in parallel.
  • a decision fusion model is constructed, and the obtained classification results of the pathological image blocks are input into the decision fusion model for fusion processing to obtain the classification results of all-digital pathological images.
  • the purpose of the computer-aided diagnosis model of digital pathology is to generate the diagnostic results of WSI.
  • the patch classification model is implemented. After the classification results of all patches in WSB are obtained, the results must be effectively fused to further obtain the final classification results of WSI.
  • Fusion processing methods include Max-pooling, Vote, and Logistic Regression.
  • Max-pooling is an ideal decision fusion method, assuming that the basic classifier of the patch has better performance. Positive samples in the whole bag are marked as 1 according to the patch base classifier, and all negative samples are marked as 0. When any patch in WSB is marked as positive, then WSB is marked as positive and WSI is positive. If all patches in WSB are negative, WSB is marked as negative, WSI is negative, and the label of a single sampling block is set to L.
  • Vote is a fusion method based on global average. By counting all patch classification results in WSB, the classification result of WSI is finally determined according to the proportion of positive samples and negative samples in WSB.
  • Logistic regression model linear regression is often used in regression analysis, not in classification problems, the combination of linear regression and logistic distribution model is logistic regression, logistic regression is a common two-class model, is a discriminant model, directly Model conditional probabilities.
  • ⁇ and ⁇ are the parameters of the logistic distribution.
  • h ⁇ (x) is the probability that the classification result is 1.
  • ⁇ is the weight vector containing the bias term.
  • the model training process adopts the stochastic gradient descent method, and iteratively optimizes with the learning rate a by traversing the training samples:
  • the logistic regression model first obtains the sample proportion result obtained by Vote through calculation, and on the basis of the proportion, the logistic regression model is used to discriminate the final classification of WSB.
  • a decision fusion model is used to combine the spatial relationships of all pathological image blocks before sampling to obtain a probability map similar to the full digital pathological image.
  • the image is corrected to obtain the classification result of the all-digital pathological image.
  • All the patches in the WSB obtained by the sliding window preprocessing have a certain spatial relationship, that is, the probability of belonging to the same class between adjacent patches in spatial position is high, and the classification is purely based on quantity. way ignores important spatial information.
  • the coordinate positions of all patches are saved, and in the process of decision model fusion, all patches are recombined according to the original arrangement, and a probability map similar to WSI can be further obtained.
  • DenseCRF fully connected conditional random fields
  • I) is the energy function, which is the objective of optimization
  • I) is the normalization factor.
  • I) consists of a one-variable potential function and a two-variable potential function, as shown in the following formula:
  • the one-variable potential function is used to measure the class probability of the point, which is the output result of the neural network.
  • a binary potential function is used to describe the relationship between points, encouraging similar points to be assigned the same label, while points that differ greatly are assigned different labels.
  • the formula of the binary potential function is as follows:
  • u(x i , x j ) is called the label compatibility term, which constrains the conditions of conduction between points. Only under the same label conditions, energy can be transmitted to each other.
  • Figure 12 is a schematic diagram of the probability correction of the fully connected conditional random field.
  • this embodiment has the following beneficial effects:
  • the first stage uses a weakly supervised algorithm of multi-instance learning, and uses coarse-grained packet labeling to realize the sampling process.
  • the classification model on the instance obtains a classification performance similar to that of the doctor's labeling result; in the second stage, the classification results of the instances in the package are decided and fused, and the spatial relationship between the sampled instances is fully utilized.
  • the classification instance is subject to result correction. In the performance test and visualization results, it is proved that the algorithm has certain effectiveness.
  • This embodiment tests the performance of the algorithm on the problems of colorectal cancer auxiliary diagnosis and microsatellite instability identification respectively.
  • the colorectal cancer pathological segmentation challenge data from MICCAI.
  • the data contains 410 positive samples and 250 negative samples, and all positive samples are marked by pathologists to the lesion area.
  • the purpose is to explore the multi-instance learning algorithm based on coarse-grained annotation, so all expert annotation masks are not used for model training, but only for model performance evaluation.
  • Resnet18 and resnet50 are used for strong supervision training, and resnet18 and resnet50 are used in combination with different data screening methods for multi-instance learning.
  • the performance is shown in the following table 3:
  • the training method based on multi-instance learning effectively improves the performance of the model on the block classification task of pathological image classification.
  • Resnet18 is better than resnet50 in overall performance, and the lack of sample size leads to a certain overfitting problem.
  • the performance results of method 3 are better than other centralized methods, and the process of randomly selecting negative slices can be regarded as data amplification and penalty to the network model.
  • the pathological image blocks in the full pathological image package are rearranged according to the position information during preprocessing, and a probability-based heat map can be obtained, as shown in Figure 14.
  • the neural network based on multi-instance learning can implement a fine-grained pathological image block classification model through an iterative multi-instance learning strategy on the coarse-grained labeled dataset. .
  • the weakly-supervised algorithm based on multi-instance learning has obvious advantages in the problem of lesion slice identification. It can well realize the effective judgment of different sampling blocks in the slice. Although all the samples in Fig. 14 can be judged as positive samples after subsequent processing, the algorithm used in this embodiment is obviously more consistent with the doctor's labeling results, so the algorithm results are more interpretable and dependable sex.
  • the recognition problem of full pathological images is a two-stage problem.
  • the above results only represent the classification performance on the sampling block. Due to the inevitable misjudgment of the model, it is hoped that through the integration of the above results, end-to-end WSI discrimination can be achieved.
  • the results need to be integrated to obtain the classification results of the entire pathological slice.
  • Table 4 The following are the performance test results of different decision fusion models on the full pathological image recognition problem, as shown in Table 4:
  • the recognition decision fusion model based on digital pathological blocks has better performance in the discrimination problem of whole pathological slices, in which DenseCRF is used to modify the results of the previous model on the generated heat map results to a certain extent.
  • the best performance is obtained by fusing the discriminative results through a logistic regression model.
  • using DenseCRF on the original probability heatmap has a certain improvement in the algorithm results.
  • MSI colorectal cancer micro-satellite instability
  • Microsatellite stability is an important molecular marker for the prognosis of colorectal cancer and other solid tumors. Microsatellite instability is due to a defect in the human body's mismatch repair mechanism. When the human DNA has microsatellite error repeats, it cannot be repaired normally, which may lead to cancerous cells. Patients with high microsatellite instability have significant benefits from PD-1 immunotherapy, and judging whether a patient has microsatellite highly instability (MSI-H) is an important part of clinical diagnosis. However, the current clinical diagnosis of microsatellite instability relies on immunohistochemical staining and genetic testing, which is time-consuming and expensive.
  • microsatellite stability It is of great significance to realize the diagnosis of microsatellite instability based on pathological images by mining potential features in pathological images through algorithms. Based on the above algorithm, the target data are divided into four types: normal, highly unstable microsatellite, low microsatellite instability (MSI-L), and microsatellite stability (micro-satellite stability).
  • the algorithm results are visualized by means of probabilistic heat maps, as shown in Figure 15, which are the results of the algorithm on the test set. It can be seen that the algorithm based on multi-instance learning has certain effectiveness in the problem of microsatellite instability.
  • the sections and tissues in which lesions did not develop were well identified. Slices judged to be highly unstable microsatellites are widely present in highly unstable data, and the two types of microsatellite stable and low microsatellite instability do not have obvious separation results, which is consistent with the results of clinical diagnosis.
  • a certain fit that is, only patients with highly unstable microsatellites are clinically significant, while the other two categories are often regarded as the same type.
  • Figure 16 shows the classification ratios in different microsatellite typing samples. It can be seen from the results that the neural network model based on weakly supervised learning can effectively identify normal slices in slices. In addition, in the slices with high microsatellite instability, the regions judged to be microsatellite unstable were significantly higher than the other categories. There is no significant difference between the two types of microsatellite low instability and microsatellite stable. This result is consistent with the clinical diagnosis of microsatellite instability. Only patients with high microsatellite instability are clinically significant to some treatment regimens and drugs.
  • test set samples are divided into three types: highly unstable microsatellites, stable microsatellites, and low microsatellites unstable and normal.
  • the AUC of the algorithm in the discrimination results of MSI-H, MSI-L+MSS, and Normal reached 0.83, 0.84, and 1, respectively. It indicates that the algorithm has certain validity in the identification of microsatellite instability in digital pathology.
  • the performance of the algorithm is tested on the identification of cancerous regions in digital pathological slices and the identification of colorectal cancer microsatellite instability.
  • the test results show that the neural network model based on multi-instance learning is effective in the identification of digital pathological images. sex.
  • This embodiment provides a method for discriminating digital pathological images of colorectal cancer based on weakly supervised learning, including:
  • step S2 includes:
  • the construction of the sampling block discrimination model based on the weakly supervised learning algorithm in the step S3 specifically includes: optimizing the training process of the deep residual network through multi-instance learning in the weakly supervised learning algorithm to obtain the sampling block discrimination model.
  • the fusion processing in the step S4 is specifically: combining all the pathological image blocks according to a preset arrangement through the decision fusion model, to obtain a probability map similar to the full digital pathological image, and introducing the probability map into the probability map.
  • the fully connected conditional random field is used to correct the classification probability map of the pathological image block to obtain the classification result of the fully digital pathological image.
  • the first stage uses a weakly supervised algorithm of multi-instance learning, and uses coarse-grained packet labeling to realize the sampling process.
  • the classification model on the instance obtains a classification performance similar to that of the doctor's labeling result; in the second stage, the classification results of the instances in the package are decided and fused, and the spatial relationship between the sampled instances is fully utilized.
  • the classification instance is subject to result correction. In the performance test and visualization results, it is proved that the algorithm has certain effectiveness.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Medical Informatics (AREA)
  • Public Health (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Epidemiology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Primary Health Care (AREA)
  • Databases & Information Systems (AREA)
  • Pathology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Investigating Or Analysing Biological Materials (AREA)

Abstract

本申请公开了基于弱监督学习的结直肠癌数字病理图像判别方法及系统，其中涉及的基于弱监督学习的结直肠癌数字病理图像判别系统，包括：采集模块，用于采集结直肠癌数字病理图像数据集；预处理模块，用于对采集的数据集中的数据进行预处理，得到预处理后的数据；第一分类模块，用于基于弱监督学习算法构建采样块判别模型，将预处理后的数据输入至构建的采样块判别模型中进行处理，得到全切片采样包中所有病理图像块的分类结果；第二分类模块，用于构建决策融合模型，将得到的病理图像块的分类结果输入至决策融合模型中进行融合处理，得到全数字病理图像的分类结果。

Description

基于弱监督学习的结直肠癌数字病理图像判别方法及系统 技术领域
本申请涉及病理图像技术领域，尤其涉及基于弱监督学习的结直肠癌数字病理图像判别方法及系统。
背景技术
肿瘤始终是危害人类健康的重大疾病之一，在全球的发病率和致死率都高居前列，在发展中国家仍不断呈现出逐年上升的趋势。结直肠癌(Colorectal Cancer，CRC)是最常见的消化道肿瘤之一，2018中国癌症统计报告显示：我国结直肠癌发病率、死亡率在全部恶性肿瘤中分别位居第三位和第五位，其中城市远高于农村，且结直肠癌发病率上升显著。有研究表明，其发病率与地区经济发达水平有关，受生活方式、饮食习惯影响较大。
肿瘤通常由影像学诊断发现可疑病灶，再进一步通过穿刺活检或手术切取，从患者体内取出3～4μm的组织样本，由病理科室通过较为复杂的染色、制片流程，制成可在显微镜下观察的单层细胞切片样本。经验丰富的病理医生在高倍光学显微镜可以通过观察细胞的形态结构，对可疑病灶进行诊断。若为病变组织，则进一步明确其分化程度、所属的病理分型等肿瘤诊断中的必要信息，这些信息会极大程度上影响患者的治疗方案和预后，所以病理检查是肿瘤诊断中的核心关键步骤，其诊断结果被称为肿瘤诊断的金标准。
在肿瘤病理图像的诊断问题上，具有一定的复杂性。在理想状态下认为肿瘤组织中所有的细胞均为异常细胞，而在实际的情况中，肿瘤组织由复杂的微环境(Tumor Micro Envirment,TME)构成，病理活检采集到的组织通常包含多种细胞。
病理图像是细胞形态的直观体现，所以在病理图像中往往包含了较为复杂的图像信息。数字病理图像与常见的医学影像不同，通常以图像金字塔的方式将不同显微放大倍数下的图像保存在文件中。其中20x放大倍数下的图像能较为清晰的观察到细胞的组织结构，能有效的完成诊断，是较为理想的放大倍数。但理想放大倍数下的图像通常具有数亿个像素点，在现有的计算机性能下，常 见的图像处理算法并不适用于如此大规模的图像，故需对数字病理图像进行有效的预处理，其特殊的预处理方法也使得基于数字病理图像的诊断问题与常规问题不同。
常规的深度学习算法，需要对训练数据进行一对一的标注，以获得好的性能，数据集的品质会很大程度上影响算法结果。由于数字病理图像格式的复杂性，使得在目标放大倍数下的数据标记任务变得困难，病理医生在数据标记问题上耗时且费力，同时在如此大规模的数据中也可能存在漏标、错标等问题。对数据进行精细标注的方式显然在医学图像领域存在一定的局限性。此外，即便是通过像素级标注的数据，在不同数据之间也存在泛化能力较差的问题。
发明内容
本申请的目的是针对现有技术的缺陷，提供了基于弱监督学习的结直肠癌数字病理图像判别方法及系统。
为了实现以上目的，本申请采用以下技术方案：
基于弱监督学习的结直肠癌数字病理图像判别系统，包括：
采集模块，用于采集结直肠癌数字病理图像数据集；
预处理模块，用于对采集的数据集中的数据进行预处理，得到预处理后的数据；
第一分类模块，用于基于弱监督学习算法构建采样块判别模型，将预处理后的数据输入至构建的采样块判别模型中进行处理，得到全切片采样包中所有病理图像块的分类结果；
第二分类模块，用于构建决策融合模型，将得到的病理图像块的分类结果输入至决策融合模型中进行融合处理，得到全数字病理图像的分类结果。
进一步的，所述预处理模块包括：
组织定位模块，用于对输入的数字病理图像中的有效组织进行定位，并生成采样数据集；
归一化模块，用于对生成采样数据集进行染色归一化处理，得到预处理后的数据。
进一步的，所述组织定位模块包括：
背景处理模块，用于对数字病理图像的下采样图像的缩略图进行前后景分割，提取缩略图中的有效图像区域；
定位模块，用于基于背景处理模块得到的结果，对数字病理图像中的有效组织进行定位；
采样模块，用于对定位后的区域进行遍历采样，生成符合算法需要的数字病理采样块。
进一步的，所述归一化模块包括：
转换模块，用于将输入的组织切片图像的RGB值转化为OD光密度值；
移除模块，用于设置阈值β，将光密度值小于β的像素点移除；
第一计算模块，用于计算OD矩阵的SVD分解；
创建模块，用于在SVD分解后的两个最大奇异值向量创建一个平面；
投影模块，用于将数据投影到创建的平面中去，并进行归一化；
第二计算模块，用于计算每个点与第一奇异值之间的夹角，并找到夹角的极值；
输出模块，用于将极值转化为OD域，输出归一化后的图像。
进一步的，所述第一分类模块中基于弱监督学习算法构建采样块判别模型具体为：通过弱监督学习算法中的多实例学习对深度残差网络的训练过程进行优化，得到采样块判别模型。
进一步的，所述第二分类模块中进行融合处理具体为：将所有病理图像块按照采样前的空间关系进行组合，得到一张与全数字病理图像相似的概率图，通过对概率图引入全连接条件随机场，对病理图像块分类概率图进行修正，得到全数字病理图像的分类结果。
相应的，还提供基于弱监督学习的结直肠癌数字病理图像判别方法，包括：
S1.采集结直肠癌数字病理图像数据集；
S2.对采集的数据集中的数据进行预处理，得到预处理后的数据；
S3.基于弱监督学习算法构建采样块判别模型，将预处理后的数据输入至构建的采样块判别模型中进行处理，得到全切片采样包中所有病理图像块的分类结果；
S4.构建决策融合模型，将得到的病理图像块的分类结果输入至决策融合 模型中进行融合处理，得到全数字病理图像的分类结果。
进一步的，所述步骤S2包括：
S21.对输入的数字病理图像中的有效组织进行定位，并生成采样数据集；
S22.对生成的采样数据集进行染色归一化处理，得到预处理后的数据。
进一步的，所述步骤S3中基于弱监督学习算法构建采样块判别模型具体为：通过弱监督学习算法中的多实例学习对深度残差网络的训练过程进行优化，得到采样块判别模型。
进一步的，所述步骤S4中进行融合处理具体为：将所有病理图像块采样前的空间关系进行组合，得到一张与全数字病理图像相似的概率图，通过对概率图引入全连接条件随机场，对病理图像块分类概率图进行修正，得到全数字病理图像的分类结果。
与现有技术相比，本申请具有以下有益效果：
(1)通过有效组织定位、染色归一化等数据预处理方式，实现了对数字全病理切片的数据采样、清洗、规范化流程，形成“包-实例”的数据结构。其中包标签与全病理切片相同，而包中样本不具有确切标记。
(2)在上述数据结构的基础上，使用两阶段的研究方法实现了全病理切片的计算机辅助诊断：第一阶段通过多实例学习的弱监督算法，使用粗粒度的包标记，实现了在采样实例上的分类模型，获得了与医生标记结果相近的分类性能；第二阶段对包内实例的分类结果进行决策融合，充分利用采样实例间的空间关系，通过全连接条件随机场对其中的误分类实例进行结果修正。在性能测试和可视化结果上，证明了算法具有一定的有效性。
附图说明
图1是实施例一提供的基于弱监督学习的结直肠癌数字病理图像判别系统结构图；
图2是实施例一提供的基于多实例学习的全病理切片判别流程图；
图3是实施例一提供的MICCAI2019挑战数据示意图；
图4是实施例一提供的数字病理图像有效组织提取示意图；
图5是实施例一提供的patch生成方式示意图；
图6是实施例一提供的染色归一化前后的病理样本示意图；
图7是实施例一提供的残差块结构示意图；
图8是实施例一提供的Basic Block和Bottleneck Block示意图；
图9是实施例一提供的全局平均池化示意图；
图10是实施例一提供的resnet网络结构示意图；
图11是实施例一提供的多实例学习的数字病理图像块分类算法训练流程示意图；
图12是实施例一提供的全连接条件随机场概率修正示意图；
图13是实施例二提供的数据标记示意图；
图14是实施例二提供的算法结果热力图；
图15是实施例二提供的微卫星不稳定识别概率图；
图16是实施例二提供的不同微卫星分型样本中的分类比例示意图；
图17是实施例二提供的微卫星不稳定识别ROC曲线图。
具体实施方式
以下通过特定的具体实例说明本申请的实施方式，本领域技术人员可由本说明书所揭露的内容轻易地了解本申请的其他优点与功效。本申请还可以通过另外不同的具体实施方式加以实施或应用，本说明书中的各项细节也可以基于不同观点与应用，在没有背离本申请的精神下进行各种修饰或改变。需说明的是，在不冲突的情况下，以下实施例及实施例中的特征可以相互组合。
本申请针对现有缺陷，提供了基于弱监督学习的结直肠癌数字病理图像判别方法及系统。
实施例一
本实施例提供的基于弱监督学习的结直肠癌数字病理图像判别系统，如图1-2所示，包括：
采集模块11，用于采集结直肠癌数字病理图像数据集；
预处理模块12，用于对采集的数据集中的数据进行预处理，得到预处理后的数据；
第一分类模块13，用于基于弱监督学习算法构建采样块判别模型，并预 处理后的数据输入至构建的采样块判别模型中进行处理，得到全切片采样包中所有病理图像块的分类结果；
第二分类模块14，用于构建决策融合模型，并将得到的病理图像块的分类结果输入至决策融合模型中进行融合处理，得到全数字病理图像的分类结果。
本实施例通过引入弱监督学习算法，能充分利用开源数据库中的海量医学图像，训练有效的分类算法。训练后的模型也可以作为预训练模型通过迁移学习的方式应用于其他病理图像问题中去。使用以弱监督学习为基础的多实例学习方式，能通过大量粗粒度标注的病理影像库有效优化算法性能，从而获得样本数据的细粒度标记，进一步提升计算机辅助判别模型的性能表现。
由于其数据的特殊性，在全数字病理图像(Whole Slide Image，WSI)的计算机辅助判别问题上，构成了一项两阶段(Two-stage)任务。
(1)第一个阶段的目标是训练一个病理图像块(Patch)的分类器。一张WSI在经过预处理流程后可以生成数千张病理图像块，由同一张WSI生成的patch构成一个全切片采样包(Whole Slide Bag，WSB)，这种数据结构构成了一个多实例学习问题。在判断WSI是否发生病变的问题上，由专家标记为正常的WSI在预处理后形成一个WSB，包中的所有patch均为正常样本；而在包含病变的WSI中，无法保证预处理后的每一张patch都包含病变组织。多实例学习的目的是希望通过只有WSI标注的数据，学习到在patch上的判别模型。
(2)第二个阶段的目标通过融合决策模型得到WSI的判别结果。一个WSB中包含多个patch，有效整合第一阶段模型在WSB上的判别结果，通过整合优化patch上的判别结果，进一步得到WSI的判别结果。
在采集模块11中，采集结直肠癌数字病理图像数据集。
本实施例使用的数据来源于MICCAI2019病理挑战和TCGA数据库，用于不同问题的研究。其中：
MICCAI2019数据是来源于4个医疗中心的结肠癌的数字病理切片。共包含410张阴性数据和250张阳性数据，所有数据均为20x放大倍数下的数字病理图像。其中250个阳性数据由病理学家对图像中的恶性病变组织进行像素级的标注，每张数据对应一张掩膜，为一张二值图像，所有病理学家选取出的病变区域的像素值为255，其余像素值均为零，如图3所示。410张阴性样本中， 不包含任何恶性病变。在本实施例中，所有病理医生的标注并未用于算法的训练，只用于对算法性能的评估。图3为阳性数据原始图像及样本标注示例。
在本实施例的实验中，将上述每组数据都随机分为70％训练集用于算法训练，15％验证集用于模型筛选，15％测试集用于性能测试。分组时注意确保同一患者的数据被划分在同一数据集中。
在预处理模块12中，对采集的数据集中的数据进行预处理，得到预处理后的数据。
具体包括组织定位模块和归一化模块。
组织定位模块，用于对输入的数字病理图像中的有效组织进行定位，并生成采样数据集；具体包括：
背景处理模块，用于对数字病理图像的下采样图像的缩略图进行前后景分割，提取缩略图中的有效图像区域；
定位模块，用于基于背景处理模块得到的结果，对数字病理图像中的有效组织进行定位；
采样模块，用于对定位后的区域进行遍历采样，生成符合算法需要的数字病理采样块。
由于数字病理图像在目标放大倍数下具有数亿个像素点，在对图像进行预处理的过程中需要对图像中的像素点进行遍历，而切片中包含了大量空白背景，占整张病理图像的30％～60％不等，遍历无效区域的过程占用了大量的计算资源。所以通过有效的目标组织定位，能去除部分无效样本和空白背景，在很大程度上提高预处理运行效率。
以特殊的图像金字塔格式保存的数字病理图像，以svs格式较为常见，无法使用常见的图像读取和处理方法。openslide是python中集成的数字病理图像处理库，通过openslide可以访问不同显微放大倍数下的病理图像，并通过坐标(x,y,w,h)的方式读取指定放大倍数下的目标区域。
一张数字病理图像中除大量的空白区域外，还存在一部分碎片和污损，为了提高图像预处理效率，需要首先对图像中的有效组织进行定位。由于目标放大倍数下的图像往往为超分辨率图像，无法直接对原始图像进行处理。所以背景滤除的预处理步骤在原始图像16倍下采样之后的缩略图上进行，之后再通 过坐标映射回原图，实现有效的区域提取。
首先通过在下采样图像上进行前后景分割，提取缩略图中的有效图像区域。OTSU算法为基于类间方差的自适应阈值算法。其假设一张图片由前景和背景组成，通过最大类间方差选取一个阈值，将前景和背景尽可能区分开。设定某个阈值T，则N 0和N 1分别为前后景像素，S像素总数，则：
Figure PCTCN2022088794-appb-000001
Figure PCTCN2022088794-appb-000002
ω 0+ω 1＝1
进一步计算前后景像素的平均值μ 0和μ 1，则灰度累计值为：
μ＝ω 0μ 0+ω 1μ 1
类间方差为：
g＝ω 0(μ 0-μ) 2+ω 1(μ 1-μ) 2
使得类间方差g最大的阈值即为最佳阈值T。
通过OTSU算法计算所得阈值T，可以有效的将缩略图分为前后景，并生成目标区域的掩膜(mask)。通过查找mask中的连通域，能快速进行组织区域定位，同时通过计算连通域的面积，能去除切片中较小的破损、污损区域。在计算连通域面积之前要先对mask进行膨胀处理，确保组织切片中的细小区域不会丢失。最终通过坐标关系映射，将缩略图中的坐标映射回指定放大倍数下的原图中去。如图4所示为数字病理图像有效组织提取流程。
得到有效区域的坐标之后，需要通过滑动窗口的方法对WSI进行采样，得到满足后续算法处理条件的patch。滑动窗口在对WSI进行采样时，可以根据数据量选择是否进行重叠(overlap)，即相邻patch之间是否存在边界像素的重复。如图5所示，左边为无重叠采样方式，右边为50％像素重叠的采样方式，有重叠的采样方式在一张WSI中能生成更多的采样数据。
对于得到的采样图像，进一步有效区分前后景图像也是非常必要的，部分采样后的图像中的细胞组织占比较少，包含了大量白色背景，无法用于模型训练。应当只选取包含有效前景内容的，即该patch中存在的组织应大于某个阈 值。通过计算patch中R、G、B三个通道像素间的方差σ来对patch进行选取，通过反复实验经验所得参数18能较好的将前后背景有效区分，当：
(σ(Red)+σ(Green)+σ(Blue))<18
则认为该张patch为缺乏有效组织的背景图像，不被纳入训练数据集中。
通过上述流程，能较为高效的处理目标数据，将全切片病理图像采样由多张patch组成的全切片采样包。
归一化模块，用于对生成采样数据集进行染色归一化处理，得到预处理后的数据。
组织切片是通过HE染色法，使用苏木精染液将细胞核内物质着蓝紫色，使用伊红将细胞外基质着红色，其染色过程受多种因素影响，无法保持一致性。不同组织机构、染色批次的病理图像，可能存在较大的色彩差异，在一定程度上影响算法性能，所以需要对不同的切片进行图像的染色归一化。归一化过程在获得切分后的病理patch上进行。染色归一化包括：输入：待归一化的图像P 1，归一化模板图像P 2；输出：归一化后图像P N。
具体包括：
转换模块，用于将输入的组织切片图像的RGB值转化为OD光密度值；
移除模块，用于设置阈值β，将光密度值小于β的像素点移除；
第一计算模块，用于计算OD矩阵的SVD分解；
创建模块，用于在SVD分解后的两个最大奇异值向量创建一个平面；
投影模块，用于将数据投影到创建的平面中去，并进行归一化；
第二计算模块，用于计算每个点与第一奇异值之间的夹角，并找到夹角的极值；
输出模块，用于将极值转化为OD域，输出归一化后的图像。
本实施例以标准图像作为染色模板，对其余数据均采用最佳染色向量处理过后，使得样本间的色彩差异得到改善。如图6所示，图6(a)为具有较大差异的染色切片采样数据，经过染色归一化后的结果如图6(b)所示。
在第一分类模块13中，基于弱监督学习算法构建采样块判别模型，并预处理后的数据输入至构建的采样块判别模型中进行处理，得到全切片采样包中所有病理图像块的分类结果。
基于弱监督学习算法构建采样块判别模型具体为：通过弱监督学习算法中的多实例学习对深度残差网络的训练过程进行优化，得到采样块判别模型。
多实例学习(multipleinstancelearning，MIL)是弱监督学习算法中的一种，在此类算法中，训练集的标注通常以包(bag)为单位，一个包中又包含若干个没有具体标记的实例(Instance)。若一个包中至少包含一个正例，则这个包被标记为正(positive)，若一个包中的所有实例都是反例，则该包被标记为反例(negative)。算法希望通过在以包为单位的数据中学习到以实例为单位的分类器。
假设X＝{X 1,X 2,…X N}是包含了N个包的训练集。其中每一个包
Figure PCTCN2022088794-appb-000003
中包含N i个实例，其中X i,j＝<x i,j,y i>表示第i个包中的第j个实例，其标签为第i个包的标签。假设所有包是独立同分布的(i.i.d)，X和隐藏变量H是由以下生成模型产生的：
Figure PCTCN2022088794-appb-000004
其中，隐藏变量H＝{H 1,H 2,…,H N}，
Figure PCTCN2022088794-appb-000005
其中H i,j是表示x i,j是否是包X i的标签y i的决定变量。进一步假设每个包中的所有x i,j是独立的，那么上式可以进一步表示为：
Figure PCTCN2022088794-appb-000006
通过迭代的最大期望(Expectation-Maximization algorithm,EM)算法优化数据的最大似然P(X)：
1)在初始化E阶段，假定所有的数据都是对包具有判别性的。
2)在M阶段，优化参数θ去最大化数据似然
Figure PCTCN2022088794-appb-000007
其中D是对包的分类有决策作用的数据集合。假设所有非决策作用的数据服从均匀分布，则上式可简化为：
Figure PCTCN2022088794-appb-000008
由此上式可描述为一个判别模型，在本实施例的实验中，该模型为卷积神经网络。
3)重复E阶段：评估所有包的隐藏变量H。通常，当且仅当P(H i,j|X)高于某个阈值时令H i,j＝1。在实验中，通过卷积神经网络在patch上的概率值作为判别标准，结合选取定量数据和高于阈值的部分数据。之后重复上述M阶段，直至算法收敛。
多实例学习算法是在已有判别模型的基础上，对算法训练流程进行优化。在本实验中，使用卷积神经网络作为待优化的判别模型。卷积神经网络在诞生之初就被主要用于图像的分类算法，经过多年研究者的不断优化，其网络结构和性能都有了极大的突破。在发展过程中，研究者们试图通过将神经网络的深度不断加深以获得更抽象的特征提取，以更多的参数量获得更好的特征学习，从而提升算法性能。而实验表明，常见的“卷积-激活-池化”神经网络结构随着深度加深，却出现了性能退化的问题。何凯明提出的Resnet深度残差网络结构通过引入残差块结构有效的解决了上述问题。本实施例使用Resnet18为判别模型，获得了较好的性能表现。
(1)残差块模型
神经网络的退化，可能是由于“卷积-激活-池化”的结构无法有效的表征恒等变换，当浅层的网络模型已经获得了最优的特征表征时，更多的隐层致使网络性能变差。Resnet中的残差块模型是解决该问题的关键，其结构如图7所示为残差块结构示意图。
如图7所示，Resnet的残差单元由两个“卷积-激活”层以及shortcut连接组成。输入为上一残差单元的输出，输出为卷积操作之后的特征图与不经过卷积操作直接shortcout连接之后的特征图相加。若残差单元的输入为x，输出为y，则输入输出的关系可以表示为：
y＝F(x)+x
则网络需要学习的映射：
F(x)＝y-x
映射关系为输入和输出之间的残差，当输入输出为最优映射时，网络参数全部置零即能表征恒等变换。
对残差模块中的添加不同的卷积层结构，可将残差模块分为Basic Block和Bottleneck Block，如图8所示。
在Basic Block中，所有的卷积核均为3×3大小的卷积核，且在残差块中始终保持特征维度的不变。而在Bottleneck Block中，通过在首层和末层引入1×1的卷积，在残差块中对通道数进行压缩和扩增。在需要进行下采样的结构中，Basic Block和Bottleneck Block均通过步长为2的卷积实现特征的下采样。在Bottleneck Block中通过引入1×1的卷积，引入较少的参数来增加残差块中的特征映射次数，常被用在层数较多的resnet结构中。
(2)全局平均池化
神经网络再不断加深的过程中，需要不断降低特征的宽度，增加特征的维度。常规的神经网络模型中常采用最大池化作为下采样的方法。而在resnet网络模型中，均未采用最大池化。在卷积层之间，采用步长为2的卷积来实现特征图的下采样。在卷积网络输出层和全连接层之间，通过全局平均池化有效降低了参数量。全局平均池化(global average pooling,GAP)如图9所示。
通常在神经网络的末端，为满足全连接层的计算需要，通常要将多维特征通过重新排列，形成一维的列向量，进一步完成目标分类任务。在resnet的卷积输出层，特征图的大小为7×7×2048维的三维张量，若将如此冗余的特征直接进行一维展开，全连接层则需要相应数量的节点，每个节点具有相同数量的参数，很大程度上会导致网络出现过拟合，模型过大等问题。
全局平均池化，将输出特征层的每一个通道内的所有特征值求算数平均值，以5*5的特征为例，通过全局平均池化的方法，将原有特征量缩小了25倍。在resnet中，全局平均池化通过对卷积输出层的每个7×7特征加权平均，最终得到了1×2048维的一维向量，有效的降低了网络参数，提高了运算效率。
(3)网络模型
对残差块进行规律性的组合叠加，可构成不同深度的resnet网络模型。在18和34层的浅层resnet中使用的是Basic Block残差块结构。50层以上的resnet引入了Bottleneck Block的残差块结构，此外，浅层的resnet卷积层的末层输 出为512维向量，而深层的resnet其末层输出为2048维向量。不同深度的resnet网络结构如图10所示。
在本实施例中，多实例模型训练流程具体为：
由于resnet具有较强的特征表示能力，并在多项机器视觉任务中获得了较好的表现，所以使用resnet作为多实例学习过程中的判别模型。
基于多实例学习的数字病理图像块分类算法训练流程如图11所示：
多实例学习算法在原有的深度学习迭代训练的基础上，通过迭代的EM过程，使得每次用于训练的样本数据在模型上有最大期望。算法迭代流程如下：
1)以滑动窗口的模式通过预处理方案对图像进行预处理，以包-实例的结构保存数据。
2)在目标模型上遍历所有训练样本，将模型softmax输出的分类概率以包为单位保存下来。
样本遍历的过程是每轮神经网络训练前的必要步骤。通过加载在上一次训练结果中保存的模型，遍历训练数据集中的所有样本，并以WSB为单位将所有patch的分类结果、分类概率保存下来。在遍历过程中，由于不需要对网络模型进行优化，故不需要保存在正向传播过程中的梯度。网络输入的批数据规模为512张图像。
3)对每个包中的样本进行筛选，最终生成实例-标签的成对数据集。
基于包-实例的数据筛选的依据为神经网络在实例上计算所得的概率值。在每次迭代的过程中，包中的实例按照在包类别上的概率进行排序。筛选数据的原则是选择模型推断所得概率值相对较高的数据。依概率也有不同的筛选方式。在本实验中分别通过以下几种方式对样本进行筛选，如表1。
Figure PCTCN2022088794-appb-000009
表1 模型训练过程中的样本筛选方式
4)通过筛选出的数据对网络模型进行优化。
5)重复2)-4)步直至网络收敛。
(3)网络训练
通过数据筛选步骤所得的数据集，均为包含标签的数据对。神经网络的不断迭代优化，依赖于计算网络在批训练数据上的分类结果与样本真实标签间的损失，并通过反向传播算法和优化器对神经网络模型进行不断优化。
交叉熵函数的本质是信息理论中的交叉熵在分类问题中的应用，能够较好的表征模型在分类问题上所表现出的性能，模型预测结果和真实标签之间的差距与交叉熵计算值之间成正比关系。交叉熵函数同样可由最大似然函数在伯努利分布的条件下推导出来，最小化交叉熵函数的过程就是对数似然函数的最大化。相比于均方损失函数，交叉熵函数具有当误差较大时，权重更新较快；误差较小时，权重更新幅度小的特性。单个样本的交叉熵损失函数，其中y为真实标签，
Figure PCTCN2022088794-appb-000010
为模型预测结果，其损失L计算方式如下：
Figure PCTCN2022088794-appb-000011
在神经网络训练的过程中还使用了学习率衰减来优化网络训练流程。网络的初始学习率为0.01，每10个epoch后对学习率进行0.1倍率的衰减。整个训练过程使用两张GPU并行训练。
在第二分类模块14中，构建决策融合模型，并将得到的病理图像块的分类结果输入至决策融合模型中进行融合处理，得到全数字病理图像的分类结果。
数字病理的计算机辅助诊断模型目的是生成WSI的诊断结果。在上一节的算法模型中，实现的是patch的分类模型。得到了WSB中所有patch的分类结果后，还要对其中的结果进行有效的融合，从而进一步得到WSI的最终分类结果。
融合处理的方法包括Max-pooling、Vote、逻辑回归模型(Logistic Regression)。
Max-pooling是较为理想的决策融合方式，假设patch的基本分类器具有较好的性能表现。根据patch基本分类器将整个包中的阳性样本标记为1，所有阴性样本标记为0。当WSB中有任意patch被标记为阳性，则WSB被标记为阳性，WSI为阳性。若WSB中的所有patch均为阴性，则WSB被标记为阴 性，WSI为阴性，设单张采样块的标签为L。
L WSI＝Max(L patch)
Vote是基于全局平均的融合方式，通过对WSB中的所有patch分类结果进行统计，根据WSB中阳性样本和阴性样本的比例，来最终决定WSI的分类结果。
Figure PCTCN2022088794-appb-000012
p WSI-neg＝1-p WSI-pos
逻辑回归模型，线性回归常用于回归分析，不用在分类问题上，将线性回归与逻辑斯蒂分布模型结合即为逻辑回归，逻辑回归是一种常见的二分类模型，是一种判别模型，直接对条件概率进行建模。
设X是连续随机变量，X服从逻辑斯蒂分布是指X具有下列分布函数：
Figure PCTCN2022088794-appb-000013
其中μ和γ为逻辑斯蒂分布的参数。
逻辑回归的表达式为参数化的逻辑斯蒂函数(μ＝0,γ＝1)，即：
Figure PCTCN2022088794-appb-000014
其中h θ(x)是分类结果为1的概率取值。θ为包含偏置项的权值向量。
模型训练的过程采用随机梯度下降法，通过遍历训练样本，以学习率a进行迭代优化：
Figure PCTCN2022088794-appb-000015
逻辑回归模型首先通过计算得到Vote所得的样本比例结果，并在比例的基础上，通过逻辑回归模型对WSB最终所属的分类进行判别。
本实施例通过决策融合模型将所有病理图像块采样前的空间关系进行组合，得到一张与全数字病理图像相似的概率图，通过对概率图引入全连接条件随机场，对病理图像块分类概率图进行修正，得到全数字病理图像的分类结果。
全连接条件随机场：
经过滑动窗口预处理所得到的WSB中的所有patch，本身是存在一定的空 间关系，即在空间位置上相邻的patch之间属于同一类的概率较大，而单纯的以数量为特征的分类方式忽略了重要的空间信息。在数据预处理的过程中，保存所有patch所在的坐标位置，在决策模型融合的过程中，将所有patch按照原有的排列方式进行重新组合，可以进一步得到一张与WSI相似的概率图。参照全连接条件随机场在图像分割领域的应用，通过对概率图引入全连接条件随机场(dense conditional random fields,DenseCRF)，对patch分类概率图进行一定的修正，使得部分离群的误判样本得到正确的分类。DenseCRF的基本原理如下：
首先，假设图像中的每一点都符合吉布斯分布：
Figure PCTCN2022088794-appb-000016
其中x是观测值，E(x|I)是能量函数，也就是优化的目标，Z(X|I)是归一化因子。能量函数E(x|I)由一元势函数和二元势函数构成，如下公式所示：
Figure PCTCN2022088794-appb-000017
其中的一元势函数用于衡量点的类别概率，为神经网络输出结果。二元势函数用于描述点与点之间的关系，鼓励相似的点分配相同的标签，而相差较大的点分配不同的标签。二元势函数公式如下：
Figure PCTCN2022088794-appb-000018
其中u(x i,x j)被称为标签兼容项，它约束了点之间传导的条件，只有相同标签条件下，能量才可以相互传导。
如图12为全连接条件随机场概率修正示意图。
与现有技术相比，本实施例具有以下有益效果：
(1)通过有效组织定位、染色归一化等数据预处理方式，实现了对数字全病理切片的数据采样、清洗、规范化流程，形成“包-实例”的数据结构。其中包标签与全病理切片相同，而包中样本不具有确切标记。
(2)在上述数据结构的基础上，使用两阶段的研究方法实现了全病理切片的计算机辅助诊断：第一阶段通过多实例学习的弱监督算法，使用粗粒度的 包标记，实现了在采样实例上的分类模型，获得了与医生标记结果相近的分类性能；第二阶段对包内实例的分类结果进行决策融合，充分利用采样实例间的空间关系，通过全连接条件随机场对其中的误分类实例进行结果修正。在性能测试和可视化结果上，证明了算法具有一定的有效性。
实施例二
本实施例提供的基于弱监督学习的结直肠癌数字病理图像判别系统与实施例一的不同之处在于：
本实施例分别在结直肠癌的病变辅助诊断和微卫星不稳定性识别问题上对算法性能进行了测试。
实验配置如表2所示:
Figure PCTCN2022088794-appb-000019
表2 实验硬件环境配置
结果与分析：
(1)结直肠癌病变辅助诊断
在病变辅助诊断任务中，使用的是来自MICCAI的结直肠癌病理分割挑战数据。数据中包含了410张阳性样本和250张阴性样本，所有阳性样本均由病理医生对病变区域进行标注。在本实验中，目的是探究基于粗粒度标注的多实例学习算法，故所有的专家标注掩膜均未用于模型训练，只用于模型性能的评估。
所有样本均采用前节所述的包-实例处理方式，对于阴性样本，包及包中实例均按照标注为阴性；而对于阳性样本，除依照包标签为实例的标注外训练标签外，需额外通过mask为实例标注其ground truth用于算法的性能评估。方法是对原始图像和mask采用同步的滑动窗口采样处理，若采样后的mask中 均为背景，则该实例的ground truth为阴性。若mask中包含前景像素，则该实例的ground truth为阳性。其示意图如图13所示。
通过实验对比了不同网络模型，不同训练方式下，病理图像块分类任务性能表现，使用resnet18、resnet50进行强监督训练，使用resnet18、resnet50结合不同的数据筛选方式进行多实例学习，其性能表现如下表3：
Figure PCTCN2022088794-appb-000020
表3 不同网络模型和训练方法下的算法性能
从表3中可以看出，与强监督学习的方法相比，基于多实例学习的训练方式，有效提升了模型在病理图像分类上块分类任务上的性能。Resnet18在整体性能上要优于resnet50，由于样本量的不足导致了一定的过拟合问题。在不同的数据筛选方法上，方法3的性能结果优于其他集中方式，随机对阴性切片选取的过程可以视为数据的扩增和对网络模型的惩罚。
将全病理图像包中的病理图像块依照预处理时的位置信息进行重新排列，可以得到一张基于概率的热力图，如图14所示。
通过对比医生在WSI中的标记结果和算法的可视化展示，发现基于多实例学习的神经网络能在粗粒度标记的数据集上，通过迭代的多实例学习策略，实现细粒度的病理图像块分类模型。对比强监督和多实例学习算法的可视化结果可以看出，在基于多实例学习的弱监督算法在对病变切片识别的问题具有明 显优势。能很好的实现对切片中不同采样块的有效判断。图14中所有样本虽然经后续处理判断均可判别为阳性样本，但本实施例使用的算法与医生的标注结果显然具有更高的一致性，从而算法结果具有更高的可解释性和可依赖性。
全病理图像的识别问题是一个两阶段问题，上述结果仅代表在采样块上的分类性能，由于模型不可避免的存在一定的误判，希望通过对上述结果的整合，实现端到端的WSI判别。在对数字病理块性能测试后，需对结果进行整合，从而得到整张病理切片的分类结果，以下是在不同的决策融合模型在全病理图像识别问题上的性能测试结果如表4：
Figure PCTCN2022088794-appb-000021
表4 决策融合模型算法性能
从结果可以看出，基于数字病理块的识别决策融合模型在全病理切片的判别问题上有较好的性能表现，其中在生成的热力图结果上通过DenseCRF对前一步模型结果进行一定修正，并通过逻辑回归模型对判别结果进行融合的方式获得了最好的性能。此外在原始概率热力图上使用DenseCRF对算法结果均有一定的提升。
进一步对全病理图像在添加DenseCRF前后的结果进行可视化可以看到。在经过DenseCRF后，部分离群的误判点在处理过后，其误判概率降低，特别是在部分被误判的正常切片中，修正后的所有结果中不再包含假阳性结果，使得基于病理图像块的决策融合模型性能得到改善。
(2)结直肠癌微卫星不稳定性辅助诊断
在明确了弱监督学习方法在癌变区域识别问题上的有效性后，探究性的将该算法应用于结直肠癌微卫星不稳定性(micro-satellite instability,MSI)问题的识别上。
微卫星稳定性是结直肠癌及其他实体瘤预后的重要分子标志物。微卫星不 稳定是由于人体错配修复机制缺陷，当人体DNA出现微卫星错误重复时，无法被正常修复，可能导致细胞癌变。微卫星高度不稳定患者对于PD-1免疫治疗获益显著，判断患者是否具备微卫星高度不稳定(microsatellite highly instability，MSI-H)，是临床诊断中的重要环节。而目前临床上微卫星不稳定性的诊断依赖于免疫组化染色和基因检测，存在耗时且费用偏高等问题。通过算法挖掘病理图像中的潜在特征，实现基于病理图像的微卫星不稳定性诊断具有重要意义。基于上述算法，将目标数据分为正常、微卫星高度不稳定、微卫星低度不稳定(micro-stallite low stability,MSI-L)、微卫星稳定(micro-satellite stability)四种类型。
通过概率热力图的方式，对算法结果进行可视化，如图15所示，是算法在测试集上的结果。可以看出，基于多实例学习的算法在微卫星不稳定性问题上具有一定的有效性。其中未发生病变的切片和组织得到了很好的识别。被判定为微卫星高度不稳定的切片在高度不稳定的数据中广泛存在，而微卫星稳定和微卫星低度不稳定两种类型不具备明显的分离结果，这个结果与临床诊断上的结果有一定契合，即只有微卫星高度不稳定的患者，在临床上才具有显著意义，而其他两类常视为同一类型。
进一步对不同类型测试集数据中算法预测结果进行了统计，在微卫星高度不稳定的样本中，被判定为微卫星高度不稳定的数字病理块比例明显高于其他几类。而在所有切片中，微卫星稳定和微卫星低度不稳定的样本数量基本持平。
如图16所示为不同微卫星分型样本中的分类比例，从结果可以看出，基于弱监督学习的神经网络模型能够有效的将切片中的正常切片识别出来。此外，在微卫星高度不稳定性的切片中，被判别为微卫星不稳定的区域明显高于其他几类。而微卫星低度不稳定和微卫星稳定的两类并没有显著的差异。这个结果与微卫星不稳定性在临床上的诊断吻合。只有微卫星高度不稳定的患者，在临床上是对部分治疗方案和药物是显著的。
对三类样本进行了端到端的分类。将测试集样本分为微卫星高度不稳定、微卫星稳定和微卫星低度不稳定、正常，分为三类，绘制其ROC曲线，并计算其AUC，结果如图17所示。
算法在MSI-H、MSI-L+MSS、Normal几类的判别结果上的AUC分别达 到了0.83、0.84、1。表示算法在数字病理的微卫星不稳定性识别上具有一定的有效性。
在数字病理切片癌变区域识别和结直肠癌微卫星不稳定性识别问题上对算法性能进行了测试，测试结果表明，基于多实例学习的神经网络模型对数字病理图像的识别问题上具有一定的有效性。
实施例三
本实施例提供基于弱监督学习的结直肠癌数字病理图像判别方法，包括：
S1.采集结直肠癌数字病理图像数据集；
S2.对采集的数据集中的数据进行预处理，得到预处理后的数据；
S3.基于弱监督学习算法构建采样块判别模型，并将预处理后的数据输入至构建的采样块判别模型中进行处理，得到全切片采样包中所有病理图像块的分类结果；
S4.构建决策融合模型，并将得到的病理图像块的分类结果输入至决策融合模型中进行融合处理，得到全数字病理图像的分类结果。
进一步的，所述步骤S2包括：
S21.对输入的数字病理图像中的有效组织进行定位；
S22.对数字病理图像中不同的组织切片进行图像的染色归一化处理，得到预处理后的数据。
进一步的，所述步骤S3中中基于弱监督学习算法构建采样块判别模型具体为：通过弱监督学习算法中的多实例学习对深度残差网络的训练过程进行优化，得到采样块判别模型。
进一步的，所述步骤S4中进行融合处理具体为：通过决策融合模型将所有病理图像块按照预设的排列方式进行组合，得到一张与全数字病理图像相似的概率图，通过对概率图引入全连接条件随机场，对病理图像块分类概率图进行修正，得到全数字病理图像的分类结果。
需要说明的是，本实施例提供的基于弱监督学习的结直肠癌数字病理图像判别方法与实施例一类似，在此不多做赘述。
与现有技术相比，本申请具有以下有益效果：
(1)通过有效组织定位、染色归一化等数据预处理方式，实现了对数字 全病理切片的数据采样、清洗、规范化流程，形成“包-实例”的数据结构。其中包标签与全病理切片相同，而包中样本不具有确切标记。
(2)在上述数据结构的基础上，使用两阶段的研究方法实现了全病理切片的计算机辅助诊断：第一阶段通过多实例学习的弱监督算法，使用粗粒度的包标记，实现了在采样实例上的分类模型，获得了与医生标记结果相近的分类性能；第二阶段对包内实例的分类结果进行决策融合，充分利用采样实例间的空间关系，通过全连接条件随机场对其中的误分类实例进行结果修正。在性能测试和可视化结果上，证明了算法具有一定的有效性。
注意，上述仅为本申请的较佳实施例及所运用技术原理。本领域技术人员会理解，本申请不限于这里所述的特定实施例，对本领域技术人员来说能够进行各种明显的变化、重新调整和替代而不会脱离本申请的保护范围。因此，虽然通过以上实施例对本申请进行了较为详细的说明，但是本申请不仅仅限于以上实施例，在不脱离本申请构思的情况下，还可以包括更多其他等效实施例，而本申请的范围由所附的权利要求范围决定。

Claims (10)

  1. 基于弱监督学习的结直肠癌数字病理图像判别系统，其特征在于，包括：
    采集模块，用于采集结直肠癌数字病理图像数据集；
    预处理模块，用于对采集的数据集中的数据进行预处理，得到预处理后的数据；
    第一分类模块，用于基于弱监督学习算法构建采样块判别模型，将预处理后的数据输入至构建的采样块判别模型中进行处理，得到全切片采样包中所有病理图像块的分类结果；
    第二分类模块，用于构建决策融合模型，将得到的病理图像块的分类结果输入至决策融合模型中进行融合处理，得到全数字病理图像的分类结果。
  2. 根据权利要求1所述的基于弱监督学习的结直肠癌数字病理图像判别系统，其特征在于，所述预处理模块包括：
    组织定位模块，用于对输入的数字病理图像中的有效组织进行定位，并生成采样数据集；
    归一化模块，用于对生成采样数据集进行染色归一化处理，得到预处理后的数据。
  3. 根据权利要求2所述的基于弱监督学习的结直肠癌数字病理图像判别系统，其特征在于，所述组织定位模块包括：
    背景处理模块，用于对数字病理图像的下采样图像的缩略图进行前后景分割，提取缩略图中的有效图像区域；
    定位模块，用于基于背景处理模块得到的结果，对数字病理图像中的有效组织进行定位；
    采样模块，用于对定位后的区域进行遍历采样，生成符合算法需要的数字病理采样块。
  4. 根据权利要求3所述的基于弱监督学习的结直肠癌数字病理图像判别系统，其特征在于，所述归一化模块包括：
    转换模块，用于将输入的组织切片图像的RGB值转化为OD光密度值；
    移除模块，用于设置阈值β，将光密度值小于β的像素点移除；
    第一计算模块，用于计算OD矩阵的SVD分解；
    创建模块，用于在SVD分解后的两个最大奇异值向量创建一个平面；
    投影模块，用于将数据投影到创建的平面中去，并进行归一化；
    第二计算模块，用于计算每个点与第一奇异值之间的夹角，并找到夹角的极值；
    输出模块，用于将极值转化为OD域，输出归一化后的图像。
  5. 根据权利要求1所述的基于弱监督学习的结直肠癌数字病理图像判别系统，其特征在于，所述第一分类模块中基于弱监督学习算法构建采样块判别模型具体为：通过弱监督学习算法中的多实例学习对深度残差网络的训练过程进行优化，得到采样块判别模型。
  6. 根据权利要求1所述的基于弱监督学习的结直肠癌数字病理图像判别系统，其特征在于，所述第二分类模块中进行融合处理具体为：将所有病理图像块按照采样前的空间关系进行组合，得到一张与全数字病理图像相似的概率图，通过对概率图引入全连接条件随机场，对病理图像块分类概率图进行修正，得到全数字病理图像的分类结果。
  7. 基于弱监督学习的结直肠癌数字病理图像判别方法，其特征在于，包括：
    S1.采集结直肠癌数字病理图像数据集；
    S2.对采集的数据集中的数据进行预处理，得到预处理后的数据；
    S3.基于弱监督学习算法构建采样块判别模型，将预处理后的数据输入至构建的采样块判别模型中进行处理，得到全切片采样包中所有病理图像块的分类结果；
    S4.构建决策融合模型，将得到的病理图像块的分类结果输入至决策融合模型中进行融合处理，得到全数字病理图像的分类结果。
  8. 根据权利要求7所述的基于弱监督学习的结直肠癌数字病理图像判别方法，其特征在于，所述步骤S2包括：
    S21.对输入的数字病理图像中的有效组织进行定位，并生成采样数据集；
    S22.对生成的采样数据集进行染色归一化处理，得到预处理后的数据。
  9. 根据权利要求7所述的基于弱监督学习的结直肠癌数字病理图像判别 方法，其特征在于，所述步骤S3中基于弱监督学习算法构建采样块判别模型具体为：通过弱监督学习算法中的多实例学习对深度残差网络的训练过程进行优化，得到采样块判别模型。
  10. 根据权利要求7所述的基于弱监督学习的结直肠癌数字病理图像判别方法，其特征在于，所述步骤S4中进行融合处理具体为：将所有病理图像块采样前的空间关系进行组合，得到一张与全数字病理图像相似的概率图，通过对概率图引入全连接条件随机场，对病理图像块分类概率图进行修正，得到全数字病理图像的分类结果。
PCT/CN2022/088794 2021-04-27 2022-04-24 基于弱监督学习的结直肠癌数字病理图像判别方法及系统 WO2022228349A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
ZA2023/10619A ZA202310619B (en) 2021-04-27 2023-11-15 Method and system for discriminating digital pathological images of colorectal cancer based on weakly supervised learning

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110460404.9A CN113221978A (zh) 2021-04-27 2021-04-27 基于弱监督学习的结直肠癌数字病理图像判别方法及系统
CN202110460404.9 2021-04-27

Publications (1)

Publication Number Publication Date
WO2022228349A1 true WO2022228349A1 (zh) 2022-11-03

Family

ID=77089246

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/088794 WO2022228349A1 (zh) 2021-04-27 2022-04-24 基于弱监督学习的结直肠癌数字病理图像判别方法及系统

Country Status (3)

Country Link
CN (1) CN113221978A (zh)
WO (1) WO2022228349A1 (zh)
ZA (1) ZA202310619B (zh)

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113221978A (zh) * 2021-04-27 2021-08-06 浙江师范大学 基于弱监督学习的结直肠癌数字病理图像判别方法及系统
CN113887377A (zh) * 2021-09-18 2022-01-04 上海申挚医疗科技有限公司 组织病理图像分析方法及系统
CN113884489B (zh) * 2021-09-29 2023-06-20 电子科技大学 一种光栅尺辅助定位的厚液层细胞自动显微成像方法
CN113723573B (zh) * 2021-11-03 2022-01-14 浙江大学 基于自适应比例学习的肿瘤组织病理分类系统及方法
CN114038051B (zh) * 2021-11-17 2023-09-08 首都医科大学附属北京同仁医院 一种原发性闭角型青光眼的房角关闭机制分型方法
CN114511523B (zh) * 2022-01-21 2024-05-31 中山大学 基于自监督学习的胃癌分子亚型分类方法及装置
CN114882008B (zh) * 2022-06-09 2024-09-10 南京工业大学 一种基于病理图像特征检测肿瘤驱动基因差异表达算法
CN114972341B (zh) * 2022-07-28 2022-12-16 华南理工大学 基于贝叶斯辅助学习的wsi图像分类方法、系统及介质
CN115620075B (zh) * 2022-12-16 2023-03-14 南昌大学 白细胞分类模型用数据集的生成方法、系统及设备
CN117038023A (zh) * 2023-10-08 2023-11-10 中国医学科学院北京协和医院 基于结直肠癌病理图像的dMMR胚系突变亚型分类方法及系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200250398A1 (en) * 2019-02-01 2020-08-06 Owkin Inc. Systems and methods for image classification
CN111985536A (zh) * 2020-07-17 2020-11-24 万达信息股份有限公司 一种基于弱监督学习的胃镜病理图像分类方法
CN113221978A (zh) * 2021-04-27 2021-08-06 浙江师范大学 基于弱监督学习的结直肠癌数字病理图像判别方法及系统

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111445488B (zh) * 2020-04-22 2023-08-04 南京大学 一种弱监督学习自动识别和分割盐体的方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200250398A1 (en) * 2019-02-01 2020-08-06 Owkin Inc. Systems and methods for image classification
CN111985536A (zh) * 2020-07-17 2020-11-24 万达信息股份有限公司 一种基于弱监督学习的胃镜病理图像分类方法
CN113221978A (zh) * 2021-04-27 2021-08-06 浙江师范大学 基于弱监督学习的结直肠癌数字病理图像判别方法及系统

Also Published As

Publication number Publication date
ZA202310619B (en) 2024-04-24
CN113221978A (zh) 2021-08-06

Similar Documents

Publication Publication Date Title
WO2022228349A1 (zh) 基于弱监督学习的结直肠癌数字病理图像判别方法及系统
Pati et al. Hierarchical graph representations in digital pathology
Man et al. Classification of breast cancer histopathological images using discriminative patches screened by generative adversarial networks
Ahmedt-Aristizabal et al. A survey on graph-based deep learning for computational histopathology
Roy et al. Patch-based system for classification of breast histology images using deep learning
Das et al. Computer-aided histopathological image analysis techniques for automated nuclear atypia scoring of breast cancer: a review
CN112101451B (zh) 一种基于生成对抗网络筛选图像块的乳腺癌组织病理类型分类方法
Janowczyk et al. Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases
Panigrahi et al. Capsule network based analysis of histopathological images of oral squamous cell carcinoma
Chang et al. Deep learning based Nucleus Classification in pancreas histological images
Banerji et al. Deep learning in histopathology: A review
Linkon et al. Deep learning in prostate cancer diagnosis and Gleason grading in histopathology images: An extensive study
CN114730463A (zh) 用于组织图像分类的多实例学习器
Hu et al. Classification of metaphase chromosomes using deep convolutional neural network
Pati et al. Reducing annotation effort in digital pathology: A Co-Representation learning framework for classification tasks
JP2023543044A (ja) 組織の画像を処理する方法および組織の画像を処理するためのシステム
Dogar et al. Attention augmented distance regression and classification network for nuclei instance segmentation and type classification in histology images
Duran-Lopez et al. Wide & Deep neural network model for patch aggregation in CNN-based prostate cancer detection systems
CN111247417A (zh) 通过利用类比例数据的卷积字典学习对对象群体进行分类
Dabass et al. A hybrid U-Net model with attention and advanced convolutional learning modules for simultaneous gland segmentation and cancer grade prediction in colorectal histopathological images
Calin et al. Object-oriented classification approach for bone metastasis mapping from whole-body bone scintigraphy
CN114864075A (zh) 一种基于病理图像的胶质瘤级别分析方法及装置
Rosales-Pérez A review on machine learning techniques for acute leukemia classification
Sunny et al. Oral epithelial cell segmentation from fluorescent multichannel cytology images using deep learning
Taheri et al. A Comprehensive Study on Classification of Breast Cancer Histopathological Images: Binary Versus Multi-Category and Magnification-Specific Versus Magnification-Independent

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22794819

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22794819

Country of ref document: EP

Kind code of ref document: A1