CN114445356A - Multi-resolution-based full-field pathological section image tumor rapid positioning method - Google Patents

Multi-resolution-based full-field pathological section image tumor rapid positioning method Download PDF

Info

Publication number
CN114445356A
CN114445356A CN202210058387.0A CN202210058387A CN114445356A CN 114445356 A CN114445356 A CN 114445356A CN 202210058387 A CN202210058387 A CN 202210058387A CN 114445356 A CN114445356 A CN 114445356A
Authority
CN
China
Prior art keywords
full
resolution
pathological section
field
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210058387.0A
Other languages
Chinese (zh)
Inventor
杨杰
王睿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jiaotong University
Original Assignee
Shanghai Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Jiaotong University filed Critical Shanghai Jiaotong University
Priority to CN202210058387.0A priority Critical patent/CN114445356A/en
Publication of CN114445356A publication Critical patent/CN114445356A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24137—Distances to cluster centroïds
    • G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Abstract

The invention provides a full-field pathological section image tumor rapid positioning method based on multi-resolution, which comprises the following steps: constructing an image block training set; obtaining two classification models by utilizing the image block training set; and (4) predicting the full-field pathological section by using the two classification models in sequence to obtain a positioning result. Under the condition of ensuring that image detail information is not sufficiently lost, two models with preset resolution ratios are respectively trained, one classification model adopts a faster full convolution network, and the other classification model is added with an efficient feature extraction module. Compare and fix a position under traditional frame adopts single resolution ratio, under the condition of guaranteeing positioning accuracy, realized quick location full field of vision pathological section goes up tumour region.

Description

Multi-resolution-based full-field pathological section image tumor rapid positioning method
Technical Field
The invention relates to the technical field of deep learning and medical image analysis, in particular to a full-field pathological section image tumor rapid positioning method based on multi-resolution.
Background
In recent years, with the continuous development of deep learning, the deep learning has more and more effects combined with other fields. In medical image analysis tasks, tumor detection on pathological images is a key step in the confirmation of a clinical pathologist. And an automatic auxiliary pathological detection system produced by combining the two can greatly reduce the misdiagnosis rate and the diagnosis time of a pathologist. With the development of digital pathology imaging system, the full-glass digital scanning technology is developed to the high-throughput fast full-field pathology slice technology (WSI), and locating a tumor region on the level of a huge digital pathology image is time-consuming, and often, in the clinical application stage, a digital pathology image doctor with 150000 x 150000 pixels takes 5-10 minutes, while the existing method is generally slow. Since pathological image detection is the "gold standard" for tumor diagnosis, great care is needed to treat this technique. Therefore, the designed system is mainly used for assisting the pathological doctors to diagnose, and a quick and accurate positioning method is needed for assisting the pathological doctors to judge the tumors, so that the workload of the pathological doctors is further reduced, and the diagnosis accuracy is improved.
Recent researches show that a convolution neural network is used for analyzing a full-view pathological section image, the convolution neural network is widely used as a characteristic extraction mode in a tumor positioning task in the full-view pathological section, the tumor characteristics under the highest magnification are trained in advance, and a sliding window mode is adopted in the inference process to cut an image block on the section image under the highest magnification and input the image block into the network for judging tumors or non-tumors. However, the highest magnification represents the resolution image of the original size, and although the detail emotion of the image is large, the number of sliding windows is large in the inference process, so that the calculated amount is large, and the inference time cannot meet the clinical requirement. Therefore, a rapid positioning method for the full-view pathological section needs to be designed, and the positioning accuracy and the positioning speed can be ensured at the same time.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a full-field pathological section image tumor rapid positioning method based on multi-resolution.
According to one aspect of the invention, a multi-resolution full-field pathological section image tumor rapid positioning method is provided, which comprises the following steps:
constructing an image block training set;
obtaining two classification models by utilizing the image block training set;
and (4) predicting the full-field pathological section by using the two classification models in sequence to obtain a positioning result.
Preferably, the constructing a training set of image blocks includes:
reading a full-field pathological section, and extracting the foreground and the background of the full-field pathological section by a large-scale method;
and extracting image blocks with preset sizes and different magnifications in the foreground and background areas, constructing image block data sets with different resolutions, and obtaining the category labels of the image block data sets. And constructing data sets with different resolutions for training, and obtaining model characteristic parameters under different magnifications.
Preferably, the two classification models have characteristic parameters of tumor and normal tissues under different magnifications; the two classification models include a low resolution model and a high resolution model.
The method includes the following steps that two classification models are successively used for predicting the full-field pathological section to obtain a positioning result, and the method includes the following steps:
firstly, predicting a full-field pathological section by using a low-resolution model to obtain a rough positioning probability map;
and judging abnormal pixel points in the rough positioning probability map by using a high-resolution model, and obtaining the probability value of the pixel position again to obtain the final positioning probability map.
Preferably, training the low resolution model comprises:
training a full convolutional classification network using the low resolution dataset, the network having Resnet18 as a backbone network;
replacing the full connection layer of the network with a full convolution layer, and controlling the number of convolution channels of the last layer as a category number;
the positive samples are weighted during training, so that the low-resolution model has higher response to the positive samples;
fusing feature maps of different levels (stages) in different convolutional networks by adopting a multi-scale feature fusion method for the low-resolution model to obtain multi-scale features;
Out＝concat(s1+s2+s3)，
where (s1, s2, s3) are feature output maps at different levels in the Resnet18 network, concat is tandem fusion, and Out is a multi-scale fusion feature.
The low-resolution network takes the resnet18 as a backbone network, and the improvement is carried out on the backbone network, so that the network is more suitable for a tumor localization task under a low-resolution image (a feature pyramid is added in the tumor localization task), local features and global features can be fused, and the discrimination capability is enhanced.
Preferably, training the high resolution model comprises:
a convolutional classification network trained using a high-resolution data set,
the network takes Resnet18 as a backbone network, an SE module is added in each basic module in Resnet18, the weight of each stage output channel is calculated, and the weight is multiplied by a feature map in a feature channel to screen out features which greatly contribute to categories. Here, the principle of the channel attention mechanism is to screen out the features with large contribution by reinforcing the channels with large feature weights.
And carrying out multi-scale feature fusion on the obtained stage output features.
The high-resolution network uses the resnet18 as a backbone network, and the improvement is carried out on the backbone network, so that the network is more suitable for a tumor localization task under a high-resolution image (a feature pyramid and a channel attention mechanism are added), and the feature expression of the fused local feature and the global feature is more comprehensive and efficient under the action of the channel attention mechanism.
Preferably, the predicting the full-field pathological section by using the low-resolution model to obtain the rough positioning probability map includes:
the original full-view pathological section is Slevel_0Acquiring a low-magnification full-field pathological section image S with preset magnification by a preset algorithmlevel_i(i＝1,2,3)；
Through a sliding window S of predetermined sizeWAnd a fixed step length SSIn low magnification full-field pathological section Slevel_i(i is 1,2,3) sliding the sample,
the samples are input into a low resolution model with a down-sampling factor of 2n；
Splicing the calculated results of the sliding window samples to generate an original size 1/2n+iMagnitude coarse positioning probability map.
Preferably, the determining, by using a high resolution model, the abnormal pixel in the rough positioning probability map to obtain the probability value of the pixel position again, so as to obtain a final positioning probability map, includes:
the probability map comprises true positive and false positive pixel points, and whether the pixel position needs to be further judged is judged through a threshold value;
the relationship between the pixel coordinate on the rough probability map and the pixel position of the point on the full-field pathological section in the original size is as follows:
(Xl0,Yl0)＝(Xln×2n+i,Yln×2n+i)
with ((X)l0,Yl0) Extracting an image block with a preset size as a sampling center, inputting the image block into a high-resolution model, wherein the down-sampling factor of the high-resolution model is 2n+i(ii) a In particular, n is the resolution scale factor of the coarse network output graph and i is the resolution scale factor of the coarse network input WSI. In the invention, the high resolution wsi is equal to the original size x40 magnification; low resolution wsi-x 20 magnification.
Updated result ((X)ln,Yln) The category of the pixel point.
According to a second aspect of the present invention, there is provided a terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute any one of the multi-resolution-based full-field pathological section image tumor fast localization methods when executing the program.
According to a third aspect of the present invention, there is provided a computer-readable storage medium having at least one instruction, at least one program, a code set, or a set of instructions stored therein, which is loaded by a processor and executes any one of the multi-resolution full-field pathology slice image tumor fast localization methods.
Compared with the prior art, the invention has the following beneficial effects:
the invention greatly shortens the prediction time of the full-field pathological section by utilizing the characteristic difference under different resolutions.
Under the condition of ensuring that image detail information is not sufficiently lost, two models with preset resolution ratios are respectively trained, a low-resolution model adopts a faster full-convolution network, and a high-efficiency feature extraction module is added to the high-resolution model. Compare and fix a position under traditional frame adopts single resolution ratio, under the condition of guaranteeing positioning accuracy, realized quick location full field of vision pathological section goes up tumour region.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a flowchart of an embodiment of a method for rapidly locating a tumor based on a multi-resolution full-field pathological section image;
FIG. 2 is a detailed structure of the Nc and Nf models in the training phase according to a preferred embodiment;
FIG. 3 is a block flow diagram of the multi-resolution model of a preferred embodiment during the prediction phase;
fig. 4 is a visualization of two WSIs (full field pathology sections) in a test set in a preferred embodiment.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the spirit of the invention. All falling within the scope of the present invention.
The invention provides an embodiment, a full-field pathological section image tumor rapid positioning method based on multi-resolution, which comprises the following steps:
s10, constructing an image block training set;
s20, training by using the image block training set to obtain two classification models;
and S30, predicting the full-field pathological section by using the two classification models in sequence to obtain a positioning result.
Based on the foregoing embodiments, optimization is performed to provide a preferred embodiment, as shown in fig. 1, which is a flowchart of a method for rapidly positioning a tumor in a multi-resolution full-view pathological section image, and the method includes:
and S1, preprocessing the digital image of the full-field pathological section by adopting a preset algorithm to obtain the foreground and the background.
In order to better obtain the foreground and the background, a preferred embodiment is provided. In this embodiment, a full-field pathological section digital image and its truth mask are obtained, and a large amount of blank background areas in the full-field pathological section image are filtered by a large law method, wherein the large law method (OTSU) uses an adaptive threshold method to filter the foreground and the background, and obtains the mask of the whole tissue area. In a lesion WSI image, the foreground includes the lesion region and the normal tissue region (where the lesion region may be determined by a truth mask: GT _ mask); in a normal WSI image, the foreground contains only normal tissue regions. The truth value mask and the whole tissue mask are amplified or reduced through different sampling magnifications.
And S2, extracting image blocks with fixed sizes from the foreground and background areas on the full-view pathological section with the original size respectively, and generating the image blocks into a data set with two types of labels.
In general, the relationship between magnification and full field slice resolution is: (X40, X20, X10, …, X0.315) (level)0,level1,level2,…,level9) And constructing image blocks of the full-field slices with different resolutions in the foreground and background areas.
In the present embodiment, a low resolution level is extracted2The image block size of the full-field slice is 163 × 163, and high-resolution level is extracted0The image block size of the full-field slice is 346, image block data sets of different resolutions are constructed, and respective class labels of the two types of image block data sets are obtained.
Specifically, after the tissue mask is obtained, the region where the tissue mask and the truth mask intersect is a tumor region, the rest of the tissue mask is a normal region, random sampling is respectively performed on the two regions, and a coordinate point (X) on the full-field slice is obtained under the maximum magnificationl0,Yl0) Sampling is performed as the center of the image block, and a high-resolution level0 image block data set is obtained when the image block has a preset size (346 x 346) and the image block has a preset foreground-background ratio (1: 1). Wherein random sampling is performed to ensure the randomness of data. If the samples are sequentially sampled according to the mask, there will be a large number of repeated images to train. These large amounts of repeated data can make the model less robust. The image blocks are extracted according to the coordinate points, and the size of the image block is a preset value (low resolution set to 163 × 163; high resolution set to 346 × 346), wherein the foreground-background ratio is set to ensure that the ratio of the foreground to the background is enough to ensure that the tumor area in the positive image block (in a single image block: 346 × 346).
And reducing all image block data in the high-resolution level0 image block data set to 4 times by a bilinear interpolation method to obtain a low-resolution level2 image block data set, so that the image blocks of the same view have different resolutions in the two data sets.
Specifically, the foreground region includes a tumor region and a normal tissue region, the image block extracted from the tumor region has a tumor type label, and the image block extracted from the normal tissue region has a normal tissue type label. The background area is a large amount of white blank areas, and a small number of image blocks are taken as normal tissue image blocks in the area.
S3 is performed on the basis of S2. S3: inputting a level2 data set into a network Nc for training to obtain model characteristic parameters under the magnification; similarly, the level0 data set is input into the network Nf, and the model parameters at this magnification are obtained.
In order to obtain the better two types of training models, a preferred embodiment is provided. In this example, the image block level2 dataset and its class label are input into the Nc network for model training, and the tumor characteristic parameters under low magnification are obtained. Wherein the network model comprises the following structure:
the Nc network model has a residual error connected convolution structure and a multi-scale feature aggregation mode, and ensures that the output result of the image block size in the preset level2 data set after Nc is 1 x1 vector.
Similarly, to obtain high magnification tumor features, the level0 dataset is used as input to the Nf network model, and the Nf model parameters at that resolution are obtained. Wherein, the network model comprises the following structure:
the Nf network is provided with a residual error connected convolution network and a multi-scale feature aggregation module and a channel attention mechanism. The network ensures that the output result of the image block size in the preset level2 data set after passing Nf is a vector of 1 x 1.
Further, the Nc network and the Nf network adopt correspondingly preset hyper-parameters during training, an optimization model is constructed through the cross entropy of the predicted values and the true values of the image blocks, and a random gradient descent method is selected for optimization.
S4 is performed on the basis of S3. S4: the network Nc trained on the level2 data set performs sliding window prediction under the level2 magnification of the full-field pathological section image to obtain a rough probability map of a tumor region.
In order to increase the speed and accuracy of the acquired coarse probability map of the tumor region, a preferred embodiment is provided. In the present embodiment, Nc model parameters are used to predict a full-field pathological section of a level2(× 10 magnification) test, sampling is performed on the full-field pathological section by overlapping sliding windows by presetting the step size and the size of the sliding windows, and the image block in each sliding window is used as an input of the Nc model. Due to the nature of the full convolutional network, the Nc model now produces a corresponding coarse probability tile.
Furthermore, probability image blocks generated by each sliding window are spliced according to a certain sequence to obtain a rough probability heat map of the full-field pathological section image downsampling size.
S5 is performed on the basis of S4. And S5, sampling candidate pixel points on the probability map obtained through positioning through a threshold value under the condition of backtracking to a level0 magnification, and obtaining accurate type judgment of the pixel points.
In order to obtain an accurate category judgment, a preferred embodiment is provided, in this embodiment, a preset threshold is used, and in this embodiment, the threshold is set to be 0.5. Marking the positions of the pixel points exceeding the threshold value in the probability heat map of the down-sampling size, and passing through the pixel positions by a formula (X)l0,Yl0)＝(Xln×2n+i,Yln×2n+i) Go back to the location on the level0 (x40 magnification) slice. The position is cut by a preset image block size (the pixel point obtained by backtracking is taken as the center, and an image block is selected from the slice with the magnification of X40 according to the size 346X 346 preset by level 0), and the type mark of the image block corresponding to the pixel point is output as the input for obtaining Nf.
And S6, obtaining a tumor positioning result based on the multi-resolution model structure.
In order to achieve an accurate and fast localization of the tumor region, the invention provides another preferred embodiment.
And S100, performing foreground extraction on the read full-field pathological section by a large law method, wherein the foreground is a tissue area stained by H & E in a tumor digital pathological image and comprises a normal tissue and a tumor tissue, and a binary mask of the tissue can be obtained by the large law method, and the binary mask is called as a mask in the following description.
In this embodiment, the true value mask is marked by the multiple open source software ASAP, and an XML file is obtained. The annotation files are provided by an official data set. And generating a truth value mask according to the annotation file, identifying a region in the truth value mask as tumor cells, identifying a non-truth value mask region in the tissue mask as normal tissue cells, extracting the image block in the region as a tumor class and a normal class of the training data set respectively, and further obtaining the image block image and the annotation information.
At this time, an image block data set Xlevel0 is obtained and is an image block acquired under default magnification of multiplied by 40 times, the length and width of each sample in the Xlevel0 data set are reduced to 4 times of the original size simultaneously through a bilinear interpolation method, an Xlevel2 data set is obtained, and the Xlevel2 data set is equivalent to an image acquired under multiplied by 10 magnification
xlevel0＝randomcrop(WSIs) xlevel0∈Rh*w*c
xlevel2＝resize(xlevel0) xlevel2∈Rh/4*w/4*c
Figure BDA0003477344520000071
Figure BDA0003477344520000072
Wherein randomcrop is randomly cut, resize is bilinear interpolation, WSIs are full-field pathological section images under the highest magnification, and xlevel0A high resolution image block data set. x is the number oflevel2Is a low resolution image block dataset where level is expressed as a full-view slice resolution level at different magnifications and level0 represents a full-view slice image at maximum magnification. h, w represents the length and width of the image block, and c represents the number of channels of the image.
After the maximum resolution WSI of the original size is downsampled by 4 times, the obtained tumor cells are not greatly lost in the aspects of semantics and details, and whether the image contains the tumor cells can be observed, so that the embodiment is carried out under the condition of ensuring that the detail information of the image is not sufficiently lost.
S200, constructing Nc and Nf models under different resolutions, wherein the training model structure is shown in FIG. 2. As can be seen, the difference between the coarse network and the fine network is that the Resnet18 is used as a backbone network, and the Resnet18 is generally different in that features of different hierarchies (L2-L4) are subjected to feature fusion (feature fusion in the form of a feature pyramid) after passing through an M module, and the final layer does not use a full-connection structure, but adopts a convolution layer plus a full-play average layer, so that image blocks of different sizes have output sizes of 1 × C and C is the number of categories (two categories, positive and negative in this example). The fine network Nf is expected to have a stronger feature expression capability at high resolution, and a channel attention module (lower right module) is added to the above structure.
In this embodiment, the Nc model is based on the Resnet18 network, the full connection layer in the network is replaced with a convolutional layer, and the input of the Nc model is Xlevel 2. The Nc model is built by stacking convolution layers with convolution kernel 3 and step size 2 with the largest pooling layer of step size 2.
Further, the Nc model can output the original image as a vector of 1 × C by stacking the convolution layer and the pooling layer, where C is a component of the original imagefRepresenting the number of classes, in this example the tumor, i.e.Cc, is located as the targetfAnd (1) calculating the characteristic value of 1 × 1 by a sigmoid function to obtain the prediction probability of each image block.
As a preferred embodiment, the sigmoid function is:
Figure BDA0003477344520000081
wherein z represents the output of the convolutional layer,
Figure BDA0003477344520000082
representing the prediction probability values of the image block. And calculating the cross entropy loss by using the predicted value of the image block and the truth label.
As a preferred embodiment, the loss function is:
Figure BDA0003477344520000083
wherein the content of the first and second substances,
Figure BDA0003477344520000084
a prediction value representing an image block is predicted,
Figure BDA0003477344520000085
representing the true value of the image block. X is to belevel2And inputting the data and the truth label thereof into an Nc model for training, carrying out coarse network iteration for 50 times at preset iteration times, wherein the learning rate is 0.001, optimizing by adopting an Adam optimizer, and updating the parameters of the Nc model.
Further, the Nf model in the embodiment is constructed, and the input of the model is xlevel0。
The model constructs the Nf classification model by stacking convolution layers with convolution kernels of 3 step size 2 with the largest pooling layer of step size 2. Wherein, a channel attention module is added behind each Pooling layer to extract key features under different scales, and the feature graph after the maximum Pooling is processed by a global average Pooling layer (Global average Pooling) to obtain 1 x 1CeIn which C iseAnd for the number of the characteristic channels of the layer, obtaining the weight of each characteristic channel for two fully-connected layers and a sigmoid function through a threshold mechanism, wherein the larger the weight value is, the larger the contribution of the channel to the classification result is, multiplying the calculated weight value of each channel by the characteristic channel to obtain a characteristic diagram with the weight value, and the size and the number of the channels of the characteristic diagram are not changed in the step of calculation. X of willlevel0Inputting data and truth labels into an Nf model for training, carrying out fine network iteration for 70 times under a preset iteration time, wherein the learning rate is 0.001, carrying out exponential decay on the learning rate, and adopting a random gradientAnd optimizing the descending method, and training the Nf model.
S300, entering the model prediction stage of the embodiment, and the testing process is shown in FIG. 3. After the training of the coarse network Nf and the fine network Nc is completed, predicting a WSI test set:
1. the scan is first performed through the coarse network at level2 level (low resolution wsi), at which point the scan sliding window size is 2883, and small probability blocks are output. And after the low-resolution wsi full image is scanned, all the small-probability fast images are spliced. A coarse probability map is obtained.
2. And further classifying all points with the area position being larger than 0.5 on the rough probability map, and finally obtaining a fine positioning probability map.
The Nc and Nf models can handle images of different magnifications. An image of a full-field pathological section × 10 magnification is obtained by a down-sampling method, and sampling is performed by a sliding window method based on the magnification as an input of the Nc model.
Specifically, in the prediction stage, since the Nc model is a full convolution structure, when the sliding window size is S W2883 probability block Inf after Ncout171, the step size of the sliding window is Sd＝16(24)，
The formula is as follows:
SW＝TIn+(Infout-Tout)×Sd
wherein T isInFor the size of the model input in the training phase, ToutThe size of the model output in the training stage, the Inf to be obtainedoutSplicing according to the sequence, generating a rough probability heat map with the size of the down-sampling proportion of the full-field pathological section image, traversing each pixel value on the heat map, and taking the size of the pixel value on the probability map as the size of the pixel value through a preset threshold t when t is>0.5 screening and recording the corresponding position coordinates ((X) of the pixel pointsln,Yln) In this embodiment, n is 4, i is the number of steps of low magnification, and i is 2. The coordinate position relation between the coordinate of the rough probability map and the coordinate of the full-field pathological section of the original size is as follows:
(Xl0,Yl0)＝(Xln×2n+i,Yln×2n+i)
and inputting the image block corresponding to the high-resolution image block at each required pixel position on the rough probability heat map into an Nf model for further judgment, updating the predicted category result in the rough probability heat map, and generating a final fine probability heat map.
The embodiment utilizes a multi-resolution network to reduce the resolution of the full-field pathological section with the original size, and utilizes the advantages of the full convolution network to extract tumor features in a low-resolution image by fusing multi-scale features, so that the Nc model has higher recall rate for tumors, and positive pixels output by the Nf model are further judged on the basis through the Nf model. In the embodiment, an improved full convolution network is utilized to adopt a process of firstly thickening and then thinning, so that the detection speed can be greatly improved while the positioning precision is ensured, and certain pathological detection requirements are met.
In order to verify the rapid positioning capability of the algorithm, the speed and the positioning accuracy of the algorithm need to be compared, a 1080Ti single card with a video memory of 11GB is used as a test, and the test is carried out based on a public data set Cameleon 16, wherein the test set comprises 128 effective full-field pathological section.
The speed measurement standard is the average time of the model for predicting 128 full-field pathological section images; and calculating the total number of false positives of the test set and the sensitivity in the threshold interval according to the prediction result, wherein the calculation formula is as follows:
Total_FPs＝FPs_thres/slide_num
Total_sensitivity＝TPs_thres/all_tumor_num
where slide _ num is the number of full field pathology slice images tested, all _ tumor _ num is the number of all tumor regions in the test slice. FPs _ thres is the number of false positives in the test full-field pathology section, and TPs _ thre is the number of true positives in the test full-field pathology section.
Further, 6 thresholds [0.25,0.5,1,2,4,8] are selected within the threshold interval, and the mean of the 6 sub-threshold sensitivities (Total _ sensitivity) is calculated as the FROC value of the model. As shown in FIG. 4, the results of the visualization of two WSIs in the test set for the present embodiment
TABLE 1 comparison of mean velocity versus FROC for different algorithms on the camelyon16 test set, with the best results shown bolded
Model (model) Speed of rotation FROC
Human - 0.73
Scan-net 28.32mins 0.7200
NCRF 44.12mins 0.8018
C2F 7.567mins 0.7700
The comparison algorithms in this embodiment are performed in the same experimental environment, and the training of the model and the selection of the model are performed by generating a training set (unified data) and a verification set. As can be seen from table 1, the rapid positioning method for full-field pathological section (C2F) provided by the present invention has a significant advantage in speed compared to other positioning algorithms, and the positioning accuracy FROC differs by 3% from the NCRF algorithm, but can exceed the positioning accuracy of human pathologists, and can provide faster and more robust auxiliary results for such labor intensive human work.
Based on the same concept of the foregoing embodiments, an embodiment of the present invention further provides a terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor, when executing the program, is configured to perform the method for rapidly locating a tumor based on a multi-resolution full-field pathological section image in any one of the foregoing embodiments.
Based on the same concept of the foregoing embodiments, an embodiment of the present invention further provides a computer-readable storage medium, where at least one instruction, at least one program, a code set, or a set of instructions is stored in the storage medium, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded by a processor and executes any one of the foregoing embodiments of the method for rapidly locating a tumor based on a multi-resolution full-field pathological section image.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The above-described preferred features may be used in any combination without conflict with each other.

Claims (10)

1. A full-field pathological section image tumor rapid positioning method based on multi-resolution is characterized by comprising the following steps: constructing an image block training set;
obtaining two classification models by utilizing the image block training set;
and (4) predicting the full-field pathological section by using the two classification models in sequence to obtain a positioning result.
2. The method for rapidly positioning tumor based on multi-resolution full-field pathological section image according to claim 1, wherein the constructing of the image block training set comprises:
reading a full-visual-field pathological section, and extracting the foreground and the background of the full-visual-field pathological section by a lawyer method;
and constructing image blocks of the full-field pathological sections with different resolutions in the foreground and background areas, obtaining image block data sets with different resolutions, and obtaining the category labels of the image block data sets.
3. The multi-resolution full-field pathological section image tumor rapid positioning method according to claim 1, wherein the constructed image block training set comprises a set of low resolution data sets and a set of high resolution data sets; the two classification models comprise a low-resolution model and a high-resolution model, and the two classification models are trained to have characteristic parameters of tumor and normal tissues under different magnifications.
4. The method for rapidly positioning tumors based on multi-resolution full-field pathological section images as claimed in claim 3, wherein training the low-resolution model comprises:
training a full convolutional classification network using the low resolution dataset, the full convolutional classification network having Resnet18 as a backbone network;
replacing the full connection layer of the full convolution type network with a full convolution layer, and controlling the number of convolution channels of the last layer to be the type number;
weighting positive samples in a low-resolution data set, and inputting the full-volume integral network training;
fusing the feature graphs of different levels (stages) in different trained convolutional networks to obtain multi-scale features;
Out＝concat(s1+s2+s3)
where (s1, s2, s3) are feature output maps at different levels in the Resnet18 network, concat is tandem fusion, and Out is a multi-scale fusion feature.
5. The method for rapidly positioning tumors based on multi-resolution full-field pathological section images as claimed in claim 3, wherein training the high-resolution model comprises:
training a convolutional classification network using the high resolution dataset, the convolutional classification network having Resnet18 as a backbone network;
adding an SE module in each basic module in the Resnet 18;
calculating the weight of each stage output channel;
multiplying the weight with a feature graph in a feature channel, and screening out features with large contribution to categories based on category contribution as stage output features; screening stage output characteristics based on the category contribution;
and performing multi-scale feature fusion on the stage output features.
6. The method for rapidly positioning tumor based on multi-resolution full-field pathological section image according to claim 3, wherein the predicting the full-field pathological section by using two classification models in sequence to obtain the positioning result comprises:
predicting the full-field pathological section by using the low-resolution model to obtain a rough positioning probability map;
obtaining abnormal pixel points in the rough positioning probability map;
and judging the abnormal pixel points through the high-resolution model, obtaining the probability value of the pixel position again, and obtaining a final positioning probability map.
7. The method for rapidly positioning tumor based on multi-resolution full-field pathological section image according to claim 6, wherein the predicting the full-field pathological section using the low resolution model to obtain the rough positioning probability map comprises:
the original full-view pathological section is Slevel_0Acquiring a low-magnification full-field pathological section image S with preset magnification by a bilinear interpolation methodlevel_i(i＝1,2,3)；
Through a sliding window S of predetermined sizeWAnd a fixed step length SSIn low magnification full-field pathological sectionSliding and sampling;
inputting the sliding samples into a low resolution model, wherein the down-sampling factor of the low resolution model is 2n；
Splicing the calculated results of the sliding window samples to generate an original size 1/2n+iMagnitude coarse positioning probability map.
8. The method for rapidly locating tumor based on multi-resolution full-field pathological section image according to claim 7, wherein the obtaining abnormal pixel points in the rough location probability map comprises: if the probability value of the pixel point is larger than the threshold value, determining the pixel point as an abnormal pixel point;
the method for determining the abnormal pixel point by using the high-resolution model to obtain the probability value of the pixel position again to obtain the final positioning probability map comprises the following steps:
the relationship between the pixel coordinate on the rough probability map and the pixel position of the point on the full-view pathological section in the original size is as follows:
(Xl0，Yl0)＝(Xln×2n+i，Yln×2n+i)
wherein (X)ln,Yln) Is an abnormal coordinate point on the coarse probability chart, (X)l0,Yl0) Is a coordinate point on the full-field slice under the maximum magnification;
with ((X)l0,Yl0) Extracting an image block with a preset size as a sampling center, and inputting the image block into a high-resolution model;
the down-sampling factor of the high resolution model is 2n+iUpdate to obtain ((X)ln,Yln) The category of the pixel point.
9. A terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor when executing the program is operable to execute the method for rapidly locating a tumor based on a full-field pathological section image with multi-resolution according to any one of claims 1-8.
10. A computer-readable storage medium, wherein at least one instruction, at least one program, a set of codes, or a set of instructions is stored in the storage medium, and the at least one instruction, the at least one program, the set of codes, or the set of instructions is loaded by a processor and executed to perform the method for rapidly locating a tumor based on a multi-resolution full-field pathological section image according to any one of claims 1 to 8.
CN202210058387.0A 2022-01-19 2022-01-19 Multi-resolution-based full-field pathological section image tumor rapid positioning method Pending CN114445356A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210058387.0A CN114445356A (en) 2022-01-19 2022-01-19 Multi-resolution-based full-field pathological section image tumor rapid positioning method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210058387.0A CN114445356A (en) 2022-01-19 2022-01-19 Multi-resolution-based full-field pathological section image tumor rapid positioning method

Publications (1)

Publication Number Publication Date
CN114445356A true CN114445356A (en) 2022-05-06

Family

ID=81368252

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210058387.0A Pending CN114445356A (en) 2022-01-19 2022-01-19 Multi-resolution-based full-field pathological section image tumor rapid positioning method

Country Status (1)

Country Link
CN (1) CN114445356A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115063592A (en) * 2022-08-16 2022-09-16 之江实验室 Multi-scale-based full-scanning pathological feature fusion extraction method and system
CN116741347A (en) * 2023-05-12 2023-09-12 中山大学附属第一医院 Pathological image patches extraction and deep learning modeling method
WO2024051655A1 (en) * 2022-09-06 2024-03-14 抖音视界有限公司 Method and apparatus for processing histopathological whole-slide image, and medium and electronic device

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115063592A (en) * 2022-08-16 2022-09-16 之江实验室 Multi-scale-based full-scanning pathological feature fusion extraction method and system
CN115063592B (en) * 2022-08-16 2022-12-06 之江实验室 Multi-scale-based full-scanning pathological feature fusion extraction method and system
WO2024051655A1 (en) * 2022-09-06 2024-03-14 抖音视界有限公司 Method and apparatus for processing histopathological whole-slide image, and medium and electronic device
CN116741347A (en) * 2023-05-12 2023-09-12 中山大学附属第一医院 Pathological image patches extraction and deep learning modeling method

Similar Documents

Publication Publication Date Title
CN111784671B (en) Pathological image focus region detection method based on multi-scale deep learning
CN111524137B (en) Cell identification counting method and device based on image identification and computer equipment
KR102108050B1 (en) Method for classifying breast cancer histology images through incremental boosting convolution networks and apparatus thereof
CN111798416B (en) Intelligent glomerulus detection method and system based on pathological image and deep learning
CN110021425B (en) Comparison detector, construction method thereof and cervical cancer cell detection method
CN111462042B (en) Cancer prognosis analysis method and system
CN114445356A (en) Multi-resolution-based full-field pathological section image tumor rapid positioning method
CN114758137B (en) Ultrasonic image segmentation method and device and computer readable storage medium
CN115909006B (en) Mammary tissue image classification method and system based on convolution transducer
CN112543934A (en) Method for determining degree of abnormality, corresponding computer readable medium and distributed cancer analysis system
CN111899259A (en) Prostate cancer tissue microarray classification method based on convolutional neural network
CN112348059A (en) Deep learning-based method and system for classifying multiple dyeing pathological images
CN115471448A (en) Artificial intelligence-based thymus tumor histopathology typing method and device
CN116884623B (en) Medical rehabilitation prediction system based on laser scanning imaging
CN115131630A (en) Model training method, microsatellite state prediction method, electronic device and storage medium
CN117015796A (en) Method for processing tissue images and system for processing tissue images
CN115210779A (en) Systematic characterization of objects in biological samples
CN115187530A (en) Method, device, terminal and medium for identifying ultrasonic automatic breast full-volume image
CN116579975A (en) Brain age prediction method and system of convolutional neural network
CN111680575A (en) Human epithelial cell staining and classifying device, equipment and storage medium
CN114332572B (en) Method for extracting breast lesion ultrasonic image multi-scale fusion characteristic parameters based on saliency map-guided hierarchical dense characteristic fusion network
CN114972202A (en) Ki67 pathological cell rapid detection and counting method based on lightweight neural network
CN116468690B (en) Subtype analysis system of invasive non-mucous lung adenocarcinoma based on deep learning
CN113705595A (en) Method, device and storage medium for predicting degree of abnormal cell metastasis
CN112488996A (en) Inhomogeneous three-dimensional esophageal cancer energy spectrum CT (computed tomography) weak supervision automatic labeling method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination