CN115690056A - Gastric cancer pathological image classification method and system based on HER2 gene detection - Google Patents
Gastric cancer pathological image classification method and system based on HER2 gene detection Download PDFInfo
- Publication number
- CN115690056A CN115690056A CN202211369435.4A CN202211369435A CN115690056A CN 115690056 A CN115690056 A CN 115690056A CN 202211369435 A CN202211369435 A CN 202211369435A CN 115690056 A CN115690056 A CN 115690056A
- Authority
- CN
- China
- Prior art keywords
- image
- gastric cancer
- pathological
- images
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000001575 pathological effect Effects 0.000 title claims abstract description 113
- 208000005718 Stomach Neoplasms Diseases 0.000 title claims abstract description 101
- 206010017758 gastric cancer Diseases 0.000 title claims abstract description 101
- 201000011549 stomach cancer Diseases 0.000 title claims abstract description 101
- 238000000034 method Methods 0.000 title claims abstract description 60
- 101150054472 HER2 gene Proteins 0.000 title claims abstract description 44
- 108700020302 erbB-2 Genes Proteins 0.000 title claims abstract description 44
- 238000001514 detection method Methods 0.000 title claims abstract description 33
- 238000012549 training Methods 0.000 claims abstract description 57
- 210000002784 stomach Anatomy 0.000 claims abstract description 36
- 238000013145 classification model Methods 0.000 claims abstract description 30
- 238000010606 normalization Methods 0.000 claims abstract description 21
- 238000000605 extraction Methods 0.000 claims abstract description 13
- 230000011218 segmentation Effects 0.000 claims abstract description 12
- 206010028980 Neoplasm Diseases 0.000 claims description 30
- 201000011510 cancer Diseases 0.000 claims description 25
- 230000006870 function Effects 0.000 claims description 25
- 238000005070 sampling Methods 0.000 claims description 21
- 230000014509 gene expression Effects 0.000 claims description 17
- 238000013527 convolutional neural network Methods 0.000 claims description 15
- 230000004083 survival effect Effects 0.000 claims description 13
- 210000004027 cell Anatomy 0.000 claims description 9
- 230000007170 pathology Effects 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 7
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 210000004907 gland Anatomy 0.000 claims description 4
- 238000003364 immunohistochemistry Methods 0.000 claims description 4
- 210000002570 interstitial cell Anatomy 0.000 claims description 4
- 238000011176 pooling Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 230000008520 organization Effects 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 238000007447 staining method Methods 0.000 claims description 2
- 229960000575 trastuzumab Drugs 0.000 abstract description 9
- 230000000694 effects Effects 0.000 abstract description 8
- 238000012545 processing Methods 0.000 abstract description 7
- 238000002560 therapeutic procedure Methods 0.000 abstract description 7
- 101001012157 Homo sapiens Receptor tyrosine-protein kinase erbB-2 Proteins 0.000 description 16
- 102100030086 Receptor tyrosine-protein kinase erbB-2 Human genes 0.000 description 16
- 230000008569 process Effects 0.000 description 11
- 230000007246 mechanism Effects 0.000 description 8
- 238000003860 storage Methods 0.000 description 5
- 238000010186 staining Methods 0.000 description 4
- 238000011460 HER2-targeted therapy Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 239000003814 drug Substances 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000002626 targeted therapy Methods 0.000 description 3
- 230000001225 therapeutic effect Effects 0.000 description 3
- 238000002512 chemotherapy Methods 0.000 description 2
- 230000034994 death Effects 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 2
- 229940079593 drug Drugs 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 230000002496 gastric effect Effects 0.000 description 2
- 238000007901 in situ hybridization Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 102000010400 1-phosphatidylinositol-3-kinase activity proteins Human genes 0.000 description 1
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 206010061818 Disease progression Diseases 0.000 description 1
- 206010062878 Gastrooesophageal cancer Diseases 0.000 description 1
- 230000004655 Hippo pathway Effects 0.000 description 1
- 102000043136 MAP kinase family Human genes 0.000 description 1
- 108091054455 MAP kinase family Proteins 0.000 description 1
- 206010027476 Metastases Diseases 0.000 description 1
- 108091007960 PI3Ks Proteins 0.000 description 1
- 102000013530 TOR Serine-Threonine Kinases Human genes 0.000 description 1
- 108010065917 TOR Serine-Threonine Kinases Proteins 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000006907 apoptotic process Effects 0.000 description 1
- 230000004900 autophagic degradation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000005750 disease progression Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 201000006974 gastroesophageal cancer Diseases 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 238000012151 immunohistochemical method Methods 0.000 description 1
- 238000001727 in vivo Methods 0.000 description 1
- 230000008595 infiltration Effects 0.000 description 1
- 238000001764 infiltration Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000009401 metastasis Effects 0.000 description 1
- 230000003278 mimic effect Effects 0.000 description 1
- 229940125645 monoclonal antibody drug Drugs 0.000 description 1
- 230000009745 pathological pathway Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000004393 prognosis Methods 0.000 description 1
- 230000019491 signal transduction Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention belongs to the technical field of computers and image processing, and particularly relates to a gastric cancer pathological image classification method and system based on HER2 gene detection, aiming at solving the problem that in the prior art, gastric cancer pathological images cannot be accurately classified, so that doctors cannot be effectively assisted in predicting the curative effect of trastuzumab therapy on patients. The invention comprises the following steps: acquiring a stomach digital pathological section image of a gastric cancer patient; performing region extraction and segmentation operation to obtain a plurality of image blocks with set pixel sizes and set tissue types; performing data enhancement normalization and data division on the image blocks to obtain training image packets; performing network iterative training through a training image packet to obtain a gastric cancer pathological image classification model; and classifying the digital pathological section images of the stomach of the patient, which are acquired in real time through the gastric cancer pathological image classification model. The method realizes accurate classification of the pathological images of the gastric cancer, thereby assisting doctors to effectively predict the curative effect of trastuzumab therapy of patients.
Description
Technical Field
The invention belongs to the technical field of computers and image processing, and particularly relates to a gastric cancer pathological image classification method and system based on HER2 gene detection.
Background
Gastric cancer is a common malignant tumor, and several important molecular pathogenic pathways known to date for gastric cancer include: PI3K/AKT/mTOR, MAPK signaling pathway ERK, JNK, p38, hippo pathways and the like are closely related to apoptosis, autophagy, tumor size, infiltration depth and distant metastasis of gastric cancer cells. However, even though the targeted therapy of gastric cancer has considerable effect in basic research, it is difficult to switch it to clinical practice, and currently, only human epidermal growth factor receptor 2 (herr 2) is approved for clinical targeted therapy of gastric cancer [1].
Trastuzumab is a specific anti-HER 2 targeted drug, and compared with simple chemotherapy, the combined scheme prolongs the total survival period to 16 months, and obviously improves the average survival time of less than one year under the traditional chemotherapy of advanced gastric cancer. Several studies have shown that anti-HER 2 therapy (mainly monoclonal antibody drugs to HER2 protein) has very significant efficacy both ex vivo and in vivo models of gastric cancer.
However, in actual clinical experiments, the objective effective rate of trastuzumab is only 47.3%, and how to develop a rapid and accurate method to predict the therapeutic effect of trastuzumab before treatment has important clinical value. The HER2 expression in the gastric cancer has higher heterogeneity, different patients also show heterogeneity of different degrees in pathological pictures, and the therapeutic effect of trastuzumab is related to the heterogeneity of the gastric cancer HER2 expression. In patients receiving trastuzumab therapy, progression Free Survival (PFS) and Overall Survival (OS) were observed for HER2 expressing gastric cancer patients with better homogeneity than for gastric cancer patients with higher heterogeneity [2] [3].
Clinically, methods for assessing gastric cancer HER2 expression heterogeneity are mainly assessed by pathological section immunohistochemistry and in situ hybridization methods. Immunohistochemistry may partially assess HER2 expression status, while in situ hybridization is the gold standard for HER2 detection. However, even if HER2 is accurately evaluated, it is difficult to accurately and effectively classify gastric cancer pathological images of patients, and thus doctors cannot be assisted in effectively predicting the efficacy of trastuzumab therapy for patients.
The following documents are background information related to the present invention:
[1] zhang Rui Hao reviewed, zhang Ming Ju, advanced gastric cancer anti-HER 2 therapy research progress [ J ] Proc. Med. Student of medicine, 2022 (035-002).
[2]Gravalos,C.,&Jimeno,A.(2008).HER2 in gastric cancer:a new prognostic factor and a novel therapeutic target.Annals of Oncology,19(9),1523-1529.
[3]Huemer,F.,Weiss,L.,Regitnig,P.,Winder,T.,Hartmann,B.,Thaler,J.,...&
E.(2020).Local and Central Evaluation of HER2 Positivity and Clinical Outcome in Advanced Gastric and Gastroesophageal Cancer—Results from the AGMT GASTRIC-5Registry.Journal of Clinical Medicine,9(4),935.
Disclosure of Invention
In order to solve the above problems in the prior art, that is, the prior art cannot accurately classify gastric cancer pathological images, and thus cannot effectively assist doctors in predicting the curative effect of trastuzumab therapy for patients, the present invention provides a gastric cancer pathological image classification method based on HER2 gene detection, which includes:
step S10, acquiring a stomach digital pathological section image of a gastric cancer patient; the slice images comprise the expression profile of cancer tissue expressing HER2 gene;
step S20, carrying out region extraction and segmentation operation on the slice image to obtain a plurality of image blocks with set pixel sizes and set tissue types;
step S30, performing data enhancement normalization and data division on the plurality of image blocks with the set pixel sizes and the set tissue types to obtain a training image packet;
step S40, carrying out iterative training on the constructed ResNet convolutional neural network through the training image packet to obtain a gastric cancer pathological image classification model;
and S50, classifying the stomach digital pathological section images of the patient based on the acquired real-time stomach digital pathological section images through a stomach cancer pathological image classification model to obtain image classification results.
In some preferred embodiments, the stomach digital pathological section image of the gastric cancer patient is an image stained by hematoxylin-eosin staining;
the expression condition of the cancer tissue expressing HER2 gene is obtained by means of immunohistochemistry, and comprises negative information and positive information.
In some preferred embodiments, step S20 includes:
step S21, connecting the cancer region boundary points of the annotation file to obtain a cancer region boundary, and converting the cancer region boundary into an image mask based on the cancer region boundary;
step S22, down-sampling the slice image and the image mask to a set level;
step S23, extracting the area of the downsampled slice image through a downsampled image mask, and dividing the extracted sub-area into image blocks with set pixel size;
and step S24, respectively calculating the proportion of the overlapping area of each image block and the ROI area in the image block, extracting the image blocks larger than a set threshold value, and obtaining a plurality of image blocks with set pixel sizes and set tissue types.
In some preferred embodiments, the down-sampling is performed by:
h and W are respectively the height and width of the slice image and the image mask after the down-sampling, height and width are respectively the height and width of the slice image and the image mask before the down-sampling, and level is the down-sampling scale.
In some preferred embodiments, the data is normalized by:
horizontally overturning, vertically overturning and randomly rotating each image block in a plurality of image blocks with set pixel sizes and containing set organization types to obtain an enhanced image block set;
and normalizing the brightness and the contrast of each image block in the enhanced image block set to obtain an enhanced normalized image block set.
In some preferred embodiments, the data is divided by:
randomly dividing the enhanced normalized image block set to obtain an image packet set consisting of a set number of image blocks;
judging the label of each image block, and executing the following steps:
if an image packet contains at least one image block with a positive label, the image packet is marked as a positive multi-example packet; otherwise, the image packet is marked as a negative class multiple instance packet.
In some preferred embodiments, the method for determining the label of each image block includes:
extracting a feature map of the image block through a ResNet convolution neural network;
and performing maximum pooling operation on the feature map, and calculating and acquiring the probability of the image block being a positive type and a negative type through a softmax normalization function.
In some preferred embodiments, the gastric cancer pathology image classification model is trained by:
b10, extracting picture lines and cell forms of image blocks in the training image packet, and removing interstitial cells and gland cells of the stomach to obtain a preprocessed training image packet;
step B20, a ResNet convolution neural network is constructed, and a single-scale multi-example learning network of a prediction bag label is trained on the basis of a preprocessed training image packet according to the down-sampling scale of each stomach digital pathological section image;
and B30, performing multi-instance learning on the multi-scale-level preprocessing training image packet based on the weight of the single-scale multi-instance learning network to obtain a gastric cancer pathological image classification model.
In some preferred embodiments, the ResNet convolutional neural network comprises a set number of residual connection blocks;
the residual connecting block comprises a 3 × 3 convolutional layer, a batch normalization layer, a ReLU activation function, a 3 × 3 convolutional layer, a batch normalization layer and a ReLU activation function which are connected in sequence.
In another aspect of the present invention, a gastric cancer pathological image classification system based on HER2 gene detection is provided, which includes:
the data acquisition module is configured to acquire a stomach digital pathological section image of a gastric cancer patient; the slice images comprise the expression profile of cancer tissue expressing HER2 gene;
the region extraction and segmentation module is configured to perform region extraction and segmentation operation on the slice image to obtain a plurality of image blocks with set pixel sizes and set tissue types;
the data sub-packaging module is configured to perform data enhancement normalization and data division on the plurality of image blocks with the set pixel sizes and the set tissue types to obtain a training image package;
the model training module is configured to perform iterative training on the constructed ResNet convolutional neural network through the training image packet to obtain a gastric cancer pathological image classification model;
and the classification module is configured to classify the stomach digital pathological section images of the patient based on the real-time acquired stomach digital pathological section images through the stomach cancer pathological image classification model to acquire an image classification result.
The invention has the beneficial effects that:
(1) The method for classifying the gastric cancer pathological images based on HER2 gene detection can analyze H & E dyed gastric cancer pathological sections, give information of HER2 gene expression and prediction information of survival time of patients, and accurately classify the gastric cancer digital pathological section images of the patients at each stage, so that doctors can be effectively assisted in judging pertinence of HER2 targeted therapy, prognosis of a part of patients needing targeted therapy is improved, and meaningless HER2 targeted therapy of a part of patients is avoided.
(2) According to the method for classifying the gastric cancer pathological images based on HER2 gene detection, in model training, a single-scale multi-example learning network for predicting bag labels is trained, and then multi-example learning is performed on the basis of the single-scale multi-example learning network, so that the performance of the trained model is improved, and the accuracy of classification of the gastric cancer pathological images is further improved.
(3) According to the method for classifying the gastric cancer pathological images based on HER2 gene detection, the attention mechanism is added in the model, and can help the model to give different weights to each input part, more key and important information is extracted, so that the model can be judged more accurately, meanwhile, more expenses are not brought to calculation and storage of the model, and the accuracy of classification of the gastric cancer pathological images is further improved.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is a schematic flow chart of the method for classifying pathological images of gastric cancer based on HER2 gene detection according to the present invention;
fig. 2 is a schematic diagram of a multi-scale pathological image bag label prediction process of the gastric cancer pathological image classification method based on HER2 gene detection according to the present invention.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not to be construed as limiting the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
The invention provides a gastric cancer pathological image classification method based on HER2 gene detection, and an accurate and effective gastric cancer pathological image classification method is developed, so that an assistant doctor can accurately, stably and conveniently predict the curative effect of anti-HER 2 targeted therapy, and the method has important clinical value and is a clinical problem to be solved urgently.
The invention discloses a gastric cancer pathological image classification method based on HER2 gene detection, which comprises the following steps:
step S10, acquiring a stomach digital pathological section image of a gastric cancer patient; the slice images comprise the expression profile of cancer tissue expressing HER2 gene;
step S20, carrying out region extraction and segmentation operation on the slice image to obtain a plurality of image blocks with set pixel sizes and set tissue types;
step S30, performing data enhancement normalization and data division on the plurality of image blocks with the set pixel sizes and the set tissue types to obtain a training image packet;
step S40, carrying out iterative training on the constructed ResNet convolutional neural network through the training image packet to obtain a gastric cancer pathological image classification model;
and S50, classifying the stomach digital pathological section images of the patient based on the acquired real-time stomach digital pathological section images through a stomach cancer pathological image classification model to obtain image classification results.
In order to more clearly describe the method for classifying pathological images of gastric cancer based on HER2 gene detection according to the present invention, the following will describe each step in the embodiment of the present invention in detail with reference to fig. 1.
The method for classifying pathological images of gastric cancer based on HER2 gene detection according to the first embodiment of the present invention includes steps S10 to S50, and each step is described in detail as follows:
step S10, acquiring a stomach digital pathological section image of a gastric cancer patient; the slice images include the expression of HER2 gene by cancer tissues.
The stomach digital pathological section image of the gastric cancer patient is an image stained by a hematoxylin-eosin staining method, and the expression condition of the cancer tissue expression HER2 gene is obtained by an immunohistochemical method and comprises negative information and positive information.
In one embodiment of the invention, the data set is from 163 WSI (digital pathological section image) pictures (H & E staining, hematoxylin-eosin staining) collected from oncology hospitals, and specifically contains four pieces of information for evaluation: ORR (objective remission rate), DCR (disease control rate), PFS (progression free survival) and OS (overall survival) information.
And step S20, carrying out region extraction and segmentation operation on the slice image to obtain a plurality of image blocks with set pixel sizes and set tissue types.
The digital pathological section image is preprocessed, and the whole large image is divided into a plurality of small images with 512 x 512 pixels, and each small image is a specific tissue type. Since the WSI image is a relatively special medical image, the width and the height of the image are usually tens of thousands to hundreds of thousands of pixels, which results in a larger whole image, so that the whole image cannot enter network training after being simply processed like other medical images, but a tumor region in the image needs to be extracted by Mask, and then the image is cut into small images (Patch) which can be used for a deep learning model.
In step S20, firstly, a labeled image Mask (Mask) is used to extract a tumor region in an image, and then the image is segmented into small images that can be used for a deep learning model, which specifically includes the following steps:
and S21, connecting the cancer region boundary points of the annotation file to obtain the cancer region boundary, and converting the cancer region boundary into an image mask based on the cancer region boundary.
The annotation file is information of one dot, which is converted into a Mask (Mask) of an image. The annotation file for the image is in the file format of XML. The file stores the boundary of each category of tissue, the concrete data is composed of a series of points, each point is position information when a professional doctor clicks the image by a mouse, and the positions of the points form a polygon delineation which the area of each tissue type is delineated.
And S22, down-sampling the slice image and the image mask to a set level.
The original WSI image is too large and requires down-sampling of the image. In an embodiment of the present invention, a down-sampling method is shown in formula (1), where level represents a down-sampling scale. And (3) according to the formula, down-sampling the image and the corresponding annotation point to a corresponding level to complete further analysis:
wherein, H and W are respectively the height and width of the slice image and the image mask after the down-sampling, and height and width are respectively the height and width of the slice image and the image mask before the down-sampling.
Step S23 is to extract a region of the down-sampled slice image through the down-sampled image mask, and divide the extracted sub-region into image blocks of a set pixel size.
For the boundary delineation polygon extracted in step S21, the boundary of the contour may be extracted first, the maximum value of the four boundaries, i.e., the top, bottom, left, and right, according to the contour is extracted, and the extracted boundary is enclosed as a sub-region of the picture to be preprocessed (to obtain a local picture of the delineation region), and then the extracted picture is divided into patch-level images of 512 × 512 size.
And step S24, respectively calculating the proportion of the overlapping area of each image block and the ROI area in the image block, extracting the image blocks larger than a set threshold value, and obtaining a plurality of image blocks with set pixel sizes and set tissue types.
For each of the divided Patch-level small images, when the area of the image overlapped with the polygon ROI extracted in step S21 is calculated to be 75% or more of the area of the Patch-level small image, the Patch is stored as valid data in the divided Patch-level image.
And step S30, performing data enhancement normalization and data division on the plurality of image blocks with the set pixel sizes and the set tissue types to obtain a training image packet.
For the training image package obtained in step S20, HER2 negative (labeled as part of HER2=0 and HER = 1) and HER2 positive (labeled as part of HER2=2 and HER = 3), patients were randomly divided into training and test sets at a ratio of 8.
The data enhancement normalization method comprises the following steps:
horizontally overturning, vertically overturning and randomly rotating each image block in a plurality of image blocks with set pixel sizes and containing set organization types to obtain an enhanced image block set;
and normalizing the brightness and the contrast of each image block in the enhanced image block set to obtain the enhanced normalized image block set.
In one embodiment of the invention, in order to increase the number of data sets and avoid model overfitting, the data is flipped horizontally or vertically during processing of the data. In addition, random 45-degree rotation is also performed on the data set pictures.
For the data normalization operation, the data set is normalized by parameters such as brightness and contrast. For the three RGB color channels of the image, the mean values are normalized to [0.6209136,0.39992052,0.68346393], respectively (the three numerical values represent the three RGB channels in the image, the same below), and the variance values are normalized to [0.26443535,0.30418476,0.19353978], respectively.
In cancer pathology, a WSI often contains pathological tissues of HER2 negative and positive regions, since tumor and non-tumor regions are mixed. Thus, the present invention is a method for Multiple Instance Learning (MIL). In particular, the present invention organizes different patch pictures from one WSI into one or several packets. In the picture organizing process, similar pathological images can be organized into a package, so that the information such as image texture, staining and the like in the package is relatively similar. If a packet contains both positive and negative pathology pictures, then this packet will be marked as a positive case; if all negative pictures are marked as negative examples. In addition, when pathologists diagnose patients, they observe the slides at different scales. To mimic this, the present invention contemplates a patch having a plurality of different dimensions.
The data division method comprises the following steps:
randomly dividing the enhanced normalized image block set to obtain an image packet set consisting of a set number of image blocks;
judging the label of each image block, and executing:
if an image packet contains at least one image block with a positive label, the image packet is marked as a positive multi-example packet; otherwise, the image packet is marked as a negative class multiple instance packet.
Judging the label of each image block, wherein the method comprises the following steps of:
extracting a feature map of the image block through a ResNet convolution neural network;
and performing maximum pooling operation on the feature map, and calculating and acquiring the probability that the image block is a positive type and a negative type through a softmax normalization function.
And S40, performing iterative training on the constructed ResNet convolutional neural network through the training image packet to obtain a gastric cancer pathological image classification model.
The stomach cancer pathological image classification model comprises the following training methods:
and B10, extracting picture lines and cell forms of the image blocks in the training image packet, and removing gastric interstitial cells and gland cells to obtain a preprocessed training image packet.
Preprocessing a Patch image to be segmented, extracting information such as picture lines, cell morphology and the like, and removing stomach interstitial cells, gland cells and the like which have larger difference with tumor tissues.
And step B20, constructing a ResNet convolutional neural network, and training a single-scale multi-example learning network of a prediction bag label based on a preprocessed training image packet according to the down-sampling scale of each stomach digital pathological section image.
The convolutional neural network used in the invention is ResNet18, and the main structure in ResNet is a residual connecting block. Each residual block has 23 × 3 convolutional layers with the same number of output channels. Each convolutional layer is followed by a bulk normalization layer and a ReLU activation function. Then, by skipping these 2 convolution operations through the cross-layer direct path, the input is added directly before the final ReLU activation function, as shown in table 1:
TABLE 1
Before the model training, the model is pre-trained to improve the model effect.
Multi-example learning is performed for patch level images of single-scale WSI:
and training a single-scale multi-example learning network to predict the label of the bag according to each WSI down-sampling scale, wherein each bag only comprises a patch on the down-sampling scale s image, and optimizing the network to obtain the bag label prediction.
As shown in fig. 2, which is a schematic diagram of a multi-scale pathological image bag label prediction process of the gastric cancer pathological image classification method based on HER2 gene detection of the present invention, a training task of single-scale multi-instance learning is expressed as a minimization problem shown in formula (2):
where this term of minimization is the loss function of the bag class label prediction, the loss function is simply defined as the cross entropy between the true class label and the predicted class label. Here, with the Attention mechanism, the final bag class label is predicted by using only the larger example of the Attention.
An Attention Mechanism (Attention Mechanism) is a data processing method in machine learning, and learns the characteristic that when people observe things, people can extract global features from some more important local features. The attention mechanism can help the model to give different weights to each input part, more key and important information is extracted, the model can be judged more accurately, and meanwhile, larger expenses cannot be brought to calculation and storage of the model. By utilizing an attention mechanism, deep learning can be more targeted when the target is observed, and the target identification and classification precision is improved.
In step B21, the 3-channel image obtained by processing the patch obtained from a certain gastric cancer pathological section image into 224 × 3 is input to the ResNet18 module, and after 4 residual connection blocks (including the batch normalization layer, the convolution layer, and the activation function) are passed, a feature image Y1 with a size of 7 × 512 is extracted.
And B22, performing maximum pooling operation on the feature map Y1, and calculating the probability of the picture as a positive class and a negative class through a softmax normalization function.
And step B23, after all the pictures in a packet are processed through the steps B21-B22, obtaining the label output P (negative or positive) of the packet through an Attention Mechanism (Attention Mechanism) according to different weights of each picture Attention in the packet.
And step B24, calculating the cross entropy loss between the output P and the gastric cancer pathological image package real label through a cross entropy calculation formula, reversely transmitting the cross entropy loss to the convolutional neural network through a gradient descent algorithm, and updating the network parameters of the convolutional neural network and the Attention network.
And B30, performing multi-instance learning on the multi-scale-level preprocessing training image packet based on the weight of the single-scale multi-instance learning network to obtain a gastric cancer pathological image classification model.
Multi-instance learning is performed on the multi-scale level patch images:
and (5) all pathological sections with different scales are contained in a bag, and the trained weight is further finely adjusted. And training a multi-scale network to predict bag labels with different scales, wherein each bag comprises a patch with different scales, and optimizing the network to obtain the bag label prediction.
The multi-scale pathological image bag label prediction process is expressed as a function shown in an equation (3):
wherein the feature extraction set
s∈[S]The method is characterized in that the method is preliminarily trained in the training process of the single-scale pathological image, and the parameters of the previous step are directly used.
Parameter set
Is expressed as a minimization problem as shown in equation (4):
and S50, classifying the stomach digital pathological section images of the patient based on the acquired real-time stomach digital pathological section images through a stomach cancer pathological image classification model to obtain image classification results.
After the classification result of the digital pathological section image of the stomach of the patient is obtained, the ORR label (CR) or Partial Response (PR) of the patient can be combined with the classification result to predict.
Predicting the objective remission rate ORR:
the last full-link layer of the gastric cancer pathological image classification model is changed into prediction of output ORR, and the optimizer adopts a cross entropy Loss function as a Loss function, as shown in formula (5):
where y represents the patient's true ORR label,
representing the predicted ORR signature.
Predictions were made for progression free survival PFS (time from the beginning of patient death or data loss observed):
the last full-link layer of the stomach cancer pathological image classification model is changed to output a numerical value, and the optimizer adopts a C-index to calculate a loss function, as shown in the formula (6):
where T represents the survival time of two patients i, j selected at random, l and
representing the values predicted at the real and model, respectively, the value of the function is 1 if the survival time of patient i is longer than that of patient j, and 0 otherwise.
After the model is evaluated and considered, the loss function of the model is further adjusted, the last full-connection layer of the gastric cancer pathological image classification model is changed to output a numerical value for outputting the predicted progression-free life cycle PFS, and an optimizer and a regularization term are unchanged as shown in a formula (7):
wherein N is E＝1 Representing the number of patients who had a recorded clinical event (in the present invention patient death or disease progression): e i =1 represents each specific case,
representing network by l 2 Regularization, λ being a parameter of the regularization,
on behalf of the output result of the network,
refers to a collection of patients that survive time t and will die or disease progress in the future.
Although the foregoing embodiments have described the steps in the foregoing sequence, those skilled in the art will understand that, in order to achieve the effect of the present embodiment, different steps are not necessarily performed in such a sequence, and may be performed simultaneously (in parallel) or in an inverse sequence, and these simple variations are within the scope of the present invention.
A gastric cancer pathological image classification system based on HER2 gene detection according to a second embodiment of the present invention includes:
the data acquisition module is configured to acquire a stomach digital pathological section image of a gastric cancer patient; the slice images comprise the expression profile of cancer tissue expressing HER2 gene;
the region extraction and segmentation module is configured to perform region extraction and segmentation operation on the slice image to obtain a plurality of image blocks with set pixel sizes and set tissue types;
the data sub-packaging module is configured to perform data enhancement normalization and data division on the plurality of image blocks with set pixel sizes and set tissue types to obtain a training image package;
the model training module is configured to carry out iterative training on the constructed ResNet convolutional neural network through the training image packet to obtain a gastric cancer pathological image classification model;
and the classification module is configured to classify the stomach digital pathological section images of the patient based on the real-time acquired stomach digital pathological section images through the stomach cancer pathological image classification model to obtain image classification results.
It can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working process and related description of the system described above may refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.
It should be noted that, the gastric cancer pathology image classification system based on HER2 gene detection provided in the above embodiment is only illustrated by the division of the above functional modules, and in practical applications, the above functions may be allocated to different functional modules according to needs, that is, the modules or steps in the embodiment of the present invention are further decomposed or combined, for example, the modules in the above embodiment may be combined into one module, or may be further split into a plurality of sub-modules, so as to complete all or part of the above described functions. Names of the modules and steps related in the embodiments of the present invention are only for distinguishing the modules or steps, and are not to be construed as unduly limiting the present invention.
An electronic apparatus according to a third embodiment of the present invention includes:
at least one processor; and
a memory communicatively coupled to at least one of the processors; wherein the content of the first and second substances,
the memory stores instructions executable by the processor for execution by the processor to implement the above-described method for HER2 gene detection-based classification of gastric cancer pathology images.
A computer-readable storage medium of a fourth embodiment of the present invention stores computer instructions for execution by the computer to implement the above-mentioned method for classifying gastric cancer pathology images based on HER2 gene detection.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes and related descriptions of the storage device and the processing device described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
Those of skill in the art would appreciate that the various illustrative modules, method steps, and modules described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that programs corresponding to the software modules, method steps may be located in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. To clearly illustrate this interchangeability of electronic hardware and software, various illustrative components and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as electronic hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The intelligent gastric cancer curative effect prediction equipment comprises pathological image acquisition equipment, prediction equipment and display equipment;
the pathological image acquisition equipment is used for acquiring a stomach digital pathological section image of a gastric cancer patient; the slice images include the expression of HER2 gene by cancer tissues.
The pathological image collecting device can be a common camera and a scanner, can also be a camera/a video camera and an image collecting card, and can also be a microscopic digital camera or a microscopic scanner, and the like, which are not detailed in detail herein.
The prediction device comprises an image processing module, a model training module, an image classification module and a prediction module:
the image processing module is used for carrying out region extraction and segmentation operation on the slice image to obtain a plurality of image blocks with set pixel sizes and set tissue types, and carrying out data enhancement normalization and data division on the plurality of image blocks with set pixel sizes and set tissue types to obtain a training image packet;
the model training module is used for carrying out iterative training on the constructed ResNet convolutional neural network through the training image packet to obtain a gastric cancer pathological image classification model;
the image classification module is used for classifying the stomach digital pathological section images of the patient based on the real-time acquired stomach digital pathological section images through a stomach cancer pathological image classification model to acquire an image classification result;
the prediction module is used for predicting an objective remission rate ORR and/or a progression-free survival time PFS based on the gastric cancer pathology image classification model and the image classification result.
ORR includes Complete Response (CR), partial Response (PR).
The display module is used for displaying the prediction result output by the prediction equipment.
The terms "first," "second," and the like are used for distinguishing between similar elements and not necessarily for describing or implying a particular order or sequence.
The terms "comprises," "comprising," or any other similar term are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is apparent to those skilled in the art that the scope of the present invention is not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can be within the protection scope of the invention.
Claims (12)
1. A gastric cancer pathological image classification method based on HER2 gene detection is characterized by comprising the following steps:
step S10, acquiring a stomach digital pathological section image of a gastric cancer patient; the slice images comprise the expression profile of cancer tissue expressing HER2 gene;
step S20, carrying out region extraction and segmentation operation on the slice image to obtain a plurality of image blocks with set pixel sizes and set tissue types;
step S30, performing data enhancement normalization and data division on the plurality of image blocks with the set pixel sizes and the set tissue types to obtain a training image packet;
s40, performing iterative training on the constructed ResNet convolutional neural network through the training image packet to obtain a gastric cancer pathological image classification model;
and S50, classifying the stomach digital pathological section images of the patient based on the acquired real-time stomach digital pathological section images through a stomach cancer pathological image classification model to obtain image classification results.
2. The method for classifying pathological images of gastric cancer based on HER2 gene detection according to claim 1, wherein the digital pathological section images of stomach of gastric cancer patient are stained by hematoxylin-eosin staining method;
the expression condition of the cancer tissue expressing HER2 gene is obtained by means of immunohistochemistry, and comprises negative information and positive information.
3. The method for classifying gastric cancer pathology images based on HER2 gene detection according to claim 1, wherein step S20 comprises:
step S21, connecting the cancer region boundary points of the annotation file to obtain a cancer region boundary, and converting the cancer region boundary into an image mask based on the cancer region boundary;
step S22, down-sampling the slice image and the image mask to a set level;
step S23, extracting the area of the downsampled slice image through a downsampled image mask, and dividing the extracted sub-area into image blocks with set pixel sizes;
and step S24, respectively calculating the proportion of the overlapping area of each image block and the ROI area in the image block, extracting the image blocks larger than a set threshold value, and obtaining a plurality of image blocks with set pixel sizes and set tissue types.
4. The method for classifying pathological images of gastric cancer based on HER2 gene detection according to claim 3, wherein the downsampling is performed by:
h and W are respectively the height and width of the slice image and the image mask after the down-sampling, height and width are respectively the height and width of the slice image and the image mask before the down-sampling, and level is the down-sampling scale.
5. The method for classifying pathological images of gastric cancer based on HER2 gene detection according to claim 1, wherein the data are normalized by enhancement by:
horizontally overturning, vertically overturning and randomly rotating each image block in a plurality of image blocks with set pixel sizes and containing set organization types to obtain an enhanced image block set;
and normalizing the brightness and the contrast of each image block in the enhanced image block set to obtain an enhanced normalized image block set.
6. The method for classifying gastric cancer pathology images based on HER2 gene detection according to claim 5, wherein said data is divided by:
randomly dividing the enhanced normalized image block set to obtain an image packet set consisting of a set number of image blocks;
judging the label of each image block, and executing:
if an image packet contains at least one image block with a positive label, the image packet is marked as a positive multi-example packet; otherwise, the image packet is marked as a negative class multi-instance packet.
7. The method for classifying pathological images of gastric cancer based on HER2 gene detection according to claim 6, wherein the label of each image block is determined by:
extracting a feature map of the image block through a ResNet convolution neural network;
and performing maximum pooling operation on the feature map, and calculating and acquiring the probability that the image block is a positive type and a negative type through a softmax normalization function.
8. The method for classifying gastric cancer pathological images based on HER2 gene detection according to claim 1, wherein the training method of the gastric cancer pathological image classification model is as follows:
b10, extracting picture lines and cell forms of image blocks in the training image packet, and removing interstitial cells and gland cells of the stomach to obtain a preprocessed training image packet;
step B20, constructing a ResNet convolutional neural network, and training a single-scale multi-example learning network of a prediction bag label based on a preprocessed training image packet aiming at the down-sampling scale of each stomach digital pathological section image;
and B30, performing multi-instance learning on the multi-scale-level preprocessing training image packet based on the weight of the single-scale multi-instance learning network to obtain a gastric cancer pathological image classification model.
9. The method for classifying pathological images of gastric cancer based on HER2 gene detection according to claim 8, wherein the ResNet convolutional neural network comprises a set number of residual connecting blocks;
the residual connecting block comprises a 3 × 3 convolutional layer, a batch normalization layer, a ReLU activation function, a 3 × 3 convolutional layer, a batch normalization layer and a ReLU activation function which are connected in sequence.
10. The method for classifying gastric cancer pathological images based on HER2 gene detection according to claim 1, wherein when the gastric cancer pathological image classification model predicts an objective remission rate ORR, the last full-link layer of the model is changed to output the prediction of the ORR, and an optimizer adopts a cross entropy Loss function as a Loss function;
the cross entropy loss function is:
where y represents the patient's true ORR label,
representing the predicted ORR signature.
11. The method for classifying gastric cancer pathological images based on HER2 gene detection according to claim 1, wherein when the gastric cancer pathological image classification model predicts a progression-free life cycle PFS, the last full-link layer of the model is changed to output a numerical value, and an optimizer calculates a loss function by using a C-index;
the C-index is as follows:
where T represents the survival time of two randomly selected i, j patients, l and
respectively representing the values predicted in the true and model, if the survival time of patient i is longer than that of patient jC-Index =1, whereas C-Index =0.
12. A gastric cancer pathological image classification system based on HER2 gene detection is characterized by comprising:
the data acquisition module is configured to acquire a stomach digital pathological section image of a gastric cancer patient; the slice images comprise the expression profile of cancer tissue expressing HER2 gene;
the region extraction and segmentation module is configured to perform region extraction and segmentation operation on the slice image to obtain a plurality of image blocks with set pixel sizes and set tissue types;
the data sub-packaging module is configured to perform data enhancement normalization and data division on the plurality of image blocks with set pixel sizes and set tissue types to obtain a training image package;
the model training module is configured to perform iterative training on the constructed ResNet convolutional neural network through the training image packet to obtain a gastric cancer pathological image classification model;
and the classification module is configured to classify the stomach digital pathological section images of the patient based on the real-time acquired stomach digital pathological section images through the stomach cancer pathological image classification model to obtain image classification results.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211369435.4A CN115690056A (en) | 2022-11-03 | 2022-11-03 | Gastric cancer pathological image classification method and system based on HER2 gene detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211369435.4A CN115690056A (en) | 2022-11-03 | 2022-11-03 | Gastric cancer pathological image classification method and system based on HER2 gene detection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115690056A true CN115690056A (en) | 2023-02-03 |
Family
ID=85047552
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211369435.4A Pending CN115690056A (en) | 2022-11-03 | 2022-11-03 | Gastric cancer pathological image classification method and system based on HER2 gene detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115690056A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117423479A (en) * | 2023-12-19 | 2024-01-19 | 神州医疗科技股份有限公司 | Prediction method and system based on pathological image data |
-
2022
- 2022-11-03 CN CN202211369435.4A patent/CN115690056A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117423479A (en) * | 2023-12-19 | 2024-01-19 | 神州医疗科技股份有限公司 | Prediction method and system based on pathological image data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Mohanakurup et al. | Breast cancer detection on histopathological images using a composite dilated Backbone Network | |
| Wan et al. | Automated grading of breast cancer histopathology using cascaded ensemble with combination of multi-level image features | |
| Gandomkar et al. | MuDeRN: Multi-category classification of breast histopathological image using deep residual networks | |
| US10235755B2 (en) | High-throughput adaptive sampling for whole-slide histopathology image analysis | |
| Dundar et al. | Computerized classification of intraductal breast lesions using histopathological images | |
| Haj-Hassan et al. | Classifications of multispectral colorectal cancer tissues using convolution neural network | |
| US20170249739A1 (en) | Computer analysis of mammograms | |
| Reshma et al. | Detection of breast cancer using histopathological image classification dataset with deep learning techniques | |
| Abdolahi et al. | Artificial intelligence in automatic classification of invasive ductal carcinoma breast cancer in digital pathology images | |
| CN112263217B (en) | Improved convolutional neural network-based non-melanoma skin cancer pathological image lesion area detection method | |
| US10956795B2 (en) | Predicting recurrence in early stage non-small cell lung cancer (NSCLC) using spatial arrangement of clusters of tumor infiltrating lymphocytes and cancer nuclei | |
| Gheonea et al. | Diagnosis system for hepatocellular carcinoma based on fractal dimension of morphometric elements integrated in an artificial neural network | |
| Ström et al. | Pathologist-level grading of prostate biopsies with artificial intelligence | |
| Cao et al. | An automatic breast cancer grading method in histopathological images based on pixel-, object-, and semantic-level features | |
| Mahmood et al. | Breast mass detection and classification using deep convolutional neural networks for radiologist diagnosis assistance | |
| Farajzadeh et al. | A fully-convolutional residual encoder-decoder neural network to localize breast cancer on histopathology images | |
| CN115690056A (en) | Gastric cancer pathological image classification method and system based on HER2 gene detection | |
| Abdollahi | Evaluating LeNet Algorithms in Classification Lung Cancer from Iraq-Oncology Teaching Hospital/National Center for Cancer Diseases | |
| Johny et al. | Optimization of CNN model with hyper parameter tuning for enhancing sturdiness in classification of histopathological images | |
| Kabir et al. | The utility of a deep learning-based approach in Her-2/neu assessment in breast cancer | |
| Sajiv et al. | A Novel and Robust Breast Cancer classification based on Histopathological Images using Naive Bayes Classifier | |
| Saranyaraj et al. | Early prediction of breast cancer based on the classification of HER‐2 and ER biomarkers using deep neural network | |
| Dey et al. | Precise segmentation and classification of epithelial rete-pegs signature in assessing lower limb wound healing progression | |
| Salsabili | Automated Segmentation of Complex Biological Structures in Histopathology Images | |
| Sameh et al. | Performance Improvement of Breast Cancer Diagnosis Using Artificial Intelligence and Image Processing Techniques |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |