CN113538435B - Pancreatic cancer pathological image classification method and system based on deep learning - Google Patents

Pancreatic cancer pathological image classification method and system based on deep learning Download PDF

Info

Publication number
CN113538435B
CN113538435B CN202111090041.0A CN202111090041A CN113538435B CN 113538435 B CN113538435 B CN 113538435B CN 202111090041 A CN202111090041 A CN 202111090041A CN 113538435 B CN113538435 B CN 113538435B
Authority
CN
China
Prior art keywords
image
pancreatic cancer
images
classification
cell
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111090041.0A
Other languages
Chinese (zh)
Other versions
CN113538435A (en
Inventor
张光磊
范广达
冯又丹
宋凡
张鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beihang University
Original Assignee
Beihang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beihang University filed Critical Beihang University
Priority to CN202111090041.0A priority Critical patent/CN113538435B/en
Publication of CN113538435A publication Critical patent/CN113538435A/en
Application granted granted Critical
Publication of CN113538435B publication Critical patent/CN113538435B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Medical Informatics (AREA)
  • Signal Processing (AREA)
  • Veterinary Medicine (AREA)
  • Computational Linguistics (AREA)
  • Public Health (AREA)
  • Software Systems (AREA)
  • Animal Behavior & Ethology (AREA)
  • Physiology (AREA)
  • Psychiatry (AREA)
  • Pathology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Computing Systems (AREA)
  • Surgery (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Fuzzy Systems (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a pancreatic cancer pathological image classification method and system based on deep learning, which comprises the following steps: preprocessing an image; constructing an image screening network model; constructing a pancreatic cancer classification network model; cascading the image screening network model and the pancreatic cancer classification network model to construct an end-to-end pancreatic cancer rapid on-site evaluation system; and (4) judging and classifying pathological images of pancreatic cancer. The method applies a transfer learning technology and an attention mechanism and cascades two different neural networks, can obtain a classification system with the precision equivalent to that of a pathologist, and provides a real-time and accurate rapid field evaluation tool for the pancreatic cancer for a clinician.

Description

Pancreatic cancer pathological image classification method and system based on deep learning
Technical Field
The invention relates to the field of artificial intelligence assisted pathological image identification, in particular to a pancreatic cancer pathological image classification method and system based on deep learning.
Background
Pancreatic cancer is a digestive system tumor with high malignancy, and has low early diagnosis rate and extremely poor prognosis, and is one of the most malignant tumors. In recent years, the incidence of pancreatic cancer has increased remarkably at home and abroad, and the latest statistical data shows that the incidence of pancreatic cancer is ninth in China in malignant tumors, the mortality rate reaches the sixth, and the mortality rate is fourth in Western countries. Because the malignancy degree of pancreatic cancer is higher, the five-year survival rate of a patient is only 6% even if the patient can be treated by an operation, so that the pancreatic cancer treatment method has important clinical significance for early accurate diagnosis and treatment of pancreatic cancer patients, and can obviously improve the survival rate of the patient.
The surgical pathological examination is the gold standard for pancreatic cancer diagnosis, but the pancreatectomy has a high risk, and the main clinical pathological diagnosis mode for pancreatic cancer at present is Endoscopic Ultrasound-guided Fine needle biopsy (EUS-FNA) with small wound, and the pathological diagnosis of pancreatic cancer is performed by sampling pathological pancreatic slices, wherein the sensitivity and specificity of the pathological slices are 85% -95% and 95% -98%. Rapid on-site evaluation (ROSE) is an important factor affecting the sensitivity of EUS-FNA pancreatic cancer diagnosis. ROSE is the on-site evaluation of the rapidly stained section obtained by EUS-FNA sampling by a pathologist, and the effectiveness and the sufficiency of the tissue section are judged in real time. The technology aims to reduce puncture times, shorten diagnosis time, realize real-time accurate diagnosis and reduce the risk of complications in the operation process. In addition, it may also be determined in ROSE whether additional puncture sampling is needed to further aid diagnosis. Although there are many studies that demonstrate that the combination of ROSE technology with EUS-FNA can improve the accuracy of pancreatic cancer diagnosis to some extent, the core problem limiting the widespread use of ROSE technology is the lack of pathologists in field evaluations.
With the rapid development of the deep learning technology and the improvement of the computing performance of the server, the artificial intelligence technology makes a breakthrough progress in the field of medical auxiliary diagnosis, and a plurality of research works have applied the artificial intelligence technology to analyze pathological images to obtain diagnosis and treatment results with the accuracy equivalent to that of clinicians. The pathological images of the pancreatic biopsy samples are automatically and intelligently analyzed by using an artificial intelligence technology, so that the problem of shortage of on-site pathology assessment doctors can be solved, and the potential of improving the pancreatic cancer diagnosis sensitivity is realized.
The Chinese patent application with the application number of CN201110063144.8 provides a method for processing digital images and classifying modes, which is applied to the computer-aided diagnosis of pancreatic cancer endoscopic ultrasound. The method is realized by extracting the textural features and the classifier of the endoscope ultrasonic image, various objective and quantitative diagnosis indexes are created, and the endoscope ultrasonic image is correctly described and explained, so that the accuracy of early diagnosis of the pancreatic cancer endoscope ultrasonic image is improved. However, the method aims at the ultrasonic image to carry out image processing and mode classification, and has limited image precision and poor accuracy.
The Chinese invention patent application with the application number of CN201911001610.2 provides a lung nodule benign and malignant classification method and system based on multi-scale migration learning, and the benign and malignant classification is realized by utilizing a migration learning network model; the Chinese patent application with the application number of CN201911415563.6 provides a thyroid papillary carcinoma pathological image classification method based on deep learning, and a VGG-f convolutional neural network is used for classification; the Chinese patent application with the application number of CN202011142354.1 provides a method and a system for classifying multiple dyeing pathological images based on deep learning, a pathological image classification model is constructed by utilizing a deep convolutional neural network, and the classification of the multiple dyeing pathological images is carried out by adopting information fused by a self-attention mechanism. However, because the clinical application scenes and the characteristics of the acquired images are different, the network model adopted by the above patent is not suitable for end-to-end classification of the pancreatic cytopathology images with high resolution and background noise region interference.
And for the pathological images of the pancreatic cells rapidly stained by EUS-FNA, the research of the pathological image classification of the pancreatic cancer based on deep learning is in an early stage of onset. The main difficulty is that the marked high-quality data is deficient and the high-resolution pathological image contains a large noise area, so that the classification performance of the model is influenced.
Disclosure of Invention
In order to solve the defects in the prior art, the invention provides a pancreatic cancer pathological image classification method based on deep learning, which is used for solving the problem of shortage of pathological doctors when a fine needle aspiration biopsy is used for diagnosing pancreatic cancer. Aiming at the problems of small data volume and more background and noise areas of the pancreatic pathological images, the method firstly uses the transfer learning technology, transfers the neural network parameters which are pre-trained well in other data sets and well-represented, and then trains in the data sets, so as to solve the over-fitting problem caused by small data volume and improve the generalization capability of the model. Subsequently, an attention mechanism is added to the pancreatic cancer classification network, so that the classification network focuses more on pancreatic cell regions, and the classification accuracy of the model is improved. In addition, the invention has cascade connection of two different neural networks, eliminates background and noise regions, and classifies pancreatic cancer after reserving cell regions, thereby providing a rapid on-site evaluation system with high accuracy for clinicians.
The invention provides a pancreatic cancer pathological image classification method based on deep learning, which comprises the following steps:
a pancreatic cancer pathological image classification method based on deep learning comprises the following steps:
s1, cutting the pathological pancreas image by using a sliding window to obtain a plurality of slice images, and screening the slice images into background noise area images and cell area images to be used as training images;
s2, constructing a lightweight image screening network model, and training the image screening network model by using the background noise area image and the cell area image;
s3, constructing a pancreatic cancer classification network model based on transfer learning, adding an attention mechanism to the classification network model, and training the classification network model by using images of cancer and non-cancer cells in the cell region images;
s4, constructing an end-to-end pancreatic cancer rapid on-site evaluation system by the image screening network model and the classification network model after cascade training;
and S5, transmitting the unlabeled pancreatic pathology image to the rapid field evaluation system of the pancreatic cancer, and judging the pancreatic pathology image to be a pancreatic cancer cell image or a normal cell image.
In step S1, the cutting out the pancreatic disease image using the sliding window specifically includes: and (3) setting a window (k) with a fixed size, sliding and cutting along the row and the column of the high-resolution pathological image, starting from the upper left corner of the image, moving (k-a) pixel points along the row direction each time for sliding for i times, then moving (k-b) pixel points along the column direction each time for sliding for j times, and cutting the original image to obtain i x j new images.
In step S2, the constructing the lightweight image screening network model specifically includes: selecting an Efficient-Net network, abandoning a full connection layer of an original network, adding a full connection layer with the neuron number of 2 at the tail end of the last convolution layer, and adding a Relu activation function and a Dropout strategy in the full connection layer.
In step S3, the constructing of the pancreatic cancer classification model based on migratory learning specifically includes: selecting a ResNet50 model pre-trained on an ImageNet data set, transferring the network structure and the pre-training parameters of the model except the full-link layer, and adding a new full-link layer with randomly initialized parameters, wherein the number of the neurons is 2.
In step S3, the attention adding mechanism specifically includes:
step one, aiming at a characteristic diagram obtained by a convolution network middle layerXOf size ofH*W*C，HAndWrepresents the length and width of the feature map,Crepresenting the number of channels of the feature map, firstly applying a global average pooling method to the feature mapXCompressed to 1 xCGlobal feature map of sizeYThe concrete formula is as follows:
Figure 334398DEST_PATH_IMAGE001
wherein the content of the first and second substances,
Figure 258491DEST_PATH_IMAGE002
represents the firstkAn original feature mapiLine ofjThe value of the column is such that,
Figure 627156DEST_PATH_IMAGE003
represents a global feature mapkThe value of each channel;
step two, calculating the weight of each characteristic channel
Figure 52321DEST_PATH_IMAGE004
Corresponding to the importance of each feature channelThe weight calculation formula is:
Figure 557252DEST_PATH_IMAGE005
wherein C1D represents a 1D convolution,mis the size of the convolution kernel and,Yis a global feature map after a global pooling operation,σrepresents a Sigmoid activation function;
step three, weighting the characteristic channel obtained in the step two
Figure 262034DEST_PATH_IMAGE004
And characteristic diagramXThe corresponding channel is subjected to multiplication operation, and the calculation formula is as follows:
Figure 383573DEST_PATH_IMAGE006
wherein the content of the first and second substances,
Figure 487796DEST_PATH_IMAGE007
representing a multiplication operation based on the corresponding channel,Zto obtain a new characteristic diagram based on a self-attention mechanism.
In step S4, the image screening network model and the classification network model after the cascade training are specifically:
(1) cutting the pathological image of the pancreatic cells to obtain i x j slice images, namely obtaining a slice sample set
Figure 971867DEST_PATH_IMAGE008
；
(2) Passing the slice sample set to the image screening network constructed at step S2; the mathematical processing function of the image screening network is
Figure 503342DEST_PATH_IMAGE009
For use inNEach sample classified as background noise or cellular regions,
Figure 954921DEST_PATH_IMAGE010
wherein, in the step (A),
Figure 862834DEST_PATH_IMAGE011
0 represents the background noise area image, 1 represents the cell area image, a new set of cell areas in the slice sample set is obtained,
Figure 342357DEST_PATH_IMAGE012
wherein, in the step (A),
Figure 169367DEST_PATH_IMAGE013
；
(3) pooling the cell regions retained by the screenN t The result is passed to step S3 to construct a pancreatic cancer classification network having a mathematical processing function off，
Figure 734341DEST_PATH_IMAGE014
Wherein, in the step (A),
Figure 180366DEST_PATH_IMAGE015
0 represents a normal cell image, 1 represents a pancreatic cancer cell image, and a cell region set is obtainedN t Each image in (a) is a prediction of pancreatic cancer or non-cancerous;
(4) recording the image set of normal cells as
Figure 655341DEST_PATH_IMAGE016
Image set of pancreatic cancer cells
Figure 528619DEST_PATH_IMAGE017
Then, then
Figure 580888DEST_PATH_IMAGE018
Wherein, in the step (A),
Figure 689659DEST_PATH_IMAGE019
；
Figure 143774DEST_PATH_IMAGE020
wherein, in the step (A),
Figure 187953DEST_PATH_IMAGE021
(ii) a Synthesizing all cell section prediction results of original pathological images by voting mechanism, and comparing normal cell sets
Figure 101420DEST_PATH_IMAGE022
And pancreatic cancer cells
Figure 889248DEST_PATH_IMAGE023
Number of middle element
Figure 322503DEST_PATH_IMAGE024
And
Figure 537584DEST_PATH_IMAGE025
if, if
Figure 564446DEST_PATH_IMAGE026
The prediction result is a normal cell image, if
Figure 765751DEST_PATH_IMAGE027
And the result is predicted to be a pancreatic cancer cell image.
The step S4 further includes: and adding a visual interface, wherein the interface receiving input end is a pancreas pathology image, and the output end displays the classification result of the pancreas pathology image.
A pancreatic cancer pathological image classification system based on deep learning is provided, and the pancreatic cancer pathological image classification method based on deep learning is applied.
The invention has the beneficial effects that:
1) the method comprises the steps of obtaining a pancreatic pathological image by a microscope, wherein the pancreatic pathological image is obtained by a pancreatic cancer classification network, the pancreatic cancer classification network is used for obtaining a pancreatic cancer image with a high quality, and the pancreatic cancer image is obtained by a microscope.
2) The invention cascades the image screening network and the pancreatic cancer classification network, ensures that the pancreatic pathological images collected by biopsy puncture can be directly classified end to end, shortens the model screening time by the light-weight image screening network, improves the model precision by the deep pancreatic cancer classification network, and ensures that the pancreatic cancer classification system can be accurately applied to the rapid field evaluation of the pancreatic puncture biopsy in real time.
3) According to the method, a transfer learning technology is introduced firstly, the network structure and parameters of the ResNet50 model pre-trained in the ImageNet data set are transferred, and the full-connection layer structure of the model is changed to meet the requirement of a two-classification task, so that the problem of lack of pathological image data of pancreatic cancer is solved, and the classification model has better generalization performance. Secondly, an attention mechanism is introduced, so that the pancreatic cell region is focused more by the model, higher weight is given to the characteristics of the region, the interference of a red blood cell noise region is inhibited, and the classification precision of the model is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flowchart of a method for constructing a pancreatic cancer pathological image classification model based on deep learning according to the present invention;
FIG. 2 is a schematic diagram of a model training and evaluation method of the present invention;
fig. 3 is a schematic diagram of a pathological image classification system for pancreatic cancer constructed in the present invention.
Detailed Description
In order that the above objects, features and effects of the present invention can be more clearly understood, a complete description of the present invention will be made below with reference to the accompanying drawings and detailed description. It should be noted that the embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
As shown in fig. 1, a method and system for classifying pathological images of pancreatic cancer based on deep learning includes the following steps:
step S1, using a microscope to acquire a pancreatic pathology image as a raw data set. As shown in fig. 2, for the original data set, 20% of the data was randomly drawn as test data and the remaining images were taken as training data.
The method comprises the steps of preprocessing a high-resolution pancreas pathological image serving as training data, cutting the high-resolution pancreas pathological image into a plurality of low-resolution fixed-size slices, increasing the training sample size on one hand, and eliminating noise interferences such as blank background areas and red blood cells of the pathological image on the other hand, so that the pancreatic cancer classification network can pay more attention to the features such as the form, arrangement mode and heterogeneity of pancreatic cells, and the interpretability and classification precision of a model are improved.
In this example, the resolution of the pathological pancreatic image acquired by the olympus BX53 microscope was 1390 × 1038 pixels. The classification network of the invention applies a transfer learning technology, a ResNet50 network pre-trained in ImageNet is transferred, the input data of the network is 224 × 224 size images, and in order to meet the input requirement of the transfer learning network, a window of 224 × 224 pixels is selected to perform sliding window clipping on the original image, and the original image moves along the rows and columns of the original image. And (3) moving (224-30) pixel points along the row direction each time from the upper left corner of the image for 7 times, then moving (224-21) pixel points along the column direction each time for 5 times, and cutting the original image to obtain 5 x 7 new image slices. After all the pancreatic pathological images are cut, the pancreatic pathological images are manually screened and classified into background noise regions and cell regions. Wherein the slice image having the following features is classified as a background noise region image: 1) at least one intact pancreatic cell is not present in the image. 2) The contrast of the image is low, and contour information between cells cannot be distinguished. 3) The images had stained spots and had a large interference with the pancreatic cell contour. The slice image with the three conditions is divided into background noise areas, and the image which has at least one complete pancreatic cell and high quality and is not interfered by contrast and stain noise in the slice image is divided into pancreatic cell area images.
And (4) normalizing, standardizing and amplifying data of the slice images obtained by cutting. Normalization, namely changing pixels of the image from 0-255 to 0-1, accelerates the convergence rate during network training, and normalization is to realize the centering of the image through the mean value and increase the generalization capability of the model. In addition, because the sample size of the medical image is small, the invention also carries out a series of data amplification on the training image so as to relieve the overfitting problem. The amplification method is to select all training images at 90 degrees and 180 degrees, and to perform horizontal turning and vertical turning operations to increase the training set to five times of the original training set. The two types of images of the background noise region and the cell region processed as described above are input data to the image screening network in step S2, and the images of the cancer and non-cancer cells in the cell region are input data to the pancreatic cancer classification network in step S3.
And step S2, constructing a lightweight image screening network.
In this example, the constructed lightweight image screening network model is based on an Efficient-Net network. The network provides a mixed scaling idea, obtains a lightweight high-precision optimal network structure by comprehensively optimizing the network width, the network depth and the resolution and utilizing a structure searching method, and obtains excellent performances in the natural image field and the medical image field. In the network design of the invention, the full connection layer of the original network is abandoned, and full connection layers with the number of the neurons being 512 and 2 are sequentially added at the tail end of the last convolution layer, thereby reducing the number of parameters, improving the operation speed and simultaneously meeting the two classification tasks of screening a background noise area and a cell area. A Relu (rectified linear unit) activation function and a Dropout strategy are added in the full connection layer, wherein the Relu activation function is a piecewise linear function which changes all negative values to 0 without changing positive values. The activation function has the effect that the gradient of the non-negative region is constant, the problem of gradient disappearance is not easy to generate, and the convergence rate of the model is maintained in a stable state. The Dropout strategy is that in each training batch, a part of feature detectors are ignored, and the activation values of corresponding neurons stop working with a certain probability p, so that the model does not depend too much on some local features in the training data, the generalization capability of the model is improved, and the model obtains more excellent performance in a small sample data set. In this example, the Dropout rate is set to 0.2.
And training an Efficient-Net image screening network by using the preprocessed images. And selecting a pyrrch framework to train the network, selecting an Adam optimizer by a training optimizer, and adopting a cross entropy loss function as a loss function. As shown in fig. 2, for the pancreatic pathology image as training data, the pancreatic pathology image is equally divided into five parts, four parts of the pancreatic pathology image are alternately used as a training set, one part of the pancreatic pathology image is used as a verification set, and a data amplification method is not applied to the verification set and the independent test set. During training, the initial learning rate is set to be 0.0001, after all training sets are input into the network for one training each time, the classification accuracy of the verification set is tested, if the accuracy of the verification set is not improved for three times continuously, the learning rate is multiplied by a coefficient of 0.7, the learning rate is reduced, the training is continued, after thirty rounds of training are completed, the model weight of the verification set in the round with the highest classification accuracy is reserved, the final model in one-time cross verification is obtained, the steps are repeated for five times in total, and the model with the highest accuracy of the verification set is selected as the image screening network in the cascade network.
And step S3, constructing a pancreatic cancer classification network. The constructed network is based on a transfer learning technology, an attention mechanism is added to the model, and the preprocessed cell images are used for training a classification network, so that the network has the function of classifying the pancreatic pathological images into cancer and non-cancer.
In the network construction of this example, the ResNet50 network structures and parameters pre-trained on the ImageNet dataset are migrated. And during migration, a feature extraction layer of the ResNet50 network is reserved, a full connection layer of the ResNet50 network is removed, a new full connection layer of random initialization parameters is added, and the number of neurons is 1024, 512 and 2, so that the binary function is realized. Adding Relu activation function and Dropout strategy in the full connection layer, the Dropout rate is set to 0.4, and the effect is better because the classification network has more parameters.
Although a large number of background and noise regions have been screened out in step S2, a significant portion of the remaining cell image contains red blood cells, which may interfere with the classification performance of the model. In order to further improve the classification precision of the model and enable the network model to pay more attention to the cell region during feature extraction, the invention adds an attention mechanism to the model.
The essence of the attention mechanism is a weighting coefficient obtained by network autonomous learning, and the characteristics of the target area are emphasized and the characteristics of the background noise area are suppressed in a weighting mode. In this example, an eca (efficient Channel assignment) module is added to the constructed ResNet50 migration learning model, and the module implements a local cross-Channel interaction strategy without dimension reduction through one-dimensional convolution, and captures cross-Channel interaction information, and the specific implementation method is as follows:
step one, aiming at a characteristic diagram obtained by a convolution network middle layerXOf a size ofH*W*C，HAndWrepresents the length and width of the feature map,Cthe number of channels representing the feature map is first compressed to 1 × 1 by applying a global average pooling methodCGlobal feature map of sizeYThe concrete formula is as follows:
Figure 194458DEST_PATH_IMAGE028
wherein the content of the first and second substances,
Figure 314861DEST_PATH_IMAGE029
represents the firstkAn original feature mapiLine ofjThe value of the column is such that,
Figure 219232DEST_PATH_IMAGE030
representing global charactersFigure characterization methodkThe value of each channel is defined asH*W*CThe original feature map of (1 x)CThe operation compresses the two-dimensional feature map into a real number representing the global feature of the feature channel layer.
Step two, calculating the weight of each characteristic channel
Figure 348862DEST_PATH_IMAGE031
Corresponding to the importance degree of each characteristic channel, the weight calculation formula is as follows:
Figure 366497DEST_PATH_IMAGE032
wherein C1D represents a 1D convolution,mis the size of the convolution kernel and,Yis a global feature map after a global pooling operation,σrepresenting a Sigmoid activation function, in the present method,kthe value is 3. The traditional SENET attention module obtains the weight parameters by applying two full-connection layers after obtaining the global feature vector
Figure 164377DEST_PATH_IMAGE033
The method has each full connection layerC*C/rThe amount of the parameter(s) of (c),rfor feature size after SENet compression operation, the 1D convolution of the added ECA attention module of the present invention only hasmA parameter quantity taking into account each channelmThe adjacent vectors are used for capturing the interaction information of local cross-channels, so that the calculation performance and efficiency of the model are ensured, and the model is proved to have more excellent performance in practice.
Step three, the weight of the characteristic channel obtained in the step two is used
Figure 962569DEST_PATH_IMAGE031
And characteristic diagramXThe corresponding channels are subjected to multiplication operation, so that different characteristic channels in the characteristic diagram have different weight coefficients, the network is helped to pay attention to the characteristics which have larger influence on the classification task so as to inhibit redundant characteristics, the classification precision of the model is improved, and the calculation formula is as follows:
Figure 427048DEST_PATH_IMAGE006
wherein the content of the first and second substances,
Figure 971293DEST_PATH_IMAGE007
representing a multiplication operation based on the corresponding channel,Zto obtain a new characteristic diagram based on a self-attention mechanism.
Subsequently, the preprocessed images were used to train the ResNet50 pancreatic cancer classification network based on the mechanism of transfer learning and attention. And selecting a pyrrch framework to train the network, selecting an Adam optimizer by a training optimizer, and adopting a cross entropy loss function as a loss function. The training data of the model are pancreatic cancer cell images and normal cell images in the cell region image set, the training and evaluation method of the model is the same as that in step S2, and after the training is finished, the model with the highest classification accuracy of the verification set is selected as a pancreatic cancer classification network in the cascade network.
And S4, cascading the image screening network in the step S2 and the pancreatic cancer classification network in the step S3 to construct an end-to-end pancreatic cancer pathological image classification system.
Cutting the high-resolution pancreatic cell pathology image after receiving the high-resolution pancreatic cell pathology image to obtain i x j slice images, namely obtaining a slice sample set
Figure 761395DEST_PATH_IMAGE034
And then, the image is passed to the image screening network constructed in step S2. The mathematical processing function of the image screening network is
Figure 781303DEST_PATH_IMAGE035
For use inNEach sample classified as background noise or cellular regions,
Figure 659261DEST_PATH_IMAGE036
wherein, in the step (A),
Figure 448225DEST_PATH_IMAGE037
and 0 represents a background noise regionImage, 1 represents a cell region image, resulting in a new set of cell regions in the slice sample set,
Figure 409228DEST_PATH_IMAGE038
wherein, in the step (A),
Figure 791799DEST_PATH_IMAGE039
. Subsequently, the remaining cell regions will be screenedN t The result is passed to step S3 to construct a pancreatic cancer classification network having a mathematical processing function off，
Figure 598081DEST_PATH_IMAGE040
Wherein, in the step (A),
Figure 507131DEST_PATH_IMAGE041
0 represents a normal cell image, 1 represents a pancreatic cancer cell image, and a cell region set is obtainedN t Each image in (a) is a predictive outcome for pancreatic cancer or non-cancer. Recording the image set of normal cells as
Figure 747357DEST_PATH_IMAGE042
Image set of pancreatic cancer cells
Figure 741858DEST_PATH_IMAGE043
Then, then
Figure 227197DEST_PATH_IMAGE044
Wherein, in the step (A),
Figure 990754DEST_PATH_IMAGE045
；
Figure 27980DEST_PATH_IMAGE046
wherein, in the step (A),
Figure 385143DEST_PATH_IMAGE047
. Synthesizing all cell section prediction results of original pathological images by voting mechanism, and comparing normal cell sets
Figure 798807DEST_PATH_IMAGE048
And pancreatic cancer cells
Figure 416870DEST_PATH_IMAGE049
Number of middle element
Figure 500363DEST_PATH_IMAGE050
And
Figure 469456DEST_PATH_IMAGE051
if, if
Figure 155653DEST_PATH_IMAGE052
The prediction result is a normal cell image, if
Figure 267703DEST_PATH_IMAGE053
And if so, the prediction result is a cancer cell image, so that the final classification result of the input high-resolution pancreatic pathology image is obtained. Specifically, the pathological image classification system for pancreatic cancer constructed in this example is shown in fig. 3. Aiming at 1390 × 1038 pixel pancreatic pathology images acquired during needle biopsy, 35 slice images are obtained by cutting the pancreatic pathology images by using a preprocessing method of step S1, the slice images are transmitted to a trained first-level image screening network for analysis, background noise region images are screened and removed, and pancreatic cell region images are reserved. And then, transmitting the retained images to a trained second-stage pancreatic cancer classification network to obtain an analysis result of each section belonging to pancreatic cancer or normal pancreatic cells. And (3) voting by combining the analysis results of all cell areas of a single image, wherein the specific voting mode is as follows: if the number of slices classified as normal cells in the image is not less than the number of slices classified as cancer cells, the image is classified as normal pancreatic cells, and conversely, pancreatic cancer cells. After the model is built, a GUI visual interface is added to the model based on Pyqt, the interface receives pancreatic pathological images acquired by clinical needle biopsy at the input end, and the classification result of the model on the images is displayed at the output end.
After the classification system is constructed, the performance of the classification system needs to be evaluated. When the performance of the classification system is evaluated, besides the classification accuracy of the test system, the sensitivity, the specificity, the positive prediction rate, the negative prediction rate and the AUC of the system are also tested on an independent test set, so that the classification performance of the system in clinical application is comprehensively evaluated.
And S5, obtaining the unmarked pancreatic pathology image collected by the microscope, transmitting the unmarked pancreatic pathology image to the pancreatic pathology image cascade classification system established in S4 for auxiliary analysis, and outputting the classification result of the pancreatic pathology image judged as the pancreatic cancer cell or the normal cell image by the classification system.
The foregoing is a more detailed description of the invention in connection with specific/preferred embodiments and is not intended to limit the practice of the invention to those descriptions. It will be apparent to those skilled in the art that various substitutions and modifications can be made to the described embodiments without departing from the spirit of the invention, and these substitutions and modifications should be considered to fall within the scope of the invention.

Claims (5)

1. A pancreatic cancer pathological image classification method based on deep learning is characterized by comprising the following steps:
s1, cutting the pathological pancreas image by using a sliding window to obtain a plurality of slice images, and screening the slice images into background noise area images and cell area images to be used as training images;
s2, constructing a lightweight image screening network model, and training the image screening network model by using the background noise area image and the cell area image;
s3, constructing a pancreatic cancer classification network model based on transfer learning, adding an attention mechanism to the classification network model, enabling the network model to pay more attention to a cell region during feature extraction, and training the classification network model by using cancer and non-cancer cell images in cell region images;
s4, constructing an end-to-end pancreatic cancer rapid on-site evaluation system by the image screening network model and the classification network model after cascade training;
s5, transmitting the unlabelled pancreatic pathology image to the rapid field evaluation system for the pancreatic cancer, and judging that the pancreatic pathology image is a pancreatic cancer cell image or a normal cell image;
in step S3, the constructing of the pancreatic cancer classification model based on migratory learning specifically includes: selecting a ResNet50 model pre-trained on an ImageNet data set, transferring a network structure and pre-training parameters of the model except a full connection layer, and adding a new full connection layer with randomly initialized parameters, wherein the number of neurons is 2;
in step S3, the attention adding mechanism specifically includes:
step one, aiming at a characteristic diagram obtained by a convolution network middle layerXOf size ofH*W*C，HAndWrepresents the length and width of the feature map,Crepresenting the number of channels of the feature map, firstly applying a global average pooling method to the feature mapXCompressed to 1 xCGlobal feature map of sizeYThe concrete formula is as follows:
Figure 730587DEST_PATH_IMAGE001
wherein the content of the first and second substances,
Figure 13801DEST_PATH_IMAGE002
represents the firstkAn original feature mapiLine ofjThe value of the column is such that,
Figure 570684DEST_PATH_IMAGE003
represents a global feature mapkThe value of each channel;
step two, calculating the weight of each characteristic channel
Figure 306559DEST_PATH_IMAGE004
Corresponding to the importance degree of each characteristic channel, the weight calculation formula is as follows:
Figure 239880DEST_PATH_IMAGE005
wherein C1D represents a 1D convolution,mis the size of the convolution kernel and,Yis a global feature map after a global pooling operation,σrepresents a Sigmoid activation function;
step three, weighting the characteristic channel obtained in the step two
Figure 643180DEST_PATH_IMAGE006
And characteristic diagramXThe corresponding channel is subjected to multiplication operation, and the calculation formula is as follows:
Figure 105385DEST_PATH_IMAGE007
wherein the content of the first and second substances,
Figure 328556DEST_PATH_IMAGE008
representing a multiplication operation based on the corresponding channel,Zobtaining a new characteristic diagram based on a self-attention mechanism;
in step S4, the image screening network model and the classification network model after the cascade training are specifically:
(1) cutting the pathological image of the pancreatic cells to obtain i x j slice images, namely obtaining a slice sample set
Figure 799989DEST_PATH_IMAGE009
；
(2) Passing the slice sample set to the image screening network constructed at step S2; the mathematical processing function of the image screening network is
Figure 792215DEST_PATH_IMAGE010
For use inNEach sample classified as background noise or cellular regions,
Figure 455016DEST_PATH_IMAGE011
wherein, in the step (A),
Figure 431062DEST_PATH_IMAGE012
0 represents the background noise area image, 1 represents the cell area image, a new set of cell areas in the slice sample set is obtained,
Figure 440606DEST_PATH_IMAGE013
wherein, in the step (A),
Figure 552919DEST_PATH_IMAGE014
；
(3) pooling the cell regions retained by the screenN t The result is passed to step S3 to construct a pancreatic cancer classification network having a mathematical processing function off，
Figure 356927DEST_PATH_IMAGE015
Wherein, in the step (A),
Figure 820269DEST_PATH_IMAGE016
0 represents a normal cell image, 1 represents a pancreatic cancer cell image, and a cell region set is obtainedN t Each image in (a) is a prediction of pancreatic cancer or non-cancerous;
(4) recording the image set of normal cells as
Figure 633504DEST_PATH_IMAGE017
Image set of pancreatic cancer cells
Figure 600323DEST_PATH_IMAGE018
Then, then
Figure 106391DEST_PATH_IMAGE019
Wherein, in the step (A),
Figure 525871DEST_PATH_IMAGE020
；
Figure 142797DEST_PATH_IMAGE021
wherein, in the step (A),
Figure 229702DEST_PATH_IMAGE022
(ii) a Synthesizing all cell section prediction results of original pathological images by voting mechanism, and comparing normal cell sets
Figure 139627DEST_PATH_IMAGE023
And pancreatic cancer cells
Figure 311982DEST_PATH_IMAGE024
Number of middle element
Figure 201441DEST_PATH_IMAGE025
And
Figure 408431DEST_PATH_IMAGE026
if, if
Figure 256301DEST_PATH_IMAGE027
The prediction result is a normal cell image, if
Figure 650374DEST_PATH_IMAGE028
And the result is predicted to be a pancreatic cancer cell image.
2. The method for classifying pathological images of pancreatic cancer based on deep learning of claim 1, wherein in step S1, the cropping pancreatic cancer images using sliding window is specifically: and (3) setting a window (k) with a fixed size, sliding and cutting along the row and the column of the high-resolution pathological image, starting from the upper left corner of the image, moving (k-a) pixel points along the row direction each time for sliding for i times, then moving (k-b) pixel points along the column direction each time for sliding for j times, and cutting the original image to obtain i x j new images.
3. The method for classifying pathological images of pancreatic cancer based on deep learning of claim 1, wherein in step S2, the constructing of the lightweight image screening network model specifically includes: selecting an Efficient-Net network, abandoning a full connection layer of an original network, adding a full connection layer with the neuron number of 2 at the tail end of the last convolution layer, and adding a Relu activation function and a Dropout strategy in the full connection layer.
4. The method for classifying pathological images of pancreatic cancer based on deep learning of claim 1, wherein said step S4 further comprises: and adding a visual interface, wherein the interface receiving input end is a pancreas pathology image, and the output end displays the classification result of the pancreas pathology image.
5. A pancreatic cancer pathological image classification system based on deep learning is characterized in that the pancreatic cancer pathological image classification method based on deep learning of any one of claims 1-4 is applied.
CN202111090041.0A 2021-09-17 2021-09-17 Pancreatic cancer pathological image classification method and system based on deep learning Active CN113538435B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111090041.0A CN113538435B (en) 2021-09-17 2021-09-17 Pancreatic cancer pathological image classification method and system based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111090041.0A CN113538435B (en) 2021-09-17 2021-09-17 Pancreatic cancer pathological image classification method and system based on deep learning

Publications (2)

Publication Number Publication Date
CN113538435A CN113538435A (en) 2021-10-22
CN113538435B true CN113538435B (en) 2022-02-18

Family

ID=78093364

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111090041.0A Active CN113538435B (en) 2021-09-17 2021-09-17 Pancreatic cancer pathological image classification method and system based on deep learning

Country Status (1)

Country Link
CN (1) CN113538435B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113935990A (en) * 2021-11-26 2022-01-14 南京鼓楼医院 Pancreas occupy-place EUS-FNA scene quick cell pathology evaluation system based on artificial intelligence
CN114565761B (en) * 2022-02-25 2023-01-17 无锡市第二人民医院 Deep learning-based method for segmenting tumor region of renal clear cell carcinoma pathological image
WO2024108425A1 (en) * 2022-11-23 2024-05-30 深圳先进技术研究院 Method for classifying pancreatic images based on hybrid attention network

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105760878A (en) * 2014-12-19 2016-07-13 西门子医疗保健诊断公司 Method and device for selecting urinary sediment microscope image with optimal focusing performance
CN109389557B (en) * 2018-10-20 2023-01-06 南京大学 Cell image super-resolution method and device based on image prior
WO2021016131A1 (en) * 2019-07-19 2021-01-28 The Jackson Laboratory Convolutional neural networks for classification of cancer histological images
US11462032B2 (en) * 2019-09-23 2022-10-04 Proscia Inc. Stain normalization for automated whole-slide image classification
US20210216745A1 (en) * 2020-01-15 2021-07-15 DeePathology Ltd. Cell Detection Studio: a system for the development of Deep Learning Neural Networks Algorithms for cell detection and quantification from Whole Slide Images
CN111415352B (en) * 2020-04-03 2023-07-28 南京信息工程大学 Cancer metastasis panoramic pathological section analysis method based on deep cascade network
CN111583204B (en) * 2020-04-27 2022-10-14 天津大学 Organ positioning method of two-dimensional sequence magnetic resonance image based on network model
CN112348059A (en) * 2020-10-23 2021-02-09 北京航空航天大学 Deep learning-based method and system for classifying multiple dyeing pathological images
CN112508850B (en) * 2020-11-10 2021-07-20 广州柏视医疗科技有限公司 Deep learning-based method for detecting malignant area of thyroid cell pathological section
CN112488234B (en) * 2020-12-10 2022-04-29 武汉大学 End-to-end histopathology image classification method based on attention pooling

Also Published As

Publication number Publication date
CN113538435A (en) 2021-10-22

Similar Documents

Publication Publication Date Title
CN113538435B (en) Pancreatic cancer pathological image classification method and system based on deep learning
CN109670510B (en) Deep learning-based gastroscope biopsy pathological data screening system
CN109272048B (en) Pattern recognition method based on deep convolutional neural network
US20170249739A1 (en) Computer analysis of mammograms
CN110889852A (en) Liver segmentation method based on residual error-attention deep neural network
CN110889853A (en) Tumor segmentation method based on residual error-attention deep neural network
CN113256641B (en) Skin lesion image segmentation method based on deep learning
CN113674253A (en) Rectal cancer CT image automatic segmentation method based on U-transducer
CN110838100A (en) Colonoscope pathological section screening and segmenting system based on sliding window
CN110111895A (en) A kind of method for building up of nasopharyngeal carcinoma far-end transfer prediction model
CN112861994B (en) Gastric seal ring cell cancer image intelligent classification system based on Unet transfer learning
CN115661029A (en) Pulmonary nodule detection and identification system based on YOLOv5
CN114565613A (en) Pancreas postoperative diabetes prediction system based on supervised deep subspace learning
CN115471701A (en) Lung adenocarcinoma histology subtype classification method based on deep learning and transfer learning
Lomacenkova et al. Whole slide pathology image patch based deep classification: an investigation of the effects of the latent autoencoder representation and the loss function form
Kumaraswamy et al. Invasive ductal carcinoma grade classification in histopathological images using transfer learning approach
CN114565601A (en) Improved liver CT image segmentation algorithm based on DeepLabV3+
CN114565786A (en) Tomography image classification device and method based on channel attention mechanism
Tenali et al. Oral Cancer Detection using Deep Learning Techniques
yahia Ibrahim et al. An enhancement technique to diagnose colon and lung cancer by using double CLAHE and deep learning
CN110084796B (en) Analysis method of complex texture CT image
CN115132275B (en) Method for predicting EGFR gene mutation state based on end-to-end three-dimensional convolutional neural network
CN116912253A (en) Lung cancer pathological image classification method based on multi-scale mixed neural network
CN111904470A (en) Dynamic interpretable reasoning auxiliary diagnosis method for medical ultrasonic examination process
Shobarani et al. Melanoma Malignancy Prognosis Using Deep Transfer Learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant