CN111462052A - Medical image analysis method and system based on graph neural network - Google Patents
Medical image analysis method and system based on graph neural network Download PDFInfo
- Publication number
- CN111462052A CN111462052A CN202010180739.0A CN202010180739A CN111462052A CN 111462052 A CN111462052 A CN 111462052A CN 202010180739 A CN202010180739 A CN 202010180739A CN 111462052 A CN111462052 A CN 111462052A
- Authority
- CN
- China
- Prior art keywords
- image
- cell
- neural network
- feature
- medical image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 73
- 238000003703 image analysis method Methods 0.000 title claims abstract description 20
- 230000001575 pathological effect Effects 0.000 claims abstract description 38
- 238000000034 method Methods 0.000 claims abstract description 33
- 238000000605 extraction Methods 0.000 claims abstract description 22
- 238000001514 detection method Methods 0.000 claims abstract description 14
- 238000000926 separation method Methods 0.000 claims abstract description 11
- 238000012549 training Methods 0.000 claims description 35
- 206010028980 Neoplasm Diseases 0.000 claims description 26
- 201000011510 cancer Diseases 0.000 claims description 24
- 238000012360 testing method Methods 0.000 claims description 18
- 238000010191 image analysis Methods 0.000 claims description 16
- 230000011218 segmentation Effects 0.000 claims description 16
- 238000010586 diagram Methods 0.000 claims description 15
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 10
- 238000005457 optimization Methods 0.000 claims description 9
- 238000011176 pooling Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 6
- 230000000717 retained effect Effects 0.000 claims description 6
- 238000012216 screening Methods 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 claims 2
- 206010058467 Lung neoplasm malignant Diseases 0.000 description 4
- 238000004195 computer-aided diagnosis Methods 0.000 description 4
- 201000005202 lung cancer Diseases 0.000 description 4
- 208000020816 lung neoplasm Diseases 0.000 description 4
- 208000009956 adenocarcinoma Diseases 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000008676 import Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 206010041823 squamous cell carcinoma Diseases 0.000 description 3
- 230000007812 deficiency Effects 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000002910 structure generation Methods 0.000 description 2
- 101000945318 Homo sapiens Calponin-1 Proteins 0.000 description 1
- 101000652736 Homo sapiens Transgelin Proteins 0.000 description 1
- 102100031013 Transgelin Human genes 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000012407 engineering method Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 208000000649 small cell carcinoma Diseases 0.000 description 1
- JJAHTWIKCUJRDK-UHFFFAOYSA-N succinimidyl 4-(N-maleimidomethyl)cyclohexane-1-carboxylate Chemical compound C1CC(CN2C(C=CC2=O)=O)CCC1C(=O)ON1C(=O)CCC1=O JJAHTWIKCUJRDK-UHFFFAOYSA-N 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
- G06F18/24133—Distances to prototypes
- G06F18/24137—Distances to cluster centroïds
- G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30061—Lung
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
Abstract
The invention provides a medical image analysis method and system based on a graph neural network, comprising the following steps: cutting the medical image of the stained pathological tissue section into a plurality of image blocks; an image background separation detection algorithm is adopted to retain image blocks displayed in an effective tissue area, wherein the effective tissue area is an area containing pathological tissues; performing cell cutting and feature extraction on the reserved image block to obtain feature information of the cell, wherein the feature information comprises position, color and geometric information of the cell; generating a feature map corresponding to the reserved image block according to the feature information of the cell, wherein the node represents the cell and the feature information of the cell, and the node represents the connection and the distance of the adjacent cells; classifying the reserved image blocks by adopting a graph neural network according to the characteristic graph to obtain the categories of the image blocks; and obtaining the type of the medical image before cutting according to the type of the reserved image block. The method and the system have strong interpretability and accurate classification.
Description
Technical Field
The invention relates to the technical field of image analysis, in particular to a medical image analysis method and system based on a graph neural network.
Background
Lung cancer is a worldwide prevalent cancer. When diagnosing cancers such as lung cancer, doctors need to analyze pathological section images of affected tissues of patients and finally judge whether the patients suffer from cancer, the types and the severity of the cancers. However, the process of diagnosis by such doctors is time consuming, takes up valuable working time of many doctors, and seriously affects the efficiency of diagnosis and the volume of patients that can be treated. Moreover, the result of slice judgment by a doctor is easily influenced by personal factors of the doctor, slice imaging effect and other factors. Applying medical image analysis to a Computer Aided Diagnosis (CAD) system can help solve these problems.
Historically, scientists have used a variety of different methods to analyze pathological image patches in order to achieve computer-aided diagnosis of cancer.
After extracting the color, texture and cell network structure features of objects such as cells in an image, the features are classified by a classifier such as Adaboost, SVM, and &lTtTtranslation = L "&gTtL &lTt/T &gTtDA, and whether the image is a cancer image or not is judged as well as other tasks.
In another prior art method, a neural network is used to analyze the pathological image blocks. Neural networks are currently the most fire-hot computer intelligence algorithms. In performing different analytical tasks on various data, including images, neural networks fulfill many tasks that traditional features and algorithms cannot address. In CAD, neural networks are a very important technology after 2009. People use neural networks such as U-Net and the like to analyze a full-size pathological scanning Image (WSI) with the resolution of 10000 × 10000; meanwhile, people put the blocks (tiles) obtained by segmenting the pathological images into deep neural networks such as VGG, ResNet, inclusion and the like for training, diagnose and judge each small area of the whole image, and segment and diagnose details such as cells and the like. The method is widely applied after the outbreak of the neural network in 2012, solves the problem which has never been solved before, and creates the accuracy rate close to that of a doctor. However, neural networks have its most serious problems: non-interpretable. The interpretability of a neural network is a drawback that has caused widespread concern since it began to propagate: even if it achieves extremely good performance, it is not completely trusted by the theoretical academia, since there is no sufficiently convincing theory to demonstrate mathematically the reliability of its performance. This is an ethical disaster for medical tasks that directly involve life and civil care.
Disclosure of Invention
The invention provides a medical image analysis method and system based on a graph neural network, which have strong interpretability and high performance.
According to one aspect of the invention, a medical image analysis method based on a graph neural network is provided, which comprises the following steps:
cutting the medical image of the stained pathological tissue section into a plurality of image blocks;
image blocks displayed in an effective tissue area are reserved by adopting an image background separation detection algorithm, and the rest image blocks are discarded, wherein the effective tissue area is an area containing pathological tissues;
performing cell cutting and feature extraction on the reserved image block to obtain feature information of the cell, wherein the feature information comprises: location, color and geometric information of the cells;
generating a feature map corresponding to the reserved image block according to the feature information of the cell, wherein the node of the feature map represents the cell and the feature information of the cell, and the edge of the feature map represents the connection and the distance of the adjacent cell;
classifying the reserved image blocks by adopting a graph neural network according to the feature graph to obtain the categories of the image blocks, wherein the categories comprise cancer feature image blocks and non-cancer feature image blocks;
and obtaining the type of the medical image before cutting according to the type of the reserved image block.
The medical image analysis method based on the graph neural network comprises a convolutional layer, a global average pooling layer and a softmax layer, wherein the convolutional layer spreads the feature information of the nodes in the feature map to adjacent nodes, the global average pooling layer averages the feature information of all the nodes to obtain the global information of the image block, and the softmax layer classifies the image block according to the global information.
The medical image analysis method based on the graph neural network is characterized in that the graph neural network is optimized through an Adam optimization method, the learning rate is set to be 0.005, and a training set is used: the test set ratio was 8: 2 the test was performed.
The medical image analysis method based on the graph neural network comprises the following steps of:
obtaining a plurality of medical image images with known classification to form an image set, dividing the image set according to a set proportion, wherein one part of the medical images are used as a training set, and the other part of the medical images are used as a testing set;
training the graph neural network through a training set, and evaluating the accuracy of classification of the graph neural network obtained through training through a test set, wherein in the process of training the graph neural network, an Adam optimization method is adopted to dynamically adjust the learning rate.
The medical image analysis method based on the graph neural network, wherein the step of generating the feature graph corresponding to the reserved image block according to the feature information of the cells comprises the following steps:
taking the identification of each cell in the image block and the characteristic information of the cell as nodes;
and obtaining adjacent nodes of each node by adopting a kNN algorithm, and taking the distance between each node and each adjacent node as an edge between each node and each adjacent node.
The medical image analysis method based on the graph neural network comprises the following steps of carrying out cell cutting and feature extraction on the reserved image block to obtain feature information of cells:
performing cell cutting on the image block by using a threshold method and a watershed segmentation algorithm;
and extracting the characteristic information of each cut cell or cell cluster by using a characteristic extraction method.
The medical image analysis method based on the graph neural network, wherein the step of retaining the image blocks displayed in the effective tissue area by adopting an image background separation detection algorithm comprises the following steps of:
obtaining a gray threshold of the background and the pathological tissue of the medical image by adopting an Otsu threshold algorithm, and obtaining an effective tissue area;
filling and removing the generated holes and isolated points of the effective tissue area by applying opening and closing operation to the effective tissue area to generate a mask of the effective tissue area;
and cutting the medical image of the effective tissue area by using the mask to obtain an image block containing pathological tissues.
The medical image analysis method based on the graph neural network, wherein the step of obtaining the classification of the medical image before cutting according to the classification of the reserved image blocks comprises the following steps:
marking the corresponding position of the original medical image according to the type of the image block;
the category of the most image blocks in the original medical image is obtained as the category of the medical image.
According to another aspect of the present invention, there is provided a medical image analysis system based on a graph neural network, including:
the cutting module is used for cutting the medical image of the stained pathological tissue section into a plurality of image blocks;
the screening module is used for retaining the image blocks displayed in the effective tissue area by adopting an image background separation detection algorithm, and discarding the rest of the image blocks, wherein the effective tissue area is an area containing pathological tissues;
the characteristic extraction module is used for carrying out cell cutting and characteristic extraction on the reserved image block to obtain characteristic information of the cell, wherein the characteristic information comprises: location, color and geometric information of the cells;
the characteristic diagram generating module is used for generating a characteristic diagram corresponding to the reserved image block according to the characteristic information of the cell, wherein the node of the characteristic diagram represents the cell and the characteristic information of the cell, and the edge of the characteristic diagram represents the connection and the distance of the adjacent cell;
the classification module is used for classifying the reserved image blocks by adopting a graph neural network according to the feature graph to obtain the categories of the image blocks, wherein the categories comprise cancer feature image blocks and non-cancer feature image blocks; and obtaining the type of the medical image before cutting according to the type of the reserved image block.
The medical image analysis system, wherein the feature map generation module includes:
a node generation unit that takes the identification of each cell in the image block and its characteristic information as a node;
and the edge generation unit is used for obtaining adjacent nodes of each node by adopting a kNN algorithm, and taking the distance between each node and the adjacent node as the edge between each node and the adjacent node.
The medical image analysis method and system based on the graph neural network use the graph neural network, and the characteristics of each cell can be directly put into the graph to be used as node characteristics. The information in the nodes is transmitted by connecting the edges of the nodes, so that the obtained information of the whole graph is classified, the interpretability is strong, and the classification is accurate.
Drawings
FIG. 1 is a schematic diagram of a flowchart of a medical image analysis method based on a graph neural network according to the present invention;
FIG. 2 is a schematic diagram of a block diagram of a medical image analysis system based on a graph neural network according to the present invention;
fig. 3 is a schematic diagram of an embodiment of a medical image analysis system based on a graph neural network according to the present invention.
Detailed Description
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It may be evident, however, that such embodiment(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more embodiments.
Various embodiments according to the present invention will be described in detail below with reference to the accompanying drawings.
Fig. 1 is a schematic view of a flowchart of a medical image analysis method based on a graph neural network according to the present invention, and as shown in fig. 1, the medical image analysis method includes:
step S1, cutting the medical image of the stained pathological tissue section into a plurality of image blocks;
step S2, image blocks displayed in an effective tissue area are retained by adopting an image background separation detection algorithm, and the rest image blocks are discarded, wherein the effective tissue area is an area containing pathological tissues;
step S3, performing cell cutting and feature extraction on the reserved image block to obtain feature information of the cell, wherein the feature information comprises the position, color, geometric information and the like of the cell, the color comprises one or more of an average value, a median, an upper quartile, a lower quartile, a maximum minimum value and a standard deviation, and the geometric information comprises one or more of an area, a perimeter and a diameter;
step S4, generating a feature map corresponding to the reserved image block according to the feature information of the cell, wherein the node of the feature map represents the cell and the feature information of the cell, and the edge of the feature map represents the connection and distance of the adjacent cells;
step S5, classifying the reserved image blocks according to the feature map by using a graph neural network to obtain categories of the image blocks, where the categories include cancer feature image blocks and non-cancer feature image blocks, and the categories may also be divided according to the type of cancer, for example, the categories may also include small cell feature image blocks, adenocarcinoma feature image blocks, squamous carcinoma feature image blocks, and the like, and preferably, the graph neural network includes a convolutional layer, a global average pooling layer, and a softmax layer, the convolutional layer spreads the feature information of the nodes in the feature map to adjacent nodes, the global average pooling layer averages the feature information of all the nodes to obtain global information of the image blocks, and the softmax layer classifies the image blocks according to the global information;
step S6, obtaining the type of the medical image before segmentation according to the type of the reserved image blocks, where the type of the medical image corresponds to the type of the image block, that is, the type of the medical image includes a cancer-characteristic medical image and a non-cancer-characteristic medical image, and the type of the medical image may be further divided according to the type of the cancer, for example, the medical image may further include a small cell-characteristic medical image, an adenocarcinoma-characteristic medical image, and a squamous carcinoma-characteristic medical image.
In one embodiment, the step of training the neural network of the graph comprises:
obtaining a plurality of medical image images with known classification to form an image set, dividing the image set according to a set proportion, wherein one part of the medical images are used as a training set, and the other part of the medical images are used as a testing set;
training the graph neural network through a training set, and evaluating the accuracy of classification of the graph neural network obtained through training through a test set, wherein in the process of training the graph neural network, an Adam optimization method is adopted to dynamically adjust the learning rate.
Preferably, the neural network of the graph is optimized by an Adam optimization method, and the learning rate is set to 0.005, so as to train the set: the test set ratio was 8: 2 the test was performed.
In the training method, an Adam optimization method is adopted to reduce the difference between the output of the graph neural network in the training set and the target output (namely the cancer type corresponding to the picture), so as to obtain the parameters of the graph neural network; the training set is used as the input and the target output of the graph neural network in the training process, and the test set is used for evaluating the classification result of the graph neural network obtained by training; the Adam optimization method dynamically adjusts the learning rate, and the effect of accelerating training is achieved.
In one embodiment, in step S2, the step of retaining the image blocks displayed in the effective tissue area by using the image background separation detection algorithm includes:
obtaining a gray threshold of the background and the pathological tissue of the medical image by adopting an Otsu threshold algorithm, and obtaining an effective tissue area;
filling and removing the generated holes and isolated points of the effective tissue area by applying opening and closing operation to the effective tissue area to generate a mask of the effective tissue area;
and cutting the medical image of the effective tissue area by using the mask to obtain an image block containing pathological tissues.
In one embodiment, in step S3, the cell segmentation and feature extraction are performed on the retained image block, and the step of obtaining feature information of the cell includes:
performing cell cutting on the image block by using a threshold method and a watershed segmentation algorithm;
and extracting the characteristic information of each cut cell or cell cluster by using a characteristic extraction method.
In one embodiment, in step S4, the step of generating a feature map corresponding to the reserved image block according to the feature information of the cell includes:
taking the identification of each cell in the image block and the characteristic information of the cell as nodes;
and obtaining adjacent nodes of each node by adopting a kNN algorithm, and taking the distance between each node and each adjacent node as an edge between each node and each adjacent node.
In one embodiment, in step S6, the step of obtaining a classification of the pre-segmentation medical image according to the classification of all the retained image blocks includes:
marking the corresponding position of the original medical image according to the type of the image block;
the category of the most image blocks in the original medical image is obtained as the category of the medical image.
Fig. 2 is a schematic diagram of a block diagram of a medical image analysis system based on a graph neural network according to the present invention, and as shown in fig. 2, the medical image analysis system includes:
the cutting module 10 is used for cutting the medical image of the stained pathological tissue section into a plurality of image blocks;
the screening module 20 is configured to retain the image blocks displayed in the effective tissue area by using an image background separation detection algorithm, and discard the rest of the image blocks, wherein the effective tissue area is an area containing pathological tissues;
the feature extraction module 30 performs cell segmentation and feature extraction on the retained image block to obtain feature information of the cell, where the feature information includes: location, color, geometric information of the cells;
the feature map generating module 40 is configured to generate a feature map corresponding to the reserved image block according to feature information of the cell, where nodes of the feature map represent the cell and feature information of the cell, and edges of the feature map represent connection and distance between adjacent cells;
the classification module 50 is configured to classify the reserved image blocks according to the feature map by using a map neural network to obtain categories of the image blocks, where the categories include cancer feature image blocks and non-cancer feature image blocks; and obtaining the type of the medical image before cutting according to the type of the reserved image block.
Preferably, the medical image analysis system further includes a training module 60, which trains the neural network before classifying the image blocks of the medical image of unknown class, where the training module 60 includes:
a training set and test set constructing unit, which is used for obtaining a plurality of known classified medical image images to form an image set, dividing the image set according to a set proportion, wherein one part of the medical images are used as a training set, and the other part of the medical images are used as a test set;
and the training unit is used for training the graph neural network through a training set and evaluating the accuracy of classification of the graph neural network obtained through training through a test set, wherein in the process of training the graph neural network, an Adam optimization method is adopted to dynamically adjust the learning rate.
In one embodiment, the screening module 20 includes:
the effective tissue area division module is used for obtaining a gray threshold of the background and the pathological tissue of the medical image by adopting an Otsu threshold algorithm and obtaining an effective tissue area;
a mask generating unit which applies open-close operation to the effective tissue area, fills and removes the generated holes and isolated points of the effective tissue area, and generates a mask of the effective tissue area;
and the cutting unit is used for cutting the medical image of the effective tissue area by using the mask to obtain an image block containing pathological tissues.
In one embodiment, the feature extraction module 30 includes:
the cell cutting unit is used for cutting cells of the image block by using a threshold method and a watershed segmentation algorithm;
and a feature information extraction unit for extracting feature information of each cut cell or cell group by using a feature extraction method.
In one embodiment, the above feature map generation module 40 includes:
a node generation unit that takes the identification of each cell in the image block and its characteristic information as a node;
and the edge generation unit is used for obtaining adjacent nodes of each node by adopting a kNN algorithm, and taking the distance between each node and the adjacent node as the edge between each node and the adjacent node.
In one embodiment, the classification module 50 includes:
the image block classifying unit is used for classifying the reserved image blocks by adopting an image neural network according to the feature map to obtain the categories of the image blocks, wherein the categories comprise cancer feature image blocks and non-cancer feature image blocks;
the marking unit marks the corresponding position of the original medical image according to the type of the image block;
and a classification unit for obtaining the most image block types in the original medical image as the medical image types.
In one embodiment of the present invention, as shown in FIG. 3, the medical image analysis system is mainly divided into two parts, a preprocessing part and a GNN part.
The preprocessing part mainly comprises three steps of image detection segmentation, feature extraction and graph structure generation, and specifically comprises the following steps of: obtaining a background of the medical image and a gray threshold of pathological tissues by adopting an Otsu threshold algorithm, and obtaining a pathological tissue area; applying opening and closing operation to the pathological tissue area, filling and removing the generated holes and isolated points of the tissue area, and generating a mask of the pathological tissue area; cutting the medical image of the pathological tissue area by using a mask to obtain an image block containing pathological tissues; using an IdentifyPrimaryObject module of CellProfiler software, and performing cell segmentation on the image by using an Otsu threshold method and a watershed segmentation algorithm in the IdentifyPrimaryObject module; extracting cell features using a measureobject intensity module and a measureobject sizeshape module of CellProfiler; adopting the characteristic information of the cells as nodes and characteristics thereof; and for each node, connecting adjacent nodes by adopting a kNN algorithm according to the position of the corresponding cell, wherein k is 6.
As shown in fig. 3, for a complete, large, labeled full-size medical image 1, they are first cut into 256 x 256 image patches 2. According to the image background separation detection algorithm which is written by people and is specially used for segmenting the images, the images displayed in the effective organization area are reserved, and the rest images are discarded.
The function of the preprocessing unit may be implemented by a combination of hardware and software, and for example, includes a memory and a processor, where the memory stores a preprocessing program, and the preprocessing program, when executed by the processor, implements the steps of image detection segmentation, feature extraction, and graph structure generation, for example, an image detection segmentation part in the preprocessing program may be set as follows:
import cv2
import numpy as np import openslide def getbackground(imgpath,level＝4,kernel_size＝(16,8)):
srcimg＝openslide.open_slide(imgpath)
assert level<＝srcimg.level_count-1and level>＝0,'Level out of range'
dim＝srcimg.level_dimensions[level]
thumbnail＝np.array(srcimg.read_region((0,0),level,dim))
thumbnail＝cv2.cvtColor(thumbnail,cv2.COLOR_RGB2GRAY)
ret,blacktile＝cv2.threshold(thumbnail,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
kernel1＝cv2.getStructuringElement(cv2.MORPH_RECT,(kernel_size[0],kernel_size[0]))
kernel2＝cv2.getStructuringElement(cv2.MORPH_RECT,(kernel_size[1],kernel_size[1]))
open＝cv2.morphologyEx(blacktile,cv2.MORPH_OPEN,kernel1)
mask＝cv2.morphologyEx(open,cv2.MORPH_CLOSE,kernel2)
return mask
after the image block is well segmented, a pipeline applied to CellProfiler software is constructed for cell segmentation and feature extraction. CellProfiler is an open source software that can be used to segment out areas of cells and to derive statistical characteristics of the color and geometry of the images in these areas. After obtaining the statistical features, a kNN graph (feature graph 3) with k equal to 8 neighbors is built and put into the GNN part.
In the GNN part, a network composed of a two-layer graph convolutional neural network (GCN network), a global mean posing layer and a log softmax layer is used for training, Adam optimizer is used in the network, the learning rate is set to 0.01, so as to train the set: and (4) verification set: the test set ratio was 8: 1: 1 the test was carried out. As shown in FIG. 3, the classification effect on a single image block can reach 91.0% initially, and the cancer type of the medical image (N: normal, SMCC: small cell carcinoma, A: adenocarcinoma, SCC: squamous carcinoma) can be obtained
The medical image analysis method based on the graph neural network extracts the position, color and geometric information of cells in the lung cancer slice pathological image by using small pictures in the lung cancer slice pathological image, and improves the classification accuracy by using the graph convolution neural network (GCN). The GNN is considered to have stronger interpretability, occupy less computing resources and have larger application potential. The characteristic engineering research method of the structure and pathological image of the GNN is matched well: the pathological image is composed of cells, and the pathological image analysis uses the characteristics of a graph composed of cell characteristics and cell positions, which correspond to nodes in the GNN and a graph formed by connecting the nodes. By combining the interpretability of the manual feature with its excellent properties of GNN, the present application makes up for both the performance deficiencies of feature engineering methods and the interpretability deficiencies of common machine learning methods.
While the foregoing disclosure shows illustrative embodiments of the invention, it should be noted that various changes and modifications could be made herein without departing from the scope of the invention as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the inventive embodiments described herein need not be performed in any particular order. Furthermore, although elements of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to a single element is explicitly stated.
Claims (10)
1. A medical image analysis method based on a graph neural network is characterized by comprising the following steps:
cutting the medical image of the stained pathological tissue section into a plurality of image blocks;
image blocks displayed in an effective tissue area are reserved by adopting an image background separation detection algorithm, and the rest image blocks are discarded, wherein the effective tissue area is an area containing pathological tissues;
performing cell cutting and feature extraction on the reserved image block to obtain feature information of the cell, wherein the feature information comprises: location, color and geometric information of the cells;
generating a feature map corresponding to the reserved image block according to the feature information of the cell, wherein the node of the feature map represents the cell and the feature information of the cell, and the edge of the feature map represents the connection and the distance of the adjacent cell;
classifying the reserved image blocks by adopting a graph neural network according to the feature graph to obtain the categories of the image blocks, wherein the categories comprise cancer feature image blocks and non-cancer feature image blocks;
and obtaining the type of the medical image before cutting according to the type of the reserved image block.
2. The medical image analysis method based on the graph neural network according to claim 1, wherein the graph neural network comprises a convolutional layer, a global averaging pooling layer and a softmax layer, the convolutional layer spreads the feature information of the nodes in the feature map to adjacent nodes, the global averaging pooling layer averages the feature information of all the nodes to obtain global information of the image block, and the softmax layer classifies the image block according to the global information.
3. The method for medical image analysis based on graph neural network of claim 1, wherein the graph neural network is optimized by Adam optimization method, learning rate is set to 0.005, so as to train set: the test set ratio was 8: 2 the test was performed.
4. The method for medical image analysis based on graph neural network of claim 3, wherein the step of training the graph neural network comprises:
obtaining a plurality of medical image images with known classification to form an image set, dividing the image set according to a set proportion, wherein one part of the medical images are used as a training set, and the other part of the medical images are used as a testing set;
training the graph neural network through a training set, and evaluating the accuracy of classification of the graph neural network obtained through training through a test set, wherein in the process of training the graph neural network, an Adam optimization method is adopted to dynamically adjust the learning rate.
5. The method for analyzing medical images based on graph neural network of claim 1, wherein the step of generating the feature map corresponding to the reserved image block according to the feature information of the cells comprises:
taking the identification of each cell in the image block and the characteristic information of the cell as nodes;
and obtaining adjacent nodes of each node by adopting a kNN algorithm, and taking the distance between each node and each adjacent node as an edge between each node and each adjacent node.
6. The method for analyzing medical images based on graph neural network of claim 1, wherein the step of performing cell segmentation and feature extraction on the retained image blocks to obtain feature information of cells comprises:
performing cell cutting on the image block by using a threshold method and a watershed segmentation algorithm;
and extracting the characteristic information of each cut cell or cell cluster by using a characteristic extraction method.
7. The method for medical image analysis based on graph neural network of claim 1, wherein the step of retaining the image blocks displayed in the effective tissue area by using the image background separation detection algorithm comprises:
obtaining a gray threshold of the background and the pathological tissue of the medical image by adopting an Otsu threshold algorithm, and obtaining an effective tissue area;
filling and removing the generated holes and isolated points of the effective tissue area by applying opening and closing operation to the effective tissue area to generate a mask of the effective tissue area;
and cutting the medical image of the effective tissue area by using the mask to obtain an image block containing pathological tissues.
8. The method for analyzing medical images based on neural network of any one of claims 1-7, wherein the step of obtaining the classification of the medical image before segmentation according to the classification of all the retained image blocks comprises:
marking the corresponding position of the original medical image according to the type of the image block;
the category of the most image blocks in the original medical image is obtained as the category of the medical image.
9. A medical image analysis system based on a graph neural network is characterized by comprising:
the cutting module is used for cutting the medical image of the stained pathological tissue section into a plurality of image blocks;
the screening module is used for retaining the image blocks displayed in the effective tissue area by adopting an image background separation detection algorithm, and discarding the rest of the image blocks, wherein the effective tissue area is an area containing pathological tissues;
the characteristic extraction module is used for carrying out cell cutting and characteristic extraction on the reserved image block to obtain characteristic information of the cell, wherein the characteristic information comprises: location, color and geometric information of the cells;
the characteristic diagram generating module is used for generating a characteristic diagram corresponding to the reserved image block according to the characteristic information of the cell, wherein the node of the characteristic diagram represents the cell and the characteristic information of the cell, and the edge of the characteristic diagram represents the connection and the distance of the adjacent cell;
the classification module is used for classifying the reserved image blocks by adopting a graph neural network according to the feature graph to obtain the categories of the image blocks, wherein the categories comprise cancer feature image blocks and non-cancer feature image blocks; and obtaining the type of the medical image before cutting according to the type of the reserved image block.
10. The medical image analysis system of claim 9, wherein the feature map generation module comprises:
a node generation unit that takes the identification of each cell in the image block and its characteristic information as a node;
and the edge generation unit is used for obtaining adjacent nodes of each node by adopting a kNN algorithm, and taking the distance between each node and the adjacent node as the edge between each node and the adjacent node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010180739.0A CN111462052A (en) | 2020-03-16 | 2020-03-16 | Medical image analysis method and system based on graph neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010180739.0A CN111462052A (en) | 2020-03-16 | 2020-03-16 | Medical image analysis method and system based on graph neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111462052A true CN111462052A (en) | 2020-07-28 |
Family
ID=71685161
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010180739.0A Pending CN111462052A (en) | 2020-03-16 | 2020-03-16 | Medical image analysis method and system based on graph neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111462052A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112132223A (en) * | 2020-09-27 | 2020-12-25 | 腾讯科技（深圳）有限公司 | Graph pooling method, device, equipment and storage medium |
| CN113256577A (en) * | 2021-05-18 | 2021-08-13 | 湖南医药学院 | Cancer auxiliary analysis system and device based on HE staining pathological image |
| CN113570001A (en) * | 2021-09-22 | 2021-10-29 | 深圳市信润富联数字科技有限公司 | Classification identification positioning method, device, equipment and computer readable storage medium |
| WO2022107435A1 (en) * | 2020-11-20 | 2022-05-27 | コニカミノルタ株式会社 | Image analysis method, image analysis system, and program |
| TWI781027B (en) * | 2021-12-22 | 2022-10-11 | 國立臺南大學 | Neural network system for staining images and image staining conversion method |
| CN115272342A (en) * | 2022-09-29 | 2022-11-01 | 深圳大学 | Cell differentiation degree evaluation method based on bright field image, storage medium and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108717554A (en) * | 2018-05-22 | 2018-10-30 | 复旦大学附属肿瘤医院 | A kind of thyroid tumors histopathologic slide image classification method and its device |
| CN108765408A (en) * | 2018-05-31 | 2018-11-06 | 杭州同绘科技有限公司 | Build the method in cancer pathology image virtual case library and the multiple dimensioned cancer detection system based on convolutional neural networks |
| US20190065817A1 (en) * | 2017-08-29 | 2019-02-28 | Konica Minolta Laboratory U.S.A., Inc. | Method and system for detection and classification of cells using convolutional neural networks |
| CN109614869A (en) * | 2018-11-10 | 2019-04-12 | 天津大学 | A kind of pathological image classification method based on multi-scale compress rewards and punishments network |
| CN110060245A (en) * | 2019-04-15 | 2019-07-26 | 青岛大学 | A kind of gastric cancer lymph node dyeing pathological image automatic identifying method based on deep neural network |
-
2020
- 2020-03-16 CN CN202010180739.0A patent/CN111462052A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190065817A1 (en) * | 2017-08-29 | 2019-02-28 | Konica Minolta Laboratory U.S.A., Inc. | Method and system for detection and classification of cells using convolutional neural networks |
| CN108717554A (en) * | 2018-05-22 | 2018-10-30 | 复旦大学附属肿瘤医院 | A kind of thyroid tumors histopathologic slide image classification method and its device |
| CN108765408A (en) * | 2018-05-31 | 2018-11-06 | 杭州同绘科技有限公司 | Build the method in cancer pathology image virtual case library and the multiple dimensioned cancer detection system based on convolutional neural networks |
| CN109614869A (en) * | 2018-11-10 | 2019-04-12 | 天津大学 | A kind of pathological image classification method based on multi-scale compress rewards and punishments network |
| CN110060245A (en) * | 2019-04-15 | 2019-07-26 | 青岛大学 | A kind of gastric cancer lymph node dyeing pathological image automatic identifying method based on deep neural network |
Non-Patent Citations (1)
| Title |
|---|
| DEEPAK ANAND等: "Histographs: Graphs in Histopathology", 《ARXIV:1908.05020V1》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112132223A (en) * | 2020-09-27 | 2020-12-25 | 腾讯科技（深圳）有限公司 | Graph pooling method, device, equipment and storage medium |
| CN112132223B (en) * | 2020-09-27 | 2024-02-27 | 腾讯科技（深圳）有限公司 | Method, device, equipment and storage medium for pooling |
| WO2022107435A1 (en) * | 2020-11-20 | 2022-05-27 | コニカミノルタ株式会社 | Image analysis method, image analysis system, and program |
| CN113256577A (en) * | 2021-05-18 | 2021-08-13 | 湖南医药学院 | Cancer auxiliary analysis system and device based on HE staining pathological image |
| CN113570001A (en) * | 2021-09-22 | 2021-10-29 | 深圳市信润富联数字科技有限公司 | Classification identification positioning method, device, equipment and computer readable storage medium |
| CN113570001B (en) * | 2021-09-22 | 2022-02-15 | 深圳市信润富联数字科技有限公司 | Classification identification positioning method, device, equipment and computer readable storage medium |
| TWI781027B (en) * | 2021-12-22 | 2022-10-11 | 國立臺南大學 | Neural network system for staining images and image staining conversion method |
| CN115272342A (en) * | 2022-09-29 | 2022-11-01 | 深圳大学 | Cell differentiation degree evaluation method based on bright field image, storage medium and system |
| CN115272342B (en) * | 2022-09-29 | 2023-02-17 | 深圳大学 | Bright field image-based cell differentiation degree evaluation method, storage medium and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111462052A (en) | Medical image analysis method and system based on graph neural network | |
| CN107886514B (en) | Mammary gland molybdenum target image lump semantic segmentation method based on depth residual error network | |
| CN110490212B (en) | Molybdenum target image processing equipment, method and device | |
| WO2020253629A1 (en) | Detection model training method and apparatus, computer device, and storage medium | |
| CN111985536B (en) | Based on weak supervised learning gastroscopic pathology image Classification method | |
| CN109087703B (en) | Peritoneal transfer marking method of abdominal cavity CT image based on deep convolutional neural network | |
| CN107665492B (en) | Colorectal panoramic digital pathological image tissue segmentation method based on depth network | |
| CN105427296B (en) | A kind of thyroid gland focus image-recognizing method based on ultrasonoscopy low rank analysis | |
| CN110309860B (en) | Method for classifying malignancy degree of lung nodule based on convolutional neural network | |
| CN112017191A (en) | Method for establishing and segmenting liver pathology image segmentation model based on attention mechanism | |
| Mi et al. | Deep learning-based multi-class classification of breast digital pathology images | |
| CN110021425B (en) | Comparison detector, construction method thereof and cervical cancer cell detection method | |
| Xu et al. | Computerized classification of prostate cancer gleason scores from whole slide images | |
| CN112734741B (en) | Image processing method and system for pneumonia CT image | |
| CN108305253A (en) | A kind of pathology full slice diagnostic method based on more multiplying power deep learnings | |
| CN113449785B (en) | Eyelid tumor digital pathological section image multi-classification method based on deep learning | |
| CN110766670A (en) | Mammary gland molybdenum target image tumor localization algorithm based on deep convolutional neural network | |
| CN114841947A (en) | Method and device for multi-scale feature extraction and prognosis analysis of H & E staining pathological image tumor region | |
| CN113077419A (en) | Information processing method and device for hip joint CT image recognition | |
| CN111815609A (en) | Pathological image classification method and system based on context awareness and multi-model fusion | |
| CN112036298A (en) | Cell detection method based on double-segment block convolutional neural network | |
| CN111784638A (en) | Pulmonary nodule false positive screening method and system based on convolutional neural network | |
| CN114511588A (en) | Method and device for judging benign and malignant breast tissue pathological image | |
| CN112927215A (en) | Automatic analysis method for digestive tract biopsy pathological section | |
| CN112446881A (en) | Pathological image segmentation system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200728 |