CN113313164A - Digital pathological image classification method and system based on superpixel segmentation and image convolution - Google Patents
Digital pathological image classification method and system based on superpixel segmentation and image convolution Download PDFInfo
- Publication number
- CN113313164A CN113313164A CN202110585365.5A CN202110585365A CN113313164A CN 113313164 A CN113313164 A CN 113313164A CN 202110585365 A CN202110585365 A CN 202110585365A CN 113313164 A CN113313164 A CN 113313164A
- Authority
- CN
- China
- Prior art keywords
- node
- graph
- pixel
- super
- segmentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 68
- 230000001575 pathological effect Effects 0.000 title claims abstract description 56
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000013528 artificial neural network Methods 0.000 claims abstract description 24
- 238000012549 training Methods 0.000 claims abstract description 18
- 230000007170 pathology Effects 0.000 claims abstract description 15
- 238000012360 testing method Methods 0.000 claims abstract description 14
- 238000012795 verification Methods 0.000 claims abstract description 13
- 239000011159 matrix material Substances 0.000 claims description 41
- 238000011176 pooling Methods 0.000 claims description 25
- 230000004913 activation Effects 0.000 claims description 7
- 230000007246 mechanism Effects 0.000 claims description 7
- 238000005192 partition Methods 0.000 claims description 7
- 238000005259 measurement Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 3
- 230000006870 function Effects 0.000 description 13
- 238000013527 convolutional neural network Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 6
- 238000003745 diagnosis Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 239000013598 vector Substances 0.000 description 3
- 238000001574 biopsy Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004195 computer-aided diagnosis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 210000004907 gland Anatomy 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000010827 pathological analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Evolutionary Biology (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Medical Informatics (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a digital pathological image classification method and a system based on superpixel segmentation and graph convolution, wherein the method comprises the following steps of: s1: performing superpixel segmentation on the digital pathological image by using a simple linear iterative clustering algorithm SLIC (linear iterative clustering algorithm) to obtain a superpixel segmentation area; s2: constructing a graph structure based on the digital pathological image by taking each super-pixel region as a node in the graph structure and taking whether the super-pixel regions share edges as the basis for distributing the edges among the nodes; s3: randomly dividing the constructed data of the graph structure into a training set, a verification set and a test set according to a preset proportion, and obtaining an optimal prediction model through a training, verification and test graph convolution neural network; s4: classifying the digital pathology image based on the prediction model. The invention realizes the classification and prediction of the digital pathological images by using the graph convolution neural network, and improves the accuracy of pathological image classification.
Description
Technical Field
The invention belongs to the technical field of computer-aided diagnosis, and particularly relates to a digital pathological image classification method and system based on superpixel segmentation and graph convolution.
Background
Normative and uniform pathology reports provide sufficient information for clinicians, but morphological assessment of pathology indicators adds workload to pathologists. Due to the maturity and standardization of the pathological biopsy technology, the diagnosis number of pathological biopsy samples is continuously increased, and the diagnosis result is influenced by factors such as fatigue and subjective experience. In recent years, rapid advances in digital pathology scanning techniques and Artificial Intelligence (AI) have provided tremendous potential for pathomorphological diagnosis. Specifically, a deep Convolutional Neural Network (CNN) is utilized to extract high-dimensional features in a pathological image, and the pathological image is predicted and classified.
However, the diagnosis of the pathologist mainly provides the final pathological analysis result according to the factors such as the morphology, the structure and the spatial distribution of abnormal cells in the pathological image. While the conventional CNN processes a structured two-dimensional array image with pixel values, the data in the form of the structured two-dimensional array is difficult to express the relationship and histological features between cells and glands in histopathology, and the spatial relationship between microscopic cells is also ignored, so that some key feature information for improving model performance is lost.
Because the digital pathological images have extremely high resolution and complex and rich histological features, how to effectively construct the graph structure of the digital pathological image and improve the classification accuracy of the digital pathological image by using a graph convolution network is a major challenge at present.
Disclosure of Invention
The invention mainly aims to overcome the defects of the prior art and provide a digital pathological image classification method and system based on superpixel segmentation and graph convolution.
According to one aspect of the present invention, there is provided a method for classifying digital pathological images based on super-pixel segmentation convolved with images, the method comprising the steps of:
s1: performing superpixel segmentation on the digital pathological image by using a simple linear iterative clustering algorithm SLIC (linear iterative clustering algorithm) to obtain a superpixel segmentation area;
s2: constructing a graph structure based on the digital pathological image by taking each super-pixel region as a node in the graph structure and taking whether the super-pixel regions share edges as the basis for distributing the edges among the nodes;
s3: randomly dividing the constructed data of the graph structure into a training set, a verification set and a test set according to a preset proportion, and obtaining an optimal prediction model through a training, verification and test graph convolution neural network;
s4: classifying the digital pathology image based on the prediction model.
Preferably, the step S1 includes:
s11: initializing seed points, namely initializing a clustering center, and setting the number of superpixels to be segmented;
s12: reselecting the seed points in the n multiplied by n neighborhood of the seed points according to the gradient values of the pixel points so as to avoid the seed points falling on the edge with larger gradient;
s13: distributing a class label to each pixel point in the neighborhood around each seed point, and determining the clustering center to which the pixel point belongs;
s14: and (4) performing distance measurement, and respectively calculating the distance from each retrieved pixel point to the seed point.
Preferably, the step S2 includes:
s21: performing super-pixel segmentation on the digital pathological image by using a SLIC algorithm to obtain a super-pixel segmentation area;
s22: taking each super-pixel segmentation area as a node in a graph structure, wherein the number of the nodes is n;
s23: performing low-dimensional coding on each partition area by using a CNN (compressed natural network) network to serve as feature information of each node, wherein the feature dimension of each node is d;
s24: on the digital pathological image after the super-pixel segmentation, whether each segmentation area shares the edge or not is used as the basis for distributing the edge between the nodes, so that an adjacent matrix A of a graph structure is established; the graph structure is defined as graph G (A, F), where A ∈ {0, 1}n×nIn the form of a contiguous matrix, the matrix,
is a node feature matrix, each node has d-dimensional features.
Preferably, the step S3 includes:
constructing a multi-scale graph convolution neural network by utilizing graph pooling, extracting characteristic information on each scale through graph convolution, using a characteristic cross layer between two continuous scales of each network layer, and fusing multi-scale characteristics in an intermediate layer; carrying out node feature combination on the graph structure output on each scale, thereby obtaining feature information of the whole graph on the corresponding scale; the concrete formula of the graph convolution neural network is as follows:
where a is the activation function and where a is the activation function,
and (3) in each layer, iterating for multiple times by using a graph convolution neural network to output node information, wherein the I is a diagonal matrix and the trainable weight matrix is a trainable weight matrix.
Preferably, the step S4 includes: and obtaining a final classification result through an MLP layer and a soft voting mechanism.
Preferably, the constructing a multi-scale graph convolution neural network by utilizing graph pooling comprises:
and carrying out node combination by calculating similarity values between the characteristics of adjacent nodes in the graph structure to obtain the graph structures with different scales, wherein the similarity values are determined according to Euclidean distances between the characteristics, and a group of new adjacency matrixes A 'and node characteristic matrixes F' are obtained after graph pooling.
The use of a feature intersection layer between two consecutive scales of each network layer includes:
and recording the position index of the node after graph pooling in the original node matrix F, and thus adding the characteristic values of the corresponding node characteristics under different scales according to the position index to realize information interaction of adjacent scales.
Preferably, the obtaining of the final classification result through the MLP layer and the soft voting mechanism includes:
performing global maximum pooling and global average pooling on the node feature matrix output on each scale, and splicing two groups of pooling results to be used as input data of an MLP layer; the MLP layer consists of a full connection layer and a Softmax function, and the class prediction probability under each scale is obtained through the Softmax function;
and carrying out weighted average on the class prediction probabilities under different scales so as to output the classification class with the maximum probability value.
According to another aspect of the present invention, there is also provided a digital pathology image classification system based on super-pixel segmentation convolved with a map, the system comprising:
the segmentation module is used for performing superpixel segmentation on the digital pathological image by using a simple linear iterative clustering algorithm SLIC (linear iterative clustering algorithm) to obtain a superpixel segmentation area;
the construction module is used for constructing a graph structure based on the digital pathological image by taking each super-pixel region as a node in the graph structure and taking whether the super-pixel regions share edges as the basis for distributing edges among the nodes;
the training module is used for randomly dividing the constructed data of the graph structure into a training set, a verification set and a test set according to a preset proportion, and obtaining an optimal prediction model through a training, verification and test graph convolution neural network;
and the classification module is used for classifying the digital pathological image based on the prediction model.
Preferably, the segmentation module is further configured to:
initializing seed points, namely initializing a clustering center, and setting the number of superpixels to be segmented;
reselecting the seed points in the n multiplied by n neighborhood of the seed points according to the gradient values of the pixel points so as to avoid the seed points falling on the edge with larger gradient;
distributing a class label to each pixel point in the neighborhood around each seed point, and determining the clustering center to which the pixel point belongs;
and (4) performing distance measurement, and respectively calculating the distance from each retrieved pixel point to the seed point.
Preferably, the building module is further configured to:
performing super-pixel segmentation on the digital pathological image by using a SLIC algorithm to obtain a super-pixel segmentation area;
taking each super-pixel segmentation area as a node in a graph structure, wherein the number of the nodes is n;
performing low-dimensional coding on each partition area by using a CNN (compressed natural network) network to serve as feature information of each node, wherein the feature dimension of each node is d;
on the digital pathological image after the super-pixel segmentation, whether each segmentation area shares the edge or not is used as the basis for distributing the edge between the nodes, so that an adjacent matrix A of a graph structure is established; the graph structure is defined as graph G (A, F), where A ∈ {0, 1}n×nIn the form of a contiguous matrix, the matrix,
is a node feature matrix, each node has d-dimensional features.
Has the advantages that: the method can directly construct the expression form of the graph structure of the digital pathological image with extremely high resolution, so that the graph structure of the whole digital pathological image is learned by utilizing the graph convolution neural network, the problem that the feature extraction cannot be directly carried out on the whole digital pathological image due to the limitation of the memory is avoided, and the accuracy of classification is improved by utilizing a multi-scale and feature intersection method.
The features and advantages of the present invention will become apparent by reference to the following drawings and detailed description of specific embodiments of the invention.
Drawings
FIG. 1 is a flow chart of a method of digital pathology image classification based on superpixel segmentation convolved with a map;
FIG. 2 is a schematic diagram of a convolutional neural network structure according to the present invention;
FIG. 3 is a schematic view of the pooling structure of the present invention;
FIG. 4 is a schematic diagram of a merge node feature of the present invention;
FIG. 5 is a result graph of the present invention after performing superpixel segmentation on a digital pathology image using the SLIC algorithm;
FIG. 6 is a diagram illustrating the result of a graph structure constructed using superpixel partition regions in accordance with the present invention;
fig. 7 is a structural diagram of a digital pathological image classification system based on super-pixel segmentation and graph convolution.
Detailed Description
The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
Fig. 1 is a flow chart of a digital pathology image classification method based on superpixel segmentation convolved with a graph. As shown in fig. 1, the present invention provides a digital pathological image classification method based on superpixel segmentation and graph convolution, which comprises the following steps:
s1: and performing superpixel segmentation on the digital pathological image by using a simple linear iterative clustering algorithm SLIC (Linear iterative clustering algorithm) to obtain a superpixel segmentation area.
Specifically, step S1 includes:
s11: initializing seed points, namely initializing a clustering center, and setting the number of superpixels to be segmented;
s12: reselecting the seed points in the n multiplied by n neighborhood of the seed points according to the gradient values of the pixel points so as to avoid the seed points falling on the edge with larger gradient;
s13: distributing a class label to each pixel point in the neighborhood around each seed point, and determining the clustering center to which the pixel point belongs;
s14: and (4) performing distance measurement, and respectively calculating the distance from each retrieved pixel point to the seed point.
Wherein performing distance metrics includes color and spatial distance.
The seed number of step S11 in the SLIC algorithm, i.e., the number of superpixels to be segmented, is characterized as a superparameter, and is manually set according to an actual digital pathology image.
S2: and constructing a graph structure based on the digital pathological image by taking each super-pixel region as a node in the graph structure and taking whether the super-pixel regions share edges as the basis for distributing edges among the nodes.
Specifically, step S2 includes:
s21: performing super-pixel segmentation on the digital pathological image by using a SLIC algorithm to obtain a super-pixel segmentation area;
s22: taking each super-pixel segmentation area as a node in a graph structure, wherein the number of the nodes is n;
s23: performing low-dimensional coding on each partition area by using a CNN (compressed natural network) network to serve as feature information of each node, wherein the feature dimension of each node is d;
s24: on the digital pathological image after the super-pixel segmentation, whether each segmentation area shares the edge or not is used as the basis for distributing the edge between the nodes, so that an adjacent matrix A of a graph structure is established; the graph structure is defined as graph G (A, F), where A ∈ {0, 1}n×nIn the form of a contiguous matrix, the matrix,
is a node feature matrix, each node has d-dimensional features. FIG. 6 shows a resulting schematic of the graph structure constructed by the present invention using superpixel segmentation.
Fig. 5 is a result diagram of the present invention after performing superpixel segmentation on a digital pathology image by using the SLIC algorithm. The segmented regions obtained by the SLIC algorithm in step S21 in the digital pathology image graph structure constructed based on the superpixel segmented regions are irregular regions, so that a circumscribed rectangle of each segmented region needs to be calculated, and the circumscribed rectangle of each segmented region is used as an input of the CNN network.
As shown in fig. 2, in the CNN network in step S23, the ResNet50 pre-trained on ImageNet is used as a backbone network to perform feature coding on the segmented regions, specifically: and (3) taking the circumscribed rectangle of each segmented region as the input of ResNet50 after pre-training, removing the Softmax function before the ResNet50 model is output, enabling the Softmax function to directly output a logic value, and replacing 1 × 1000 class outputs with 1 × 1024 feature vectors, thereby completing feature coding of each segmented region in the WSI.
S3: and randomly dividing the constructed data of the graph structure into a training set, a verification set and a test set according to a preset proportion, and obtaining an optimal prediction model through training, verifying and testing a graph convolution neural network.
Preferably, the step S3 includes:
constructing a multi-scale graph convolution neural network by utilizing graph pooling, extracting characteristic information on each scale through graph convolution, using a characteristic cross layer between two continuous scales of each network layer, and fusing multi-scale characteristics in an intermediate layer; carrying out node feature combination on the graph structure output on each scale, thereby obtaining feature information of the whole graph on the corresponding scale; the concrete formula of the graph convolution neural network is as follows:
where a is the activation function and where a is the activation function,
and (3) in each layer, iterating for multiple times by using a graph convolution neural network to output node information, wherein the I is a diagonal matrix and the trainable weight matrix is a trainable weight matrix.
Specifically, a multi-scale graph convolution neural network is designed by utilizing graph pooling, as shown in fig. 3, feature information is extracted by graph convolution on each scale, and in order to further enhance information exchange across scales, a feature crossing layer is used between two continuous scales of each network layer, so that multi-scale features are allowed to be fused in the middle layer.
Then, referring to fig. 4, node feature merging is performed on the graph structure output on each scale, so as to obtain feature information of the whole graph on the corresponding scale.
Defining Z ═ GCN (a, F) to represent a process of multiple iterations in the graph convolution neural network described in the above step 3), where a is an adjacency matrix and F is a node feature matrix of the initial input.
Preferably, the constructing a multi-scale graph convolution neural network by utilizing graph pooling comprises:
and carrying out node combination by calculating similarity values between the characteristics of adjacent nodes in the graph structure to obtain the graph structures with different scales, wherein the similarity values are determined according to Euclidean distances between the characteristics, and a group of new adjacency matrixes A 'and node characteristic matrixes F' are obtained after graph pooling. And the threshold value of the similarity is manually set as a super parameter according to the actual condition.
The use of a feature intersection layer between two consecutive scales of each network layer includes:
and recording the position index of the node after graph pooling in the original node matrix F, and thus adding the characteristic values of the corresponding node characteristics under different scales according to the position index to realize information interaction of adjacent scales.
S4: classifying the digital pathology image based on the prediction model.
Preferably, the step S4 includes: and obtaining a final classification result through an MLP layer and a soft voting mechanism.
Preferably, the obtaining of the final classification result through the MLP layer and the soft voting mechanism includes:
performing global maximum pooling and global average pooling on the node feature matrix output on each scale, and splicing two groups of pooling results to be used as input data of an MLP layer; the MLP layer is composed of a full connection layer and a Softmax function, and the class prediction probability under each scale is obtained through the Softmax function.
Specifically, under each scale, the result of pooling the global maximum and the result of pooling the global average are transversely spliced to form a one-dimensional feature vector, and the one-dimensional feature vector is used as the input of the MLP layer; the MLP layer is a multilayer perceptron and comprises a plurality of hidden layers and a Softmax function, wherein each hidden layer comprises a full connection layer and an activation function, and the class prediction probability under the scale can be obtained after the output of the hidden layers passes through the Softmax function.
The soft voting mechanism in the above steps is specifically: and carrying out weighted average on the class prediction probabilities under different scales so as to output the classification class with the maximum probability value.
The method can directly construct the expression form of the graph structure of the digital pathological image with extremely high resolution, so that the graph structure of the whole digital pathological image is learned by utilizing the graph convolution neural network, the problem that the feature extraction cannot be directly carried out on the whole digital pathological image due to the limitation of the memory is avoided, and meanwhile, the classification accuracy is improved by utilizing a multi-scale and feature intersection method.
Example 2
Fig. 7 is a structural diagram of a digital pathological image classification system based on super-pixel segmentation and graph convolution. As shown in fig. 7, the present invention also provides a digital pathological image classification system based on super-pixel segmentation and graph convolution, the system comprising:
the segmentation module is used for performing superpixel segmentation on the digital pathological image by using a simple linear iterative clustering algorithm SLIC (linear iterative clustering algorithm) to obtain a superpixel segmentation area;
the construction module is used for constructing a graph structure based on the digital pathological image by taking each super-pixel region as a node in the graph structure and taking whether the super-pixel regions share edges as the basis for distributing edges among the nodes;
the training module is used for randomly dividing the constructed data of the graph structure into a training set, a verification set and a test set according to a preset proportion, and obtaining an optimal prediction model through a training, verification and test graph convolution neural network;
and the classification module is used for classifying the digital pathological image based on the prediction model.
Preferably, the segmentation module is further configured to:
initializing seed points, namely initializing a clustering center, and setting the number of superpixels to be segmented;
reselecting the seed points in the n multiplied by n neighborhood of the seed points according to the gradient values of the pixel points so as to avoid the seed points falling on the edge with larger gradient;
distributing a class label to each pixel point in the neighborhood around each seed point, and determining the clustering center to which the pixel point belongs;
and (4) performing distance measurement, and respectively calculating the distance from each retrieved pixel point to the seed point.
Preferably, the building module is further configured to:
performing super-pixel segmentation on the digital pathological image by using a SLIC algorithm to obtain a super-pixel segmentation area;
taking each super-pixel segmentation area as a node in a graph structure, wherein the number of the nodes is n;
performing low-dimensional coding on each partition area by using a CNN (compressed natural network) network to serve as feature information of each node, wherein the feature dimension of each node is d;
on the digital pathological image after the super-pixel segmentation, whether each segmentation area shares the edge or not is used as the basis for distributing the edge between the nodes, so that an adjacent matrix A of a graph structure is established; the figures areThe structure is defined as graph G (A, F), where A ∈ {0, 1}n×nIn the form of a contiguous matrix, the matrix,
is a node feature matrix, each node has d-dimensional features.
The specific implementation process of the method steps executed by each module in embodiment 2 of the present invention is the same as the implementation process of each step in embodiment 1, and is not described herein again.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all modifications and equivalents of the present invention, which are made by the contents of the present specification and the accompanying drawings, or directly/indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A digital pathological image classification method based on super-pixel segmentation and graph convolution is characterized by comprising the following steps:
s1: performing superpixel segmentation on the digital pathological image by using a simple linear iterative clustering algorithm SLIC (linear iterative clustering algorithm) to obtain a superpixel segmentation area;
s2: constructing a graph structure based on the digital pathological image by taking each super-pixel region as a node in the graph structure and taking whether the super-pixel regions share edges as the basis for distributing the edges among the nodes;
s3: randomly dividing the constructed data of the graph structure into a training set, a verification set and a test set according to a preset proportion, and obtaining an optimal prediction model through a training, verification and test graph convolution neural network;
s4: classifying the digital pathology image based on the prediction model.
2. The method according to claim 1, wherein the step S1 includes:
s11: initializing seed points, namely initializing a clustering center, and setting the number of superpixels to be segmented;
s12: reselecting the seed points in the n multiplied by n neighborhood of the seed points according to the gradient values of the pixel points so as to avoid the seed points falling on the edge with larger gradient;
s13: distributing a class label to each pixel point in the neighborhood around each seed point, and determining the clustering center to which the pixel point belongs;
s14: and (4) performing distance measurement, and respectively calculating the distance from each retrieved pixel point to the seed point.
3. The method according to claim 1, wherein the step S2 includes:
s21: performing super-pixel segmentation on the digital pathological image by using a SLIC algorithm to obtain a super-pixel segmentation area;
s22: taking each super-pixel segmentation area as a node in a graph structure, wherein the number of the nodes is n;
s23: performing low-dimensional coding on each partition area by using a CNN (compressed natural network) network to serve as feature information of each node, wherein the feature dimension of each node is d;
s24: on the digital pathological image after the super-pixel segmentation, whether each segmentation area shares the edge or not is used as the basis for distributing the edge between the nodes, so that an adjacent matrix A of a graph structure is established; the graph structure is defined as graph G (A, F), where A ∈ {0, 1}n×nIn the form of a contiguous matrix, the matrix,
is a node feature matrix, each node has d-dimensional features.
4. The method according to claim 3, wherein the step S3 includes:
constructing a multi-scale graph convolution neural network by utilizing graph pooling, extracting characteristic information on each scale through graph convolution, using a characteristic cross layer between two continuous scales of each network layer, and fusing multi-scale characteristics in an intermediate layer; carrying out node feature combination on the graph structure output on each scale, thereby obtaining feature information of the whole graph on the corresponding scale; the concrete formula of the graph convolution neural network is as follows:
where a is the activation function and where a is the activation function,
and (3) in each layer, iterating for multiple times by using a graph convolution neural network to output node information, wherein the I is a diagonal matrix and the trainable weight matrix is a trainable weight matrix.
5. The method according to claim 4, wherein the step S4 includes: and obtaining a final classification result through an MLP layer and a soft voting mechanism.
6. The method of claim 4, wherein constructing a multi-scale atlas neural network using atlas pooling comprises:
and carrying out node combination by calculating similarity values between the characteristics of adjacent nodes in the graph structure to obtain the graph structures with different scales, wherein the similarity values are determined according to Euclidean distances between the characteristics, and a group of new adjacency matrixes A 'and node characteristic matrixes F' are obtained after graph pooling.
The use of a feature intersection layer between two consecutive scales of each network layer includes:
and recording the position index of the node after graph pooling in the original node matrix F, and thus adding the characteristic values of the corresponding node characteristics under different scales according to the position index to realize information interaction of adjacent scales.
7. The method of claim 5, wherein the obtaining the final classification result through the MLP layer and the soft voting mechanism comprises:
performing global maximum pooling and global average pooling on the node feature matrix output on each scale, and splicing two groups of pooling results to be used as input data of an MLP layer; the MLP layer consists of a full connection layer and a Softmax function, and the class prediction probability under each scale is obtained through the Softmax function;
and carrying out weighted average on the class prediction probabilities under different scales so as to output the classification class with the maximum probability value.
8. A digital pathology image classification system based on superpixel segmentation convolved with a map, the system comprising:
the segmentation module is used for performing superpixel segmentation on the digital pathological image by using a simple linear iterative clustering algorithm SLIC (linear iterative clustering algorithm) to obtain a superpixel segmentation area;
the construction module is used for constructing a graph structure based on the digital pathological image by taking each super-pixel region as a node in the graph structure and taking whether the super-pixel regions share edges as the basis for distributing edges among the nodes;
the training module is used for randomly dividing the constructed data of the graph structure into a training set, a verification set and a test set according to a preset proportion, and obtaining an optimal prediction model through a training, verification and test graph convolution neural network;
and the classification module is used for classifying the digital pathological image based on the prediction model.
9. The system of claim 8, wherein the segmentation module is further configured to:
initializing seed points, namely initializing a clustering center, and setting the number of superpixels to be segmented;
reselecting the seed points in the n multiplied by n neighborhood of the seed points according to the gradient values of the pixel points so as to avoid the seed points falling on the edge with larger gradient;
distributing a class label to each pixel point in the neighborhood around each seed point, and determining the clustering center to which the pixel point belongs;
and (4) performing distance measurement, and respectively calculating the distance from each retrieved pixel point to the seed point.
10. The system of claim 8, wherein the build module is further configured to:
performing super-pixel segmentation on the digital pathological image by using a SLIC algorithm to obtain a super-pixel segmentation area;
taking each super-pixel segmentation area as a node in a graph structure, wherein the number of the nodes is n;
performing low-dimensional coding on each partition area by using a CNN (compressed natural network) network to serve as feature information of each node, wherein the feature dimension of each node is d;
on the digital pathological image after the super-pixel segmentation, whether each segmentation area shares the edge or not is used as the basis for distributing the edge between the nodes, so that an adjacent matrix A of a graph structure is established; the graph structure is defined as graph G (A, F), where A ∈ {0, 1}n×nIn the form of a contiguous matrix, the matrix,
is a node feature matrix, each node has d-dimensional features.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110585365.5A CN113313164A (en) | 2021-05-27 | 2021-05-27 | Digital pathological image classification method and system based on superpixel segmentation and image convolution |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110585365.5A CN113313164A (en) | 2021-05-27 | 2021-05-27 | Digital pathological image classification method and system based on superpixel segmentation and image convolution |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113313164A true CN113313164A (en) | 2021-08-27 |
Family
ID=77375604
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110585365.5A Pending CN113313164A (en) | 2021-05-27 | 2021-05-27 | Digital pathological image classification method and system based on superpixel segmentation and image convolution |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113313164A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113555091A (en) * | 2021-09-22 | 2021-10-26 | 中国科学院自动化研究所 | Parkinson sMRI image-based disease classification system, method and device |
| CN114399002A (en) * | 2022-01-21 | 2022-04-26 | 平安科技（深圳）有限公司 | Image scene classification method and device based on graph neural network |
| CN115995024A (en) * | 2023-03-22 | 2023-04-21 | 成都理工大学 | Image classification method based on class diagram neural network |
| CN116664240A (en) * | 2023-06-05 | 2023-08-29 | 山东睿政信息科技有限公司 | Shopping behavior prediction method, system, storage medium and terminal based on graphic neural network |
| CN116934754A (en) * | 2023-09-18 | 2023-10-24 | 四川大学华西第二医院 | Liver image identification method and device based on graph neural network |
| CN117115241A (en) * | 2023-09-06 | 2023-11-24 | 北京透彻未来科技有限公司 | Method for searching central focus of digital pathological image in zooming process |
| CN117408997A (en) * | 2023-12-13 | 2024-01-16 | 安徽省立医院(中国科学技术大学附属第一医院) | Auxiliary detection system for EGFR gene mutation in non-small cell lung cancer histological image |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106446914A (en) * | 2016-09-28 | 2017-02-22 | 天津工业大学 | Road detection based on superpixels and convolution neural network |
| CN108573491A (en) * | 2017-03-10 | 2018-09-25 | 南京大学 | A kind of three-dimensional ultrasound pattern dividing method based on machine learning |
| CN108717554A (en) * | 2018-05-22 | 2018-10-30 | 复旦大学附属肿瘤医院 | A kind of thyroid tumors histopathologic slide image classification method and its device |
| WO2019001208A1 (en) * | 2017-06-28 | 2019-01-03 | 苏州比格威医疗科技有限公司 | Segmentation algorithm for choroidal neovascularization in oct image |
| US20190087726A1 (en) * | 2017-08-30 | 2019-03-21 | The Board Of Regents Of The University Of Texas System | Hypercomplex deep learning methods, architectures, and apparatus for multimodal small, medium, and large-scale data representation, analysis, and applications |
| WO2019108888A1 (en) * | 2017-11-30 | 2019-06-06 | The Research Foundation For The State University Of New York | SYSTEM AND METHOD TO QUANTIFY TUMOR-INFILTRATING LYMPHOCYTES (TILs) FOR CLINICAL PATHOLOGY ANALYSIS |
| CN110659692A (en) * | 2019-09-26 | 2020-01-07 | 重庆大学 | Pathological image automatic labeling method based on reinforcement learning and deep neural network |
| US20200179082A1 (en) * | 2018-12-10 | 2020-06-11 | Dentsply Sirona Inc. | Method for the design and manufacture of a dental component |
| CN111274903A (en) * | 2020-01-15 | 2020-06-12 | 合肥工业大学 | Cervical cell image classification method based on graph convolution neural network |
| AU2020103901A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field |
-
2021
- 2021-05-27 CN CN202110585365.5A patent/CN113313164A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106446914A (en) * | 2016-09-28 | 2017-02-22 | 天津工业大学 | Road detection based on superpixels and convolution neural network |
| CN108573491A (en) * | 2017-03-10 | 2018-09-25 | 南京大学 | A kind of three-dimensional ultrasound pattern dividing method based on machine learning |
| WO2019001208A1 (en) * | 2017-06-28 | 2019-01-03 | 苏州比格威医疗科技有限公司 | Segmentation algorithm for choroidal neovascularization in oct image |
| US20190087726A1 (en) * | 2017-08-30 | 2019-03-21 | The Board Of Regents Of The University Of Texas System | Hypercomplex deep learning methods, architectures, and apparatus for multimodal small, medium, and large-scale data representation, analysis, and applications |
| WO2019108888A1 (en) * | 2017-11-30 | 2019-06-06 | The Research Foundation For The State University Of New York | SYSTEM AND METHOD TO QUANTIFY TUMOR-INFILTRATING LYMPHOCYTES (TILs) FOR CLINICAL PATHOLOGY ANALYSIS |
| CN108717554A (en) * | 2018-05-22 | 2018-10-30 | 复旦大学附属肿瘤医院 | A kind of thyroid tumors histopathologic slide image classification method and its device |
| US20200179082A1 (en) * | 2018-12-10 | 2020-06-11 | Dentsply Sirona Inc. | Method for the design and manufacture of a dental component |
| CN110659692A (en) * | 2019-09-26 | 2020-01-07 | 重庆大学 | Pathological image automatic labeling method based on reinforcement learning and deep neural network |
| CN111274903A (en) * | 2020-01-15 | 2020-06-12 | 合肥工业大学 | Cervical cell image classification method based on graph convolution neural network |
| AU2020103901A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113555091B (en) * | 2021-09-22 | 2022-02-08 | 中国科学院自动化研究所 | Parkinson sMRI image-based disease classification system, method and device |
| CN113555091A (en) * | 2021-09-22 | 2021-10-26 | 中国科学院自动化研究所 | Parkinson sMRI image-based disease classification system, method and device |
| CN114399002A (en) * | 2022-01-21 | 2022-04-26 | 平安科技（深圳）有限公司 | Image scene classification method and device based on graph neural network |
| WO2023137916A1 (en) * | 2022-01-21 | 2023-07-27 | 平安科技（深圳）有限公司 | Graph neural network-based image scene classification method and apparatus |
| CN115995024A (en) * | 2023-03-22 | 2023-04-21 | 成都理工大学 | Image classification method based on class diagram neural network |
| CN116664240B (en) * | 2023-06-05 | 2024-02-02 | 谢馥聪 | Shopping behavior prediction method, system, storage medium and terminal based on graphic neural network |
| CN116664240A (en) * | 2023-06-05 | 2023-08-29 | 山东睿政信息科技有限公司 | Shopping behavior prediction method, system, storage medium and terminal based on graphic neural network |
| CN117115241A (en) * | 2023-09-06 | 2023-11-24 | 北京透彻未来科技有限公司 | Method for searching central focus of digital pathological image in zooming process |
| CN117115241B (en) * | 2023-09-06 | 2024-03-29 | 北京透彻未来科技有限公司 | Method for searching central focus of digital pathological image in zooming process |
| CN116934754A (en) * | 2023-09-18 | 2023-10-24 | 四川大学华西第二医院 | Liver image identification method and device based on graph neural network |
| CN116934754B (en) * | 2023-09-18 | 2023-12-01 | 四川大学华西第二医院 | Liver image identification method and device based on graph neural network |
| CN117408997A (en) * | 2023-12-13 | 2024-01-16 | 安徽省立医院(中国科学技术大学附属第一医院) | Auxiliary detection system for EGFR gene mutation in non-small cell lung cancer histological image |
| CN117408997B (en) * | 2023-12-13 | 2024-03-08 | 安徽省立医院(中国科学技术大学附属第一医院) | Auxiliary detection system for EGFR gene mutation in non-small cell lung cancer histological image |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113313164A (en) | Digital pathological image classification method and system based on superpixel segmentation and image convolution | |
| CN112396002B (en) | SE-YOLOv 3-based lightweight remote sensing target detection method | |
| CN111612008B (en) | Image segmentation method based on convolution network | |
| CN112347970B (en) | Remote sensing image ground object identification method based on graph convolution neural network | |
| CN115017418B (en) | Remote sensing image recommendation system and method based on reinforcement learning | |
| CN108596240B (en) | Image semantic segmentation method based on discriminant feature network | |
| CN110188827A (en) | A kind of scene recognition method based on convolutional neural networks and recurrence autocoder model | |
| CN111639607A (en) | Model training method, image recognition method, model training device, image recognition device, electronic equipment and storage medium | |
| CN113160265A (en) | Construction method of prediction image for brain corpus callosum segmentation for corpus callosum state evaluation | |
| CN114332473A (en) | Object detection method, object detection device, computer equipment, storage medium and program product | |
| CN115063786A (en) | High-order distant view fuzzy license plate detection method | |
| CN114863348A (en) | Video target segmentation method based on self-supervision | |
| CN114783604A (en) | Method, system and storage medium for predicting sentinel lymph node metastasis of breast cancer | |
| CN115240024A (en) | Method and system for segmenting extraterrestrial pictures by combining self-supervised learning and semi-supervised learning | |
| CN115732034A (en) | Identification method and system of spatial transcriptome cell expression pattern | |
| CN111161244A (en) | Industrial product surface defect detection method based on FCN + FC-WXGboost | |
| CN116012332A (en) | Hierarchical graph-based pathological image primary tumor stage multi-example learning method, frame, equipment and medium | |
| CN116091946A (en) | Yolov 5-based unmanned aerial vehicle aerial image target detection method | |
| CN117237733A (en) | Breast cancer full-slice image classification method combining self-supervision and weak supervision learning | |
| CN114299344A (en) | Low-cost automatic searching method of neural network structure for image classification | |
| CN116403071B (en) | Method and device for detecting few-sample concrete defects based on feature reconstruction | |
| CN116758349A (en) | Hyperspectral image classification method based on multi-scale super-pixel node aggregate graph convolution residual network | |
| CN115393378B (en) | Low-cost and efficient cell nucleus image segmentation method | |
| CN109583584B (en) | Method and system for enabling CNN with full connection layer to accept indefinite shape input | |
| CN114494284B (en) | Scene analysis model and method based on explicit supervision area relation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |