CN116012332A - Hierarchical graph-based pathological image primary tumor stage multi-example learning method, frame, equipment and medium - Google Patents
Hierarchical graph-based pathological image primary tumor stage multi-example learning method, frame, equipment and medium Download PDFInfo
- Publication number
- CN116012332A CN116012332A CN202211714879.7A CN202211714879A CN116012332A CN 116012332 A CN116012332 A CN 116012332A CN 202211714879 A CN202211714879 A CN 202211714879A CN 116012332 A CN116012332 A CN 116012332A
- Authority
- CN
- China
- Prior art keywords
- patch
- tissue
- graph
- nodes
- chart
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Abstract
A hierarchical graph-based pathological image primary tumor stage multi-example learning method, framework, equipment and medium, wherein the method comprises the following steps: constructing a structure perception hierarchy chart through data preprocessing, feature extraction and composition; the key mode of primary tumor stage is captured by learning the cross-scale spatial characteristics, and a layered attention-based graph characterization network is constructed, so that pathological images can be accurately classified; the multi-granularity interpretability is provided by identifying the deepest penetration region, so that the clinical identification and judgment of pathology are assisted, and the working efficiency of pathologists is effectively improved; the learning framework, the equipment and the medium are used for realizing a primary tumor stage multi-example learning method in the tissue pathology image based on the structure perception hierarchy chart; the most critical example features of the corresponding stage can be captured efficiently across the scale space.
Description
Technical Field
The invention belongs to the technical field of medical image processing and computer vision, and relates to a primary tumor stage multi-example learning method, frame, equipment and medium in pathological images based on a hierarchical graph.
Background
Primary tumor stage (pT stage) is an important indicator for cancer research, which is useful for judging patient prognosis and guiding selection of clinical treatment regimen. In clinical routine examinations, primary tumor stage mainly involves measurement of tumor size and identification of tumor infiltrating normal tissue. The former can be easily measured by macro examination, but the latter requires extensive and time-consuming microscopic examination of the pathological section by the pathologist. Typically, a pathologist needs to first identify the tissue hierarchy under a low power scope and then look for the deepest infiltration area by examining the junction of the tumor with other normal tissue under a high power scope. In addition, some small clusters of cancer cells tend to be distant from the primary tumor, which also presents great difficulties in locating and identifying stage critical features.
With the rapid development of digital pathology, many pathology recognition decision processes can be modeled as machine learning tasks handled by deep learning algorithms. For primary tumor staging, it can be considered a classification task of a full-field digital pathological section, the classification of which is determined by the infiltration relationship between the tumor and other normal tissues. One of the most straightforward solutions is to segment different types of tissue first and then model the spatial infiltration relationships between the tissues, but the segmentation of different types of tissue requires a large number of pixel-level fine labels. And a large number of pixel-level labels, which are often difficult to obtain, are dependent on the pathological knowledge of the pathologist's specialty.
Currently, multi-example learning is a popular weakly supervised learning paradigm that relies solely on slice-level labels, and has been widely used for classification tasks of full-field digital slices, e.g., classification and parting, etc. Classical multi-example learning frameworks are generally divided into three steps: firstly, cutting the full-view digital pathological section into a series of non-overlapping small image blocks, which are also called patches; each patch is considered an example, all examples of a slice forming a package; the features of each instance in the package are then extracted and input into the pooling layer to aggregate the instances into feature representations of the entire package, and finally the package features are classified by a classifier.
However, the direct application of the classical multiple-example framework to primary tumor staging tasks has the following two disadvantages: 1) The most distinctive area of primary tumor staging is the interface between the tumor and other normal tissue. However, these regions are sparse in large-sized full-field digital pathological slices, making it difficult for existing multi-example learning-based methods to effectively capture the most critical example features for staging; 2) The primary tumor stage is mainly based on the infiltration relation among tissues, and needs to be checked step by step from low to high multiple. However, existing methods based on multi-instance learning are generally based on single magnification extraction of morphological features, and cannot capture cross-scale spatial relationships between tissues.
Patent application CN112488234a discloses an attention pooling-based end-to-end histopathological image classification method, which comprises the following steps: s1: slicing the histopathological image into slices with specified sizes, removing the slices with excessive background areas, and forming a bag by the rest slices;
s2: taking the packet obtained in the step S1 as input, and training a deep neural network by using a standard multi-example learning method; s3: scoring all the sections by using the trained deep neural network, taking m Zhang Qiepian with highest score and m Zhang Qiepian with lowest score of each whole glass slide image, and collecting the whole glass slide images into a new package; s4: building a deep neural network comprising an attention module, and training the network by using the new packet obtained in the step S3; s5: and (3) processing the histopathological images to be classified by S1 and S3, and classifying by using the model obtained by S4. The invention can obtain better classification effect under the condition that only a small amount of samples exist at present, provides an auxiliary diagnosis mechanism for doctors, and relieves the problem of medical resource shortage; however, the invention does not consider that patches with key effects on classification in the primary tumor stage only occupy a small part, the patches can not be accurately positioned, and the most key example characteristics for the stage are difficult to capture; meanwhile, the method only extracts the characteristics of pathological sections under single multiple, and loses some necessary information, which is unfavorable for capturing the cross-scale spatial relationship between tissues.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a primary tumor stage multi-example learning method, a frame, equipment and a medium in a pathological image based on a hierarchical graph, wherein the full-view pathological section with a pyramid structure is represented by constructing a structural perception hierarchical graph based on an example organization method of the graph, so that key characteristics related to primary tumor stage capturing by a model are effectively promoted; on the basis of the structural perception hierarchical graph, constructing a hierarchical attention-based graph characterization network, and capturing a key mode of primary tumor staging by learning cross-scale spatial features; the pathological image can be accurately classified, the multi-granularity interpretability is provided by identifying the deepest penetration area, the pathological clinical identification and judgment are assisted, and the working efficiency of a pathologist is effectively improved.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
the primary tumor stage multi-example learning method based on the pathology image of the hierarchical graph comprises the following specific steps:
step one, constructing a structure perception hierarchy chart, and representing a full-view digital pathological section of a pyramid structure as a heterogeneous structure perception hierarchy chart, wherein the structure perception hierarchy chart comprises two types of subgraphs: a tissue map at the top layer and a plurality of patch maps at the lower layer are respectively established on the low-magnification slice and the relatively high-magnification slice; in the switching process between the fields of different amplification factors, determining the subordinate relation between patch graph nodes and corresponding tissue graph nodes under different amplification factors through the incidence matrix structure;
step two, constructing a hierarchical attention-based graph characterization network: the patch graph convolutional neural network is used for the structural perception hierarchy graph constructed in the step one, and local context characteristics of nodes in the high-magnification patch graph are encoded; fusing the patch characteristics with high magnification into the corresponding low magnification organization chart nodes by using a hierarchical attention layer and a cross attention mechanism; the fused tissue diagram is input into a tissue diagram convolutional neural network and used for coding the overall spatial tissue structural characteristics and capturing the infiltration relation between the tumor and the normal tissue; finally, the global attention layer aggregates all nodes of the tissue map to obtain a slice-level characteristic representation for prediction of the stage class.
The first step is to construct a structure perception hierarchy chart, and the specific implementation steps are as follows:
1) Data preprocessing
1.1 For a full-field digital pathological section with pyramid structure, the low resolution to the high resolution are respectively denoted as w t ，w 1 ，w 2 ，…w n ,w n+1 Will w t The resolution of the image is sampled to half, the tissue is separated from the background by adopting an Ojin binarization algorithm, a binary division mask is obtained, and then the division mask is respectively processed for 2 1 、2 2 、2 3 ，…，2 n ，2 n+1 ，2 n+2 Up-sampling multiple times to match w respectively t ，w 1 ，w 2 ，…，w n ,w n+1 Resolution of (2);
1.2 Cutting the slices with different multiples obtained in the step 1.1) into non-overlapping patches by adopting a sliding window method to respectively generate patch sets P t ，P 1 ，P 2 ，…P n ,P n+1 The method comprises the steps of carrying out a first treatment on the surface of the Under the guidance of the segmentation mask, the background of the slice is ignored, and the patch only comprises a foreground region containing tissues;
1.3 At w) t Dividing the image processed in the step 1.2) into different tissue blocks by using a simple linear iterative clustering algorithm;
2) Feature extraction
Extraction step 1.2The patch set P obtained in the step t ，P 1 ，P 2 ，…P n ,P n+1 Characteristics of the nodes; will P t ，P 1 ，P 2 ，…P n ,P n+1 All patches in the patch are spliced together and input into a feature encoder to obtain initialized node features X of n+1 patch graphs 1 ，X 2 ，…X n ,X n+1 The method comprises the steps of carrying out a first treatment on the surface of the For the organizational chart, due to each organization block t therein i All cover patch P t Several patches in (c) may be used to make t i The covered patch is input into a feature encoder, and then the average pooling operation is carried out, so that the tissue block t can be obtained i Is characterized by (2); finally obtaining node characteristic X of the organization chart t ；
3) Building a structure-aware hierarchical graph G
Constructing a structure perception hierarchy chart G according to the node characteristics of the organization blocks and the organization charts obtained in the step 2); graph G contains three types of edges, E t For modeling the adjacency of adjacent tissue blocks in an organizational chart,
modeling the adjacency of the patch and its eight neighbors in each patch graph,/>
Modeling the affiliation of nodes in the organization chart and the patch chart; meanwhile, generating an adjacency matrix and an association matrix At according to the connection relation of the edges,
at A t In the case of tissue t i ，t j With edges in between, the element +.>
Otherwise, 0, and the rest matrixes are the same; finally, the structure-aware hierarchical graph G may be expressed as +.>
Step two, constructing a hierarchical attention-based graph characterization network, which comprises the following specific steps:
1) Updating patch node characteristics through a patch convolution neural network:
after the structural perception hierarchy graph G is built in the step one, for each patch graph
Respectively inputting the patch images into the patch image convolution neural network; the structure of each patch is the same, each patch convolve neural network and has L layers, each layer can update the characteristics of each node, and for each node, the first layer (L E [0, L]) The characteristic updating process of (a) is as follows:
1.1 For node i, its current characteristics are
Finding all neighbor nodes N (i), for each neighbor node j epsilon N (i), adding the characteristics of the neighbor nodes with the characteristics of the node i, and activating the added characteristics through a ReLU function;
1.2 All the added features are accumulated into one fusion feature through an attention mechanism;
1.3 Fusion features to be obtained
Residual connection is carried out and then the residual connection is input into a multi-layer perceptron to obtain updated characteristics
Finishing the updating of the first layer characteristics;
after the L layers are all updated, splicing the outputs of all the layers together to serve as node characteristics of a final patch diagram; after the neural network is rolled up through the patch map,the patch map is updated to
2) The patch graph node features are fused into the tissue graph node features through a hierarchical attention layer:
inputting the patch diagram into a hierarchical attention layer according to the incidence matrix
Patches contained in each organization in the organization chart are found, under the guidance of the organization characteristics, the patch node characteristics contained in the organization are weighted and summed by utilizing a cross attention mechanism, and detail perceived organization characteristics are generated>
Wherein the weight is the similarity of the patch and its corresponding organization; splicing the detail perception features of the two scales to obtain an organization chart feature fused with the fine-granularity patch node feature, wherein the fused organization chart is expressed as G f ；
3) Updating the node characteristics of the tissue map through the convolutional neural network of the tissue map:
will G f Inputting the tissue map convolution neural network, and encoding the global space relation of tissue map nodes into each tissue node characteristic to obtain an updated tissue map G '' f The method comprises the steps of carrying out a first treatment on the surface of the The design of the tissue map convolution neural network is the same as that of the patch map convolution neural network, and the characteristic updating process is the same as that of the steps 1.1) to 1.3);
4) The node features in the organizational chart are fused into packet-level features for staging through the global attention layer:
will G' f Inputting a global attention layer, and weighting and fusing the characteristics of the tissue nodes according to the importance of the tissue nodes on the slices to obtain a package-level characteristic representation x bag The method comprises the steps of carrying out a first treatment on the surface of the And then x is bag Inputting into the full-connecting layer, and obtaining the final primary tumor stage prediction knot after softmax operationAnd (5) fruits.
The primary tumor stage multi-example learning framework based on the pathology image of the hierarchy chart comprises a structural perception hierarchy chart construction module and a hierarchy attention-based chart characterization network module:
the structure perception hierarchy diagram construction module is used for representing the full-view digital slice of the pyramid structure as a heterogeneous structure perception hierarchy diagram, and comprises two types of subgraphs: a tissue map at the top layer and a plurality of patch maps at the lower layer are respectively established on the low-magnification slice and the relatively high-magnification slice; specifically, in the tissue map, different tissues and nearest neighbor relations among the different tissues are respectively expressed as nodes and edges, so that spatial relations among the different tissues are clearly expressed, and the network model is beneficial to perceiving the infiltration relation between tumors and normal tissues; in the patch diagram, patches are taken as nodes, eight neighborhood relations among the patches are taken as edges, and modeling of fine granularity characteristics under high multiplying power is realized; in addition, in the switching process between fields of different magnification, determining the subordinate relation between patch map nodes and corresponding tissue map nodes under different magnification through the incidence matrix structure;
the hierarchical attention-based graph characterization network module includes four parts: patch map convolutional neural network, hierarchical attention layer, organizational chart convolutional neural network, and global attention layer; the patch graph convolutional neural network is used for encoding local context characteristics of nodes in the high-magnification patch graph; the hierarchical attention layer fuses the patch characteristics of high magnification into the corresponding low magnification organization chart nodes by using a cross attention mechanism; the fused tissue diagram is input into a tissue diagram convolutional neural network and used for coding the overall spatial tissue structural characteristics and capturing the infiltration relation between the tumor and the normal tissue; finally, the global attention layer aggregates all nodes of the tissue map to obtain a slice-level characteristic representation for prediction of the stage class.
A hierarchical graph-based pathology image primary tumor stage multi-example learning device, comprising:
a memory for storing a computer program;
and the processor is used for realizing the primary tumor stage multi-example learning method in the histopathological image based on the structure perception hierarchy chart in the first step to the second step when executing the computer program.
A computer readable storage medium storing a computer program which when executed by a processor enables a user to stage a primary tumor of an image of an histopathology using a multi-example learning method based on a structural perception hierarchy.
Compared with the prior art, the invention has the following beneficial technical effects:
1. the multi-example learning framework simulates the resolution judgment process of pathologists, and provides accurate and efficient interpretation primary tumor stage resolution judgment tasks by constructing a structural perception hierarchy chart and a hierarchical attention-based chart characterization network.
2. The invention represents the full-view pathological section with the pyramid structure by constructing the structure perception hierarchy chart, and effectively promotes the multi-example learning model to capture key characteristics related to primary tumor stage.
3. On the basis of the structural perception hierarchical graph, the hierarchical attention-based graph characterization network is provided, and key modes of primary tumor stage are captured by learning cross-scale spatial features, so that pathological images can be accurately classified.
4. The provided framework can accurately classify pathological images, provide multi-granularity interpretability by identifying the deepest penetration area, assist pathological clinical identification and judgment, and effectively improve the working efficiency of pathologists.
Drawings
Fig. 1 is an overall process flow diagram.
FIG. 2 is a flowchart showing a specific construction of a structure-aware hierarchical graph, wherein 5×,10×,20× each represents a pathological section magnification of 5 times, 10 times, 20 times, and corresponding to w t ，w 1 ，w 2 。
Fig. 3 is a block diagram of a hierarchical attention-based graph characterization network.
Detailed Description
The present invention is described in further detail below:
referring to fig. 1, a hierarchical graph-based pathological image primary tumor stage multi-example learning method specifically includes the steps of:
step one, constructing a structure perception hierarchy chart, wherein the specific implementation steps are as follows: see FIG. 2
1) Data preprocessing
1.1 For a full field digital pathology with a pyramidal structure, three resolutions are exemplified. Represented by w from low resolution to high resolution t ，w 1 ，w 2 Will w t The resolution of the image is sampled to half, tissue is separated from the background by adopting an Ojin binarization algorithm, a binary segmentation mask is obtained, and up-sampling is respectively carried out on the segmentation mask for 2 times, 4 times and 8 times so as to respectively match w t ，w 1 ，w 2 Resolution of (2);
1.2 Cutting the three times of slices obtained in the step 1.1) into non-overlapping patches by adopting a sliding window method to respectively generate three patch sets P t ，P 1 ，P 2 The method comprises the steps of carrying out a first treatment on the surface of the Under the guidance of the segmentation mask, the background of the slice is ignored, and the patch only comprises a foreground region containing tissues;
1.3 At w) t Dividing the image processed in the step 1.2) into different tissue blocks by using a simple linear iterative clustering algorithm;
2) Feature extraction
Extracting three patch sets P generated in step 1.2) t ，P 1 ，P 2 Characteristics of the nodes; will P 1 ，P 2 All patches in the patch are spliced together and input into a feature encoder to obtain initialized node features X of two patch graphs 1 ，X 2 The method comprises the steps of carrying out a first treatment on the surface of the For the organizational chart, due to each organization block t therein i All cover patch P t Several patches in (c) may be used to make t i The covered patch is input into a feature encoder, and then the average pooling operation is carried out, so that the tissue block t can be obtained i Is characterized by (2); finally obtaining the node characteristics of the organization chartSign X t ；
3) Building a structure-aware hierarchical graph G
Constructing a structure perception hierarchy chart G according to the node characteristics of the organization blocks and the organization charts obtained in the step 2); graph G contains three types of edges, E t For modeling the adjacency of adjacent tissue blocks in an organizational chart,
modeling the adjacency of the patch and its eight neighbors in each patch graph,/>
Dependencies of nodes in the organizational chart and patch chart are modeled. Meanwhile, generating an adjacent matrix and an associated matrix A according to the connection relation of the edges t ，
At A t In the case of tissue t i ，t j With edges in between, the element +.>
Otherwise, 0, and the rest matrixes are the same; finally, the structure-aware hierarchical graph G may be represented as
Step two, constructing a hierarchical attention-based graph characterization network, which comprises the following specific steps: see FIG. 3
1) Updating patch node characteristics through a patch convolution neural network:
after the structural perception hierarchy graph G is built in the step one, the two patch graphs are subjected to
Respectively input it intoInto two patch-map convolution neural networks. The two patch images have the same structure, each patch image convolution neural network has L layers, each layer can update the characteristics of each node, and for each node, the first layer (L E [0, L]) The characteristic updating process of (a) is as follows:
1.1 For node i, its current characteristics are
Finding all neighbor nodes N (i), for each neighbor node j epsilon N (i), adding the characteristics of the neighbor nodes with the characteristics of the node i, and activating the added characteristics through a ReLU function;
1.2 All the added features are accumulated into one fusion feature through an attention mechanism;
1.3 Fusion features to be obtained
Residual connection is carried out and then the residual connection is input into a multi-layer perceptron to obtain updated characteristics
Finishing the updating of the first layer characteristics;
after the L layers are all updated, splicing the outputs of all the layers together to serve as node characteristics of a final patch diagram; after the neural network is rolled up by the patch chart, the patch chart is updated as
2) Fusing the characteristics of the patches in the patch graph into the characteristics of the nodes in the organization graph through the hierarchical attention layer:
inputting the patch diagram into a hierarchical attention layer according to the incidence matrix
Patches contained in each organization in the organization chart are found, under the guidance of the organization characteristics, the patch node characteristics contained in the organization are weighted and summed by utilizing a cross attention mechanism, and detail perceived organization characteristics are generated>
Wherein the weight is the similarity of the patch and its corresponding organization; finally, splicing the detail perception features of the two scales to obtain an organization chart feature fused with the fine-granularity patch node feature, wherein the fused organization chart is expressed as G f ；
3) Updating the node characteristics of the tissue map through the convolutional neural network of the tissue map:
will G f Inputting the tissue map convolution neural network, and encoding the global space relation of tissue map nodes into each tissue node characteristic to obtain an updated tissue map G '' f The method comprises the steps of carrying out a first treatment on the surface of the The design of the tissue map convolution neural network is the same as that of the patch map convolution neural network, and the characteristic updating process is the same as that of the steps 1.1) to 1.3);
4) The node features in the organizational chart are fused into packet-level features for staging through the global attention layer:
will G' f Inputting a global attention layer, and weighting and fusing the characteristics of the tissue nodes according to the importance of the tissue nodes on the slices to obtain a package-level characteristic representation x bag The method comprises the steps of carrying out a first treatment on the surface of the And then x is bag Inputting the full-connecting layer, and obtaining a final primary tumor stage prediction result after softmax operation.
The whole multi-example learning framework is iteratively trained by minimizing the cross entropy loss function of the prediction result and the real label.
The invention discloses a hierarchical graph-based pathological image primary tumor stage multi-example learning framework, which comprises two modules: a structure-aware hierarchical graph construction module and a hierarchical attention-based graph networking module.
The structure perception hierarchy diagram construction module is used for representing the full-view digital slice of the pyramid structure as a heterogeneous structure perception hierarchy diagram and comprises two types of subgraphs: the tissue map at the top layer and the plurality of patch maps at the lower layer are built on the low magnification and relatively higher magnification slices, respectively. Specifically, in the tissue map, different tissues and nearest neighbor relations between the different tissues are respectively expressed as nodes and edges, so that spatial relations between the different tissues can be clearly expressed, and the network model is beneficial to sensing the infiltration relation between tumors and normal tissues. In the patch diagram, patches are taken as nodes, eight neighborhood relations among the patches are taken as edges, and modeling of fine granularity characteristics under high multiplying power is realized; furthermore, to simulate the switching process between fields of view at different magnifications when a pathologist is working, we propose an associative matrix structure to determine the affiliations between patch map nodes and corresponding tissue map nodes at different magnifications.
The hierarchical attention based graph characterization network module includes four parts: patch map convolutional neural networks, hierarchical attention layers, organizational map convolutional neural networks, and global attention layers. The patch graph convolutional neural network is used for encoding local context characteristics of nodes in the high-magnification patch graph; the hierarchical attention layer fuses the patch characteristics of high magnification into the corresponding low magnification organization chart nodes by using a cross attention mechanism; the fused tissue diagram is input into a tissue diagram convolutional neural network and used for coding the overall spatial tissue structural characteristics and capturing the infiltration relation between the tumor and the normal tissue; finally, the global attention layer aggregates all nodes of the tissue map to obtain a slice-level characteristic representation for prediction of the stage class.
Claims (6)
1. The primary tumor stage multi-example learning method based on the pathological image of the hierarchical map is characterized by comprising the following steps of: the method comprises the following specific steps:
step one, constructing a structure perception hierarchy chart, and representing a full-view digital pathological section of a pyramid structure as a heterogeneous structure perception hierarchy chart, wherein the structure perception hierarchy chart comprises two types of subgraphs: a tissue map at the top layer and a plurality of patch maps at the lower layer are respectively established on the low-magnification slice and the relatively high-magnification slice; in the switching process between the fields of different amplification factors, determining the subordinate relation between patch graph nodes and corresponding tissue graph nodes under different amplification factors through the incidence matrix structure;
step two, constructing a hierarchical attention-based graph characterization network: the patch graph convolutional neural network is used for the structural perception hierarchy graph constructed in the step one, and local context characteristics of nodes in the high-magnification patch graph are encoded; fusing the patch characteristics with high magnification into the corresponding low magnification organization chart nodes by using a hierarchical attention layer and a cross attention mechanism; the fused tissue diagram is input into a tissue diagram convolutional neural network and used for coding the overall spatial tissue structural characteristics and capturing the infiltration relation between the tumor and the normal tissue; finally, the global attention layer aggregates all nodes of the tissue map to obtain a slice-level characteristic representation for prediction of the stage class.
2. The hierarchical graph-based pathological image primary tumor stage multi-example learning method according to claim 1, wherein the method is characterized by comprising the following steps of: the first step is to construct a structure perception hierarchy chart, and the specific implementation steps are as follows:
1) Data preprocessing
1.1 For a full-field digital pathological section with pyramid structure, the low resolution to the high resolution are respectively denoted as w t ，w 1 ，w 2 ，…w n ,w n+1 Will w t The resolution of the image is sampled to half, the tissue is separated from the background by adopting an Ojin binarization algorithm, a binary division mask is obtained, and then the division mask is respectively processed for 2 1 、2 2 、2 3 ，…2 n ，2 n+1 ，2 n+2 Up-sampling multiple times to match w respectively t ，w 1 ，w 2 ，…w n ,w n+1 Resolution of (2);
1.2 Cutting the slices with different multiples obtained in the step 1.1) into non-overlapping patches by adopting a sliding window method to respectively generate patch sets P t ，P 1 ，P 2 ，…P n ,P n+1 The method comprises the steps of carrying out a first treatment on the surface of the Under the guidance of the segmentation mask, the background of the slice is ignored, and the patch only comprises a foreground region containing tissues;
1.3 At w) t Dividing the image processed in the step 1.2) into different tissue blocks by using a simple linear iterative clustering algorithm;
2) Feature extraction
Extraction step oneThe patch set P obtained in step 1.2) t ，P 1 ，P 2 ，…P n ,P n+1 Characteristics of the nodes; will P t ，P 1 ，P 2 ，…P n ,P n +1 All patches in the patch are spliced together and input into a feature encoder to obtain initialized node features X of n+1 patch graphs 1 ，X 2 ，…X n ,X n+1 The method comprises the steps of carrying out a first treatment on the surface of the For the organizational chart, due to each organization block t therein i All cover patch P t Several patches in (c) may be used to make t i The covered patch is input into a feature encoder, and then the average pooling operation is carried out, so that the tissue block t can be obtained i Is characterized by (2); finally obtaining node characteristic X of the organization chart t ；
3) Building a structure-aware hierarchical graph G
Constructing a structure perception hierarchy chart G according to the node characteristics of the organization blocks and the organization charts obtained in the step 2); graph G contains three types of edges, E t For modeling the adjacency of adjacent tissue blocks in an organizational chart,
modeling the adjacency of the patch and its eight neighbors in each patch graph,/>
Modeling the affiliation of nodes in the organization chart and the patch chart; meanwhile, generating an adjacent matrix and an associated matrix A according to the connection relation of the edges t ，/>
At A t In the case of tissue t i ，t j With edges in between, the element +.>
Otherwise, 0, and the rest matrixes are the same; finally, the structure-aware hierarchical graph G may be expressed as +.>
/>
3. The hierarchical graph-based pathological image primary tumor stage multi-example learning method according to claim 1, wherein the method is characterized by comprising the following steps of: step two, constructing a hierarchical attention-based graph characterization network, which comprises the following specific steps:
1) Updating patch node characteristics through a patch convolution neural network:
after the structural perception hierarchy graph G is built in the step one, for each patch graph
Respectively inputting the patch images into the patch image convolution neural network; the structure of each patch is the same, each patch convolve neural network and has L layers, each layer can update the characteristics of each node, and for each node, the first layer (L E [0, L]) The characteristic updating process of (a) is as follows:
1.1 For node i, its current characteristics are
Finding all neighbor nodes N (i), for each neighbor node j epsilon N (i), adding the characteristics of the neighbor nodes with the characteristics of the node i, and activating the added characteristics through a ReLU function;
1.2 All the added features are accumulated into one fusion feature through an attention mechanism;
1.3 Fusion features to be obtained
Residual connection is carried out and then the residual connection is input into a multi-layer perceptron to obtain updated characteristics
Finishing the updating of the first layer characteristics;
after the L layers are all updated, splicing the outputs of all the layers together to serve as node characteristics of a final patch diagram; after the neural network is rolled up by the patch chart, the patch chart is updated as
2) The patch graph node features are fused into the tissue graph node features through a hierarchical attention layer:
inputting the patch diagram into a hierarchical attention layer according to the incidence matrix
Patches contained in each organization in the organization chart are found, under the guidance of the organization characteristics, the patch node characteristics contained in the organization are weighted and summed by utilizing a cross attention mechanism, and detail perceived organization characteristics are generated>
Wherein the weight is the similarity of the patch and its corresponding organization; splicing the detail perception features of the two scales to obtain an organization chart feature fused with the fine-granularity patch node feature, wherein the fused organization chart is expressed as G f ；
3) Updating the node characteristics of the tissue map through the convolutional neural network of the tissue map:
will G f Inputting the tissue map convolution neural network, and encoding the global space relation of tissue map nodes into each tissue node characteristic to obtain an updated tissue map G '' f The method comprises the steps of carrying out a first treatment on the surface of the Tissue ofThe design of the graph convolution neural network is the same as that of the patch graph convolution neural network, and the characteristic updating process is the same as that of the steps 1.1) to 1.3);
4) The node features in the organizational chart are fused into packet-level features for staging through the global attention layer:
will G' f Inputting a global attention layer, and weighting and fusing the characteristics of the tissue nodes according to the importance of the tissue nodes on the slices to obtain a package-level characteristic representation x bag The method comprises the steps of carrying out a first treatment on the surface of the And then x is bag Inputting the full-connecting layer, and obtaining a final primary tumor stage prediction result after softmax operation.
4. The primary tumor stage multi-example learning framework based on the pathological image of the hierarchical map is characterized in that: the system comprises a structure perception hierarchy diagram construction module and a diagram characterization network module based on hierarchical attention:
the structure perception hierarchy diagram construction module is used for representing the full-view digital slice of the pyramid structure as a heterogeneous structure perception hierarchy diagram, and comprises two types of subgraphs: a tissue map at the top layer and a plurality of patch maps at the lower layer are respectively established on the low-magnification slice and the relatively high-magnification slice; specifically, in the tissue map, different tissues and nearest neighbor relations among the different tissues are respectively expressed as nodes and edges, so that spatial relations among the different tissues are clearly expressed, and the network model is beneficial to perceiving the infiltration relation between tumors and normal tissues; in the patch diagram, patches are taken as nodes, eight neighborhood relations among the patches are taken as edges, and modeling of fine granularity characteristics under high multiplying power is realized; in addition, in the switching process between fields of different magnification, determining the subordinate relation between patch map nodes and corresponding tissue map nodes under different magnification through the incidence matrix structure;
the hierarchical attention-based graph characterization network module includes four parts: patch map convolutional neural network, hierarchical attention layer, organizational chart convolutional neural network, and global attention layer; the patch graph convolutional neural network is used for encoding local context characteristics of nodes in the high-magnification patch graph; the hierarchical attention layer fuses the patch characteristics of high magnification into the corresponding low magnification organization chart nodes by using a cross attention mechanism; the fused tissue diagram is input into a tissue diagram convolutional neural network and used for coding the overall spatial tissue structural characteristics and capturing the infiltration relation between the tumor and the normal tissue; finally, the global attention layer aggregates all nodes of the tissue map to obtain a slice-level characteristic representation for prediction of the stage class.
5. The primary tumor stage multi-example learning device based on the pathological image of the hierarchical map is characterized in that: comprising the following steps:
a memory for storing a computer program;
a processor for implementing the hierarchical graph based pathology image primary tumor stage multi-example learning method of any one of claims 1 to 3 when executing the computer program.
6. A computer-readable storage medium storing a computer program, characterized in that: the computer program, when executed by the processor, enables a user to stage primary tumor of the histopathological image based on the structural perception hierarchy chart by utilizing a multi-example learning method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211714879.7A CN116012332A (en) | 2022-12-29 | 2022-12-29 | Hierarchical graph-based pathological image primary tumor stage multi-example learning method, frame, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211714879.7A CN116012332A (en) | 2022-12-29 | 2022-12-29 | Hierarchical graph-based pathological image primary tumor stage multi-example learning method, frame, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116012332A true CN116012332A (en) | 2023-04-25 |
Family
ID=86031360
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211714879.7A Pending CN116012332A (en) | 2022-12-29 | 2022-12-29 | Hierarchical graph-based pathological image primary tumor stage multi-example learning method, frame, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116012332A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116485792A (en) * | 2023-06-16 | 2023-07-25 | 中南大学 | Histopathological subtype prediction method and imaging method |
| CN117455906A (en) * | 2023-12-20 | 2024-01-26 | 东南大学 | Digital pathological pancreatic cancer nerve segmentation method based on multi-scale cross fusion and boundary guidance |
-
2022
- 2022-12-29 CN CN202211714879.7A patent/CN116012332A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116485792A (en) * | 2023-06-16 | 2023-07-25 | 中南大学 | Histopathological subtype prediction method and imaging method |
| CN116485792B (en) * | 2023-06-16 | 2023-09-15 | 中南大学 | Histopathological subtype prediction method and imaging method |
| CN117455906A (en) * | 2023-12-20 | 2024-01-26 | 东南大学 | Digital pathological pancreatic cancer nerve segmentation method based on multi-scale cross fusion and boundary guidance |
| CN117455906B (en) * | 2023-12-20 | 2024-03-19 | 东南大学 | Digital pathological pancreatic cancer nerve segmentation method based on multi-scale cross fusion and boundary guidance |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108830326B (en) | Automatic segmentation method and device for MRI (magnetic resonance imaging) image | |
| CN109697460B (en) | Object detection model training method and target object detection method | |
| CN116012332A (en) | Hierarchical graph-based pathological image primary tumor stage multi-example learning method, frame, equipment and medium | |
| JP7026826B2 (en) | Image processing methods, electronic devices and storage media | |
| CN109978918A (en) | A kind of trajectory track method, apparatus and storage medium | |
| CN105574859A (en) | Liver tumor segmentation method and device based on CT (Computed Tomography) image | |
| CN110310253B (en) | Digital slice classification method and device | |
| Pan et al. | Cell detection in pathology and microscopy images with multi-scale fully convolutional neural networks | |
| CN108229576B (en) | Cross-magnification pathological image feature learning method | |
| CN111488921A (en) | Panoramic digital pathological image intelligent analysis system and method | |
| CN110264454B (en) | Cervical cancer histopathological image diagnosis method based on multi-hidden-layer conditional random field | |
| CN113313164A (en) | Digital pathological image classification method and system based on superpixel segmentation and image convolution | |
| CN111402267A (en) | Segmentation method, device and terminal for epithelial cell nucleus in prostate cancer pathological image | |
| CN110348435A (en) | A kind of object detection method and system based on clipping region candidate network | |
| CN115909006B (en) | Mammary tissue image classification method and system based on convolution transducer | |
| CN112348059A (en) | Deep learning-based method and system for classifying multiple dyeing pathological images | |
| Cao et al. | An automatic breast cancer grading method in histopathological images based on pixel-, object-, and semantic-level features | |
| CN113378792A (en) | Weak supervision cervical cell image analysis method fusing global and local information | |
| CN113469119A (en) | Cervical cell image classification method based on visual converter and graph convolution network | |
| CN113902669A (en) | Method and system for reading urine exfoliative cell fluid-based smear | |
| Wang et al. | A region-line primitive association framework for object-based remote sensing image analysis | |
| CN112927215A (en) | Automatic analysis method for digestive tract biopsy pathological section | |
| CN112488996A (en) | Inhomogeneous three-dimensional esophageal cancer energy spectrum CT (computed tomography) weak supervision automatic labeling method and system | |
| Ye et al. | Multitask classification of breast cancer pathological images using SE-DenseNet | |
| CN115393378B (en) | Low-cost and efficient cell nucleus image segmentation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |