WO2024051655A1 - 全视野组织学图像的处理方法、装置、介质和电子设备 - Google Patents

全视野组织学图像的处理方法、装置、介质和电子设备 Download PDF

Info

Publication number
WO2024051655A1
WO2024051655A1 PCT/CN2023/116820 CN2023116820W WO2024051655A1 WO 2024051655 A1 WO2024051655 A1 WO 2024051655A1 CN 2023116820 W CN2023116820 W CN 2023116820W WO 2024051655 A1 WO2024051655 A1 WO 2024051655A1
Authority
WO
WIPO (PCT)
Prior art keywords
causal
subgraph
image
resolution
node
Prior art date
Application number
PCT/CN2023/116820
Other languages
English (en)
French (fr)
Inventor
边成
张志诚
李永会
Original Assignee
抖音视界有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 抖音视界有限公司 filed Critical 抖音视界有限公司
Publication of WO2024051655A1 publication Critical patent/WO2024051655A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/136—Segmentation; Edge detection involving thresholding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present disclosure relates to the field of image processing technology, and specifically, to a processing method, device, medium and electronic device for full-field histological images.
  • Deep neural networks have been widely used in the field of image processing technology.
  • deep neural networks can handle smaller image sizes, such as 255*255 or 512*512.
  • Histopathological Whole-slide Image English: Histopathological Whole-slide Image, abbreviation: WSI
  • WSI Histopathological Whole-slide Image
  • the gold standard for many diagnoses often takes up 100M to 10G of storage space due to the excessive number of pixels (such as 80000*80000, or even 200000*2000000). ), which cannot be processed directly using deep neural networks.
  • the present disclosure provides a method for processing full-field histological images, the method including:
  • Segment multiple full-field histological images of the specified site to obtain multiple image blocks corresponding to each full-field histological image, and each full-field histological image has a different resolution
  • a heterogeneous graph is generated based on a plurality of image blocks corresponding to all the full-field histological images.
  • the heterogeneous graph includes a node set and an edge set, and the node set includes an image corresponding to each of the image blocks. Nodes composed of features and resolutions corresponding to the image blocks.
  • the edge set includes spatial edges used to characterize the spatial relationship between each node, and resolution edges used to characterize the resolution relationship between each node;
  • Extract a causal subgraph from the heterogeneous graph the causal subgraph includes features that are not related to the distribution of the environmental subgraph, and the environmental subgraph is the heterogeneous graph except the causal subgraph. Area;
  • the indication information corresponding to the designated part is determined according to the causal subgraph, and the indication information is used to characterize the state of the designated part and/or the target area in the designated part.
  • the present disclosure provides a full-field histological image processing device, which device includes:
  • a segmentation module used to segment multiple full-field histological images of a designated part to obtain multiple image blocks corresponding to each full-field histological image, and each full-field histological image has a different resolution.
  • a generation module configured to generate a heterogeneous graph based on the plurality of image blocks corresponding to all the full-field histological images.
  • the heterogeneous graph includes a node set and an edge set, and the node set includes each of the A node composed of the image features corresponding to the image block and the resolution corresponding to the image block.
  • the edge set includes spatial edges used to characterize the spatial relationship between the nodes, and resolutions used to represent the resolution relationship between the nodes. side;
  • An extraction module configured to extract a causal subgraph from the heterogeneous graph.
  • the characteristics included in the causal subgraph are not related to the distribution of the environmental subgraph.
  • the environmental subgraph is the heterogeneous graph except for the above mentioned ones. cause and effect Area outside the picture;
  • a processing module configured to determine indication information corresponding to the designated part according to the causal subgraph, where the indication information is used to characterize the state of the designated part and/or the target area in the designated part.
  • the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processing device, the steps of the method described in the first aspect of the present disclosure are implemented.
  • an electronic device including:
  • a processing device configured to execute the computer program in the storage device to implement the steps of the method described in the first aspect of the present disclosure.
  • the present disclosure first segments multiple full-field histological images of different resolutions at a designated site to obtain multiple image blocks corresponding to each full-field histological image, and then based on the corresponding corresponding images of all full-field histological images.
  • Multiple image blocks to generate a heterogeneous graph, where the heterogeneous graph includes a node set and an edge set, and then extract the causal subgraph that is not related to the distribution of the environmental subgraph from the heterogeneous graph, and finally determine the specified design based on the causal subgraph Instruction information corresponding to the part, where the indication information is used to characterize the status of the specified part and/or the target area in the specified part.
  • This disclosure achieves essentially interpretable full-field histology by constructing heterogeneous graphs that can characterize image features, spatial relationships, and resolution relationships, extracting causal subgraphs that satisfy distribution invariance, and thereby determining indication information.
  • Figure 1 is a flow chart of a method for processing full-field histological images according to an exemplary embodiment
  • Figure 2 is a flow chart of another method for processing full-field histological images according to an exemplary embodiment
  • Figure 3 is a structural diagram of a processing model according to an exemplary embodiment
  • Figure 4 is a flow chart of another method for processing full-field histological images according to an exemplary embodiment
  • Figure 5 is a flow chart of another method for processing full-field histological images according to an exemplary embodiment
  • Figure 6 is a structural diagram of another processing model according to an exemplary embodiment
  • Figure 7 is a flow chart of a training processing model according to an exemplary embodiment
  • Figure 8 is a flow chart of another training processing model according to an exemplary embodiment
  • Figure 9 is a block diagram of a full-field histological image processing device according to an exemplary embodiment
  • Figure 10 is a block diagram of another full-field histological image processing device according to an exemplary embodiment
  • Figure 11 is a block diagram of another full-field histological image processing device according to an exemplary embodiment
  • FIG. 12 is a block diagram of an electronic device according to an exemplary embodiment.
  • the term “include” and its variations are open-ended, that is, “including but not limited to.”
  • the term “based on” means “based at least in part on.”
  • the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; and the term “some embodiments” means “at least some embodiments”. Relevant definitions of other terms will be given in the description below.
  • a prompt message is sent to the user to clearly remind the user that the operation requested will require the acquisition and use of the user's personal information. Therefore, users can autonomously choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that perform the operations of the technical solution of the present disclosure based on the prompt information.
  • the method of sending prompt information to the user may be, for example, a pop-up window, and the prompt information may be presented in the form of text in the pop-up window.
  • the pop-up window can also contain a selection control for the user to choose "agree” or "disagree” to provide personal information to the electronic device.
  • Figure 1 is a flow chart of a method for processing full-field histological images according to an exemplary embodiment. As shown in Figure 1, the method includes:
  • Step 101 Segment multiple full-field histological images of the specified site to obtain multiple image blocks corresponding to each full-field histological image.
  • Each full-field histological image has a different resolution.
  • a full-field histological image scanner to scan the designated site to obtain a full-field histological image reflecting the designated site.
  • the designated part may be an organ of the human body, such as the liver, heart, stomach, etc., or it may be a partial region of the organ, which is not specifically limited in this disclosure.
  • the full-field histological image can then be enlarged or reduced to obtain multiple full-field histological images with different resolutions. For example, 3 full-field histology images can be obtained with resolutions of Thumbnail, 5X, and 10X respectively.
  • Each full-field histological image can be segmented according to preset rules to obtain multiple image patches (English: patches) corresponding to each full-field histological image.
  • each full-field tissue can be segmented according to the sliding-windows method. Learn to segment the image into multiple image blocks with a size of 512*512 pixels.
  • the background part in the full-field histological image can be filtered out before segmenting the full-field histological image.
  • the RGB value can be The area whose variance is less than the preset threshold is determined as the background part, and then the background part in the full-field histological image is deleted and then segmented. The resulting multiple image blocks will not contain the background part, which avoids the interference of the background part on the recognition.
  • Step 102 Generate a heterogeneous graph based on multiple image blocks corresponding to all full-field histological images.
  • the heterogeneous graph includes a node set and an edge set.
  • the node set includes the image features corresponding to each image block and the image features corresponding to the image block.
  • the edge set consists of nodes composed of resolutions, including space edges used to characterize the spatial relationship between nodes, and resolution edges used to characterize the resolution relationship between nodes.
  • a heterogeneous map can be generated based on multiple image blocks corresponding to all full-field histological images, as well as the spatial relationship and resolution relationship between the image blocks.
  • the heterogeneous graph includes a node set and an edge set.
  • the node set includes multiple nodes, and the nodes correspond to the image blocks one-to-one, that is, each node corresponds to an image block, and the image features and resolution corresponding to the image block can be combined into a node.
  • Image features can be understood as feature vectors or feature maps (English: Feature Map) used to characterize image blocks. For example, if the image feature is a 1024-dimensional feature vector, then the image feature and the corresponding resolution can be combined into a 1025-dimensional feature vector as a node.
  • the edge set includes multiple edges.
  • the edges can be space edges used to characterize the spatial relationship between nodes, and resolution edges used to characterize the resolution relationship between nodes.
  • the spatial relationship is used to describe any two images. The relationship between blocks in the same full-field histological image. If two image blocks are in the same full-field histological image, and the two image blocks are adjacent (that is, spatially adjacent), then it indicates that the relationship between the two image blocks is There is a spatial relationship, and there is a spatial edge between the nodes corresponding to the two image blocks.
  • the resolution relationship is used to describe the relationship between any two image blocks in two different full-field histological images.
  • Resolution adjacency can be understood as sorting the resolutions of multiple full-field histology images. Adjacent resolutions are resolution adjacencies. For example, there are three types: Thumbnail, 5X and 10X. resolution, then Thumbnail and 5X are adjacent resolutions, and 5X and 10X are adjacent resolutions. Heterogeneous maps can effectively characterize the image features, spatial relationships, and resolution relationships of multiple full-field histological images.
  • Step 103 Extract the causal subgraph from the heterogeneous graph.
  • the characteristics included in the causal subgraph are not related to the distribution of the environmental subgraph.
  • the environmental subgraph is the area in the heterogeneous graph except the causal subgraph.
  • Step 104 Determine the indication information corresponding to the specified part according to the causal subgraph.
  • the indication information is used to characterize the status of the specified part and/or the target area in the specified part.
  • the heterogeneous graph can be learned to extract the causal subgraph and environmental subgraph from it.
  • the environment subgraph is the area in the heterogeneous graph except the causal subgraph.
  • the characteristics included in the causal subgraph are not related to the distribution of the environmental subgraph, that is, the characteristics included in the causal subgraph will not be affected by the distribution of the environmental subgraph. It can also be understood that the causal subgraph has distribution invariance. In other words, the characteristics included in the causal subgraph are essentially different from the characteristics included in the environmental subgraph.
  • a cat's whiskers are a characteristic of a cat. This characteristic will not be affected by the environment. That is, whether the cat is on the lawn, on the sofa, or on the floor, the characteristic of the whiskers remains unchanged.
  • the causal subgraph can be used to determine the indication information corresponding to the specified part.
  • the indication information may include information used to characterize the state of the specified part, and may also include information used to characterize the target area in the specified part.
  • the information and indication information may also include information used to characterize the state of the designated part and information used to characterize the target area in the designated part.
  • the status of a designated site can include two states: good and diseased, or it can also include three states: primary, intermediate, and advanced.
  • the state of a designated site can also be used to describe the 5-year survival rate.
  • the state of a designated site can also be used to describe the 5-year survival rate.
  • the prognosis is described, which is not specifically limited in this disclosure.
  • the target area in the designated part can be understood as the area that needs attention in the full-field histological image, which can intuitively and effectively help professionals make judgments on the designated part. Since the indication information is directly determined based on the causal subgraph, the target region is also determined based on the causal subgraph, that is, the target region is the region corresponding to the causal subgraph in the full-field histological image.
  • the indication information is determined directly based on the causal subgraph, that is to say, the indication information is completely determined based on the causal subgraph, that is, there is a direct causal relationship between the indication information and the causal subgraph.
  • the indication information in this disclosure has direct cause and effect with the causal subgraph. Relational and causal subgraphs can provide intrinsic interpretability for indicator information, enabling the recognition of intrinsically interpretable full-field histological images.
  • FIG. 2 is a flow chart of another method for processing full-field histological images according to an exemplary embodiment. As shown in Figure 2, the implementation of step 102 may include:
  • Step 1021 Perform feature extraction on each image block to obtain the image features corresponding to the image block.
  • Step 1022 Use the image feature corresponding to each image block and the resolution corresponding to the image block as a node to generate a node set.
  • feature extraction can be performed on each image block to obtain image features that can characterize the image block.
  • the ResNet network, the KimiaNet network, or the Encoder in the Transformer can be used to extract features from the image blocks, and this disclosure does not specifically limit this.
  • the image features corresponding to each image block and the resolution corresponding to the image block can be concatenated (English: Concat) as the node corresponding to the image block, thereby obtaining a node set V composed of nodes corresponding to all image blocks. .
  • Step 1023 Construct a spatial edge between two nodes corresponding to any two adjacent image blocks in the same full-field histological image, and construct a spatial edge between any two image blocks corresponding to the same area of the designated part and with adjacent resolutions.
  • a resolution edge is constructed between two nodes to generate an edge set.
  • Step 1024 Generate a heterogeneous graph based on the node set and edge set.
  • an edge set is constructed based on the spatial relationship and resolution relationship between any two image patches. Specifically, for any two adjacent image blocks in the same full-field histological image, in these two images A spatial edge is constructed between the nodes corresponding to the image blocks. That is to say, the two image blocks corresponding to the two nodes connected by the spatial edge have the same resolution and are spatially adjacent. Multiple spatial edges can be expressed as a spatial adjacency matrix (denoted as A spacial ). The number of image patches is
  • a resolution edge is constructed between the nodes corresponding to the two image blocks. That is to say, the two nodes corresponding to the two nodes connected by the resolution edge
  • the image blocks correspond to the same area in the specified part and are adjacent in resolution.
  • Multiple resolution edges can be expressed as a resolution adjacent matrix (expressed as A scale ).
  • the number of image blocks
  • the size of A scale
  • which corresponds to the same area of the specified part.
  • which corresponds to the same area of the specified part.
  • which corresponds to the same area of the specified part.
  • which corresponds to the same area of the specified part.
  • which corresponds to the same area of the specified part.
  • the corresponding elements of two image blocks with adjacent resolutions are 1, and other elements are 0.
  • the edge set E consisting of all space edges and resolution edges is obtained.
  • G (V,
  • a processing model can be pre-trained to implement the full-field histological image processing method provided by the present disclosure.
  • the structure of the processing model is shown in Figure 3 and can include: an extractor (represented as Rational Extractor), The encoder (denoted as Heterogeous Encoder) and the classifier (denoted as Classifier), where the connection relationship between the extractor, encoder and classifier is: the input of the extractor is used as the input of the processing model, and the output of the extractor is used as the encoder The input of the sum, the output of the encoder, serves as the input of the classifier, and the output of the classifier serves as the output of the processing model.
  • FIG. 4 is a flow chart of another method for processing full-field histological images according to an exemplary embodiment. As shown in Figure 4, step 103 can be implemented as follows:
  • the heterogeneous graph is fed into the extractor in the pre-trained processing model to obtain the causal subgraph.
  • step 104 may include:
  • Step 1041 Input the causal subgraph into the encoder in the processing model to obtain causal image features used to characterize the causal subgraph.
  • Step 1042 Input the causal image features into the classifier in the processing model to obtain the specified part status. and/or, identify target regions in full-field histology images based on causal subgraphs.
  • the causal subgraph obtained in step 102 can be input to the extractor.
  • the node set, the spatial adjacency matrix, and the resolution adjacency matrix can be input to the extractor, and the extractor can extract the causal subgraph therefrom.
  • the extractor can first learn from the heterogeneous graph the contribution of each spatial edge to determining the state of the specified part (that is, the influence mentioned later), and the contribution of each resolution edge to determining the state of the specified part. contribution, and then generate a causal subgraph based on the spatial edges and resolution edges with the largest contribution of a specified proportion (for example, 20%).
  • the causal subgraph can then be input to the encoder in the processing model to obtain causal image features used to characterize the causal subgraph.
  • the causal image feature can be a feature vector or a feature map.
  • the encoder can extract features of heterogeneous graphs, such as GNN (English: Graph Neural Network, Chinese: Graph Neural Network), or HGAT (English: Heterogeneous Graph Attention Networks, Chinese: Heterogeneous Graph Attention Network), This disclosure does not specifically limit this.
  • GNN English: Graph Neural Network, Chinese: Graph Neural Network
  • HGAT English: Heterogeneous Graph Attention Networks, Chinese: Heterogeneous Graph Attention Network
  • the target area can be determined in the full-field histological image based on the causal subgraph. That is, the target region is the region in the full-field histological image corresponding to the causal subgraph.
  • FIG. 5 is a flow chart of another method for processing full-field histological images according to an exemplary embodiment. As shown in Figure 5, step 103 can be implemented by the following steps:
  • Step 1031 Input the heterogeneous graph into the extractor and determine the influence of each spatial edge and the influence of each resolution edge.
  • Step 1032 Sort the influence degree of each space edge to determine the causal space edge, and sort the influence degree of each resolution edge to determine the causal resolution edge.
  • Step 1033 Determine the causal subgraph based on the causal space edges and causal resolution edges.
  • the heterogeneous graph can be input into the extractor, and the extractor can be used to learn the influence of each spatial edge and the influence of each resolution edge in the heterogeneous graph.
  • the extractor can include graph neural network network, the first multi-layer perceptron and the second multi-layer perceptron.
  • the heterogeneous graph can be input into the graph neural network, and the graph neural network can extract the graph structure features corresponding to the heterogeneous graph.
  • the graph structure features can be input into the first multi-layer perceptron and the second multi-layer perceptron respectively, and the influence of each spatial edge output by the first multi-layer perceptron and the influence of each spatial edge output by the second multi-layer perceptron can be obtained.
  • the influence of the resolution edge can be understood as the contribution of the edge to determining the state of the specified part, and can also be understood as the importance of the edge.
  • Two multi-layer perceptrons (English: Multi-Layer Perception, abbreviation: MLP) can be trained separately, and then the sigmoid function is used to determine the influence of each spatial edge and the influence of each resolution edge.
  • MLP Multi-Layer Perception
  • MLP spacial represents the first multi-layer perceptron
  • MLP scale represents the second multi-layer perceptron
  • ⁇ represents the sigmoid function
  • Z spacial represents the output of the first multi-layer perceptron
  • Z scale represents the output of the second multi-layer perceptron.
  • M spatial represents the influence of each spatial edge
  • M scale represents the influence of each resolution edge.
  • M spacial can be understood as a matrix corresponding one-to-one to the elements in the spatial adjacency matrix, where each element is used to represent the influence of the corresponding spatial edge.
  • M scale can be understood as the resolution adjacency matrix. A matrix with one-to-one correspondence between elements, where each element is used to represent the influence of the corresponding resolution edge.
  • the influence degree of each spatial edge can be sorted to determine the causal space edge to construct the causal subgraph, and the influence degree of each resolution edge can be sorted to determine the causal subgraph to be constructed.
  • the causal resolution edge of Specifically, the influence degree of each spatial edge can be arranged in descending order, and the top 20% of the spatial edges can be selected as the causal space edges. At the same time, the influence degree of each resolution edge can be arranged in descending order, and the top 20% of the spatial edges can be selected as the causal space edges. The first 20% of the resolution edges serve as causal resolution edges.
  • E c_spa Top r1 (M spacial ⁇ A spacial )
  • E c_sca Top r2 (M scale ⁇ A scale )
  • E c E c_spa ⁇ E c_sca
  • E c_spa represents the causal space edge
  • E c_sca represents the causal resolution edge
  • Top r1 represents the selection of the largest r1 edges
  • Top r2 represents the selection of the largest r2 edges
  • ⁇ represents element-wise multiplication
  • E c represents the causal space edge. and the union of causal resolution edges.
  • u represents a node
  • V c represents the set of nodes connected by E c
  • V s represents the set of nodes other than V c
  • C represents the causal subgraph
  • E represents the environment subgraph, that is, the area in the heterogeneous graph except the causal subgraph.
  • step 1033 can be implemented through the following steps:
  • Step 4) Use the causal resolution edges and causal space edges that satisfy the resolution causality condition as the causal edge set, and the nodes connected by the causal edge set as the causal node set.
  • the resolution causality condition is: for each causal node, if the The causal node is not the root node. There is a causal resolution edge belonging to the causal edge set connecting the causal node and the corresponding parent node. The root node is the node with the lowest resolution. The parent node The resolution of the point is smaller than the resolution of the causal node.
  • Step 5 Determine the causal subgraph based on the causal edge set and the causal node set.
  • the preset resolution causality conditions can be used to filter the causal resolution edges and causal space edges to obtain the causal edge set, and then the causal edge set includes each edge (including the causal resolution edge and the causal space edge).
  • the connected nodes are used as causal nodes, and a causal node set including all causal nodes is obtained.
  • a causal subgraph is generated based on the causal edge set and the causal node set.
  • the resolution causal condition is: for each causal node, if the causal node is not a root node, there is a causal resolution edge belonging to the causal edge set connecting the causal node to the parent node, and the root node has the lowest resolution. node, the resolution of the parent node is smaller than the resolution of the causal node.
  • the causal edge set should include a causal resolution edge connecting the causal node and a causal node with a resolution of 5X.
  • the causal edge set should include a causal resolution edge connecting the causal node and a causal node with a resolution of Thumbnail. If a causal node does not satisfy the resolution causality condition, then the causal node and all edges connected to the causal node are deleted.
  • V' c ⁇ u ⁇ V:u ⁇ E' c ⁇
  • V' s ⁇ u ⁇ V:u ⁇ E-E' c ⁇
  • C (V' c ,E' c )
  • E (V' s ,E-E' c )
  • ScaleClean represents the resolution causal condition
  • E' c represents the causal edge set
  • V' c represents the causal point set
  • V' s represents the set of nodes other than V' c .
  • the resolution causality condition can be understood as, since the two nodes connected by the resolution edge correspond to the same area of the specified part (that is, represent the same part of cells), if the node with lower resolution among the two nodes does not Belongs to the causal subgraph, then the nodes with corresponding high resolution should not belong to the causal subgraph. This can avoid dividing the characteristics of the same cell into causal subgraphs and environmental subgraphs at the same time. It should be noted that if the node with high resolution does not belong to the causal subgraph, it will not affect the node with low resolution. For example, a node with a resolution of 5X can represent a cell, and a corresponding node with a resolution of 10X can represent a part of the cell. The cell can be treated as a causal subgraph while excluding some normal areas in the cell.
  • the encoder can include: a resolution-based attention convolution layer and an iterative pooling layer.
  • the input of the resolution-based attention convolution layer is used as the input of the encoder
  • the output of the resolution-based attention convolution layer is used as the input of the iterative pooling layer
  • the output of the iterative pooling layer is used as the output of the encoder.
  • the resolution-based attention convolution layer and the iterative pooling layer can be used as one unit, and the encoder can include multiple (for example, 2) units connected in sequence.
  • Step 1041 may include:
  • Step 6 Input the causal subgraph into the attention convolution layer, so that the attention convolution layer determines the characteristics of the node based on each node in the causal subgraph and the resolution of the node with an edge between the node.
  • Step 7) Input the structural features corresponding to each node in the causal subgraph into the iterative pooling layer, so that the iterative pooling layer pools the features of nodes with spatial edges between them to obtain causal image features.
  • the causal subgraph is input to a resolution-based attention convolution layer.
  • the attention convolution layer determines the node based on the resolution of each node in the causal subgraph and the node with an edge between it and the node. Node characteristics.
  • the traditional attention convolution layer calculates the attention score of each edge in the same way, while the resolution-based attention convolution layer calculates the attention score of each edge based on the spatial edge sum.
  • ⁇ vv' represents the attention score of the edge between node v and node v'
  • ⁇ r represents the resolution score
  • ⁇ v' represents the initial attention score
  • ⁇ represents the activation function, for example, it can be ReLu
  • h v represents the feature (ie embedding) of node v
  • h r represents the mean of the features of all nodes with resolution r
  • h r' represents the resolution of The mean value of the features of all nodes of r'
  • R represents the resolution set
  • h v' represents the characteristics of node v'
  • represents splicing (i.e.
  • N r represents the node with resolution r and adjacent to node v
  • the set of v" represents any node with resolution r and adjacent to node v
  • h v" represents the characteristics of v
  • V T represents the learnable attention layer for all neighbor nodes of node v.
  • the structural features corresponding to each node in the causal subgraph can be input into the iterative pooling layer, so that the iterative pooling layer pools the features of nodes with spatial edges between them to obtain causal image features. That is, nodes with similar semantic features and similar spatial distributions are aggregated while preserving the structure of the heterogeneous graph, thereby reducing the computational burden.
  • the structure of the graph neural network in the extractor mentioned above can be the same as that of the encoder, that is, the graph neural network includes a resolution-based attention convolution layer and an iterative pooling layer.
  • the input of the rate-based attention convolution layer is used as the input of the graph neural network
  • the output of the resolution-based attention convolution layer is used as the input of the iterative pooling layer
  • the output of the iterative pooling layer is used as the output of the graph neural network.
  • the resolution-based attention convolution layer and the iterative pooling layer can be used as a unit, and the graph neural network can include multiple (for example, 2) sequentially connected units.
  • the structure of the processing model consists of extractors, encoders, and classifiers. On the basis of , it can also include a sampler (denoted as Sampler) and an environment classifier (denoted as Environment Classifier), as shown in Figure 6.
  • Figure 7 is a flow chart of training a processing model according to an exemplary embodiment. As shown in Figure 7, the processing model is trained in the following manner:
  • Step A Obtain a sample input set and a sample output set.
  • the sample input set includes multiple sample inputs.
  • the sample input includes training heterogeneous maps corresponding to multiple full-field histological training images of the specified site.
  • the sample output set includes the training heterogeneous images corresponding to each sample. Input corresponding sample output, and each sample output includes training instruction information corresponding to the training heterogeneous graph.
  • the sample input set includes multiple sample inputs
  • the sample output set includes the sample output corresponding to each sample input.
  • the sample input may include training heterogeneous images corresponding to multiple full-field histological training images of the specified site, and the resolutions corresponding to the multiple full-field histological training images are different.
  • the construction method of training heterogeneous graph is the same as step 102, and will not be described again this time.
  • each sample output includes training instruction information corresponding to the training heterogeneous graph, where the training instruction information is used to characterize the state of the specified part.
  • Step B For each sample input, input the training heterogeneous graph included in the sample input into the extractor to obtain the training causal subgraph.
  • Step C Determine the training environment subgraph based on the training heterogeneous graph and the training causal subgraph.
  • the training heterogeneous graph can be learned to extract the training causal subgraph and the training environment subgraph.
  • the training heterogeneous graph can be input to the extractor, so that the extractor learns from the training heterogeneous graph the contribution of each spatial edge to determining the state of the specified part, and the contribution of each resolution edge to determining the state of the specified part. contribution, and then generate a training causal subgraph based on the specified proportion of spatial edges and resolution edges that contribute the most. Then the area in the training heterogeneous graph except the training causal subgraph can be used as the training environment subgraph. You can also store the training environment subgraph obtained from each previous training in the cache, and then randomly select one from it as the training environment subgraph for this training.
  • step D the training causal subgraph and the training environment subgraph are input into the encoder to obtain the training causal image features used to characterize the training causal subgraph, and the training environment image features used to characterize the training environment subgraph.
  • step E the trained causal image features are input into the classifier to obtain predicted indication information
  • the trained environmental image features are input into the environment classifier in the processing model to obtain predicted environmental indication information.
  • Step F Determine the environmental loss based on the predicted environment indication information and the training indication information corresponding to the sample input to train the environment classifier.
  • Step G Determine the causal loss based on the prediction instruction information and the training instruction information corresponding to the sample input to train the extractor, encoder and classifier.
  • the training causal subgraph and the training environment subgraph are input into the encoder to obtain the training causal image features used to characterize the training causal subgraph, and the training environment image features used to characterize the training environment subgraph. Then input the trained causal image features into the classifier to obtain the predicted indication information (expressed as y' c ), and input the trained environmental image features into the environment classifier in the processing model to obtain the predicted environmental indication information (expressed as y' s ) .
  • the environmental loss can be determined based on the predicted environmental indication information and the training indication information corresponding to the sample input, and with the goal of reducing the environmental loss, the back propagation algorithm is used to train the parameters of the neurons in the environment classifier.
  • the parameters of the neurons For example, it can be the weight (English: Weight) and bias (English: Bias) of the neuron.
  • the environmental loss can be, for example, l(y' s ,y), where l represents the cross-entropy loss, and y represents the training instruction information corresponding to the sample input. That is, the environment loss is only used to train the environment classifier and does not affect the training of the extractor, encoder and classifier.
  • the causal loss is determined based on the prediction instruction information and the training instruction information corresponding to the sample input, and with the goal of reducing the causal loss, the back propagation algorithm is used to train the parameters of the neurons in the extractor, encoder, and classifier.
  • the causal loss can be, for example, l(y' c ,y). That is, causal losses are used to train extractors, encoders, and classifiers without affecting the training of environment classifiers.
  • FIG. 8 is a flow chart of another training processing model according to an exemplary embodiment. As shown in FIG. As shown in 8, step C may include:
  • Step C1 use the area in the training heterogeneous graph except the training causal subgraph as the sample environment subgraph.
  • Step C2 Store the sample environment subgraph into a sample environment subgraph set.
  • the sample environment subgraph set is used to store the sample environment subgraph corresponding to each sample input.
  • Step C3 Randomly select a sample environment subgraph from the sample environment subgraph set as the training environment subgraph.
  • the sample environment sub-image is stored in the sample environment sub-image set.
  • the sample environment sub-image set stores the sample environment sub-image obtained during the entire training process.
  • the sample environment sub-image set reflects the samples in the sample input set. Distribution of environment subgraphs.
  • the environment may affect the final recognition results.
  • the target area is the area where type A cells are located, and the areas where type B cells, type C cells, and type D cells are all located are the environment.
  • the processing model may judge whether type A cells exist based on type B cells, and cannot learn the essential causal relationship (that is, judge whether type A cells exist based on type A cells).
  • the processing model can learn a variety of environments.
  • the sampler is used to randomly select a sample environment subgraph from the sample environment subgraph set as the training environment subgraph, which can avoid the processing model to determine the prediction indication information based on the training environment subgraph, thus improving the accuracy of identification and ensuring the essential interpretive.
  • the present disclosure first segments multiple full-field histological images of different resolutions at a designated site to obtain multiple image blocks corresponding to each full-field histological image, and then based on the corresponding images of all full-field histological images Multiple image blocks to generate a heterogeneous graph, where the heterogeneous graph includes a node set and an edge set, and then extract the causal subgraph that is not related to the distribution of the environmental subgraph from the heterogeneous graph, and finally determine the specified design based on the causal subgraph Instruction information corresponding to the part, where the instruction information is used to characterize the specified part status, and/or the target area in the specified part.
  • the present disclosure achieves essentially interpretable full-field histology by constructing heterogeneous graphs that can characterize image features, spatial relationships, and resolution relationships, extracting causal subgraphs that satisfy distribution invariance, and thereby determining indication information.
  • Image recognition is essentially interpretable full-field histology by constructing heterogeneous graphs that can characterize image features, spatial
  • Figure 9 is a block diagram of a full-field histological image processing device according to an exemplary embodiment. As shown in Figure 9, the device 200 includes:
  • the segmentation module 201 is used to segment multiple full-field histological images of a designated site to obtain multiple image blocks corresponding to each full-field histological image. Each full-field histological image has a different resolution.
  • the generation module 202 is configured to generate a heterogeneous graph based on multiple image blocks corresponding to all full-field histological images.
  • the heterogeneous graph includes a node set and an edge set.
  • the node set includes image features corresponding to each image block and the image.
  • the nodes are composed of the resolution corresponding to the block.
  • the edge set includes the spatial edges used to characterize the spatial relationship between each node, and the resolution edges used to characterize the resolution relationship between each node.
  • the extraction module 203 is used to extract the causal subgraph from the heterogeneous graph.
  • the characteristics included in the causal subgraph are not related to the distribution of the environmental subgraph.
  • the environmental subgraph is the area in the heterogeneous graph except the causal subgraph.
  • the processing module 204 is configured to determine the indication information corresponding to the specified part according to the causal subgraph, and the indication information is used to characterize the status of the specified part and/or the target area in the specified part.
  • FIG. 10 is a block diagram of another full-field histological image processing device according to an exemplary embodiment.
  • the generation module 202 may include:
  • the extraction sub-module 2021 is used to extract features of each image block to obtain the image features corresponding to the image block.
  • the first generation sub-module 2022 is used to use the image feature corresponding to each image block and the resolution corresponding to the image block as a node to generate a node set.
  • the second generation sub-module 2023 is used to construct a spatial edge between two nodes corresponding to any two adjacent image blocks in the same full-field histological image. Any two corresponding designated parts are in the same area and have the same resolution. Construct a resolution edge between two nodes corresponding to adjacent image blocks to generate an edge set.
  • the third generation sub-module 2024 is used to generate a heterogeneous graph according to the node set and the edge set.
  • the extraction module 203 can be used to:
  • the heterogeneous graph is fed into the extractor in the pre-trained processing model to obtain the causal subgraph.
  • Processing module 204 may be used to:
  • the causal subgraph is input to the encoder in the processing model to obtain causal image features used to characterize the causal subgraph.
  • the causal image features are input to the classifier in the processing model to obtain the status of the specified part. and/or, identify target regions in full-field histology images based on causal subgraphs.
  • FIG 11 is a block diagram of another full-field histological image processing device according to an exemplary embodiment.
  • the extraction module 203 may include:
  • the influence degree extraction sub-module 2031 is used to input the heterogeneous graph into the extractor and determine the influence degree of each spatial edge and the influence degree of each resolution edge.
  • the sorting sub-module 2032 is used to sort the influence degree of each space edge to determine the causal space edge, and to sort the influence degree of each resolution edge to determine the causal resolution edge.
  • the determination sub-module 2033 is used to determine the causal subgraph according to the causal space edges and the causal resolution edges.
  • the extractor may include a graph neural network, a first multi-layer perceptron and a second multi-layer perceptron.
  • the extraction sub-module 2031 can be used to perform the following steps:
  • Step 1) Input the heterogeneous graph into the graph neural network to obtain the graph structure characteristics corresponding to the heterogeneous graph.
  • Step 2) Input the graph structure features into the first multi-layer perceptron to obtain the influence of each spatial edge.
  • Step 3 Input the graph structure features into the second multi-layer perceptron to obtain the influence of each resolution edge.
  • the determining sub-module 2033 can be used to perform the following steps:
  • Step 4) Use the causal resolution edges and causal space edges that satisfy the resolution causality condition as the causal edge set, and the nodes connected by the causal edge set as the causal node set.
  • the resolution causality condition is: for For each causal node, if the causal node is not a root node, there is a causal resolution edge belonging to the causal edge set connecting the causal node and the corresponding parent node.
  • the root node is the node with the lowest resolution, and the resolution of the parent node is less than The resolution of this causal node.
  • Step 5 Determine the causal subgraph based on the causal edge set and the causal node set.
  • the encoder can include: a resolution-based attention convolution layer and an iterative pooling layer.
  • Encoding sub-module 2041 can be used to perform the following steps:
  • Step 6 Input the causal subgraph into the attention convolution layer, so that the attention convolution layer determines the characteristics of the node based on each node in the causal subgraph and the resolution of the node with an edge between the node.
  • Step 7) Input the structural features corresponding to each node in the causal subgraph into the iterative pooling layer, so that the iterative pooling layer pools the features of nodes with spatial edges between them to obtain causal image features.
  • the processing model is trained by:
  • Step A Obtain a sample input set and a sample output set.
  • the sample input set includes multiple sample inputs.
  • the sample input includes training heterogeneous maps corresponding to multiple full-field histological training images of the specified site.
  • the sample output set includes the training heterogeneous images corresponding to each sample. Input corresponding sample output, and each sample output includes training instruction information corresponding to the training heterogeneous graph.
  • Step B For each sample input, input the training heterogeneous graph included in the sample input into the extractor to obtain the training causal subgraph.
  • Step C Determine the training environment subgraph based on the training heterogeneous graph and the training causal subgraph.
  • step D the training causal subgraph and the training environment subgraph are input into the encoder to obtain the training causal image features used to characterize the training causal subgraph, and the training environment image features used to characterize the training environment subgraph.
  • step E the trained causal image features are input into the classifier to obtain predicted indication information
  • the trained environmental image features are input into the environment classifier in the processing model to obtain predicted environmental indication information.
  • Step F Determine the environmental loss based on the predicted environment indication information and the training indication information corresponding to the sample input to train the environment classifier.
  • Step G Determine the causal loss based on the prediction instruction information and the training instruction information corresponding to the sample input to train the extractor, encoder and classifier.
  • step C may include:
  • Step C1 use the area in the training heterogeneous graph except the training causal subgraph as the sample environment subgraph.
  • Step C2 Store the sample environment subgraph into a sample environment subgraph set.
  • the sample environment subgraph set is used to store the sample environment subgraph corresponding to each sample input.
  • Step C3 Randomly select a sample environment subgraph from the sample environment subgraph set as the training environment subgraph.
  • the present disclosure first segments multiple full-field histological images of different resolutions at a designated site to obtain multiple image blocks corresponding to each full-field histological image, and then based on the corresponding images of all full-field histological images Multiple image blocks to generate a heterogeneous graph, where the heterogeneous graph includes a node set and an edge set, and then extract the causal subgraph that is not related to the distribution of the environmental subgraph from the heterogeneous graph, and finally determine the specified design based on the causal subgraph Instruction information corresponding to the part, where the indication information is used to characterize the status of the specified part and/or the target area in the specified part.
  • This disclosure achieves essentially interpretable full-field histology by constructing heterogeneous graphs that can characterize image features, spatial relationships, and resolution relationships, extracting causal subgraphs that satisfy distribution invariance, and thereby determining indication information.
  • Image recognition is essentially interpretable full-field histology by constructing heterogeneous graphs that can characterize image features, spatial
  • Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablets), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as Mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers, etc.
  • the electronic device shown in FIG. 12 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.
  • the electronic device 300 may include a processing device (eg, central processing unit, graphics processor, etc.) 301, which may be loaded into a random access device according to a program stored in a read-only memory (ROM) 302 or from a storage device 308.
  • the program in the memory (RAM) 303 executes various appropriate actions and processes.
  • various programs and data required for the operation of the electronic device 300 are also stored.
  • the processing device 301, ROM 302 and RAM 303 are connected to each other via a bus 304.
  • An input/output (I/O) interface 305 is also connected to bus 304.
  • the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 307 such as a computer; a storage device 308 including a magnetic tape, a hard disk, etc.; and a communication device 309.
  • the communication device 309 may allow the electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data.
  • FIG. 12 illustrates electronic device 300 with various means, it should be understood that implementation or availability of all illustrated means is not required. More or fewer means may alternatively be implemented or provided.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via communication device 309, or from storage device 308, or from ROM 302.
  • the processing device 301 When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, Hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic memory parts, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
  • Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the above.
  • terminal devices and servers can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium.
  • Communications e.g., communications network
  • communications networks include local area networks (“LAN”), wide area networks (“WAN”), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or developed in the future network of.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
  • the above-mentioned computer-readable medium carries one or more programs.
  • the electronic device causes the electronic device to: segment multiple full-field histological images of the designated part to obtain each A plurality of image blocks corresponding to the full-field histology image, each of the full-field histology images has a different resolution; according to all the full-field histology images, Corresponding to a plurality of the image blocks, a heterogeneous graph is generated.
  • the heterogeneous graph includes a node set and an edge set.
  • the node set includes the image features corresponding to each of the image blocks and the resolution corresponding to the image block.
  • the edge set includes space edges used to characterize the spatial relationship between the nodes, and resolution edges used to characterize the resolution relationship between the nodes;
  • the causal subgraph is extracted from the heterogeneous graph. , the characteristics included in the causal subgraph are not related to the distribution of the environmental subgraph, and the environmental subgraph is the area in the heterogeneous graph except the causal subgraph; determine the said causal subgraph according to the causal subgraph Instruction information corresponding to the designated part, the indication information is used to characterize the status of the designated part and/or the target area in the designated part.
  • Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as "C" or similar programming languages.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider). connected via the Internet).
  • LAN local area network
  • WAN wide area network
  • Internet service provider such as an Internet service provider
  • each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved.
  • each block in the block diagram and/or flowchart illustration, and the block diagram and/or flowchart The combinations of blocks in the figures may be implemented by dedicated hardware-based systems that perform specified functions or operations, or by a combination of dedicated hardware and computer instructions.
  • the modules involved in the embodiments of the present disclosure can be implemented in software or hardware.
  • the name of the module does not constitute a limitation on the module itself under certain circumstances.
  • the segmentation module can also be described as "a module for segmenting full-field histological images.”
  • FPGAs Field Programmable Gate Arrays
  • ASICs Application Specific Integrated Circuits
  • ASSPs Application Specific Standard Products
  • SOCs Systems on Chips
  • CPLD Complex Programmable Logical device
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include electrical connections based on one or more wires, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • CD-ROM portable compact disk read-only memory
  • magnetic storage device or any suitable combination of the above.
  • Example 1 provides a method for processing full-field histological images, including: segmenting multiple full-field histological images of a designated site to obtain each full-field tissue.
  • the composition includes a node set and an edge set.
  • the node set includes nodes composed of the image features corresponding to each image block and the resolution corresponding to the image block.
  • the edge set includes nodes using The spatial edges used to characterize the spatial relationship between each node, and the resolution edges used to characterize the resolution relationship between each node; a causal subgraph is extracted from the heterogeneous graph, and the causal subgraph includes features and The distribution of the environment subgraph is irrelevant, and the environment subgraph is the area in the heterogeneous graph except the causal subgraph; the indication information corresponding to the designated part is determined according to the causal subgraph, and the indication The information is used to characterize the status of the designated part and/or the target area in the designated part.
  • Example 2 provides the method of Example 1, wherein generating a heterogeneous image based on a plurality of image blocks corresponding to all the full-field histological images includes: Perform feature extraction on the image block to obtain the image features corresponding to the image block; use the image features corresponding to each image block and the resolution corresponding to the image block as a node to generate the node set; in the same A spatial edge is constructed between two nodes corresponding to any two adjacent image blocks in the full-field histological image, and two nodes corresponding to any two image blocks corresponding to the same area of the specified part and with adjacent resolutions are constructed. A resolution edge is constructed between them to generate the edge set; the heterogeneous graph is generated according to the node set and the edge set.
  • Example 3 provides the method of Example 1, wherein extracting the causal subgraph from the heterogeneous graph includes: inputting the heterogeneous graph into a pre-trained processing model. extractor to obtain the causal subgraph; determining the indication information corresponding to the designated part according to the causal subgraph includes: inputting the causal subgraph into an encoder in the processing model to obtain Causal image features used to characterize the causal subgraph; input the causal image features into a classifier in the processing model to obtain the state of the designated part; and/or, according to the causal subgraph, in The target area is determined in the full field histology image.
  • Example 4 provides the method of Example 3, which inputs the heterogeneous graph into an extractor in a pre-trained processing model to obtain the causal subgraph, including: The heterogeneous graph is input into the extractor, and the influence degree of each spatial edge and the influence degree of each resolution edge are determined; the influence degree of each spatial edge is sorted to determine the causal space edge, Sort the influence degree of each resolution edge to determine the causal resolution edge; according to the cause The causal subgraph is determined by using the causal space edge and the causal resolution edge.
  • Example 5 provides the method of Example 4, the extractor includes a graph neural network, a first multi-layer perceptron and a second multi-layer perceptron, and the heterogeneous
  • the graph is input into the extractor, and the influence degree of each spatial edge and the influence degree of each resolution edge are determined, including: inputting the heterogeneous graph into the graph neural network to obtain the heterogeneous graph.
  • Corresponding graph structure features input the graph structure features into the first multi-layer perceptron to obtain the influence degree of each spatial edge; input the graph structure features into the second multi-layer perceptron to obtain The degree of influence of each of the resolution edges.
  • Example 5 provides the method of Example 2,
  • Example 6 provides the method of Example 4, wherein determining the causal subgraph according to the causal space edge and the causal resolution edge includes: satisfying resolution causality.
  • the causal resolution edge and the causal space edge of the condition are used as a causal edge set, and the nodes connected by the causal edge set are used as a causal node set.
  • the resolution causal condition is: for each causal node, If the causal node is not a root node, there is a causal resolution edge belonging to the causal edge set connecting the causal node and the corresponding parent node.
  • the root node is the node with the lowest resolution, and the resolution of the parent node is is smaller than the resolution of the causal node; determine the causal subgraph according to the causal edge set and the causal node set.
  • Example 7 provides the method of Example 3.
  • the encoder includes: a resolution-based attention convolution layer and an iterative pooling layer; and inputting the causal subgraph
  • the encoder in the processing model to obtain causal image features used to characterize the causal subgraph includes: inputting the causal subgraph into the attention convolution layer, so that the attention convolution layer Based on each node in the causal subgraph and the resolution of the node with an edge between the node and the node, determine the characteristics of the node; input the structural characteristics corresponding to each node in the causal subgraph into the iteration pool layer, so that the iterative pooling layer pools the features of nodes with spatial edges between them to obtain the causal image features.
  • Example 8 provides the method of Example 3.
  • the processing model is trained in the following manner: obtaining a sample input set and the sample output set, where the sample input set includes multiple sample inputs, the sample inputs include training heterogeneous images corresponding to multiple full-field histological training images of a designated part, and the sample output set includes sample outputs corresponding to each of the sample inputs, and each of the samples
  • the output includes training instruction information corresponding to the training heterogeneous graph; for each sample input, input the training heterogeneous graph included in the sample input into the extractor to obtain a training causal subgraph; according to the Train the heterogeneous graph and the training causal subgraph, and determine the training environment subgraph; input the training causal subgraph and the training environment subgraph into the encoder to obtain the training causal subgraph used to characterize the training causal subgraph.
  • Training causal image features, and training environment image features used to characterize the training environment subgraph input the training causal image features into the classifier to obtain prediction indication information, and input the training environment subgraph into the Process the environment classifier in the model to obtain the predicted environment indication information; determine the environmental loss according to the predicted environment indication information and the training indication information corresponding to the sample input to train the environment classifier; according to the predicted indication Information and the training instruction information corresponding to the sample input determine causal losses to train the extractor, the encoder and the classifier.
  • Example 9 provides the method of Example 8, wherein determining a training environment subgraph according to the training heterogeneous graph and the training causal subgraph includes: converting the training heterogeneous graph The area in the composition except the training causal subgraph is used as a sample environment subgraph; the sample environment subgraph is stored in a sample environment subgraph set, and the sample environment subgraph set is used to store the input corresponding to each sample The sample environment subgraph of the sample environment subgraph; randomly select a sample environment subgraph from the sample environment subgraph set as the training environment subgraph.
  • Example 10 provides a device for processing full-field histological images, including: a segmentation module for segmenting multiple full-field histological images of a specified site to obtain each A plurality of image blocks corresponding to the full-field histological image, each of the full-field histological image corresponding to a different resolution; a generation module configured to generate the full-field histological image according to all of the full-field histological images.
  • a plurality of the image blocks corresponding to the weaving image are generated to generate a heterogeneous graph.
  • the heterogeneous graph includes a node set and an edge set.
  • the node set includes the image features corresponding to each of the image blocks and the corresponding image blocks.
  • the edge set includes spatial edges used to characterize the spatial relationship between nodes, and resolution edges used to characterize the resolution relationship between nodes;
  • the extraction module is used to extract from the different Extract the causal subgraph from the composition, the characteristics included in the causal subgraph are not related to the distribution of the environmental subgraph, and the environmental subgraph is the area in the heterogeneous graph except the causal subgraph;
  • process A module configured to determine indication information corresponding to the specified part according to the causal subgraph, where the indication information is used to characterize the state of the specified part and/or the target area in the specified part.
  • Example 11 provides a computer-readable medium having a computer program stored thereon, which implements the steps of the methods described in Examples 1 to 9 when executed by a processing device.
  • Example 12 provides an electronic device, including: a storage device having a computer program stored thereon; and a processing device configured to execute the computer program in the storage device, to Implement the steps of the methods described in Example 1 to Example 9.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Quality & Reliability (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Image Processing (AREA)

Abstract

一种全视野组织学图像的处理方法、装置、介质和电子设备，涉及图像处理技术领域，该方法包括：对指定部位的多个全视野组织学图像进行分割，得到每个全视野组织学图像对应的多个图像块，根据所有全视野组织学图像对应的多个图像块，生成异构图，异构图包括节点集和边集，节点集包括由每个图像块对应的图像特征和该图像块对应的分辨率组成的节点，边集包括用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边，从异构图中提取出因果子图，因果子图包括的特征与环境子图的分布不相关，根据因果子图确定指定部位对应的指示信息，指示信息用于表征指定部位的状态，和/或指定部位中的目标区域。

Description

全视野组织学图像的处理方法、装置、介质和电子设备
本申请要求于2022年9月6日提交中国专利局、申请号为202211086109.2、发明名称为“全视野组织学图像的处理方法、装置、介质和电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开涉及图像处理技术领域，具体地，涉及一种全视野组织学图像的处理方法、装置、介质和电子设备。
背景技术
随着人工智能相关技术的不断发展，深度神经网络在图像处理技术领域中得到了广泛的应用。通常情况下，深度神经网络能够处理的图像尺寸较小，例如255*255或者512*512。然而全视野组织学图像(英文：Histopathological Whole-slide Image，缩写：WSI)作为很多诊断的黄金标准，往往由于像素数量过大(例如80000*80000，甚至200000*2000000，占据100M到10G的存储空间)，而无法直接使用深度神经网络来处理。
发明内容
提供该发明内容部分以便以简要的形式介绍构思，这些构思将在后面的具体实施方式部分被详细描述。该发明内容部分并不旨在标识要求保护的技术方案的关键特征或必要特征，也不旨在用于限制所要求的保护的技术方案的范围。
第一方面，本公开提供一种全视野组织学图像的处理方法，所述方法包括：
对指定部位的多个全视野组织学图像进行分割，得到每个所述全视野组织学图像对应的多个图像块，每个所述全视野组织学图像对应的分辨率不同；
根据所有所述全视野组织学图像对应的多个所述图像块，生成异构图，所述异构图包括节点集和边集，所述节点集包括由每个所述图像块对应的图像特征和该图像块对应的分辨率组成的节点，所述边集包括用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边；
从所述异构图中提取出因果子图，所述因果子图包括的特征与环境子图的分布不相关，所述环境子图为所述异构图中除所述因果子图之外的区域；
根据所述因果子图确定所述指定部位对应的指示信息，所述指示信息用于表征所述指定部位的状态，和/或所述指定部位中的目标区域。
第二方面，本公开提供一种全视野组织学图像的处理装置，所述装置包括：
分割模块，用于对指定部位的多个全视野组织学图像进行分割，得到每个所述全视野组织学图像对应的多个图像块，每个所述全视野组织学图像对应的分辨率不同；
生成模块，用于根据所有所述全视野组织学图像对应的多个所述图像块，生成异构图，所述异构图包括节点集和边集，所述节点集包括由每个所述图像块对应的图像特征和该图像块对应的分辨率组成的节点，所述边集包括用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边；
提取模块，用于从所述异构图中提取出因果子图，所述因果子图包括的特征与环境子图的分布不相关，所述环境子图为所述异构图中除所述因果子 图之外的区域；
处理模块，用于根据所述因果子图确定所述指定部位对应的指示信息，所述指示信息用于表征所述指定部位的状态，和/或所述指定部位中的目标区域。
第三方面，本公开提供一种计算机可读介质，其上存储有计算机程序，该程序被处理装置执行时实现本公开第一方面所述方法的步骤。
第四方面，本公开提供一种电子设备，包括：
存储装置，其上存储有计算机程序；
处理装置，用于执行所述存储装置中的所述计算机程序，以实现本公开第一方面所述方法的步骤。
通过上述技术方案，本公开首先对指定部位的多个不同分辨率的全视野组织学图像进行分割，得到每个全视野组织学图像对应的多个图像块，之后根据所有全视野组织学图像对应的多个图像块，生成异构图，其中异构图包括节点集和边集，再从异构图中提取出与环境子图的分布不相关的因果子图，最后根据因果子图确定指定部位对应的指示信息，其中指示信息用于表征指定部位的状态，和/或指定部位中的目标区域。本公开通过构建能够表征图像特征、空间关系和分辨率关系的异构图，从中提取出满足分布不变性的因果子图，并以此确定指示信息，实现了本质上可解释的全视野组织学图像的识别。
本公开的其他特征和优点将在随后的具体实施方式部分予以详细说明。
附图说明
结合附图并参考以下具体实施方式，本公开各实施例的上述和其他特征、优点及方面将变得更加明显。贯穿附图中，相同或相似的附图标记表示相同或相似的元素。应当理解附图是示意性的，原件和元素不一定按照比例 绘制。在附图中：
图1是根据一示例性实施例示出的一种全视野组织学图像的处理方法的流程图；
图2是根据一示例性实施例示出的另一种全视野组织学图像的处理方法的流程图；
图3是根据一示例性实施例示出的一种处理模型的结构图；
图4是根据一示例性实施例示出的另一种全视野组织学图像的处理方法的流程图；
图5是根据一示例性实施例示出的另一种全视野组织学图像的处理方法的流程图；
图6是根据一示例性实施例示出的另一种处理模型的结构图；
图7是根据一示例性实施例示出的一种训练处理模型的流程图；
图8是根据一示例性实施例示出的另一种训练处理模型的流程图；
图9是根据一示例性实施例示出的一种全视野组织学图像的处理装置的框图；
图10是根据一示例性实施例示出的另一种全视野组织学图像的处理装置的框图；
图11是根据一示例性实施例示出的另一种全视野组织学图像的处理装置的框图；
图12是根据一示例性实施例示出的一种电子设备的框图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加 透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
应当理解，本公开的方法实施方式中记载的各个步骤可以按照不同的顺序执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
需要注意，本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
需要注意，本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有明确指出，否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的，而并不是用于对这些消息或信息的范围进行限制。
可以理解的是，在使用本公开各实施例公开的技术方案之前，均应当依据相关法律法规通过恰当的方式对本公开所涉及个人信息的类型、使用范围、使用场景等告知用户并获得用户的授权。
例如，在响应于接收到用户的主动请求时，向用户发送提示信息，以明确地提示用户，其请求执行的操作将需要获取和使用到用户的个人信息。从而，使得用户可以根据提示信息来自主地选择是否向执行本公开技术方案的操作的电子设备、应用程序、服务器或存储介质等软件或硬件提供个人信息。
作为一种可选的但非限定性的实现方式，响应于接收到用户的主动请求，向用户发送提示信息的方式例如可以是弹窗的方式，弹窗中可以以文字的方式呈现提示信息。此外，弹窗中还可以承载供用户选择“同意”或者“不同意”向电子设备提供个人信息的选择控件。
可以理解的是，上述通知和获取用户授权过程仅是示意性的，不对本公开的实现方式构成限定，其它满足相关法律法规的方式也可应用于本公开的实现方式中。
同时，可以理解的是，本技术方案所涉及的数据(包括但不限于数据本身、数据的获取或使用)应当遵循相应法律法规及相关规定的要求。
图1是根据一示例性实施例示出的一种全视野组织学图像的处理方法的流程图，如图1所示，该方法包括：
步骤101，对指定部位的多个全视野组织学图像进行分割，得到每个全视野组织学图像对应的多个图像块，每个全视野组织学图像对应的分辨率不同。
举例来说，当专业人员需要对指定部位进行诊断时，可以利用全视野组织学图像扫描仪对指定部位进行扫描，以获得反映指定部位的全视野组织学图像。指定部位可以是人体的某个器官，例如：肝脏、心脏、胃等，也可以是器官的部分区域，本公开对此不做具体限定。之后可以对全视野组织学图像进行放大或缩小操作，得到多个分辨率不同的全视野组织学图像。例如，可以得到3个全视野组织学图像，分辨率分别为Thumbnail、5X和10X。可以按照预设规则对每个全视野组织学图像进行分割，得到每个全视野组织学图像对应的多个图像块(英文：patch)，例如可以按照sliding-windows的方式将每个全视野组织学图像分割为多个512*512像素大小的图像块。为了降低待处理的数据量，同时减少干扰，在对全视野组织学图像进行分割之前，可以先过滤掉全视野组织学图像中的背景部分。具体的，可以将RGB值的 方差小于预设阈值的区域确定为背景部分，然后删除全视野组织学图像中的背景部分，再做分割，得到的多个图像块中不会包含背景部分，避免了背景部分对识别的干扰。
步骤102，根据所有全视野组织学图像对应的多个图像块，生成异构图，异构图包括节点集和边集，节点集包括由每个图像块对应的图像特征和该图像块对应的分辨率组成的节点，边集包括用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边。
示例的，可以根据全部的全视野组织学图像对应的多个图像块，以及各个图像块之间的空间关系、分辨率关系，生成异构图。其中，异构图包括节点集和边集，异构图可以表示为G，节点集表示为V，边集表示为E，G＝(V,E)。节点集包括多个节点，节点与图像块一一对应，即每个节点对应一个图像块，可以将该图像块对应的图像特征以及分辨率组成一个节点。图像特征可以理解为用于表征图像块的特征向量或者特征图(英文：Feature Map)。例如，图像特征为一个1024维度的特征向量，那么可以将图像特征和对应的分辨率组合成一个1025维度的特征向量作为节点。
边集包括多个边，边可以是用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边，其中，空间关系用于说明任意两个图像块在同一全视野组织学图像中的关系，若两个图像块在同一全视野组织学图像中，且两个图像块相邻(即空间上相邻)，那么表明这两个图像块之间具有空间关系，这两个图像块对应的节点之间具有一个空间边。分辨率关系用于说明任意两个图像块在不同两个全视野组织学图像中的关系，若两个图像块对应指定部位的同一区域，且两个图像块分辨率相邻，那么表明这两个图像块之间具有分辨率关系，这两个图像块对应的节点之间具有一个分辨率边。分辨率相邻可以理解为对多个全视野组织学图像的分辨率进行排序，相邻的分辨率即为分辨率相邻，例如共有Thumbnail、5X和10X三种 分辨率，那么Thumbnail和5X为分辨率相邻，5X和10X为分辨率相邻。异构图能够有效表征多个全视野组织学图像的图像特征、空间关系和分辨率关系。
步骤103，从异构图中提取出因果子图，因果子图包括的特征与环境子图的分布不相关，环境子图为异构图中除因果子图之外的区域。
步骤104，根据因果子图确定指定部位对应的指示信息，指示信息用于表征指定部位的状态，和/或指定部位中的目标区域。
示例的，在得到异构图之后，可以对异构图进行学习，以从中抽取出因果子图和环境子图。其中，环境子图为异构图中除因果子图之外的区域。因果子图包括的特征与环境子图的分布不相关，即因果子图包括的特征不会受到环境子图的分布的影响，也可以理解为因果子图具有分布不变性。也就是说，因果子图包括的特征和环境子图包括的特征具有本质上的区别。例如，猫的胡须是猫的一个特征，这个特征不会收到环境的影响，即不论猫在草坪上、沙发上或者地板上，胡须这个特征不变。
在得到因果子图之后，可以利用因果子图来确定指定部位对应的指示信息，其中，指示信息可以包括用于表征指定部位的状态的信息，还可以包括用于表征指定部位中的目标区域的信息，指示信息也可以同时包括用于表征指定部位的状态的信息，和用于表征指定部位中的目标区域的信息。指定部位的状态例如可以包括：良好、病变两种状态，也可以包括：初级、中级、高级三种状态，指定部位的状态也可以用于描述5年生存率，指定部位的状态还可以用于描述预后情况，本公开对此不做具体限定。指定部位中的目标区域可以理解为全视野组织学图像中需要关注的区域，能够直观、有效地帮助专业人员对指定部位进行判断。由于指示信息是直接根据因果子图确定的，因此目标区域也是根据因果子图确定，也就是说，目标区域是因果子图对应在全视野组织学图像中的区域。
这样，通过对多个全视野组织学图像的分割，生成了能够有效表征图像特征、空间关系和分辨率关系的异构图，并从中提取出具有分布不变性的因果子图。最后直接根据因果子图来确定指示信息，也就是说指示信息完全根据因果子图确定，即指示信息与因果子图之间具有直接的因果关系。相比于其他基于热力图、注意力图等提供外在解释性的方案，仅能说明预测结果可能与热力图、注意力图相关，而本公开中的指示信息与因果子图之间具有直接的因果关系，因果子图可以为指示信息提供本质上的解释性，实现了本质上可解释的全视野组织学图像的识别。
图2是根据一示例性实施例示出的另一种全视野组织学图像的处理方法的流程图，如图2所示，步骤102的实现方式可以包括：
步骤1021，对每个图像块进行特征提取，得到该图像块对应的图像特征。
步骤1022，将每个图像块对应的图像特征和该图像块对应的分辨率作为一个节点，生成节点集。
示例的，可以对每个图像块进行特征提取，得到能够表征该图像块的图像特征。具体的，可以利用ResNet网络、KimiaNet网络或者Transformer中的Encoder对图像块进行特征提取，本公开对此不做具体限定。之后，可以将每个图像块对应的图像特征和该图像块对应的分辨率进行拼接(英文：Concat)，作为该图像块对应的节点，从而得到由所有图像块对应的节点组成的节点集V。
步骤1023，在同一全视野组织学图像中任意两个相邻的图像块对应的两个节点之间构建一个空间边，在任意两个对应指定部位同一区域且分辨率相邻的图像块对应的两个节点之间构建一个分辨率边，生成边集。
步骤1024，根据节点集和边集生成异构图。
示例的，根据任意两个图像块之间的空间关系和分辨率关系，构建边集。具体的，针对同一全视野组织学图像中任意两个相邻的图像块，在这两个图 像块对应的节点之间构建一个空间边，也就是说空间边连接的两个节点对应的两个图像块对应的分辨率相同，且空间上相邻。多个空间边可以表示为一个空间相邻矩阵(表示为Aspacial)，图像块的数量为|V|，那么Aspacial的大小为|V|*|V|，其中同一全视野组织学图像中两个相邻的图像块对应的元素为1，其他元素为0。针对对应指定部位同一区域且分辨率相邻的任意两个图像块，在这两个图像块对应的节点之间构建一个分辨率边，也就是说分辨率边连接的两个节点对应的两个图像块对应在指定部位的区域相同，且分辨率上相邻。多个分辨率边可以表示为一个分辨率相邻矩阵(表示为Ascale)，图像块的数量为|V|，那么Ascale的大小为|V|*|V|，其中对应指定部位同一区域且分辨率相邻的两个图像块对应的元素为1，其他元素为0。这样得到了由全部空间边和分辨率边组成的边集E。最后可以将节点集V和边集E生成异构图G，即G＝(V,E)。
在一种应用场景中，可以预先训练一个处理模型来实现本公开提供的全视野组织学图像的处理方法，处理模型的结构如图3所示，可以包括：提取器(表示为Rational Extractor)、编码器(表示为Heterogeous Encoder)和分类器(表示为Classifier)，其中，提取器、编码器和分类器的连接关系为：提取器的输入作为处理模型的输入，提取器的输出，作为编码器和的输入，编码器的输出，作为分类器的输入，分类器的输出作为处理模型的输出。
图4是根据一示例性实施例示出的另一种全视野组织学图像的处理方法的流程图，如图4所示，步骤103的实现方式可以为：
将异构图输入预先训练的处理模型中的提取器，以得到因果子图。
步骤104的实现方式可以包括：
步骤1041，将因果子图输入处理模型中的编码器，以得到用于表征因果子图的因果图像特征。
步骤1042，将因果图像特征输入处理模型中的分类器，以得到指定部位 的状态。和/或，根据因果子图，在全视野组织学图像中确定目标区域。
举例来说，可以将步骤102中得到的因果子图输入提取器，例如可以将节点集、空间相邻矩阵和分辨率相邻矩阵输入提取器，提取器可以从中提取出因果子图。具体的，提取器可以先从异构图中学习每个空间边对于确定指定部位的状态的贡献(即后文所提及的影响度)，以及每个分辨率边的对于确定指定部位的状态的贡献，然后根据贡献最大的指定比例(例如20％)的空间边和分辨率边生成因果子图。
之后可以将因果子图输入处理模型中的编码器，以得到用于表征因果子图的因果图像特征，因果图像特征可以是特征向量或者特征图。编码器能够提取异构图的特征，例如可以是GNN(英文：Graph Neural Network，中文：图神经网络)，也可以是HGAT(英文：Heterogeneous Graph Attention Networks，中文：异构图注意力网络)，本公开对此不做具体限定。最后，若指示信息包括用于表征指定部位的状态的信息，可以将因果图像特征输入分类器，以得到指定部位的状态。若指示信息包括用于表征指定部位中的目标区域的信息，那么可以根据因果子图，在全视野组织学图像中确定目标区域。也就是说，目标区域是因果子图对应在全视野组织学图像中的区域。
图5是根据一示例性实施例示出的另一种全视野组织学图像的处理方法的流程图，如图5所示，步骤103可以通过以下步骤来实现：
步骤1031，将异构图输入提取器，确定每个空间边的影响度和每个分辨率边的影响度。
步骤1032，对每个空间边的影响度进行排序确定因果空间边，对每个分辨率边的影响度进行排序确定因果分辨率边。
步骤1033，根据因果空间边和因果分辨率边，确定因果子图。
示例的，可以将异构图输入提取器，利用提取器来学习异构图中每个空间边的影响度和每个分辨率边的影响度。具体的，提取器可以包括图神经网 络、第一多层感知机和第二多层感知机。
首先可以将异构图输入图神经网络，由图神经网络提取异构图对应的图结构特征。图结构特征可以理解为包含图结构信息的节点表示，也就是说，图结构特征中包括了每个节点对应的图结构表示。例如，以Z来表示图结构特征，那么Z＝GNN(G)，其中GNN表示图神经网络的处理。
之后可以分别将图结构特征输入第一多层感知机和第二多层感知机，得到第一多层感知机输出的每个空间边的影响度，和第二多层感知机输出的每个分辨率边的影响度。每个边(包括空间边和分辨率边)的影响度可以理解为该边对于确定指定部位的状态的贡献大小，也可以理解为该边的重要程度。可以分别训练两个多层感知机(英文：Multi-Layer Perception，缩写：MLP)，再利用sigmoid函数来确定每个空间边的影响度和每个分辨率边的影响度。具体过程可以表示为：
Zspacial＝MLPspacial(Z)
Zscale＝MLPscale(Z)
Mspacial＝σ(ZT spacial,Zspacial)
Mscale＝σ(ZT scale,Zscale)
其中，MLPspacial表示第一多层感知机，MLPscale表示第二多层感知机，σ表示sigmoid函数，Zspacial表示第一多层感知机的输出，Zscale表示第二多层感知机的输出，Mspacial表示每个空间边的影响度，Mscale表示每个分辨率边的影响度。可以将Mspacial理解为与空间相邻矩阵中的元素一一对应的矩阵，其中每个元素用于表示对应的空间边的影响度，同样的，可以将Mscale理解与分辨率相邻矩阵中的元素一一对应的矩阵，其中每个元素用于表示对应的分辨率边的影响度。
然后可以对每个空间边的影响度进行排序，从而确定构建因果子图的因果空间边，同时对每个分辨率边的影响度进行排序，从而确定构建因果子图 的因果分辨率边。具体的，可以对每个空间边的影响度进行降序排列，选取排在最前面的20％的空间边作为因果空间边，同时对每个分辨率边的影响度进行降序排列，选取排在最前面的20％的分辨率边作为因果分辨率边。具体过程可以表示为：
Ec_spa＝Topr1(Mspacial⊙Aspacial)
Ec_sca＝Topr2(Mscale⊙Ascale)
Ec＝Ec_spa∪Ec_sca
其中，Ec_spa表示因果空间边，Ec_sca表示因果分辨率边，Topr1表示选择最大的r1个边，Topr2表示选择最大的r2个边，⊙表示element-wise multiplication，Ec表示因果空间边和因果分辨率边的并集。
最后，可以根据因果空间边和因果分辨率边，确定因果子图。具体的，因果子图可以表示为：
Vc＝{u∈V:u∈Ec}
Vs＝{u∈V:u∈E-Ec}
C＝(Vc,Ec)
E＝(Vs,E-Ec)
其中，u表示节点，Vc表示Ec连接的节点的集合，Vs表示除Vc以外的节点的集合。C表示因果子图，E表示环境子图，即异构图中除因果子图之外的区域。
为了进一步保证因果子图的合理性，步骤1033可以通过以下步骤来实现：
步骤4)将满足分辨率因果条件的因果分辨率边和因果空间边，作为因果边集，将因果边集连接的节点作为因果节点集，分辨率因果条件为：针对每个因果节点，若该因果节点不为根节点，存在一个属于因果边集的因果分辨率边连接该因果节点与对应的母节点，根节点为分辨率最低的节点，母节 点的分辨率小于该因果节点的分辨率。
步骤5)根据因果边集和因果节点集确定因果子图。
示例的，可以利用预设的分辨率因果条件对因果分辨率边和因果空间边进行筛选，得到因果边集，然后将因果边集中包括的每个边(包括因果分辨率边和因果空间边)所连接的节点作为因果节点，得到包括全部因果节点的因果节点集。最后根据因果边集和因果节点集生成因果子图。具体的，分辨率因果条件为：针对每个因果节点，若该因果节点不为根节点，存在一个属于因果边集中的因果分辨率边将该因果节点与母节点相连，根节点为辨率最低的节点，母节点的分辨率小于该因果节点的分辨率。例如，某个因果节点的分辨率为10X，那么因果边集中应该包括一个因果分辨率边连接该因果节点和一个分辨率为5X的因果节点。再比如，某个因果节点的分辨率为5X，那么因果边集中应该包括一个因果分辨率边连接该因果节点和一个分辨率为Thumbnail的因果节点。若某个因果节点不满足分辨率因果条件，那么删除该因果节点和所有与该因果节点相连的边。
具体的，可以表示为：
E’c＝ScaleClean(Ec)
相应的：
V’c＝{u∈V:u∈E’c}
V’s＝{u∈V:u∈E-E’c}
C＝(V’c,E’c)
E＝(V’s,E-E’c)
其中，ScaleClean表示分辨率因果条件，E’c表示因果边集，V’c表示因果点集，V’s表示除V’c以外的节点的集合。
分辨率因果条件可以理解为，由于分辨率边连接的两个节点对应指定部位的同一区域(即代表同一部分细胞)，如果两个节点中分辨率低的节点不 属于因果子图，那么相应分辨率高的节点也不应当属于因果子图。这样可以避免把同一个细胞的特征同时划分为因果子图和环境子图。需要注意的是，若分辨率高的节点不属于因果子图，不会影响到分辨率低的节点。例如，分辨率为5X的一个节点可以表征一个细胞，对应的分辨率为10X的节点可以表征细胞里的一部分，可以把细胞当成因果子图的同时排除细胞中一些正常的区域。
在另一种应用场景中，编码器可以包括：基于分辨率的注意力卷积层和迭代池化层。基于分辨率的注意力卷积层的输入作为编码器的输入，基于分辨率的注意力卷积层的输出作为迭代池化层的输入，迭代池化层的输出作为编码器的输出。进一步的，基于分辨率的注意力卷积层和迭代池化层可以作为一个单元，编码器中可以包括多个(例如2个)依次相连的单元。步骤1041可以包括：
步骤6)将因果子图输入注意力卷积层，以使注意力卷积层基于因果子图中每个节点，和与该节点之间具有边的节点的分辨率，确定该节点的特征。
步骤7)将因果子图中每个节点对应的结构特征输入迭代池化层，以使迭代池化层对之间具有空间边的节点的特征进行池化，得到因果图像特征。
举例来说，将因果子图输入基于分辨率的注意力卷积层，注意力卷积层根据因果子图中每个节点，和与该节点之间具有边的节点的分辨率，来确定该节点的特征。传统的注意力卷积层，在计算每个边的注意力分数时计算方式是相同，而基于分辨率的注意力卷积层，在计算每个边的注意力分数时，会针对空间边和分辨率边采用不同的计算方式，以使处理模型能够区分空间边和分辨率边。具体的，基于分辨率的注意力卷积层计算每个边的注意力分数的方式可以是：


αvv'＝αr·αv'
其中，αvv'表示节点v和节点v’之间的边的注意力分数，αr表示分辨率分数，αv'表示初始注意力分数，β表示激活函数，例如可以是ReLu，表示针对分辨率为r的节点的可学习的注意力层，hv表示节点v的特征(即embedding)，hr表示分辨率为r的所有节点的特征的均值，hr'表示分辨率为r’的所有节点的特征的均值，R表示分辨率集合，hv'表示节点v’的特征，||表示拼接(即Concat)，Nr表示分辨率为r且与节点v相邻的节点的集合，v”表示任一分辨率为r且与节点v相邻的节点，hv”表示v”的特征，VT表示针对节点v的全部邻居节点的可学习的注意力层。
之后，可以将因果子图中每个节点对应的结构特征输入迭代池化层，以使迭代池化层对之间具有空间边的节点的特征进行池化，得到因果图像特征。也就是说，聚合具有语义特征相似和空间分布相似的节点的同时保留异构图的结构，从而减少计算负担。
需要说明的是，上文提及的提取器中的图神经网络的结构可以和编码器的结构相同，即图神经网络包括了基于分辨率的注意力卷积层和迭代池化层，基于分辨率的注意力卷积层的输入作为图神经网络的输入，基于分辨率的注意力卷积层的输出作为迭代池化层的输入，迭代池化层的输出作为图神经网络的输出。进一步的，基于分辨率的注意力卷积层和迭代池化层可以作为一个单元，图神经网络中可以包括多个(例如2个)依次相连的单元。
在对处理模型进行训练时，处理模型的结构在提取器、编码器和分类器 的基础上，还可以包括采样器(表示为Sampler)和环境分类器(表示为Environment Classifier)，如图6所示。
图7是根据一示例性实施例示出的一种训练处理模型的流程图，如图7所示，处理模型是通过以下方式训练得到的：
步骤A，获取样本输入集和样本输出集，样本输入集包括多个样本输入，样本输入包括指定部位的多个全视野组织学训练图像对应的训练异构图，样本输出集中包括与每个样本输入对应的样本输出，每个样本输出包括训练异构图对应的训练指示信息。
举例来说，在对处理模型进行训练之前，需要先获取用于训练的样本输入集和样本输出集，样本输入集包括了多个样本输入，样本输出集中包括与每个样本输入对应的样本输出。样本输入可以包括指定部位的多个全视野组织学训练图像对应的训练异构图，多个全视野组织学训练图像对应的分辨率均不相同。训练异构图的构建方式与步骤102相同，此次不再赘述。相应的，每个样本输出包括训练异构图对应的训练指示信息，其中，训练指示信息用于表征指定部位的状态。
步骤B，针对每个样本输入，将该样本输入包括的训练异构图输入提取器，以得到训练因果子图。
步骤C，根据训练异构图和训练因果子图，确定训练环境子图。
示例的，可以对训练异构图进行学习，以从中抽取出训练因果子图和训练环境子图。具体的，可以将训练异构图输入提取器，以使提取器从训练异构图中学习每个空间边对于确定指定部位的状态的贡献，以及每个分辨率边的对于确定指定部位的状态的贡献，然后根据贡献最大的指定比例的空间边和分辨率边生成训练因果子图。然后可以将训练异构图中除训练因果子图之外的区域作为训练环境子图。也可以将之前每次训练得到的训练环境子图存储在缓存中，然后从中随机抽取一个作为本次训练的训练环境子图。
步骤D，将训练因果子图和训练环境子图输入编码器，以得到用于表征训练因果子图的训练因果图像特征，和用于表征训练环境子图的训练环境图像特征。
步骤E，将训练因果图像特征输入分类器，以得到预测指示信息，将训练环境图像特征输入处理模型中的环境分类器，以得到预测环境指示信息。
步骤F，根据预测环境指示信息和该样本输入对应的训练指示信息确定环境损失，以训练环境分类器。
步骤G，根据预测指示信息和该样本输入对应的训练指示信息确定因果损失，以训练提取器、编码器和分类器。
示例的，将训练因果子图和训练环境子图输入编码器，得到用于表征训练因果子图的训练因果图像特征，和用于表征训练环境子图的训练环境图像特征。再将训练因果图像特征输入分类器，以得到预测指示信息(表示为y’c)，将训练环境图像特征输入处理模型中的环境分类器，以得到预测环境指示信息(表示为y’s)。
最后，可以根据预测环境指示信息和该样本输入对应的训练指示信息确定环境损失，并以降低环境损失为目标，利用反向传播算法来训练环境分类器中的神经元的参数，神经元的参数例如可以是神经元的权重(英文：Weight)和偏置量(英文：Bias)。环境损失例如可以是l(y’s,y)，l表示交叉熵损失，y表示该样本输入对应的训练指示信息。也就是说，环境损失仅用于训练环境分类器，而不会影响到提取器、编码器和分类器的训练。同时，根据预测指示信息和该样本输入对应的训练指示信息确定因果损失，并以降低因果损失为目标，利用反向传播算法来训练提取器、编码器和分类器中的神经元的参数。因果损失例如可以是l(y’c,y)。也就是说，因果损失用于训练提取器、编码器和分类器，而不会影响到环境分类器的训练。
图8是根据一示例性实施例示出的另一种训练处理模型的流程图，如图 8所示，步骤C可以包括：
步骤C1，将训练异构图中除训练因果子图的区域作为样本环境子图。
步骤C2，将样本环境子图存入样本环境子图集，样本环境子图集用于存储每个样本输入对应的样本环境子图。
步骤C3，从样本环境子图集中随机抽取一个样本环境子图作为训练环境子图。
示例的，为了避免训练环境子图对处理模型训练过程的干扰，在利用每个样本输入进行训练时，可以先将训练异构图中除训练因果子图的区域作为样本环境子图，然后将样本环境子图存入样本环境子图集，可以理解为，样本环境子图集中存储了整个训练过程中获得的样本环境子图，也就是说样本环境子图集反映的是样本输入集中的样本环境子图的分布。环境可能会影响到最终的识别结果。例如目标区域为A类细胞所在的区域，B类细胞、C类细胞和D类细胞所在的区域均为环境，若训练过程中某个批次中90％的训练异构图只包括A类细胞和B类细胞，那么处理模型可能根据B类细胞来判断是否存在A类细胞，并不能学习到本质的因果关系(即根据A类细胞来判断是否存在A类细胞)。通过建立样本环境子图集，其中将会包括B类细胞、C类细胞和D类细胞，使得处理模型可以学习到多种环境。最后，利用采样器从样本环境子图集中随机抽取一个样本环境子图作为训练环境子图，能够避免处理模型根据训练环境子图来确定预测指示信息，从而提高识别的准确度，保证本质上的解释性。
综上所述，本公开首先对指定部位的多个不同分辨率的全视野组织学图像进行分割，得到每个全视野组织学图像对应的多个图像块，之后根据所有全视野组织学图像对应的多个图像块，生成异构图，其中异构图包括节点集和边集，再从异构图中提取出与环境子图的分布不相关的因果子图，最后根据因果子图确定指定部位对应的指示信息，其中指示信息用于表征指定部位 的状态，和/或指定部位中的目标区域。本公开通过构建能够表征图像特征、空间关系和分辨率关系的异构图，从中提取出满足分布不变性的因果子图，并以此确定指示信息，实现了本质上可解释的全视野组织学图像的识别。
图9是根据一示例性实施例示出的一种全视野组织学图像的处理装置的框图，如图9所示，该装置200包括：
分割模块201，用于对指定部位的多个全视野组织学图像进行分割，得到每个全视野组织学图像对应的多个图像块，每个全视野组织学图像对应的分辨率不同。
生成模块202，用于根据所有全视野组织学图像对应的多个图像块，生成异构图，异构图包括节点集和边集，节点集包括由每个图像块对应的图像特征和该图像块对应的分辨率组成的节点，边集包括用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边。
提取模块203，用于从异构图中提取出因果子图，因果子图包括的特征与环境子图的分布不相关，环境子图为异构图中除因果子图之外的区域。
处理模块204，用于根据因果子图确定指定部位对应的指示信息，指示信息用于表征指定部位的状态，和/或指定部位中的目标区域。
图10是根据一示例性实施例示出的另一种全视野组织学图像的处理装置的框图，如图10所示，生成模块202可以包括：
提取子模块2021，用于对每个图像块进行特征提取，得到该图像块对应的图像特征。
第一生成子模块2022，用于将每个图像块对应的图像特征和该图像块对应的分辨率作为一个节点，生成节点集。
第二生成子模块2023，用于在同一全视野组织学图像中任意两个相邻的图像块对应的两个节点之间构建一个空间边，在任意两个对应指定部位同一区域且分辨率相邻的图像块对应的两个节点之间构建一个分辨率边，生成边 集。
第三生成子模块2024，用于根据节点集和边集生成异构图。
在一种实现方式中，提取模块203可以用于：
将异构图输入预先训练的处理模型中的提取器，以得到因果子图。
处理模块204可以用于：
将因果子图输入处理模型中的编码器，以得到用于表征因果子图的因果图像特征。
将因果图像特征输入处理模型中的分类器，以得到指定部位的状态。和/或，根据因果子图，在全视野组织学图像中确定目标区域。
图11是根据一示例性实施例示出的另一种全视野组织学图像的处理装置的框图，如图11所示，提取模块203可以包括：
影响度提取子模块2031，用于将异构图输入提取器，确定每个空间边的影响度和每个分辨率边的影响度。
排序子模块2032，用于对每个空间边的影响度进行排序确定因果空间边，对每个分辨率边的影响度进行排序确定因果分辨率边。
确定子模块2033，用于根据因果空间边和因果分辨率边，确定因果子图。
在一种应用场景中，提取器可以包括图神经网络、第一多层感知机和第二多层感知机。相应的，提取子模块2031可以用于执行以下步骤：
步骤1)将异构图输入图神经网络，得到异构图对应的图结构特征。
步骤2)将图结构特征输入第一多层感知机，得到每个空间边的影响度。
步骤3)将图结构特征输入第二多层感知机，得到每个分辨率边的影响度。
在另一种应用场景中，确定子模块2033可以用于执行以下步骤：
步骤4)将满足分辨率因果条件的因果分辨率边和因果空间边，作为因果边集，将因果边集连接的节点作为因果节点集，分辨率因果条件为：针对 每个因果节点，若该因果节点不为根节点，存在一个属于因果边集的因果分辨率边连接该因果节点与对应的母节点，根节点为分辨率最低的节点，母节点的分辨率小于该因果节点的分辨率。
步骤5)根据因果边集和因果节点集确定因果子图。
在另一种应用场景中，编码器可以包括：基于分辨率的注意力卷积层和迭代池化层。编码子模块2041可以用于执行以下步骤：
步骤6)将因果子图输入注意力卷积层，以使注意力卷积层基于因果子图中每个节点，和与该节点之间具有边的节点的分辨率，确定该节点的特征。
步骤7)将因果子图中每个节点对应的结构特征输入迭代池化层，以使迭代池化层对之间具有空间边的节点的特征进行池化，得到因果图像特征。
在一种实现方式中，处理模型是通过以下方式训练得到的：
步骤A，获取样本输入集和样本输出集，样本输入集包括多个样本输入，样本输入包括指定部位的多个全视野组织学训练图像对应的训练异构图，样本输出集中包括与每个样本输入对应的样本输出，每个样本输出包括训练异构图对应的训练指示信息。
步骤B，针对每个样本输入，将该样本输入包括的训练异构图输入提取器，以得到训练因果子图。
步骤C，根据训练异构图和训练因果子图，确定训练环境子图。
步骤D，将训练因果子图和训练环境子图输入编码器，以得到用于表征训练因果子图的训练因果图像特征，和用于表征训练环境子图的训练环境图像特征。
步骤E，将训练因果图像特征输入分类器，以得到预测指示信息，将训练环境图像特征输入处理模型中的环境分类器，以得到预测环境指示信息。
步骤F，根据预测环境指示信息和该样本输入对应的训练指示信息确定环境损失，以训练环境分类器。
步骤G，根据预测指示信息和该样本输入对应的训练指示信息确定因果损失，以训练提取器、编码器和分类器。
在另一种实现方式中，步骤C可以包括：
步骤C1，将训练异构图中除训练因果子图的区域作为样本环境子图。
步骤C2，将样本环境子图存入样本环境子图集，样本环境子图集用于存储每个样本输入对应的样本环境子图。
步骤C3，从样本环境子图集中随机抽取一个样本环境子图作为训练环境子图。
关于上述实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。
综上所述，本公开首先对指定部位的多个不同分辨率的全视野组织学图像进行分割，得到每个全视野组织学图像对应的多个图像块，之后根据所有全视野组织学图像对应的多个图像块，生成异构图，其中异构图包括节点集和边集，再从异构图中提取出与环境子图的分布不相关的因果子图，最后根据因果子图确定指定部位对应的指示信息，其中指示信息用于表征指定部位的状态，和/或指定部位中的目标区域。本公开通过构建能够表征图像特征、空间关系和分辨率关系的异构图，从中提取出满足分布不变性的因果子图，并以此确定指示信息，实现了本质上可解释的全视野组织学图像的识别。
下面参考图12，其示出了适于用来实现本公开实施例的电子设备(例如本公开实施例的执行主体)300的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图12示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图12所示，电子设备300可以包括处理装置(例如中央处理器、图形处理器等)301，其可以根据存储在只读存储器(ROM)302中的程序或者从存储装置308加载到随机访问存储器(RAM)303中的程序而执行各种适当的动作和处理。在RAM 303中，还存储有电子设备300操作所需的各种程序和数据。处理装置301、ROM 302以及RAM 303通过总线304彼此相连。输入/输出(I/O)接口305也连接至总线304。
通常，以下装置可以连接至I/O接口305：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置306；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置307；包括例如磁带、硬盘等的存储装置308；以及通信装置309。通信装置309可以允许电子设备300与其他设备进行无线或有线通信以交换数据。虽然图12示出了具有各种装置的电子设备300，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置309从网络上被下载和安装，或者从存储装置308被安装，或者从ROM 302被安装。在该计算机程序被处理装置301执行时，执行本公开实施例的方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、 硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。
在一些实施方式中，终端设备、服务器可以利用诸如HTTP(HyperText Transfer Protocol，超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)，广域网(“WAN”)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：对指定部位的多个全视野组织学图像进行分割，得到每个所述全视野组织学图像对应的多个图像块，每个所述全视野组织学图像对应的分辨率不同；根据所有所述全视野组织学图像对 应的多个所述图像块，生成异构图，所述异构图包括节点集和边集，所述节点集包括由每个所述图像块对应的图像特征和该图像块对应的分辨率组成的节点，所述边集包括用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边；从所述异构图中提取出因果子图，所述因果子图包括的特征与环境子图的分布不相关，所述环境子图为所述异构图中除所述因果子图之外的区域；根据所述因果子图确定所述指定部位对应的指示信息，所述指示信息用于表征所述指定部位的状态，和/或所述指定部位中的目标区域。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言——诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)——连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程 图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的模块可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，模块的名称在某种情况下并不构成对该模块本身的限定，例如，分割模块还可以被描述为“对全视野组织学图像进行分割的模块”。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上系统(SOC)、复杂可编程逻辑设备(CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的一个或多个实施例，示例1提供了一种全视野组织学图像的处理方法，包括：对指定部位的多个全视野组织学图像进行分割，得到每个所述全视野组织学图像对应的多个图像块，每个所述全视野组织学图像对应的分辨率不同；根据所有所述全视野组织学图像对应的多个所述图像块，生成异构图，所述异构图包括节点集和边集，所述节点集包括由每个所述图像块对应的图像特征和该图像块对应的分辨率组成的节点，所述边集包括用 于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边；从所述异构图中提取出因果子图，所述因果子图包括的特征与环境子图的分布不相关，所述环境子图为所述异构图中除所述因果子图之外的区域；根据所述因果子图确定所述指定部位对应的指示信息，所述指示信息用于表征所述指定部位的状态，和/或所述指定部位中的目标区域。
根据本公开的一个或多个实施例，示例2提供了示例1的方法，所述根据所有所述全视野组织学图像对应的多个所述图像块，生成异构图，包括：对每个所述图像块进行特征提取，得到该图像块对应的图像特征；将每个所述图像块对应的图像特征和该图像块对应的分辨率作为一个节点，生成所述节点集；在同一所述全视野组织学图像中任意两个相邻的图像块对应的两个节点之间构建一个空间边，在任意两个对应所述指定部位同一区域且分辨率相邻的图像块对应的两个节点之间构建一个分辨率边，生成所述边集；根据所述节点集和所述边集生成所述异构图。
根据本公开的一个或多个实施例，示例3提供了示例1的方法，所述从所述异构图中提取出因果子图，包括：将所述异构图输入预先训练的处理模型中的提取器，以得到所述因果子图；所述根据所述因果子图确定所述指定部位对应的指示信息，包括：将所述因果子图输入所述处理模型中的编码器，以得到用于表征所述因果子图的因果图像特征；将所述因果图像特征输入所述处理模型中的分类器，以得到所述指定部位的状态；和/或，根据所述因果子图，在所述全视野组织学图像中确定所述目标区域。
根据本公开的一个或多个实施例，示例4提供了示例3的方法，所述将所述异构图输入预先训练的处理模型中的提取器，以得到所述因果子图，包括：将所述异构图输入所述提取器，确定每个所述空间边的影响度和每个所述分辨率边的影响度；对每个所述空间边的影响度进行排序确定因果空间边，对每个所述分辨率边的影响度进行排序确定因果分辨率边；根据所述因 果空间边和所述因果分辨率边，确定所述因果子图。
根据本公开的一个或多个实施例，示例5提供了示例4的方法，所述提取器包括图神经网络、第一多层感知机和第二多层感知机，所述将所述异构图输入所述提取器，确定每个所述空间边的影响度和每个所述分辨率边的影响度，包括：将所述异构图输入所述图神经网络，得到所述异构图对应的图结构特征；将所述图结构特征输入所述第一多层感知机，得到每个所述空间边的影响度；将所述图结构特征输入所述第二多层感知机，得到每个所述分辨率边的影响度。根据本公开的一个或多个实施例，示例5提供了示例2的方法，
根据本公开的一个或多个实施例，示例6提供了示例4的方法，所述根据所述因果空间边和所述因果分辨率边，确定所述因果子图，包括：将满足分辨率因果条件的所述因果分辨率边和所述因果空间边，作为因果边集，将所述因果边集连接的节点作为因果节点集，所述分辨率因果条件为：针对每个所述因果节点，若该因果节点不为根节点，存在一个属于所述因果边集的因果分辨率边连接该因果节点与对应的母节点，所述根节点为分辨率最低的节点，所述母节点的分辨率小于该因果节点的分辨率；根据所述因果边集和所述因果节点集确定所述因果子图。
根据本公开的一个或多个实施例，示例7提供了示例3的方法，所述编码器包括：基于分辨率的注意力卷积层和迭代池化层；所述将所述因果子图输入所述处理模型中的编码器，以得到用于表征所述因果子图的因果图像特征，包括：将所述因果子图输入所述注意力卷积层，以使所述注意力卷积层基于所述因果子图中每个节点，和与该节点之间具有边的节点的分辨率，确定该节点的特征；将所述因果子图中每个节点对应的结构特征输入所述迭代池化层，以使所述迭代池化层对之间具有空间边的节点的特征进行池化，得到所述因果图像特征。
根据本公开的一个或多个实施例，示例8提供了示例3的方法，所述处理模型是通过以下方式训练得到的：获取样本输入集和所述样本输出集，所述样本输入集包括多个样本输入，所述样本输入包括指定部位的多个全视野组织学训练图像对应的训练异构图，所述样本输出集中包括与每个所述样本输入对应的样本输出，每个所述样本输出包括所述训练异构图对应的训练指示信息；针对每个所述样本输入，将该样本输入包括的所述训练异构图输入所述提取器，以得到训练因果子图；根据所述训练异构图和所述训练因果子图，确定训练环境子图；将所述训练因果子图和所述训练环境子图输入所述编码器，以得到用于表征所述训练因果子图的训练因果图像特征，和用于表征所述训练环境子图的训练环境图像特征；将所述训练因果图像特征输入所述分类器，以得到预测指示信息，将所述训练环境子图输入所述处理模型中的环境分类器，以得到预测环境指示信息；根据所述预测环境指示信息和该样本输入对应的所述训练指示信息确定环境损失，以训练所述环境分类器；根据所述预测指示信息和该样本输入对应的所述训练指示信息确定因果损失，以训练所述提取器、所述编码器和所述分类器。
根据本公开的一个或多个实施例，示例9提供了示例8的方法，所述根据所述训练异构图和所述训练因果子图，确定训练环境子图，包括：将所述训练异构图中除所述训练因果子图的区域作为样本环境子图；将所述样本环境子图存入样本环境子图集，所述样本环境子图集用于存储每个所述样本输入对应的所述样本环境子图；从所述样本环境子图集中随机抽取一个样本环境子图作为所述训练环境子图。
根据本公开的一个或多个实施例，示例10提供了一种全视野组织学图像的处理装置，包括：分割模块，用于对指定部位的多个全视野组织学图像进行分割，得到每个所述全视野组织学图像对应的多个图像块，每个所述全视野组织学图像对应的分辨率不同；生成模块，用于根据所有所述全视野组 织学图像对应的多个所述图像块，生成异构图，所述异构图包括节点集和边集，所述节点集包括由每个所述图像块对应的图像特征和该图像块对应的分辨率组成的节点，所述边集包括用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边；提取模块，用于从所述异构图中提取出因果子图，所述因果子图包括的特征与环境子图的分布不相关，所述环境子图为所述异构图中除所述因果子图之外的区域；处理模块，用于根据所述因果子图确定所述指定部位对应的指示信息，所述指示信息用于表征所述指定部位的状态，和/或所述指定部位中的目标区域。
根据本公开的一个或多个实施例，示例11提供了一种计算机可读介质，其上存储有计算机程序，该程序被处理装置执行时实现示例1至示例9中所述方法的步骤。
根据本公开的一个或多个实施例，示例12提供了一种电子设备，包括：存储装置，其上存储有计算机程序；处理装置，用于执行所述存储装置中的所述计算机程序，以实现示例1至示例9中所述方法的步骤。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单 个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。关于上述实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。

Claims (12)

  1. 一种全视野组织学图像的处理方法，其特征在于，所述方法包括：
    对指定部位的多个全视野组织学图像进行分割，得到每个所述全视野组织学图像对应的多个图像块，每个所述全视野组织学图像对应的分辨率不同；
    根据所有所述全视野组织学图像对应的多个所述图像块，生成异构图，所述异构图包括节点集和边集，所述节点集包括由每个所述图像块对应的图像特征和该图像块对应的分辨率组成的节点，所述边集包括用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边；
    从所述异构图中提取出因果子图，所述因果子图包括的特征与环境子图的分布不相关，所述环境子图为所述异构图中除所述因果子图之外的区域；
    根据所述因果子图确定所述指定部位对应的指示信息，所述指示信息用于表征所述指定部位的状态，和/或所述指定部位中的目标区域。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所有所述全视野组织学图像对应的多个所述图像块，生成异构图，包括：
    对每个所述图像块进行特征提取，得到该图像块对应的图像特征；
    将每个所述图像块对应的图像特征和该图像块对应的分辨率作为一个节点，生成所述节点集；
    在同一所述全视野组织学图像中任意两个相邻的图像块对应的两个节点之间构建一个空间边，在任意两个对应所述指定部位同一区域且分辨率相邻的图像块对应的两个节点之间构建一个分辨率边，生成所述边集；
    根据所述节点集和所述边集生成所述异构图。
  3. 根据权利要求1所述的方法，其特征在于，所述从所述异构图中提取出因果子图，包括：
    将所述异构图输入预先训练的处理模型中的提取器，以得到所述因果子图；
    所述根据所述因果子图确定所述指定部位对应的指示信息，包括：
    将所述因果子图输入所述处理模型中的编码器，以得到用于表征所述因果子图的因果图像特征；
    将所述因果图像特征输入所述处理模型中的分类器，以得到所述指定部位的状态；和/或，根据所述因果子图，在所述全视野组织学图像中确定所述目标区域。
  4. 根据权利要求3所述的方法，其特征在于，所述将所述异构图输入预先训练的处理模型中的提取器，以得到所述因果子图，包括：
    将所述异构图输入所述提取器，确定每个所述空间边的影响度和每个所述分辨率边的影响度；
    对每个所述空间边的影响度进行排序确定因果空间边，对每个所述分辨率边的影响度进行排序确定因果分辨率边；
    根据所述因果空间边和所述因果分辨率边，确定所述因果子图。
  5. 根据权利要求4所述的方法，其特征在于，所述提取器包括图神经网络、第一多层感知机和第二多层感知机，所述将所述异构图输入所述提取器，确定每个所述空间边的影响度和每个所述分辨率边的影响度，包括：
    将所述异构图输入所述图神经网络，得到所述异构图对应的图结构特征；
    将所述图结构特征输入所述第一多层感知机，得到每个所述空间边的影 响度；
    将所述图结构特征输入所述第二多层感知机，得到每个所述分辨率边的影响度。
  6. 根据权利要求4所述的方法，其特征在于，所述根据所述因果空间边和所述因果分辨率边，确定所述因果子图，包括：
    将满足分辨率因果条件的所述因果分辨率边和所述因果空间边，作为因果边集，将所述因果边集连接的节点作为因果节点集，所述分辨率因果条件为：针对每个所述因果节点，若该因果节点不为根节点，存在一个属于所述因果边集的因果分辨率边连接该因果节点与对应的母节点，所述根节点为分辨率最低的节点，所述母节点的分辨率小于该因果节点的分辨率；
    根据所述因果边集和所述因果节点集确定所述因果子图。
  7. 根据权利要求3所述的方法，其特征在于，所述编码器包括：基于分辨率的注意力卷积层和迭代池化层；所述将所述因果子图输入所述处理模型中的编码器，以得到用于表征所述因果子图的因果图像特征，包括：
    将所述因果子图输入所述注意力卷积层，以使所述注意力卷积层基于所述因果子图中每个节点，和与该节点之间具有边的节点的分辨率，确定该节点的特征；
    将所述因果子图中每个节点对应的结构特征输入所述迭代池化层，以使所述迭代池化层对之间具有空间边的节点的特征进行池化，得到所述因果图像特征。
  8. 根据权利要求3所述的方法，其特征在于，所述处理模型是通过以下方式训练得到的：
    获取样本输入集和所述样本输出集，所述样本输入集包括多个样本输入，所述样本输入包括指定部位的多个全视野组织学训练图像对应的训练异构图，所述样本输出集中包括与每个所述样本输入对应的样本输出，每个所述样本输出包括所述训练异构图对应的训练指示信息；
    针对每个所述样本输入，将该样本输入包括的所述训练异构图输入所述提取器，以得到训练因果子图；
    根据所述训练异构图和所述训练因果子图，确定训练环境子图；
    将所述训练因果子图和所述训练环境子图输入所述编码器，以得到用于表征所述训练因果子图的训练因果图像特征，和用于表征所述训练环境子图的训练环境图像特征；
    将所述训练因果图像特征输入所述分类器，以得到预测指示信息，将所述训练环境图像特征输入所述处理模型中的环境分类器，以得到预测环境指示信息；
    根据所述预测环境指示信息和该样本输入对应的所述训练指示信息确定环境损失，以训练所述环境分类器；
    根据所述预测指示信息和该样本输入对应的所述训练指示信息确定因果损失，以训练所述提取器、所述编码器和所述分类器。
  9. 根据权利要求8所述的方法，其特征在于，所述根据所述训练异构图和所述训练因果子图，确定训练环境子图，包括：
    将所述训练异构图中除所述训练因果子图的区域作为样本环境子图；
    将所述样本环境子图存入样本环境子图集，所述样本环境子图集用于存储每个所述样本输入对应的所述样本环境子图；
    从所述样本环境子图集中随机抽取一个样本环境子图作为所述训练环境子图。
  10. 一种全视野组织学图像的处理装置，其特征在于，所述装置包括：
    分割模块，用于对指定部位的多个全视野组织学图像进行分割，得到每个所述全视野组织学图像对应的多个图像块，每个所述全视野组织学图像对应的分辨率不同；
    生成模块，用于根据所有所述全视野组织学图像对应的多个所述图像块，生成异构图，所述异构图包括节点集和边集，所述节点集包括由每个所述图像块对应的图像特征和该图像块对应的分辨率组成的节点，所述边集包括用于表征各节点之间空间关系的空间边，和用于表征各节点之间分辨率关系的分辨率边；
    提取模块，用于从所述异构图中提取出因果子图，所述因果子图包括的特征与环境子图的分布不相关，所述环境子图为所述异构图中除所述因果子图之外的区域；
    处理模块，用于根据所述因果子图确定所述指定部位对应的指示信息，所述指示信息用于表征所述指定部位的状态，和/或所述指定部位中的目标区域。
  11. 一种计算机可读介质，其上存储有计算机程序，其特征在于，该程序被处理装置执行时实现权利要求1-9中任一项所述方法的步骤。
  12. 一种电子设备，其特征在于，包括：
    存储装置，其上存储有计算机程序；
    处理装置，用于执行所述存储装置中的所述计算机程序，以实现权利要求1-9中任一项所述方法的步骤。
PCT/CN2023/116820 2022-09-06 2023-09-04 全视野组织学图像的处理方法、装置、介质和电子设备 WO2024051655A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211086109.2A CN115439449B (zh) 2022-09-06 2022-09-06 全视野组织学图像的处理方法、装置、介质和电子设备
CN202211086109.2 2022-09-06

Publications (1)

Publication Number Publication Date
WO2024051655A1 true WO2024051655A1 (zh) 2024-03-14

Family

ID=84246535

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/116820 WO2024051655A1 (zh) 2022-09-06 2023-09-04 全视野组织学图像的处理方法、装置、介质和电子设备

Country Status (2)

Country Link
CN (1) CN115439449B (zh)
WO (1) WO2024051655A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118379565A (zh) * 2024-06-25 2024-07-23 中国科学院自动化研究所 基于信息瓶颈与因果学习的因果子图提取方法、装置

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115439449B (zh) * 2022-09-06 2023-05-09 抖音视界有限公司 全视野组织学图像的处理方法、装置、介质和电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112419151A (zh) * 2020-11-19 2021-02-26 北京有竹居网络技术有限公司 图像退化处理方法、装置、存储介质及电子设备
CN114445356A (zh) * 2022-01-19 2022-05-06 上海交通大学 基于多分辨率的全视野病理切片图像肿瘤快速定位方法
US20220164946A1 (en) * 2020-11-20 2022-05-26 International Business Machines Corporation Interpretation of whole-slide images in digital pathology
CN114549896A (zh) * 2022-01-24 2022-05-27 清华大学 用于生存预测的全视野图像的异构高阶表示方法及装置
CN115439449A (zh) * 2022-09-06 2022-12-06 抖音视界有限公司 全视野组织学图像的处理方法、装置、介质和电子设备

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11113293B2 (en) * 2019-01-18 2021-09-07 Adobe Inc. Latent network summarization
GB201913616D0 (en) * 2019-09-20 2019-11-06 Univ Oslo Hf Histological image analysis
US20210372938A1 (en) * 2020-05-29 2021-12-02 The Board Of Trustee Of The University Of Alabama Deep learning-based crack segmentation through heterogeneous image fusion
CN112215837B (zh) * 2020-10-26 2023-01-06 北京邮电大学 多属性图像语义分析方法和装置
US20220198813A1 (en) * 2020-12-17 2022-06-23 Sri International System and method for efficient visual navigation
CN113658178B (zh) * 2021-10-14 2022-01-25 北京字节跳动网络技术有限公司 组织图像的识别方法、装置、可读介质和电子设备
CN114565919B (zh) * 2022-01-19 2024-06-07 深圳先进技术研究院 基于数字病理图像的肿瘤微环境空间关系建模系统与方法
CN114708347B (zh) * 2022-03-22 2024-09-17 江门市中心医院 一种基于自适应选择的双源域异构迁移学习的肺结节ct图像分类方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112419151A (zh) * 2020-11-19 2021-02-26 北京有竹居网络技术有限公司 图像退化处理方法、装置、存储介质及电子设备
US20220164946A1 (en) * 2020-11-20 2022-05-26 International Business Machines Corporation Interpretation of whole-slide images in digital pathology
CN114445356A (zh) * 2022-01-19 2022-05-06 上海交通大学 基于多分辨率的全视野病理切片图像肿瘤快速定位方法
CN114549896A (zh) * 2022-01-24 2022-05-27 清华大学 用于生存预测的全视野图像的异构高阶表示方法及装置
CN115439449A (zh) * 2022-09-06 2022-12-06 抖音视界有限公司 全视野组织学图像的处理方法、装置、介质和电子设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118379565A (zh) * 2024-06-25 2024-07-23 中国科学院自动化研究所 基于信息瓶颈与因果学习的因果子图提取方法、装置

Also Published As

Publication number Publication date
CN115439449B (zh) 2023-05-09
CN115439449A (zh) 2022-12-06

Similar Documents

Publication Publication Date Title
US20230196117A1 (en) Training method for semi-supervised learning model, image processing method, and device
WO2024051655A1 (zh) 全视野组织学图像的处理方法、装置、介质和电子设备
WO2021218471A1 (zh) 一种用于图像处理的神经网络以及相关设备
CN111382868A (zh) 神经网络结构搜索方法和神经网络结构搜索装置
CN112668588B (zh) 车位信息生成方法、装置、设备和计算机可读介质
WO2023231954A1 (zh) 一种数据的去噪方法以及相关设备
US20240232575A1 (en) Neural network obtaining method, data processing method, and related device
US20240312252A1 (en) Action recognition method and apparatus
KR20220047228A (ko) 이미지 분류 모델 생성 방법 및 장치, 전자 기기, 저장 매체, 컴퓨터 프로그램, 노변 장치 및 클라우드 제어 플랫폼
WO2023231753A1 (zh) 一种神经网络的训练方法、数据的处理方法以及设备
US20240233358A9 (en) Image classification method, model training method, device, storage medium, and computer program
CN111062431A (zh) 图像聚类方法、图像聚类装置、电子设备及存储介质
WO2021139351A1 (zh) 图像分割方法、装置、介质及电子设备
CN116452810A (zh) 一种多层次语义分割方法、装置、电子设备及存储介质
WO2022012178A1 (zh) 用于生成目标函数的方法、装置、电子设备和计算机可读介质
WO2024114659A1 (zh) 一种摘要生成方法及其相关设备
WO2024001653A9 (zh) 特征提取方法、装置、存储介质及电子设备
WO2023197857A1 (zh) 一种模型切分方法及其相关设备
WO2023197910A1 (zh) 一种用户行为预测方法及其相关设备
CN116362301A (zh) 一种模型的量化方法以及相关设备
CN115587616A (zh) 网络模型训练方法、装置、存储介质及计算机设备
WO2022052647A1 (zh) 一种数据处理方法、神经网络的训练方法以及相关设备
CN112509052B (zh) 黄斑中心凹的检测方法、装置、计算机设备和存储介质
CN115186738A (zh) 模型训练方法、装置和存储介质
WO2023231796A1 (zh) 一种视觉任务处理方法及其相关设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23862349

Country of ref document: EP

Kind code of ref document: A1