CN113269724A - Fine-grained cancer subtype classification method - Google Patents
Fine-grained cancer subtype classification method Download PDFInfo
- Publication number
- CN113269724A CN113269724A CN202110467023.3A CN202110467023A CN113269724A CN 113269724 A CN113269724 A CN 113269724A CN 202110467023 A CN202110467023 A CN 202110467023A CN 113269724 A CN113269724 A CN 113269724A
- Authority
- CN
- China
- Prior art keywords
- patch
- classification
- grained
- fine
- cell nucleus
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 206010028980 Neoplasm Diseases 0.000 title claims abstract description 30
- 238000000034 method Methods 0.000 title claims abstract description 28
- 201000011510 cancer Diseases 0.000 title claims abstract description 27
- 210000003855 cell nucleus Anatomy 0.000 claims abstract description 27
- 230000011218 segmentation Effects 0.000 claims abstract description 17
- 230000001575 pathological effect Effects 0.000 claims abstract description 13
- 238000000605 extraction Methods 0.000 claims description 7
- 238000013136 deep learning model Methods 0.000 claims description 3
- 210000004027 cell Anatomy 0.000 description 8
- 239000010410 layer Substances 0.000 description 7
- 238000010586 diagram Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000004940 nucleus Anatomy 0.000 description 2
- 230000007170 pathology Effects 0.000 description 2
- 210000001519 tissue Anatomy 0.000 description 2
- 101000611614 Homo sapiens Proline-rich protein PRCC Proteins 0.000 description 1
- 102100040829 Proline-rich protein PRCC Human genes 0.000 description 1
- 208000006265 Renal cell carcinoma Diseases 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 201000010279 papillary renal cell carcinoma Diseases 0.000 description 1
- 239000002356 single layer Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H70/00—ICT specially adapted for the handling or processing of medical references
- G16H70/60—ICT specially adapted for the handling or processing of medical references relating to pathologies
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
Abstract
The invention discloses a fine-grained cancer subtype classification method, which comprises the following steps: the method comprises the following steps: obtaining a cell nucleus segmentation and classification result in the pathological picture; step two: extracting instance patch according to the result of the cell nucleus segmentation and classification prediction; step three: extracting convolution characteristics of the instance patch; step four: the classification of cancer subtypes is completed by generating picture-level features of the used samples patch through a Transformer model. The invention can assist pathological doctors in classifying cancer subtypes and improve the working efficiency of doctors.
Description
Technical Field
The invention belongs to the field of intelligent pathological classification, and particularly relates to a fine-grained cancer subtype classification method.
Background
With the development of digital pathology techniques, tissue sections are stored in the form of digital images, making automatic identification of pathological patterns possible, while typing of cancer subtypes attracts a large number of researchers as a fundamental task for digital pathology analysis, many CNN-based models are currently proposed for typing of different cancers of different tissues, these methods rely primarily on extracting some specific histological features that can be easily extracted by CNN to identify subtypes of cancer, such as structural features of tumors, however, for some cancers requiring more fine-grained characterization, traditional models based on CNN do not perform well in classifying cancer subtypes, such as papillary renal cell carcinoma pRCC, two subtypes of which share a similar structural feature, both papillary structures, while in the pathological classification of pRCC, two subtypes of pRCC classification rely mainly on two more fine-grained features: cellular level features and cell layer structure features. Type 1 PRCC is composed primarily of small basophilic cubocytes, arranged in a monolayer. The nuclei are small and uniform, round or oval, the nucleoli are not evident, while the high grade nuclei with cells in pseudo-stratified arrangement and significant nucleoli are the hallmark feature of type 2 pRCC. The traditional CNN-based model cannot well capture the fine-grained characteristics, and cannot well classify the cancer subtypes based on the fine-grained characteristics. In addition, some methods manually construct the cell level features through a prediction result of segmentation and classification of cell nuclei, and although the methods can extract fine-grained cell level information, the methods cannot acquire the cell level features and the methods are not very universal for specific tasks.
Disclosure of Invention
The invention aims to provide a fine-grained cancer subtype classification method to overcome the defects of the prior art.
In order to achieve the purpose, the invention adopts the following technical scheme:
a fine-grained cancer subtype classification method comprising the steps of:
the method comprises the following steps: obtaining a cell nucleus segmentation and classification result in the pathological picture;
step two: extracting instance patch according to the result of the cell nucleus segmentation and classification prediction;
step three: extracting convolution characteristics of the instance patch;
step four: the classification of cancer subtypes is completed by generating picture-level features of the used samples patch through a Transformer model.
The invention further improves the method that in the step one, pathological pictures of cancer regions are input to an example extraction module, and cell nucleus segmentation and classification prediction results are obtained through the existing cell nucleus segmentation and classification deep learning model with high accuracy, such as U-Net.
The further improvement of the invention is that in the second step, the instance patch is extracted, specifically: and according to the acquired result of the segmentation and classification prediction of the cell nucleus, cutting out a region with a fixed size by taking the center of the cell nucleus as a cutting center, and recording the coordinates of the center point and the level of the cell nucleus to acquire a series of instances patch.
The further improvement of the invention is that in the third step, the example patch convolution feature extraction specifically comprises: and inputting the instance patch into a small CNN network with two layers of convolution to extract convolution characteristics.
The invention has the further improvement that in the fourth step, the sample patch is used to generate picture-level characteristics through a Transformer model to complete the classification of cancer subtypes, and the concrete steps are as follows:
acquiring corresponding embedded information through the position information and the core level information of the instances patch, combining the embedded information with the convolution characteristics acquired in the step three, tiling all the instances patch characteristics finally acquired to form a sequence, and adding a cls _ token to the forefront of the sequence to integrate the information of all the patches so as to generate a sequence S0The sequence S0The input is processed by a Transfomer encoder, and then the cls _ token integrating all patch information is input into a Linear _ Layer for final subtype classification.
The invention is further improved in that the sequence S input to the Transfomer encoder0The specific generation method is as follows:
wherein
For added cls _ token, generate image-level classification feature for integrating all patch information, τ (x)i) Where i is 1,2, …, N is the embedding feature obtained by the mini-CNN of the ith instance patch, [ s ]X,sY]For position embedding, X, Y are coordinate information, sgradeIs core level embedded.
Compared with the prior art, the invention has at least the following beneficial technical effects:
1. the classification of cancer subtypes is assisted by pathologists, and the working efficiency of the doctors is improved.
2. The method is used for learning based on the examples, the characteristics of the cells and the surrounding environment are more concerned, a meaningless background area is omitted, the calculation complexity is reduced, and the classification accuracy is improved.
3. Based on a Transformer model, fine-grained pathological features such as cell level and cell layer level can be captured and fused better, and subtype classification can be carried out more accurately.
Drawings
FIG. 1 is a diagram showing a method for classifying fine-grained cancer subtypes.
The example extracting module is used for segmenting and grading cell nuclei of the input pathological image, then cutting out cell nucleus regions according to segmentation results, recording corresponding cell nucleus levels and position information, and generating an example patch as input of the subtype classifying module. 2. Is a subtype classification module, and is characterized in that,
convolution feature extraction is carried out on the instance patch through a small CNN, then kernel-level embedded information and kernel position embedded information are added and input into a Transformer encoder together, and finally subtype classification is carried out through a linear layer.
Fig. 2 is a schematic diagram of a mini-CNN.
FIG. 3 is a schematic diagram of a transform encoder.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
The invention relates to a fine-grained cancer subtype classification method based on an example, which mainly comprises an example extraction module and a subtype classification module;
the specific implementation steps are as follows:
the method comprises the following steps: and inputting the pathological picture into an example extraction module, and obtaining a cell nucleus segmentation and classification prediction result through a cell nucleus segmentation and classification deep learning model such as U-Net.
Step two: according to the obtained result of the segmentation and classification prediction of the cell nucleus, a series of example patches are obtained by cutting out patch areas with fixed size (for example, 64 × 64) by taking the center of the cell nucleus as a cutting center and recording the coordinates of the center point and the level of the cell nucleus.
Step three: and inputting the sample patch into a small CNN network with two layers of convolution to extract convolution characteristics, wherein the concrete structure of the network is shown in FIG. 2.
Step four: acquiring corresponding embedded information through the position information and the core level information of the instances patch, combining the embedded information with the convolution characteristics acquired in the step three, tiling all the instances patch characteristics finally acquired to form a sequence, and adding a cls _ token to the forefront of the sequence to integrate the information of all the patches so as to generate a sequence S0It is expressed specifically as the following equation:
wherein
The added cls _ token is used for integrating all patch information to generate image-level classification features; τ (x)i) I is 1,2, …, N is the embedded feature obtained by the mini CNN of the ith instance patch; [ s ] ofX,sY]For position embedding, X and Y are coordinate information, and in practical application, because the size of a histopathology image is large, learning redundant position embedding has no meaning, an image coordinate grid can be reduced to a certain extent, for example, 1/20 with the original size, and the grid position where the center of a cell nucleus is located is used as the position of the cell nucleus; sgradeIs core level embedded.
Will sequence S0The input is processed by a Transfomer encoder, and then the cls _ token is input into a Linear _ Layer for final subtype classification. Because the input requirements of the Transformer are consistent in length, and the number of cores in different pathological pictures is inconsistent, the input patch is arranged according to the core level and size information (from high to low and from large to small), the first N cores are taken, and if the number of cores is less than 0, the input patch is filled with the data. And the specific structure of the transform encoder is shown in fig. 3.
Although the invention has been described in detail hereinabove with respect to a general description and specific embodiments thereof, it will be apparent to those skilled in the art that modifications or improvements may be made thereto based on the invention. Accordingly, such modifications and improvements are intended to be within the scope of the invention as claimed.
Claims (6)
1. A method for classifying fine-grained cancer subtypes, comprising the steps of:
the method comprises the following steps: obtaining a cell nucleus segmentation and classification result in the pathological picture;
step two: extracting instance patch according to the result of the cell nucleus segmentation and classification prediction;
step three: extracting convolution characteristics of the instance patch;
step four: the classification of cancer subtypes is completed by generating picture-level features of the used samples patch through a Transformer model.
2. The fine-grained cancer subtype classification method according to claim 1, wherein in the first step, pathological pictures of cancer regions are input to an instance extraction module, and cell nucleus segmentation and classification prediction results are obtained through an existing cell nucleus segmentation and classification deep learning model with high accuracy, such as U-Net.
3. The fine-grained cancer subtype classification method according to claim 1, wherein in step two, the instance patch is extracted by: and according to the acquired result of the segmentation and classification prediction of the cell nucleus, cutting out a region with a fixed size by taking the center of the cell nucleus as a cutting center, and recording the coordinates of the center point and the level of the cell nucleus to acquire a series of instances patch.
4. The fine-grained cancer subtype classification method according to claim 1, characterized in that in step three, the example patch convolution feature extraction specifically comprises: and inputting the instance patch into a small CNN network with two layers of convolution to extract convolution characteristics.
5. The fine-grained cancer subtype classification method according to claim 1, wherein in the fourth step, the classification of cancer subtypes is completed by generating picture-level features of the used instances patch through a transform model, and the concrete steps are as follows:
acquiring corresponding embedded information through the position information and the core level information of the instances patch, combining the embedded information with the convolution characteristics acquired in the step three, tiling all the instances patch characteristics finally acquired to form a sequence, and adding a cls _ token to the forefront of the sequence to integrate the information of all the patches so as to generate a sequence S0The sequence S0Input to a Transfomer encoder for processing, and then all the components are integratedThe cls _ token of patch information is input into a Linear _ Layer for final subtype classification.
6. The fine-grained cancer subtype classification method according to claim 5, wherein the sequence S input to the Transfomer coder0The specific generation method is as follows:
wherein
For added cls _ token, generate image-level classification feature for integrating all patch information, τ (x)i) Where i is 1,2, …, N is the embedding feature obtained by the mini-CNN of the ith instance patch, [ s ]X，sY]For position embedding, X, Y are coordinate information, sgradeIs core level embedded.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110467023.3A CN113269724A (en) | 2021-04-28 | 2021-04-28 | Fine-grained cancer subtype classification method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110467023.3A CN113269724A (en) | 2021-04-28 | 2021-04-28 | Fine-grained cancer subtype classification method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113269724A true CN113269724A (en) | 2021-08-17 |
Family
ID=77229653
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110467023.3A Pending CN113269724A (en) | 2021-04-28 | 2021-04-28 | Fine-grained cancer subtype classification method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113269724A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113951866A (en) * | 2021-10-28 | 2022-01-21 | 北京深睿博联科技有限责任公司 | Deep learning-based uterine fibroid diagnosis method and device |
| CN114119585A (en) * | 2021-12-01 | 2022-03-01 | 昆明理工大学 | Method for identifying key feature enhanced gastric cancer image based on Transformer |
| CN115909006A (en) * | 2022-10-27 | 2023-04-04 | 武汉兰丁智能医学股份有限公司 | Mammary tissue image classification method and system based on convolution Transformer |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112365471A (en) * | 2020-11-12 | 2021-02-12 | 哈尔滨理工大学 | Cervical cancer cell intelligent detection method based on deep learning |
| CN112434172A (en) * | 2020-10-29 | 2021-03-02 | 西安交通大学 | Pathological image prognosis feature weight calculation method and system |
-
2021
- 2021-04-28 CN CN202110467023.3A patent/CN113269724A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112434172A (en) * | 2020-10-29 | 2021-03-02 | 西安交通大学 | Pathological image prognosis feature weight calculation method and system |
| CN112365471A (en) * | 2020-11-12 | 2021-02-12 | 哈尔滨理工大学 | Cervical cancer cell intelligent detection method based on deep learning |
Non-Patent Citations (2)
| Title |
|---|
| ALEXEY DOSOVITSKIY等: "AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE", 《ARXIV》 * |
| JUN CHENG等: "Computational analysis of pathological images enables a better diagnosis of TFE3 Xp11.2 translocation renal cell carcinoma", 《NATURE COMMUNICATIONS》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113951866A (en) * | 2021-10-28 | 2022-01-21 | 北京深睿博联科技有限责任公司 | Deep learning-based uterine fibroid diagnosis method and device |
| CN114119585A (en) * | 2021-12-01 | 2022-03-01 | 昆明理工大学 | Method for identifying key feature enhanced gastric cancer image based on Transformer |
| CN114119585B (en) * | 2021-12-01 | 2022-11-29 | 昆明理工大学 | Method for identifying key feature enhanced gastric cancer image based on Transformer |
| CN115909006A (en) * | 2022-10-27 | 2023-04-04 | 武汉兰丁智能医学股份有限公司 | Mammary tissue image classification method and system based on convolution Transformer |
| CN115909006B (en) * | 2022-10-27 | 2024-01-19 | 武汉兰丁智能医学股份有限公司 | Mammary tissue image classification method and system based on convolution transducer |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110472616B (en) | Image recognition method and device, computer equipment and storage medium | |
| CN113269724A (en) | Fine-grained cancer subtype classification method | |
| CN108664996B (en) | Ancient character recognition method and system based on deep learning | |
| CN110175613A (en) | Street view image semantic segmentation method based on Analysis On Multi-scale Features and codec models | |
| CN109191472A (en) | Based on the thymocyte image partition method for improving U-Net network | |
| CN109191471A (en) | Based on the pancreatic cell image partition method for improving U-Net network | |
| Chen et al. | Shadow-based Building Detection and Segmentation in High-resolution Remote Sensing Image. | |
| Kadam et al. | Detection and localization of multiple image splicing using MobileNet V1 | |
| CN109409384A (en) | Image-recognizing method, device, medium and equipment based on fine granularity image | |
| CN109086777A (en) | A kind of notable figure fining method based on global pixel characteristic | |
| CN112132827A (en) | Pathological image processing method and device, electronic equipment and readable storage medium | |
| CN108648256A (en) | A kind of gray scale core three-dimensional method for reconstructing based on super dimension | |
| Rao | Mitos-rcnn: A novel approach to mitotic figure detection in breast cancer histopathology images using region based convolutional neural networks | |
| CN110349170B (en) | Full-connection CRF cascade FCN and K mean brain tumor segmentation algorithm | |
| CN112801182A (en) | RGBT target tracking method based on difficult sample perception | |
| CN113610024B (en) | Multi-strategy deep learning remote sensing image small target detection method | |
| Qian et al. | Spatial contextual noise removal for post classification smoothing of remotely sensed images | |
| CN112508860B (en) | Artificial intelligence interpretation method and system for positive check of immunohistochemical image | |
| CN109117841B (en) | Scene text detection method based on stroke width transformation and convolutional neural network | |
| Dong et al. | Building Extraction from High Spatial Resolution Remote Sensing Images of Complex Scenes by Combining Region-Line Feature Fusion and OCNN | |
| CN113392852B (en) | Vehicle detection method and system based on deep learning | |
| CN115512220A (en) | Remote sensing image road segmentation method based on improved Unet network model | |
| CN112926670A (en) | Garbage classification system and method based on transfer learning | |
| CN113222887A (en) | Deep learning-based nano-iron labeled neural stem cell tracing method | |
| CN113205526B (en) | Distribution line accurate semantic segmentation method based on multi-source information fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210817 |