CN117405644B - Three-level lymphoid structure maturity identification method based on multicolor immunofluorescence - Google Patents
Three-level lymphoid structure maturity identification method based on multicolor immunofluorescence Download PDFInfo
- Publication number
- CN117405644B CN117405644B CN202311714748.3A CN202311714748A CN117405644B CN 117405644 B CN117405644 B CN 117405644B CN 202311714748 A CN202311714748 A CN 202311714748A CN 117405644 B CN117405644 B CN 117405644B
- Authority
- CN
- China
- Prior art keywords
- level
- module
- cell
- tertiary
- slice
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000010166 immunofluorescence Methods 0.000 title claims abstract description 41
- 238000000034 method Methods 0.000 title claims abstract description 22
- 208000033878 Tertiary Lymphoid Structures Diseases 0.000 claims abstract description 99
- 230000001926 lymphatic effect Effects 0.000 claims abstract description 68
- 238000000605 extraction Methods 0.000 claims abstract description 64
- 239000012188 paraffin wax Substances 0.000 claims abstract description 16
- 206010028980 Neoplasm Diseases 0.000 claims abstract description 14
- 238000012216 screening Methods 0.000 claims abstract description 8
- 210000004027 cell Anatomy 0.000 claims description 90
- 210000001519 tissue Anatomy 0.000 claims description 74
- 210000001744 T-lymphocyte Anatomy 0.000 claims description 50
- 210000003719 b-lymphocyte Anatomy 0.000 claims description 50
- 102100022005 B-lymphocyte antigen CD20 Human genes 0.000 claims description 49
- 101000897405 Homo sapiens B-lymphocyte antigen CD20 Proteins 0.000 claims description 49
- 230000003325 follicular Effects 0.000 claims description 33
- 102100032768 Complement receptor type 2 Human genes 0.000 claims description 29
- 101000941929 Homo sapiens Complement receptor type 2 Proteins 0.000 claims description 29
- 210000004698 lymphocyte Anatomy 0.000 claims description 28
- 230000002776 aggregation Effects 0.000 claims description 27
- 238000004220 aggregation Methods 0.000 claims description 27
- 238000005070 sampling Methods 0.000 claims description 25
- 230000004927 fusion Effects 0.000 claims description 17
- 210000002751 lymph Anatomy 0.000 claims description 14
- 238000004458 analytical method Methods 0.000 claims description 13
- 238000003125 immunofluorescent labeling Methods 0.000 claims description 10
- 238000010186 staining Methods 0.000 claims description 10
- 230000006835 compression Effects 0.000 claims description 7
- 238000007906 compression Methods 0.000 claims description 7
- FWBHETKCLVMNFS-UHFFFAOYSA-N 4',6-Diamino-2-phenylindol Chemical compound C1=CC(C(=N)N)=CC=C1C1=CC2=CC=C(C(N)=N)C=C2N1 FWBHETKCLVMNFS-UHFFFAOYSA-N 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 238000004043 dyeing Methods 0.000 claims description 5
- 238000012482 interaction analysis Methods 0.000 claims description 4
- 238000012758 nuclear staining Methods 0.000 claims description 4
- 238000011176 pooling Methods 0.000 claims description 3
- 239000011800 void material Substances 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 17
- 238000001220 thermal lens spectroscopy Methods 0.000 description 17
- 230000011637 translesion synthesis Effects 0.000 description 17
- 230000009286 beneficial effect Effects 0.000 description 10
- 239000007850 fluorescent dye Substances 0.000 description 10
- 230000004083 survival effect Effects 0.000 description 5
- 230000000875 corresponding effect Effects 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 230000002596 correlated effect Effects 0.000 description 3
- 210000003855 cell nucleus Anatomy 0.000 description 2
- 230000008614 cellular interaction Effects 0.000 description 2
- 230000000052 comparative effect Effects 0.000 description 2
- 238000012757 fluorescence staining Methods 0.000 description 2
- 230000028993 immune response Effects 0.000 description 2
- 230000035800 maturation Effects 0.000 description 2
- 238000004393 prognosis Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 208000001333 Colorectal Neoplasms Diseases 0.000 description 1
- 208000007097 Urinary Bladder Neoplasms Diseases 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 239000000427 antigen Substances 0.000 description 1
- 230000030741 antigen processing and presentation Effects 0.000 description 1
- 102000036639 antigens Human genes 0.000 description 1
- 108091007433 antigens Proteins 0.000 description 1
- 230000006907 apoptotic process Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 210000004204 blood vessel Anatomy 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000001684 chronic effect Effects 0.000 description 1
- 208000037976 chronic inflammation Diseases 0.000 description 1
- 230000006020 chronic inflammation Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 210000001280 germinal center Anatomy 0.000 description 1
- 230000012178 germinal center formation Effects 0.000 description 1
- 206010073071 hepatocellular carcinoma Diseases 0.000 description 1
- 231100000844 hepatocellular carcinoma Toxicity 0.000 description 1
- 210000002865 immune cell Anatomy 0.000 description 1
- 230000002757 inflammatory effect Effects 0.000 description 1
- 210000004072 lung Anatomy 0.000 description 1
- 208000020816 lung neoplasm Diseases 0.000 description 1
- 210000001165 lymph node Anatomy 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 210000005212 secondary lymphoid organ Anatomy 0.000 description 1
- 239000012128 staining reagent Substances 0.000 description 1
- 210000000130 stem cell Anatomy 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
- G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
- G01N21/64—Fluorescence; Phosphorescence
- G01N21/6428—Measuring fluorescence of fluorescent products of reactions or of fluorochrome labelled reactive substances, e.g. measuring quenching effects, using measuring "optrodes"
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N1/00—Sampling; Preparing specimens for investigation
- G01N1/02—Devices for withdrawing samples
- G01N1/04—Devices for withdrawing samples in the solid state, e.g. by cutting
- G01N1/06—Devices for withdrawing samples in the solid state, e.g. by cutting providing a thin slice, e.g. microtome
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N1/00—Sampling; Preparing specimens for investigation
- G01N1/28—Preparing specimens for investigation including physical details of (bio-)chemical methods covered elsewhere, e.g. G01N33/50, C12Q
- G01N1/30—Staining; Impregnating ; Fixation; Dehydration; Multistep processes for preparing samples of tissue, cell or nucleic acid material and the like for analysis
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
- G01N33/48—Biological material, e.g. blood, urine; Haemocytometers
- G01N33/50—Chemical analysis of biological material, e.g. blood, urine; Testing involving biospecific ligand binding methods; Immunological testing
- G01N33/53—Immunoassay; Biospecific binding assay; Materials therefor
- G01N33/531—Production of immunochemical test materials
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
- G01N33/48—Biological material, e.g. blood, urine; Haemocytometers
- G01N33/50—Chemical analysis of biological material, e.g. blood, urine; Testing involving biospecific ligand binding methods; Immunological testing
- G01N33/53—Immunoassay; Biospecific binding assay; Materials therefor
- G01N33/531—Production of immunochemical test materials
- G01N33/532—Production of labelled immunochemicals
- G01N33/533—Production of labelled immunochemicals with fluorescent label
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
- G06V20/693—Acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
- G06V20/695—Preprocessing, e.g. image segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
- G06V20/698—Matching; Classification
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Immunology (AREA)
- Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Chemical & Material Sciences (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- Pathology (AREA)
- Hematology (AREA)
- Urology & Nephrology (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Biotechnology (AREA)
- Cell Biology (AREA)
- Microbiology (AREA)
- Food Science & Technology (AREA)
- Medicinal Chemistry (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Optics & Photonics (AREA)
- Chemical Kinetics & Catalysis (AREA)
- Investigating Or Analysing Biological Materials (AREA)
Abstract
The invention discloses a three-level lymphoid structure maturity identification method based on multicolor immunofluorescence, which belongs to the technical field of image identification and comprises the following steps: obtaining paraffin sample sections of tumor patients; dying the paraffin sample slice of the tumor patient to obtain a first dyed slice and a second dyed slice; performing panoramic scanning and feature registration on the first dyed slice and the second dyed slice to obtain a first full-view slice and a second full-view slice after feature registration; the method comprises the steps that three-level lymphoid structure areas of a first full-view picture and a second full-view picture are obtained based on a first feature extraction network and a second feature extraction network after corresponding feature registration, and a multicolor immunofluorescence image to be identified, which is marked with a plurality of three-level lymphoid structure areas, is obtained through screening out error feature areas; and carrying out tertiary lymphoid structure maturity identification based on the standby multicolor immunofluorescence graph to obtain a tertiary lymphoid structure maturity identification result. The invention solves the problem that the mature tertiary lymphatic structure is difficult to be identified rapidly and accurately.
Description
Technical Field
The invention belongs to the technical field of tertiary lymphoid structure maturity identification, and particularly relates to a tertiary lymphoid structure maturity identification method based on polychromatic immunofluorescence.
Background
Tertiary lymphoid structures (Tertiary lymphoid structures, TLS) are ectopic lymphoid structures that are located at sites of chronic inflammation and drive specific antigen immune responses. Unlike secondary lymphoid organs such as lymph nodes, TLS lacks the envelope and has its unique features and functions. The primary role of TLS is to promote immune responses in chronic inflammatory environments, including antigen presentation, T cell and B cell interactions, lymphocyte proliferation and activation, germinal center formation, etc., the formation and maturation process of which involves multiple types of immune cell involvement. Based on the structure and cellular composition of TLS, mature and immature TLS can be distinguished, and researchers have found that the degree of TLS maturation correlates with the prognosis of tumors. In untreated lung, colorectal and bladder cancers, mature TLS with germinal centers is positively correlated with survival, while immature TLS is either not correlated with survival or is less correlated with survival. In hepatocellular carcinoma, mature TLS is associated with improved survival, while immature TLS acts as a survival region for tumor progenitor cells and results in a poorer prognosis. Therefore, how to resolve mature TLS is a very important research direction.
In the current research, multicolor immunofluorescent staining is often used to mark different cell populations, then a panoramic scanning image is taken by a fluorescent microscope, TLS and maturity thereof are identified according to the staining condition of each part in the image, but the work still has great challenges, mainly because of two aspects: on one hand, the method is easy to be overlooked, namely, a panoramic scanning image dyed by multicolor immunofluorescence is large, and a scanning image can have a plurality of TLSs to hundreds of TLSs and can be scattered in different areas, and the method is totally manually distinguished, so that the workload is huge, and the overlooking is easy to occur; another aspect is the inefficiency, i.e., the high and complex requirements for judgment of mature and immature TLS.
Disclosure of Invention
Aiming at the defects in the prior art, the three-stage lymphoid structure maturity identification method based on multicolor immunofluorescence provided by the invention firstly carries out panoramic scanning and feature registration on a paraffin sample slice and a dyed slice of a tumor patient, so as to obtain a first full-view slice and a second full-view slice, the image features in the second full-view slice extracted through a feature extraction network screen out the error feature region in the first full-view slice, a multicolor immunofluorescence image to be identified is obtained, and finally the problem that the mature three-stage lymphoid structure is difficult to quickly and accurately identify is solved by carrying out three-stage lymphoid structure maturity identification on the multicolor immunofluorescence image to be used.
In order to achieve the aim of the invention, the invention adopts the following technical scheme:
the invention provides a three-level lymphoid structure maturity identification method based on multicolor immunofluorescence, which comprises the following steps:
s1, obtaining paraffin sample sections of tumor patients;
s2, staining a paraffin sample slice of a tumor patient to obtain a first stained slice and a second stained slice;
s3, observing the second dyed slice, judging whether a crack exists in the second dyed slice, returning to S1 if the crack exists in the second dyed slice, and entering S4 if the crack exists in the second dyed slice;
s4, carrying out panoramic scanning and feature registration on the first dyed slice and the second dyed slice to obtain a first full-view slice and a second full-view slice after feature registration;
s5, three-level lymphoid structure areas of the first full-view picture and the second full-view picture are screened out through error feature areas based on the first feature extraction network and the second feature extraction network which correspond to the extracted features after registration, and a multicolor immunofluorescence image to be identified, which is marked with a plurality of three-level lymphoid structure areas, is obtained;
s6, identifying the maturity of the tertiary lymphoid structure based on the to-be-used multicolor immunofluorescence graph to obtain an identification result of the maturity of the tertiary lymphoid structure.
The beneficial effects of the invention are as follows: according to the three-stage lymphoid structure maturity identification method based on multicolor immunofluorescence, provided by the invention, the paraffin sample of a tumor patient is sliced and dyed in different modes, so that a basis is provided for quality control of the paraffin sample of the tumor patient through a second dyed slice, and the effectiveness of three-stage lymphoid structure maturity identification is ensured; according to the invention, panoramic scanning and feature registration are carried out on the first dyed slice and the second dyed slice, and defective error feature areas are rapidly and efficiently screened out on the basis of three-level lymphoid structure areas of the first full-view slice and the second full-view slice extracted by the first feature extraction network and the second feature extraction network, so that the accuracy and high efficiency of three-level lymphoid structure maturity identification are ensured; according to the invention, through multi-aspect analysis such as cell focusing trend, tissue structure classification, tissue structure form and interaction, the identification of the tertiary lymphoid structure maturity of the multi-color immunofluorescence image to be used is realized, and the accuracy of the identification result of the tertiary lymphoid structure maturity is fully realized.
Further, the step S2 includes the following steps:
s21, slicing the paraffin sample according to a preset thickness, and arbitrarily selecting two continuous slices, wherein the value range of the preset thickness is 4-5 mu m;
s22, carrying out multicolor immunofluorescence staining on the selected first section to obtain a first stained section;
s23, hematoxylin-eosin staining is carried out on the selected second section to obtain a second stained section.
The beneficial effects of adopting the further scheme are as follows: according to the invention, the paraffin sample is sliced according to the preset thickness, so that the microscopic observation effect is ensured, and two continuous slices are selected at will, so that the microscopic deconstruction between the slices is ensured to be the same as possible; the invention is based on the hematoxylin-eosin stained second staining section, ensures the quality of paraffin samples of tumor patients, and provides a basis for screening out error tertiary lymphoid structure areas which are invalid for identifying maturity on the first full-scope film; the invention is based on the first staining section of multicolor immunofluorescence staining, and can be convenient for distinguishing and identifying cells stained with different colors, observing aggregation trend, observing distribution condition, counting and measuring interval when identifying the maturity of the tertiary lymphoid structure, especially when analyzing the multiple aspects such as focusing trend, tissue structure classification, tissue structure form and interaction.
Further, the matching features adopted in the feature registration in S4 include a tissue outer edge shape feature, a tissue inner vessel shape, a cavity shape and a cell cluster shape feature.
The beneficial effects of adopting the further scheme are as follows: according to the invention, through structural shape characteristics such as tissue outer edge shape characteristics, tissue inner vessel shape, cavity shape, cell cluster shape characteristics and the like, the angles of the first dyed slice and the second dyed slice are adjusted, so that characteristic registration is realized, and a foundation is provided for error characteristic screening of the digital characteristics extracted from the first full-view slice corresponding to the first dyed slice by utilizing the digital characteristics extracted from the second full-view slice corresponding to the second dyed slice, namely, a foundation is provided for rapid screening of a split tertiary lymphoid structure area which is ineffective in maturity identification.
Further, the step S5 includes the following steps:
s51, acquiring a dyed slice full-view film dataset, wherein the dyed slice full-view film dataset comprises a first dyed slice panoramic film sub-dataset formed by a plurality of first dyed slice panoramic films and a second dyed slice panoramic film sub-dataset formed by a plurality of second dyed slice panoramic films;
S52, respectively constructing a first feature extraction network and a second feature extraction network;
s53, training a first feature extraction network by using the first dyed slice panoramic sheet data set to obtain a trained first feature extraction network;
s54, training a second feature extraction network by using the second dyeing slice full-scene data set to obtain a trained second feature extraction network;
s55, extracting a three-level lymphatic structure region of the first full-scene film after feature registration by using the trained first feature extraction network, and taking the three-level lymphatic structure region as a first digital feature;
s57, searching an incomplete tertiary lymphatic structure area in the second digital characteristic, and taking the area in the first digital characteristic corresponding to the searched result as an error characteristic;
s58, screening out error features in the first digital features to obtain a multicolor immunofluorescence image to be identified, wherein the multicolor immunofluorescence image is marked with a plurality of tertiary lymphoid structure areas.
The beneficial effects of adopting the further scheme are as follows: the invention provides a first feature extraction network and a second feature extraction network, and the first feature extraction network and the second feature extraction network are trained through a first dyeing slice panoramic film sub-data set and a second dyeing slice panoramic film sub-data set respectively so as to extract a first digital feature and a second digital feature respectively, and error feature screening is carried out on a cracked tertiary lymph structure area which is invalid for maturity identification in the first digital feature based on the second digital feature, so that the maturity of the tertiary lymph structure area is accurately identified, the error feature is screened, and the accuracy degree of a tertiary lymph structure maturity identification object is ensured.
Further, the first feature extraction network and the second feature extraction network in S52 each include:
the image feature extraction sub-network is used for receiving the full-view image of the first dyed slice or the full-view image of the second dyed slice and acquiring three-level lymphatic structure image feature information in the full-view image of the first dyed slice or the full-view image of the second dyed slice through the lightweight network module;
the multi-scale feature extraction sub-network is used for receiving the three-level lymphatic structure image feature information and extracting the multi-scale feature information of the three-level lymphatic structure in the three-level lymphatic structure image feature information through the pyramid module;
the characteristic fusion prediction output sub-network is used for receiving the three-level lymphatic structure image characteristic information and the multi-scale characteristic information of the three-level lymphatic structure, and predicting and outputting a panoramic image marked with the three-level lymphatic structure area through multi-scale characteristic fusion, sampling, convolution and splicing treatment, wherein the panoramic image marked with the three-level lymphatic structure area predicted and output by the first characteristic extraction network is a first digital characteristic, and the panoramic image marked with the three-level lymphatic structure area predicted and output by the second characteristic extraction network is a second digital characteristic.
The beneficial effects of adopting the further scheme are as follows: the sub-network structures of the first feature extraction network and the second feature extraction network provided by the invention enable the network model to be lighter by introducing the lightweight network module, the depth separable rolling and the inverted residual structure, have more advantages in identifying small sample problems of the type in the three-level lymphoid structure area, realize accurately identifying the three-level lymphoid structure area by acquiring and analyzing the features of the dyed cell nuclei DAPI, the T cell CD3, the B cell CD20 and the follicular cell CD21, and can complete the prediction of the three-level lymphoid structure area in the panorama based on the dyed slice under the condition of the small sample.
Further, the lightweight network module is of a straight-barrel structure and comprises a convolution sub-module, a first residual pouring sub-module, a second residual pouring sub-module, a third residual pouring sub-module, a fourth residual pouring sub-module, a fifth residual pouring sub-module, a sixth residual pouring sub-module and a seventh residual pouring sub-module which are connected in sequence;
the input end of the convolution sub-module is the input end of the image feature extraction sub-network and is used for correspondingly inputting the panoramic film of the first dyed slice or the panoramic film of the second dyed slice; the convolution kernel of the convolution sub-module is 3 multiplied by 3, the step length is 2, and the channel number is 32; the step length of the first reverse residual sub-module is 1, and the channel number is 16; the second residual pouring submodule comprises a first residual pouring unit with a step length of 2 and a channel number of 24 and a second residual pouring unit with a step length of 1 and a channel number of 24 which are connected in sequence; the third residual pouring submodule comprises a 3 rd residual pouring unit with the step length of 2 and the channel number of 32, a fourth residual pouring unit with the step length of 1 and the channel number of 32 and a fifth residual pouring unit which are connected in sequence; the fourth residual pouring submodule comprises a sixth residual pouring unit, a seventh residual pouring unit, an eighth residual pouring unit and a ninth residual pouring unit which are sequentially connected, wherein the step length is 1, and the channel number is 64; the fifth residual pouring submodule comprises a tenth residual pouring unit, an eleventh residual pouring unit and a twelfth residual pouring unit which are sequentially connected, wherein the step length is 1, and the channel number is 96; the sixth residual pouring submodule comprises a thirteenth residual pouring unit, a fourteenth residual pouring unit and a fifteenth residual pouring unit which are sequentially connected, wherein the step length is 1, and the channel number is 160; the seventh reverse residual sub-module is an output end of the lightweight network module; the step length of the seventh reverse residual sub-module is 1, and the channel number is 320; the lightweight network module outputs three-level lymphatic structure image characteristic information to the pyramid module and the secondary downsampling module;
The pyramid module convolves the three-level lymphatic structure image characteristic information by adopting parallel 1X 1 convolution, 3 groups of depth separable convolutions with void ratios of 6, 12 and 18 respectively and a global pooling layer to obtain characteristics with different receptive fields, and carries out 1X 1 convolution operation on the characteristics of each different receptive field to obtain multi-scale characteristic information of the three-level lymphatic structure;
the characteristic fusion prediction output sub-network comprises a multi-scale characteristic fusion module, a secondary up-sampling module, a secondary down-sampling module, a low-level characteristic module, a Concat splicing module, a convolution module and a four-time up-sampling module;
the multi-scale feature fusion module compresses the multi-scale feature information of the three-level lymphatic structure through convolution of 1 multiplied by 1, and carries out up-sampling for 2 times on the channel compression result sequentially through the secondary up-sampling module to obtain up-sampled three-level lymphatic structure feature information;
the secondary downsampling module performs 2 times downsampling on the three-level lymphatic structure image characteristic information to obtain low-level characteristic information of the three-level lymphatic structure image characteristic information, and performs 1X 1 convolution operation on the low-level characteristic information through the low-level characteristic module to obtain downsampled three-level lymphatic structure characteristic information;
The up-sampling three-level lymph structure characteristic information and the down-sampling three-level lymph structure characteristic information are subjected to Concat splicing through a Concat splicing module, so that spliced three-level lymph structure characteristic information is obtained;
the convolution module carries out channel compression on the spliced three-level lymphatic structure characteristic information through 3 multiplied by 3 convolution to obtain an initial predicted three-level lymphatic structure region;
and the four-time upsampling module performs 4-time upsampling on the initial predicted tertiary lymphoid structure area through a plurality of convolutions to obtain a panoramic image with the tertiary lymphoid structure area marked with the same size as the input panoramic sheet.
The beneficial effects of adopting the further scheme are as follows: the invention provides a specific structure of a feature extraction network for extracting a first digitized feature or a second digitized feature, and realizes high-accuracy, low-cost and rapid and successful prediction of a three-level lymphatic structure area in a panoramic image through repeated sampling, compression, splicing and convolution processing of image feature information and multi-scale feature information thereof.
Further, the first digitization characteristic and the second digitization characteristic are areas positive in nuclear staining of at least two indexes through DAPI, wherein the areas are three chromogenic indexes of T cell CD3, B cell CD20 and follicular cell CD 21.
The beneficial effects of adopting the further scheme are as follows: the invention provides a method for identifying tertiary lymphoid structure areas, namely the first digital characteristic and the second digital characteristic, which are used for identifying and dividing the staining result areas of cell nuclei DAPI, T cell CD3, B cell CD20 and follicular cell CD21, so that the tertiary lymphoid structure areas can be effectively identified.
Further, the step S6 includes the steps of:
s61, respectively carrying out aggregation trend analysis on positive T cell CD3, B cell CD20 and follicular cell CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence image to be identified to obtain cell aggregation trend information in each marked tertiary lymphocyte structure region;
s62, respectively classifying tissue structures of positive T cells CD3, B cells CD20 and follicular cells CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence map to be identified to obtain a plurality of tertiary lymphocyte structure regions with a first type of tissue structures and a plurality of tertiary lymphocyte structure regions with a second type of tissue structures;
s63, respectively carrying out tissue structure form analysis on positive T cell CD3, B cell CD20 and follicular cell CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence image to be identified, and obtaining various cell quantity information in each marked tertiary lymphocyte structure region;
S64, respectively carrying out interaction analysis on positive T cell CD3, B cell CD20 and follicular cell CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence image to be identified to obtain distance information among various cells in each marked tertiary lymphocyte structure region;
s65, based on the marked cell aggregation trend information, the number information of various cells and the distance information among various cells in each tertiary lymphoid structure area, the tertiary lymphoid structure maturity recognition is carried out on each tertiary lymphoid structure area with the first type of tissue structure and each tertiary lymphoid structure area with the second type of tissue structure, and a tertiary lymphoid structure maturity recognition result is obtained.
The beneficial effects of adopting the further scheme are as follows: the invention provides a characteristic analysis content and a characteristic analysis method for identifying the maturity of a tertiary lymphoid structure, realizes analysis of multiple aspects including cell focusing trend, tissue structure classification, tissue structure form, interaction and the like, provides specific analysis content and to-be-judged division characteristics for accurately identifying a tertiary lymphoid structure area, such as cell aggregation trend information, intercellular tissue structure information, various cell quantity information and various intercellular distance information, and can realize rapid acquisition of the analysis results by adopting multicolor fluorescence staining.
Further, the first tissue structure in S62 includes positive follicular cell CD21, positive B cell CD20, and positive T cell CD3;
the second tissue structure in S62 is a structure comprising only positive B cell CD20 and positive T cell CD3.
The beneficial effects of adopting the further scheme are as follows: the invention provides the cell tissue structure characteristics of the first tissue structure and the second tissue structure, provides a large-class identification division judgment basis for the maturity analysis of the tertiary lymphoid structure area, and can rapidly acquire the first tissue structure and the second tissue structure based on the multicolor fluorescence staining result.
Further, the step S65 includes the following steps:
s651, aiming at three-level lymphoid structure areas of the first tissue structure, if the number of positive follicular cells CD21 in the area exceeds a first preset cell number threshold, the three-level lymphoid structure areas of the first tissue structure are mature three-level lymphoid structure areas, wherein the value range of the first preset cell number threshold is 20-28, and preferably 25;
s652, aiming at the tertiary lymphoid structure area of each first type of tissue structure, if the number of the positive follicular cells CD21 in the area does not exceed a first preset cell number threshold, judging whether the number of the positive T cells CD3 and the number of the positive B cells CD20 in the area are respectively in a remarkable aggregation trend and larger than a second preset cell number threshold, wherein the average distance between the T cells CD3 and the average distance between the B cells CD20 do not exceed a first preset distance threshold, if so, the tertiary lymphoid structure area of the first type of tissue structure is a mature tertiary lymphoid structure area, otherwise, the tertiary lymphoid structure area of the first type of tissue structure is an immature tertiary lymphoid structure area, wherein the range of the second preset cell number threshold is 95-105, the range of the first preset distance threshold is 6-10 mu m, the second preset cell number threshold is preferably 100, and the first preset distance threshold is preferably 8 mu m;
S653, aiming at the tertiary lymphoid structure area of each second type of tissue structure, if positive T cells CD3 and B cells CD20 in the area are in aggregation trend, judging whether the numbers of the T cells CD3 and the B cells CD20 are larger than a second preset cell number threshold value, and the average distance between the T cells CD3 and the average distance between the B cells CD20 are not larger than a first preset distance threshold value, if so, the tertiary lymphoid structure area of the second type of tissue structure is a mature tertiary lymphoid structure area, otherwise, the tertiary lymphoid structure area of the second type of tissue structure is an immature tertiary lymphoid structure area;
s654, aiming at the tertiary lymphoid structure area of each second type of tissue structure, if the positive T cell CD3 and B cell CD20 in the area do not have aggregation trend, the tertiary lymphoid structure area of the second type of tissue structure is an immature tertiary lymphoid structure area.
The beneficial effects of adopting the further scheme are as follows: the invention provides a specific method for identifying the maturity of the tertiary lymph structure area based on the acquired cell aggregation trend information, various cell number information and various cell distance information, and realizes the rapid and accurate identification of the mature tertiary lymph structure area based on multicolor fluorescent staining and image identification results.
Other advantages that are also present with respect to the present invention will be more detailed in the following examples.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart showing steps of a method for identifying maturity of tertiary lymphoid structure based on polychromatic immunofluorescence according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of the fluorescent staining and fusion of positive T cell CD3, B cell CD20 and follicular cell CD21 in the embodiment of the present invention, wherein (a) is a schematic diagram of the fluorescent staining of positive T cell CD3 and orange, (B) is a schematic diagram of the fluorescent staining of positive B cell CD20 and red, (c) is a schematic diagram of the fluorescent staining of positive follicular cell CD21 and (d) is a schematic diagram of the fluorescent staining and fusion of positive T cell CD3, B cell CD20 and follicular cell CD 21.
Fig. 3 is a schematic diagram of a feature extraction network for extracting a first digitized feature or a second digitized feature according to an embodiment of the invention.
FIG. 4 is a schematic representation of tertiary lymphoid structure area extracted by a first feature extraction network according to an embodiment of the present invention.
FIG. 5 is a comparative schematic of the aggregation tendency of cells according to the embodiment of the present invention, wherein (a) is a schematic of the aggregation tendency of cells, and (b) is a schematic of the discrete non-aggregation tendency of cells.
FIG. 6 is a schematic diagram of a cell interaction analysis in an embodiment of the present invention.
FIG. 7 is a graph showing the comparison of images of multicolor immunofluorescent staining with images of TLS maturity recognition of the completed tertiary lymphoid structure region according to the present invention, wherein (a) is a schematic representation of multicolor immunofluorescent staining and (b) is a schematic representation of the recognized mature tertiary lymphoid structure region.
Wherein, A, the first tertiary lymphoid structure area; B. a second tertiary lymphoid structure region.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present invention.
As shown in fig. 1, in one embodiment of the present invention, the present invention provides a method for identifying the maturity of a tertiary lymphoid structure based on polychromatic immunofluorescence, comprising the steps of:
s1, obtaining paraffin sample sections of tumor patients;
s2, staining a paraffin sample slice of a tumor patient to obtain a first stained slice and a second stained slice;
the step S2 comprises the following steps:
s21, slicing the paraffin sample according to a preset thickness, and randomly selecting two continuous slices; in this embodiment, the preset thickness is 4The method comprises the steps of carrying out a first treatment on the surface of the The two continuous slices ensure the structural consistency in the two slices to the greatest extent;
s22, carrying out multicolor immunofluorescence staining on the selected first section to obtain a first stained section;
as shown in fig. 2, the first section is a schematic diagram of multicolor immunofluorescent staining according to the present invention, wherein a) is a schematic diagram of positive T cell CD3 stained with orange fluorescent staining, (B) is a schematic diagram of positive B cell CD20 stained with red fluorescent staining, (c) is a schematic diagram of positive follicular cell CD21 stained with yellow fluorescent staining, and (d) is a schematic diagram of positive T cell CD3, B cell CD20 and follicular cell CD21 stained with fluorescent staining and fused.
S23, hematoxylin-eosin staining is carried out on the selected second section to obtain a second stained section.
S3, observing the second dyed slice, judging whether a crack exists in the second dyed slice, returning to S1 if the crack exists in the second dyed slice, and entering S4 if the crack exists in the second dyed slice;
s4, carrying out panoramic scanning and feature registration on the first dyed slice and the second dyed slice to obtain a first full-view slice and a second full-view slice after feature registration;
the matching features adopted in the feature registration in the step S4 comprise tissue outer edge shape features, tissue inner blood vessel shapes, cavity shapes and cell cluster shape features.
S5, three-level lymphoid structure areas of the first full-view picture and the second full-view picture are screened out through error feature areas based on the first feature extraction network and the second feature extraction network which correspond to the extracted features after registration, and a multicolor immunofluorescence image to be identified, which is marked with a plurality of three-level lymphoid structure areas, is obtained;
the step S5 comprises the following steps:
s51, acquiring a dyed slice full-view film dataset, wherein the dyed slice full-view film dataset comprises a first dyed slice panoramic film sub-dataset formed by a plurality of first dyed slice panoramic films and a second dyed slice panoramic film sub-dataset formed by a plurality of second dyed slice panoramic films;
S52, respectively constructing a first feature extraction network and a second feature extraction network;
the first feature extraction network and the second feature extraction network in S52 each include:
the image feature extraction sub-network is used for receiving the full-view image of the first dyed slice or the full-view image of the second dyed slice and acquiring three-level lymphatic structure image feature information in the full-view image of the first dyed slice or the full-view image of the second dyed slice through the lightweight network module;
the multi-scale feature extraction sub-network is used for receiving the three-level lymphatic structure image feature information and extracting the multi-scale feature information of the three-level lymphatic structure in the three-level lymphatic structure image feature information through the pyramid module;
the characteristic fusion prediction output sub-network is used for receiving the three-level lymphatic structure image characteristic information and the multi-scale characteristic information of the three-level lymphatic structure, and predicting and outputting a panoramic image marked with the three-level lymphatic structure area through multi-scale characteristic fusion, sampling, convolution and splicing treatment, wherein the panoramic image marked with the three-level lymphatic structure area predicted and output by the first characteristic extraction network is a first digital characteristic, and the panoramic image marked with the three-level lymphatic structure area predicted and output by the second characteristic extraction network is a second digital characteristic.
As shown in fig. 3, the lightweight network module is in a straight-barrel structure, and comprises a convolution sub-module, a first residual pouring sub-module, a second residual pouring sub-module, a third residual pouring sub-module, a fourth residual pouring sub-module, a fifth residual pouring sub-module, a sixth residual pouring sub-module and a seventh residual pouring sub-module which are sequentially connected;
the input end of the convolution sub-module is the input end of the image feature extraction sub-network and is used for correspondingly inputting the panoramic film of the first dyed slice or the panoramic film of the second dyed slice; the convolution kernel of the convolution sub-module is 3 multiplied by 3, the step length is 2, and the channel number is 32; the step length of the first reverse residual sub-module is 1, and the channel number is 16; the second residual pouring submodule comprises a first residual pouring unit with a step length of 2 and a channel number of 24 and a second residual pouring unit with a step length of 1 and a channel number of 24 which are connected in sequence; the third residual pouring submodule comprises a 3 rd residual pouring unit with the step length of 2 and the channel number of 32, a fourth residual pouring unit with the step length of 1 and the channel number of 32 and a fifth residual pouring unit which are connected in sequence; the fourth residual pouring submodule comprises a sixth residual pouring unit, a seventh residual pouring unit, an eighth residual pouring unit and a ninth residual pouring unit which are sequentially connected, wherein the step length is 1, and the channel number is 64; the fifth residual pouring submodule comprises a tenth residual pouring unit, an eleventh residual pouring unit and a twelfth residual pouring unit which are sequentially connected, wherein the step length is 1, and the channel number is 96; the sixth residual pouring submodule comprises a thirteenth residual pouring unit, a fourteenth residual pouring unit and a fifteenth residual pouring unit which are sequentially connected, wherein the step length is 1, and the channel number is 160; the seventh reverse residual sub-module is an output end of the lightweight network module; the step length of the seventh reverse residual sub-module is 1, and the channel number is 320; the lightweight network module outputs three-level lymphatic structure image characteristic information to the pyramid module and the secondary downsampling module;
The pyramid module convolves the three-level lymphatic structure image characteristic information by adopting parallel 1X 1 convolution, 3 groups of depth separable convolutions with void ratios of 6, 12 and 18 respectively and a global pooling layer to obtain characteristics with different receptive fields, and carries out 1X 1 convolution operation on the characteristics of each different receptive field to obtain multi-scale characteristic information of the three-level lymphatic structure;
the characteristic fusion prediction output sub-network comprises a multi-scale characteristic fusion module, a secondary up-sampling module, a secondary down-sampling module, a low-level characteristic module, a Concat splicing module, a convolution module and a four-time up-sampling module;
the multi-scale feature fusion module compresses the multi-scale feature information of the three-level lymphatic structure through convolution of 1 multiplied by 1, and carries out up-sampling for 2 times on the channel compression result sequentially through the secondary up-sampling module to obtain up-sampled three-level lymphatic structure feature information;
the secondary downsampling module performs 2 times downsampling on the three-level lymphatic structure image characteristic information to obtain low-level characteristic information of the three-level lymphatic structure image characteristic information, and performs 1X 1 convolution operation on the low-level characteristic information through the low-level characteristic module to obtain downsampled three-level lymphatic structure characteristic information;
The up-sampling three-level lymph structure characteristic information and the down-sampling three-level lymph structure characteristic information are subjected to Concat splicing through a Concat splicing module, so that spliced three-level lymph structure characteristic information is obtained;
the convolution module carries out channel compression on the spliced three-level lymphatic structure characteristic information through 3 multiplied by 3 convolution to obtain an initial predicted three-level lymphatic structure region;
and the four-time upsampling module performs 4-time upsampling on the initial predicted tertiary lymphoid structure area through a plurality of convolutions to obtain a panoramic image with the tertiary lymphoid structure area marked with the same size as the input panoramic sheet.
S53, training a first feature extraction network by using the first dyed slice panoramic sheet data set to obtain a trained first feature extraction network;
s54, training a second feature extraction network by using the second dyeing slice full-scene data set to obtain a trained second feature extraction network;
s55, extracting a three-level lymphatic structure region of the first full-scene film after feature registration by using the trained first feature extraction network, and taking the three-level lymphatic structure region as a first digital feature;
as shown in the figure4, which is a schematic diagram of providing a first digitized feature in the present embodiment, extracting the first digitized feature through a first feature extraction network, where the first digitized feature is a tertiary lymphoid structure area marked by a dashed circle, and a and B are a first tertiary lymphoid structure area and a second tertiary lymphoid structure area marked by a dashed circle, respectively; in this example, the first tertiary lymphoid structure region and the second tertiary lymphoid structure region are numbered as TLS-matched HE20 and TLS-matched HE46, respectively, and the areas of the first tertiary lymphoid structure region and the second tertiary lymphoid structure region are 0.1813mm, respectively 2 And 0.1007mm 2 I.e. the area of the extracted tertiary lymphoid structure region numbered 20 is 0.1813mm 2 The area of the tertiary lymphoid structure region numbered 46 is 0.1007mm 2 。
S56, extracting a tertiary lymphoid structure area of the second full-scene film after feature registration by using the trained second feature extraction network, and taking the tertiary lymphoid structure area as a second digital feature;
the first digitization characteristic and the second digitization characteristic are areas with positive nuclear staining of at least two indexes through DAPI, wherein the areas are three color development indexes of T cell CD3, B cell CD20 and follicular cell CD 21. DAPI is a nuclear staining reagent that stains DNA, commonly used for apoptosis detection.
S57, searching an incomplete tertiary lymphatic structure area in the second digital characteristic, and taking the area in the first digital characteristic corresponding to the searched result as an error characteristic;
s58, screening out error features in the first digital features to obtain a multicolor immunofluorescence image to be identified, wherein the multicolor immunofluorescence image is marked with a plurality of tertiary lymphoid structure areas.
S6, identifying the maturity of the tertiary lymphoid structure based on the to-be-used multicolor immunofluorescence graph to obtain an identification result of the maturity of the tertiary lymphoid structure.
The step S6 comprises the following steps:
S61, respectively carrying out aggregation trend analysis on positive T cell CD3, B cell CD20 and follicular cell CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence image to be identified to obtain cell aggregation trend information in each marked tertiary lymphocyte structure region;
as shown in fig. 5, a comparative diagram of the aggregation trend of cells is shown, wherein (a) is a diagram showing the aggregation trend of cells, and (b) is a diagram showing the discrete non-aggregation trend of cells.
S62, respectively classifying tissue structures of positive T cells CD3, B cells CD20 and follicular cells CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence map to be identified to obtain a plurality of tertiary lymphocyte structure regions with a first type of tissue structures and a plurality of tertiary lymphocyte structure regions with a second type of tissue structures;
the first tissue structure in the S62 comprises positive follicular cell CD21, positive B cell CD20 and positive T cell CD3;
the second tissue structure in S62 is a structure comprising only positive B cell CD20 and positive T cell CD3.
S63, respectively carrying out tissue structure form analysis on positive T cell CD3, B cell CD20 and follicular cell CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence image to be identified, and obtaining various cell quantity information in each marked tertiary lymphocyte structure region;
S64, respectively carrying out interaction analysis on positive T cell CD3, B cell CD20 and follicular cell CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence image to be identified to obtain distance information among various cells in each marked tertiary lymphocyte structure region;
as shown in fig. 6, a schematic diagram of interaction between T cell CD3, B cell CD20 and follicular cell CD21 is shown, wherein the coiled follicular substance is a stained cell, and distance information between various cells can be obtained through a connection line between the stained cells;
s65, based on the marked cell aggregation trend information, the number information of various cells and the distance information among various cells in each tertiary lymphoid structure area, the tertiary lymphoid structure maturity recognition is carried out on each tertiary lymphoid structure area with the first type of tissue structure and each tertiary lymphoid structure area with the second type of tissue structure, and a tertiary lymphoid structure maturity recognition result is obtained.
The step S65 includes the steps of:
s651, aiming at each tertiary lymphoid structure area with a first tissue structure, if the number of positive follicular cells CD21 in the area exceeds a first preset cell number threshold value, the tertiary lymphoid structure area with the first tissue structure is a mature tertiary lymphoid structure area; in this embodiment, the first preset cell number threshold is 25;
S652, aiming at three-level lymphoid structure areas with the first tissue structures, if the number of the positive follicular cells CD21 in the area does not exceed a first preset cell number threshold, judging whether the positive T cells CD3 and B cells CD20 in the area respectively have obvious aggregation trend and the number of the positive T cells CD3 and B cells CD20 are respectively larger than a second preset cell number threshold, and the average distance between the T cells CD3 and the average distance between the B cells CD20 do not exceed the first preset distance threshold, if so, the three-level lymphoid structure area with the first tissue structures is a mature three-level lymphoid structure area, otherwise, the three-level lymphoid structure area with the first tissue structures is an immature three-level lymphoid structure area; in this embodiment, the second preset cell number threshold is 100, and the first preset distance threshold is 8 μm;
s653, aiming at the tertiary lymphoid structure area of each second type of tissue structure, if positive T cells CD3 and B cells CD20 in the area are in aggregation trend, judging whether the numbers of the T cells CD3 and the B cells CD20 are larger than a second preset cell number threshold value, and the average distance between the T cells CD3 and the average distance between the B cells CD20 are not larger than a first preset distance threshold value, if so, the tertiary lymphoid structure area of the second type of tissue structure is a mature tertiary lymphoid structure area, otherwise, the tertiary lymphoid structure area of the second type of tissue structure is an immature tertiary lymphoid structure area;
S654, aiming at the tertiary lymphoid structure area of each second type of tissue structure, if the positive T cell CD3 and B cell CD20 in the area do not have aggregation trend, the tertiary lymphoid structure area of the second type of tissue structure is an immature tertiary lymphoid structure area.
As shown in fig. 7, the image of multicolor immunofluorescent staining is compared with the TLS maturity identification image of the completed tertiary lymphoid structure area in the embodiment of the present invention, wherein (a) is a first stained section obtained by multicolor immunofluorescent staining, and (b) is a mature tertiary lymphoid structure area identified by the multicolor immunofluorescent-based tertiary lymphoid structure maturity identification method provided by the present invention.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention.
Claims (5)
1. The three-level lymphoid structure maturity identification method based on multicolor immunofluorescence is characterized by comprising the following steps of:
s1, obtaining paraffin sample sections of tumor patients;
S2, staining a paraffin sample slice of a tumor patient to obtain a first stained slice and a second stained slice;
the step S2 comprises the following steps:
s21, slicing the paraffin sample according to a preset thickness, and randomly selecting two continuous slices;
s22, carrying out multicolor immunofluorescence staining on the selected first section to obtain a first stained section;
s23, carrying out hematoxylin-eosin staining on the selected second section to obtain a second stained section;
s3, observing the second dyed slice, judging whether a crack exists in the second dyed slice, returning to S1 if the crack exists in the second dyed slice, and entering S4 if the crack exists in the second dyed slice;
s4, carrying out panoramic scanning and feature registration on the first dyed slice and the second dyed slice to obtain a first full-view slice and a second full-view slice after feature registration;
s5, three-level lymphoid structure areas of the first full-view picture and the second full-view picture are screened out through error feature areas based on the first feature extraction network and the second feature extraction network which correspond to the extracted features after registration, and a multicolor immunofluorescence image to be identified, which is marked with a plurality of three-level lymphoid structure areas, is obtained;
the step S5 comprises the following steps:
s51, acquiring a dyed slice full-view film dataset, wherein the dyed slice full-view film dataset comprises a first dyed slice panoramic film sub-dataset formed by a plurality of first dyed slice panoramic films and a second dyed slice panoramic film sub-dataset formed by a plurality of second dyed slice panoramic films;
S52, respectively constructing a first feature extraction network and a second feature extraction network;
s53, training a first feature extraction network by using the first dyed slice panoramic sheet data set to obtain a trained first feature extraction network;
s54, training a second feature extraction network by using the second dyeing slice full-scene data set to obtain a trained second feature extraction network;
s55, extracting a three-level lymphatic structure region of the first full-scene film after feature registration by using the trained first feature extraction network, and taking the three-level lymphatic structure region as a first digital feature;
s56, extracting a tertiary lymphoid structure area of the second full-scene film after feature registration by using the trained second feature extraction network, and taking the tertiary lymphoid structure area as a second digital feature;
s57, searching an incomplete tertiary lymphatic structure area in the second digital characteristic, and taking the area in the first digital characteristic corresponding to the searched result as an error characteristic;
s58, screening error features in the first digital features to obtain a multicolor immunofluorescence image to be identified, wherein the multicolor immunofluorescence image is marked with a plurality of tertiary lymphoid structure areas;
s6, performing tertiary lymphoid structure maturity identification based on the to-be-used multicolor immunofluorescence image to obtain a tertiary lymphoid structure maturity identification result;
The step S6 comprises the following steps:
s61, respectively carrying out aggregation trend analysis on positive T cell CD3, B cell CD20 and follicular cell CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence image to be identified to obtain cell aggregation trend information in each marked tertiary lymphocyte structure region;
s62, respectively classifying tissue structures of positive T cells CD3, B cells CD20 and follicular cells CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence map to be identified to obtain a plurality of tertiary lymphocyte structure regions with a first type of tissue structures and a plurality of tertiary lymphocyte structure regions with a second type of tissue structures;
the first tissue structure in the S62 comprises positive follicular cell CD21, positive B cell CD20 and positive T cell CD3;
the second tissue structure in S62 is that only positive B cell CD20 and positive T cell CD3 are included;
s63, respectively carrying out tissue structure form analysis on positive T cell CD3, B cell CD20 and follicular cell CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence image to be identified, and obtaining various cell quantity information in each marked tertiary lymphocyte structure region;
S64, respectively carrying out interaction analysis on positive T cell CD3, B cell CD20 and follicular cell CD21 in each tertiary lymphocyte structure region marked by the multicolor immunofluorescence image to be identified to obtain distance information among various cells in each marked tertiary lymphocyte structure region;
s65, based on the marked cell aggregation trend information, the cell number information and the distance information among various cells in the three-level lymphoid structure areas, respectively carrying out three-level lymphoid structure maturity identification on the three-level lymphoid structure areas with the first type of tissue structures and the three-level lymphoid structure areas with the second type of tissue structures to obtain three-level lymphoid structure maturity identification results;
the step S65 includes the steps of:
s651, aiming at each tertiary lymphoid structure area with a first tissue structure, if the number of positive follicular cells CD21 in the area exceeds a first preset cell number threshold value, the tertiary lymphoid structure area with the first tissue structure is a mature tertiary lymphoid structure area;
s652, aiming at three-level lymphoid structure areas with the first tissue structures, if the number of the positive follicular cells CD21 in the area does not exceed a first preset cell number threshold, judging whether the positive T cells CD3 and B cells CD20 in the area respectively have obvious aggregation trend and the number of the positive T cells CD3 and B cells CD20 are respectively larger than a second preset cell number threshold, and the average distance between the T cells CD3 and the average distance between the B cells CD20 do not exceed the first preset distance threshold, if so, the three-level lymphoid structure area with the first tissue structures is a mature three-level lymphoid structure area, otherwise, the three-level lymphoid structure area with the first tissue structures is an immature three-level lymphoid structure area;
S653, aiming at the tertiary lymphoid structure area of each second type of tissue structure, if positive T cells CD3 and B cells CD20 in the area are in aggregation trend, judging whether the numbers of the T cells CD3 and the B cells CD20 are larger than a second preset cell number threshold value, and the average distance between the T cells CD3 and the average distance between the B cells CD20 are not larger than a first preset distance threshold value, if so, the tertiary lymphoid structure area of the second type of tissue structure is a mature tertiary lymphoid structure area, otherwise, the tertiary lymphoid structure area of the second type of tissue structure is an immature tertiary lymphoid structure area;
s654, aiming at the tertiary lymphoid structure area of each second type of tissue structure, if the positive T cell CD3 and B cell CD20 in the area do not have aggregation trend, the tertiary lymphoid structure area of the second type of tissue structure is an immature tertiary lymphoid structure area.
2. The method of claim 1, wherein the matching features used in the feature registration in S4 include tissue outer edge shape features, tissue inner vessel shape, cavity shape and cell cluster shape features.
3. The method of claim 1, wherein the first and second feature extraction networks in S52 each comprise:
the image feature extraction sub-network is used for receiving the full-view image of the first dyed slice or the full-view image of the second dyed slice and acquiring three-level lymphatic structure image feature information in the full-view image of the first dyed slice or the full-view image of the second dyed slice through the lightweight network module;
the multi-scale feature extraction sub-network is used for receiving the three-level lymphatic structure image feature information and extracting the multi-scale feature information of the three-level lymphatic structure in the three-level lymphatic structure image feature information through the pyramid module;
the characteristic fusion prediction output sub-network is used for receiving the three-level lymphatic structure image characteristic information and the multi-scale characteristic information of the three-level lymphatic structure, and predicting and outputting a panoramic image marked with the three-level lymphatic structure area through multi-scale characteristic fusion, sampling, convolution and splicing treatment, wherein the panoramic image marked with the three-level lymphatic structure area predicted and output by the first characteristic extraction network is a first digital characteristic, and the panoramic image marked with the three-level lymphatic structure area predicted and output by the second characteristic extraction network is a second digital characteristic.
4. The method for identifying the maturity of the three-stage lymphoid structure based on multicolor immunofluorescence according to claim 3, wherein the lightweight network module is of a straight-barrel structure and comprises a convolution sub-module, a first residual-pouring sub-module, a second residual-pouring sub-module, a third residual-pouring sub-module, a fourth residual-pouring sub-module, a fifth residual-pouring sub-module, a sixth residual-pouring sub-module and a seventh residual-pouring sub-module which are sequentially connected;
the input end of the convolution sub-module is the input end of the image feature extraction sub-network and is used for correspondingly inputting the panoramic film of the first dyed slice or the panoramic film of the second dyed slice; the convolution kernel of the convolution sub-module is 3 multiplied by 3, the step length is 2, and the channel number is 32; the step length of the first reverse residual sub-module is 1, and the channel number is 16; the second residual pouring submodule comprises a first residual pouring unit with a step length of 2 and a channel number of 24 and a second residual pouring unit with a step length of 1 and a channel number of 24 which are connected in sequence; the third residual pouring submodule comprises a 3 rd residual pouring unit with the step length of 2 and the channel number of 32, a fourth residual pouring unit with the step length of 1 and the channel number of 32 and a fifth residual pouring unit which are connected in sequence; the fourth residual pouring submodule comprises a sixth residual pouring unit, a seventh residual pouring unit, an eighth residual pouring unit and a ninth residual pouring unit which are sequentially connected, wherein the step length is 1, and the channel number is 64; the fifth residual pouring submodule comprises a tenth residual pouring unit, an eleventh residual pouring unit and a twelfth residual pouring unit which are sequentially connected, wherein the step length is 1, and the channel number is 96; the sixth residual pouring submodule comprises a thirteenth residual pouring unit, a fourteenth residual pouring unit and a fifteenth residual pouring unit which are sequentially connected, wherein the step length is 1, and the channel number is 160; the seventh reverse residual sub-module is an output end of the lightweight network module; the step length of the seventh reverse residual sub-module is 1, and the channel number is 320; the lightweight network module outputs three-level lymphatic structure image characteristic information to the pyramid module and the secondary downsampling module;
The pyramid module convolves the three-level lymphatic structure image characteristic information by adopting parallel 1X 1 convolution, 3 groups of depth separable convolutions with void ratios of 6, 12 and 18 respectively and a global pooling layer to obtain characteristics with different receptive fields, and carries out 1X 1 convolution operation on the characteristics of each different receptive field to obtain multi-scale characteristic information of the three-level lymphatic structure;
the characteristic fusion prediction output sub-network comprises a multi-scale characteristic fusion module, a secondary up-sampling module, a secondary down-sampling module, a low-level characteristic module, a Concat splicing module, a convolution module and a four-time up-sampling module;
the multi-scale feature fusion module compresses the multi-scale feature information of the three-level lymphatic structure through convolution of 1 multiplied by 1, and carries out up-sampling for 2 times on the channel compression result sequentially through the secondary up-sampling module to obtain up-sampled three-level lymphatic structure feature information;
the secondary downsampling module performs 2 times downsampling on the three-level lymphatic structure image characteristic information to obtain low-level characteristic information of the three-level lymphatic structure image characteristic information, and performs 1X 1 convolution operation on the low-level characteristic information through the low-level characteristic module to obtain downsampled three-level lymphatic structure characteristic information;
The up-sampling three-level lymph structure characteristic information and the down-sampling three-level lymph structure characteristic information are subjected to Concat splicing through a Concat splicing module, so that spliced three-level lymph structure characteristic information is obtained;
the convolution module carries out channel compression on the spliced three-level lymphatic structure characteristic information through 3 multiplied by 3 convolution to obtain an initial predicted three-level lymphatic structure region;
and the four-time upsampling module performs 4-time upsampling on the initial predicted tertiary lymphoid structure area through a plurality of convolutions to obtain a panoramic image with the tertiary lymphoid structure area marked with the same size as the input panoramic sheet.
5. The method of claim 1, wherein the first and second digitization features are three color indicators of T cell CD3, B cell CD20 and follicular cell CD21, at least two of which are areas positive for nuclear staining by DAPI.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311714748.3A CN117405644B (en) | 2023-12-14 | 2023-12-14 | Three-level lymphoid structure maturity identification method based on multicolor immunofluorescence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311714748.3A CN117405644B (en) | 2023-12-14 | 2023-12-14 | Three-level lymphoid structure maturity identification method based on multicolor immunofluorescence |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117405644A CN117405644A (en) | 2024-01-16 |
| CN117405644B true CN117405644B (en) | 2024-02-09 |
Family
ID=89494720
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311714748.3A Active CN117405644B (en) | 2023-12-14 | 2023-12-14 | Three-level lymphoid structure maturity identification method based on multicolor immunofluorescence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117405644B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001061626A1 (en) * | 2000-02-15 | 2001-08-23 | Niczyporuk Marek A | Information processing system and method of using same |
| WO2017087847A1 (en) * | 2015-11-20 | 2017-05-26 | Oregon Health & Science University | Multiplex immunohistochemistry image cytometry |
| CN112004459A (en) * | 2018-02-02 | 2020-11-27 | 大学健康网络 | Devices, systems, and methods for tumor visualization and ablation |
| CN114341937A (en) * | 2019-09-05 | 2022-04-12 | 徕卡生物系统成像股份有限公司 | User-assisted iteration of cellular image segmentation |
| CN114841320A (en) * | 2022-05-07 | 2022-08-02 | 西安邮电大学 | Organ automatic segmentation method based on laryngoscope medical image |
| CN116188474A (en) * | 2023-05-05 | 2023-05-30 | 四川省肿瘤医院 | Three-level lymphatic structure identification method and system based on image semantic segmentation |
| CN116400080A (en) * | 2023-03-28 | 2023-07-07 | 复旦大学附属肿瘤医院 | Application of tertiary lymphoid structure in preparation of kidney cancer parting diagnosis or prognosis evaluation |
| CN116912240A (en) * | 2023-09-11 | 2023-10-20 | 南京理工大学 | Mutation TP53 immunology detection method based on semi-supervised learning |
-
2023
- 2023-12-14 CN CN202311714748.3A patent/CN117405644B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001061626A1 (en) * | 2000-02-15 | 2001-08-23 | Niczyporuk Marek A | Information processing system and method of using same |
| WO2017087847A1 (en) * | 2015-11-20 | 2017-05-26 | Oregon Health & Science University | Multiplex immunohistochemistry image cytometry |
| CN112004459A (en) * | 2018-02-02 | 2020-11-27 | 大学健康网络 | Devices, systems, and methods for tumor visualization and ablation |
| CN114341937A (en) * | 2019-09-05 | 2022-04-12 | 徕卡生物系统成像股份有限公司 | User-assisted iteration of cellular image segmentation |
| CN114841320A (en) * | 2022-05-07 | 2022-08-02 | 西安邮电大学 | Organ automatic segmentation method based on laryngoscope medical image |
| CN116400080A (en) * | 2023-03-28 | 2023-07-07 | 复旦大学附属肿瘤医院 | Application of tertiary lymphoid structure in preparation of kidney cancer parting diagnosis or prognosis evaluation |
| CN116188474A (en) * | 2023-05-05 | 2023-05-30 | 四川省肿瘤医院 | Three-level lymphatic structure identification method and system based on image semantic segmentation |
| CN116912240A (en) * | 2023-09-11 | 2023-10-20 | 南京理工大学 | Mutation TP53 immunology detection method based on semi-supervised learning |
Non-Patent Citations (2)
| Title |
|---|
| Image-based correction of continuous and discontinuous non-planar axial distortion in serial section microscopy;Hanslovsky P等;Bioinformatics;20171231;第 33 卷(第 9 期);第1379-1386页 * |
| 面向深度学习的胰腺医学图像分割方法研究进展;曹路洋等;小型微型计算机系统;20221231;第 43 卷(第 12 期);第2591-2604页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117405644A (en) | 2024-01-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109447977B (en) | Visual defect detection method based on multispectral deep convolutional neural network | |
| Vaickus et al. | Automating the Paris System for urine cytopathology—A hybrid deep‐learning and morphometric approach | |
| JP5113227B2 (en) | Image pattern recognition system and method | |
| US10586376B2 (en) | Automated method of predicting efficacy of immunotherapy approaches | |
| CN110473167B (en) | Deep learning-based urinary sediment image recognition system and method | |
| JP2013541767A (en) | System and method for digital evaluation of cell block preparations | |
| Chibuta et al. | Real-time malaria parasite screening in thick blood smears for low-resource setting | |
| Parab et al. | Red blood cell classification using image processing and CNN | |
| Shu et al. | Artificial‐intelligence‐enabled reagent‐free imaging hematology analyzer | |
| CN1140498A (en) | Automatic cytological specimen classification system and method | |
| CN113222944B (en) | Cell nucleus segmentation method and cancer auxiliary analysis system and device based on pathological image | |
| Lin et al. | Digital pathology and artificial intelligence as the next chapter in diagnostic hematopathology | |
| Mattie et al. | PathMaster: content-based cell image retrieval using automated feature extraction | |
| CN117405644B (en) | Three-level lymphoid structure maturity identification method based on multicolor immunofluorescence | |
| EP3729053B1 (en) | Fast and robust fourier domain-based cell differentiation | |
| JP2003500664A (en) | Methods and systems for universal analysis of experimental data | |
| Frost | I. Cellular Morphology Bespeaks Biologic Behavior | |
| Liu et al. | Fast noninvasive morphometric characterization of free human sperms using deep learning | |
| US7257243B2 (en) | Method for analyzing a biological sample | |
| Jagannadh et al. | Microfluidic microscopy-assisted label-free approach for cancer screening: automated microfluidic cytology for cancer screening | |
| Laosai et al. | Deep-Learning-based Acute Leukemia classification using imaging flow cytometry and morphology | |
| CN115050024B (en) | Intelligent real-time identification method and system for granulocytes with interpretability | |
| Koudounas et al. | Three‐dimensional tissue volume generation in conventional brightfield microscopy | |
| Zordan et al. | Cellular image classification workflow for real-time image based sort decisions | |
| Chernyshova | Criteria and Method for Detection of Circulating Tumor Cells |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |