CN116434226B - Circulating tumor cell analyzer - Google Patents
Circulating tumor cell analyzer Download PDFInfo
- Publication number
- CN116434226B CN116434226B CN202310671528.0A CN202310671528A CN116434226B CN 116434226 B CN116434226 B CN 116434226B CN 202310671528 A CN202310671528 A CN 202310671528A CN 116434226 B CN116434226 B CN 116434226B
- Authority
- CN
- China
- Prior art keywords
- feature
- fish
- circulating tumor
- feature vector
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 208000005443 Circulating Neoplastic Cells Diseases 0.000 title claims abstract description 198
- 239000013598 vector Substances 0.000 claims description 250
- 230000004927 fusion Effects 0.000 claims description 83
- 210000004027 cell Anatomy 0.000 claims description 40
- 239000011159 matrix material Substances 0.000 claims description 35
- 238000001514 detection method Methods 0.000 claims description 31
- 238000010586 diagram Methods 0.000 claims description 25
- 238000010606 normalization Methods 0.000 claims description 21
- 238000000605 extraction Methods 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 15
- 238000005457 optimization Methods 0.000 claims description 14
- 230000011218 segmentation Effects 0.000 claims description 12
- 230000004913 activation Effects 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 4
- 238000011176 pooling Methods 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 4
- 238000009826 distribution Methods 0.000 abstract description 22
- 210000004881 tumor cell Anatomy 0.000 abstract description 21
- 238000004458 analytical method Methods 0.000 abstract description 14
- 238000005516 engineering process Methods 0.000 abstract description 13
- 238000005065 mining Methods 0.000 abstract description 10
- 238000013135 deep learning Methods 0.000 abstract description 8
- 238000013473 artificial intelligence Methods 0.000 abstract description 4
- 238000000034 method Methods 0.000 description 36
- 230000000694 effects Effects 0.000 description 12
- 238000004590 computer program Methods 0.000 description 10
- 238000003753 real-time PCR Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 9
- 230000014509 gene expression Effects 0.000 description 9
- 206010028980 Neoplasm Diseases 0.000 description 8
- 238000013527 convolutional neural network Methods 0.000 description 8
- 125000004122 cyclic group Chemical group 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000008901 benefit Effects 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 5
- 238000007792 addition Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 210000003463 organelle Anatomy 0.000 description 4
- 210000005259 peripheral blood Anatomy 0.000 description 4
- 239000011886 peripheral blood Substances 0.000 description 4
- 238000003860 storage Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 230000003252 repetitive effect Effects 0.000 description 3
- 239000000523 sample Substances 0.000 description 3
- 108020004711 Nucleic Acid Probes Proteins 0.000 description 2
- 210000004369 blood Anatomy 0.000 description 2
- 239000008280 blood Substances 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 210000000349 chromosome Anatomy 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 239000007850 fluorescent dye Substances 0.000 description 2
- 238000009396 hybridization Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 230000000873 masking effect Effects 0.000 description 2
- 230000015654 memory Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000003058 natural language processing Methods 0.000 description 2
- 239000002853 nucleic acid probe Substances 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 206010059866 Drug resistance Diseases 0.000 description 1
- 206010027476 Metastases Diseases 0.000 description 1
- 238000012408 PCR amplification Methods 0.000 description 1
- 241000700605 Viruses Species 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 238000009098 adjuvant therapy Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000010170 biological method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 238000013399 early diagnosis Methods 0.000 description 1
- 238000010195 expression analysis Methods 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 210000000265 leukocyte Anatomy 0.000 description 1
- 238000011528 liquid biopsy Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000009401 metastasis Effects 0.000 description 1
- 230000001394 metastastic effect Effects 0.000 description 1
- 206010061289 metastatic neoplasm Diseases 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002980 postoperative effect Effects 0.000 description 1
- 238000004393 prognosis Methods 0.000 description 1
- 108090000623 proteins and genes Proteins 0.000 description 1
- 102000004169 proteins and genes Human genes 0.000 description 1
- 238000001959 radiotherapy Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 230000008961 swelling Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
- G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
- G01N21/64—Fluorescence; Phosphorescence
- G01N21/6486—Measuring fluorescence of biological material, e.g. DNA, RNA, cells
-
- C—CHEMISTRY; METALLURGY
- C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
- C12Q—MEASURING OR TESTING PROCESSES INVOLVING ENZYMES, NUCLEIC ACIDS OR MICROORGANISMS; COMPOSITIONS OR TEST PAPERS THEREFOR; PROCESSES OF PREPARING SUCH COMPOSITIONS; CONDITION-RESPONSIVE CONTROL IN MICROBIOLOGICAL OR ENZYMOLOGICAL PROCESSES
- C12Q1/00—Measuring or testing processes involving enzymes, nucleic acids or microorganisms; Compositions therefor; Processes of preparing such compositions
- C12Q1/68—Measuring or testing processes involving enzymes, nucleic acids or microorganisms; Compositions therefor; Processes of preparing such compositions involving nucleic acids
- C12Q1/6844—Nucleic acid amplification reactions
- C12Q1/6851—Quantitative amplification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
- G06V20/693—Acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
- G06V20/695—Preprocessing, e.g. image segmentation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Abstract
A circulating tumor cell analyzer that acquires FISH images acquired by a fluorescence microscope; and mining implicit characteristic distribution information about CTC cells in the FISH image by adopting an artificial intelligence technology based on deep learning, and detecting the number of the CTC cells based on the implicit characteristic distribution information of the CTC cells. Thus, the number of the circulating tumor cells can be intelligently counted and detected, and further the analysis of the tumor cells is facilitated.
Description
Technical Field
The present application relates to the field of intelligent analysis technology, and more particularly, to a circulating tumor cell analyzer.
Background
The Circulating Tumor Cells (CTC) are tumor cells which fall off from primary or metastatic lesions and enter peripheral blood, the tumor cells are used as a real-time liquid biopsy means to reflect whether the tumor is invasively metastasized, blood ring tumor cells in blood are directly related to the occurrence and development of cancers, single or small quantity of circulating tumor cell clusters in the peripheral blood are detected and analyzed, and the accuracy and the effectiveness assessment of early diagnosis of the tumor, stage and typing of tumor diagnosis, preoperative assessment and postoperative adjuvant treatment guidance and assessment, patient response to treatment (chemotherapeutics and radiotherapy) assessment, tumor recurrence and metastasis prediction, tumor treatment effect implementation monitoring, tumor individuation accurate treatment guidance, tumor drug resistance monitoring, prognosis judgment and prediction are very critical. However, compared with a large number of white blood cells in peripheral blood, the circulating tumor cells are rare cells in peripheral blood, and detection is very difficult.
Fluorescent quantitative PCR (qPCR) is a molecular biological method based on PCR technology, which can quantitatively determine the amount of DNA, RNA or protein. Unlike conventional PCR techniques, qPCR techniques can monitor the fluorescent signal during PCR amplification in real time, and determine the number of products of a PCR reaction by measuring the intensity of the fluorescent signal. The qPCR technology can be applied to the fields of virus detection, gene expression analysis, SNP typing and the like, and has the advantages of high efficiency, rapidness, sensitivity, accuracy and the like.
However, the current fluorescent quantitative PCR technology needs to detect the product quantity of PCR reaction in a manual observation counting mode, the manual observation mode is inconvenient and high in cost, and has the defects of multiple uncertain factors, high requirements on personnel, large human errors, difficulty in realizing clinical batch and accurate detection and standardized detection, and great limitation on the application of the fluorescent quantitative PCR technology in actual clinic.
Thus, an optimized circulating tumor cell analyzer is desired.
Disclosure of Invention
The present application has been made in order to solve the above technical problems. Embodiments of the present application provide a circulating tumor cell analyzer that acquires FISH images acquired by a fluorescence microscope; and mining implicit characteristic distribution information about CTC cells in the FISH image by adopting an artificial intelligence technology based on deep learning, and detecting the number of the CTC cells based on the implicit characteristic distribution information of the CTC cells. Thus, the number of the circulating tumor cells can be intelligently counted and detected, and further the analysis of the tumor cells is facilitated.
In a first aspect, there is provided a circulating tumor cell analyzer comprising:
the FISH image acquisition module is used for acquiring the FISH image acquired by the fluorescence microscope;
the image feature extraction module is used for enabling the FISH image to pass through an image feature extractor based on a pyramid network to obtain a FISH feature map, wherein the feature map output by the last convolution layer of the pyramid network is the depth FISH feature map;
the dynamic filtering module is used for enabling the FISH feature map to pass through a dynamic filter based on a convolution kernel so as to obtain a filtered FISH feature map;
the feature map expansion module is used for expanding each feature matrix of the filtered FISH feature map along the channel dimension into feature vectors to obtain a sequence of the FISH feature vectors;
the context Wen Yuyi associated coding module is used for enabling the sequence of the FISH feature vectors to pass through a context encoder based on a converter to obtain a plurality of FISH image global context semantic associated feature vectors;
the image semantic segmentation module is used for carrying out image semantic segmentation after the global context semantic association feature vectors of the plurality of FISH images are two-dimensionally arranged into a two-dimensional feature matrix so as to obtain a circulating tumor cell prediction graph;
The first full-connection module is used for enabling the circulating tumor cell prediction graph to pass through the first full-connection layer so as to obtain a circulating tumor cell full-connection feature vector;
the second full-connection module is used for enabling the depth FISH feature map to pass through a second full-connection layer to obtain a FISH depth feature vector;
the feature fusion module is used for fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fusion feature vector;
the feature optimization module is used for carrying out directional distance normalization on the tangential plane of the feature manifold curved surface based on the neighborhood points on the fusion feature vector to obtain an optimized fusion feature vector; and
and the quantity detection module is used for enabling the optimized fusion characteristic vector to pass through a decoder to obtain a decoding value, wherein the decoding value is used for representing the quantity of the circulating tumor cells contained in the FISH image.
In the above-mentioned circulating tumor cell analyzer, the image feature extraction module is configured to: each layer of the pyramid network-based image feature extractor is used for respectively carrying out input data in forward transfer of the layer: carrying out convolution processing on the input data to obtain a convolution characteristic diagram; carrying out mean pooling treatment based on a feature matrix on the convolution feature map to obtain a pooled feature map; performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the pyramid network-based image feature extractor is the FISH feature map, and the input of the first layer of the pyramid network-based image feature extractor is the FISH image.
In the above-mentioned circulating tumor cell analyzer, the upper and lower Wen Yuyi related coding modules comprise: the one-dimensional arrangement unit is used for one-dimensionally arranging the sequence of the FISH feature vectors to obtain global FISH feature vectors; a self-attention unit, configured to calculate a product between the global FISH feature vector and a transpose vector of each FISH feature vector in the sequence of FISH feature vectors to obtain a plurality of self-attention correlation matrices; the normalization unit is used for respectively performing normalization processing on each self-attention correlation matrix in the plurality of self-attention correlation matrices to obtain a plurality of normalized self-attention correlation matrices; the activation unit is used for enabling each normalized self-attention correlation matrix in the normalized self-attention correlation matrices to obtain a plurality of probability values through a Softmax classification function; and a weighting unit, configured to weight each FISH feature vector in the sequence of FISH feature vectors with each probability value in the plurality of probability values as a weight, so as to obtain global context semantic association feature vectors of the plurality of FISH images.
In the above-mentioned circulating tumor cell analyzer, the first full-connection module comprises: the image pixel unfolding unit is used for unfolding the circulating tumor cell prediction graph into a cell prediction one-dimensional pixel feature vector; and the full-connection association coding unit is used for carrying out full-connection coding on the cell prediction one-dimensional pixel feature vector by using the first full-connection layer so as to obtain the circulating tumor cell full-connection feature vector.
In the above-mentioned circulating tumor cell analyzer, the feature fusion module is configured to: fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fusion feature vector by using the following fusion formula; wherein, the fusion formula is:
wherein,for the fusion feature vector,/a. About.>A feature vector is fully connected for the circulating tumor cells,/>for the FISH depth feature vector, +.>Representing addition by position +.>Is a weighting parameter for controlling the balance between the circulating tumor cell full junction feature vector and the FISH depth feature vector.
In the above-mentioned circulating tumor cell analyzer, the feature optimization module is configured to: carrying out directional distance normalization on the tangential plane of the feature manifold curved surface based on the neighborhood points on the fusion feature vector by using the following optimization formula to obtain the optimized fusion feature vector; wherein, the optimization formula is:
wherein,is the +.o of the fusion feature vector>Characteristic value of individual position->And->Is the mean and standard deviation of the respective sets of position feature values in the fusion feature vector, and +.>Is the +.f of the optimized fusion feature vector>Characteristic values of the individual positions.
Swelling in the above cycleIn the tumor cell analyzer, the number detection module is used for: performing decoding regression on the optimized fusion feature vector by using the decoder according to the following decoding formula to obtain the decoding value; wherein, the decoding formula is:wherein->Representing the optimized fusion feature vector, +.>Representing the decoded value->Representing a weight matrix, +.>Representing the bias vector +_>Representing a matrix multiplication.
Compared with the prior art, the circulating tumor cell analyzer provided by the application acquires the FISH image acquired by the fluorescence microscope; and mining implicit characteristic distribution information about CTC cells in the FISH image by adopting an artificial intelligence technology based on deep learning, and detecting the number of the CTC cells based on the implicit characteristic distribution information of the CTC cells. Thus, the number of the circulating tumor cells can be intelligently counted and detected, and further the analysis of the tumor cells is facilitated.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is an application scenario diagram of a circulating tumor cell analyzer according to an embodiment of the present application.
Fig. 2 is a block diagram of a circulating tumor cell analyzer according to an embodiment of the present application.
Fig. 3 is a block diagram of the upper and lower Wen Yuyi associated coding modules in a circulating tumor cell analyzer according to an embodiment of the present application.
Fig. 4 is a block diagram of the first fully connected module in a circulating tumor cell analyzer according to an embodiment of the present application.
Fig. 5 is a flow chart of a method of circulating tumor cell analysis according to an embodiment of the present application.
Fig. 6 is a schematic diagram of a system architecture of a method of circulating tumor cell analysis according to an embodiment of the present application.
Detailed Description
The following description of the technical solutions in the embodiments of the present application will be made with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
Unless defined otherwise, all technical and scientific terms used in the examples of this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to limit the scope of the present application.
In the description of the embodiments of the present application, unless otherwise indicated and defined, the term "connected" should be construed broadly, and for example, may be an electrical connection, may be a communication between two elements, may be a direct connection, or may be an indirect connection via an intermediary, and it will be understood by those skilled in the art that the specific meaning of the term may be understood according to the specific circumstances.
It should be noted that, the term "first\second\third" in the embodiments of the present application is merely to distinguish similar objects, and does not represent a specific order for the objects, it is to be understood that "first\second\third" may interchange a specific order or sequence where allowed. It is to be understood that the "first\second\third" distinguishing objects may be interchanged where appropriate such that the embodiments of the present application described herein may be implemented in sequences other than those illustrated or described herein.
As described above, the current fluorescent quantitative PCR technology needs to detect the product quantity of PCR reaction in a manual observation counting mode, the manual observation mode is inconvenient and high in cost, and has the defects of multiple uncertain factors, high requirements on personnel, large human error, difficult realization of clinical batch and accurate detection and standardized detection, and greatly limits the application of the fluorescent quantitative PCR technology in actual clinic. Thus, an optimized circulating tumor cell analyzer is desired.
It will be appreciated that FISH techniques can utilize fluorescent-labeled specific nucleic acid probes to hybridize to corresponding target DNA molecules or RNA molecules within cells, and by observing fluorescent signals under a fluorescent microscope or confocal laser scanner to determine the morphology and distribution of the stained cells or organelles after hybridization to the specific probes, or the localization of DNA regions or RNA molecules that bind fluorescent probes within chromosomes or other organelles, thereby facilitating the amount of products for subsequent PCR reactions.
Based on this, in the technical solution of the present application, it is desirable to realize the detection of the number of CTC cells by analyzing CTC FISH images acquired by a fluorescence microscope. However, since the characteristic information about CTC cells in the FISH image is a hidden characteristic with a small scale, that is, the proportion of the characteristic information in the occupied image is small, the fluorescence characteristic information is weak, and it is difficult to capture and extract the characteristic information in a traditional manner, so that the data detection accuracy of CTC cells is low. Therefore, in this process, it is difficult to mine implicit characteristic distribution information about CTC cells in the FISH image and perform CTC cell number detection based on the implicit characteristic distribution information of CTC cells.
In recent years, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, text signal processing, and the like. Deep learning and development of neural networks provide new solutions and schemes for mining implicit feature distribution information about CTC cells in the FISH image and for performing the number detection of CTC cells.
Specifically, in the technical solution of the present application, first, FISH images are acquired by a fluorescence microscope. Next, feature mining is performed on the FISH image using a convolutional neural network model having excellent performance in terms of implicit feature extraction, and in particular, focusing on shallow feature information such as appearance, detail, edges, contours, and positions of circulating tumor cells is more required in consideration of not only deep semantic feature information about circulating tumor cells in the FISH image when feature extraction of the FISH image is performed. The pyramid network mainly solves the multi-scale problem in target detection, and can simultaneously utilize the high resolution of low-layer features and the high semantic information of high-layer features to achieve a good effect by fusing the features of different layers. Therefore, in the technical scheme of the application, the FISH image is encoded by an image feature extractor based on a pyramid network to obtain a FISH feature map. In particular, the encoder based on the pyramid network adopts the first to fifth convolution layers with different depths to perform feature mining on the FISH image respectively, so as to extract deep implicit features about circulating tumor cells in the FISH image, and simultaneously retain abundant feature information such as appearance, edge, contour, details, positions and the like of a shallow layer, thereby improving the detection accuracy when the number of circulating tumor cells is detected subsequently, wherein the feature map output by the last convolution layer of the pyramid network is the deep FISH feature map. It should be understood that the pyramid network mainly solves the multi-scale problem in target detection, and can independently detect on different feature layers by simply changing network connection under the condition of basically not increasing the calculation amount of the original model, thereby greatly improving the performance of detecting small targets of the circulating tumor cells.
In the technical scheme of the application, in order to improve the robustness and the anti-interference capability for detecting the number of the circulating tumor cells, the FISH feature map is filtered in a dynamic filter based on a convolution kernel to obtain a filtered FISH feature map.
Then, the correlation of the characteristic distribution information about the circulating tumor cells due to the circulating tumor cells being distributed at the respective positions in the FISH image, that is, at the respective positions in the FISH image, is considered. Because of the inherent limitations of convolution operations, pure CNN methods have difficulty learning explicit global and remote semantic information interactions. Therefore, in the technical solution of the present application, in order to enable the full expression of the implicit feature distribution information of the circulating tumor cells, after each feature matrix of the filtered FISH feature map along the channel dimension is further expanded into feature vectors to obtain a sequence of FISH feature vectors, the sequence of FISH feature vectors is encoded in a context encoder based on a converter, so as to extract the context semantic association feature information based on the image global about the implicit features of the circulating tumor cells at each local position in the FISH image, thereby obtaining a plurality of FISH image global context semantic association feature vectors.
Further, the global context semantic association feature vectors of the multiple FISH images are two-dimensionally arranged into a two-dimensional feature matrix, and then image semantic segmentation is carried out, so that corresponding masking operation is carried out after the circulating tumor cell position area is detected, and a circulating tumor cell prediction graph is obtained. And determining the number of circulating tumor cells contained in the FISH image based on the circulating tumor cell prediction map. That is, specifically, after the circulating tumor cell prediction map is obtained, the circulating tumor cell prediction map and the deep semantic features of the tumor cells are input into a decoder in a combined manner, so as to obtain the number of circulating tumor cells contained in the FISH image. It should be understood that the deep FISH feature map has high-dimensional semantic feature information of tumor cells, and the integration of the feature can effectively count the number of circulating tumor cells.
More specifically, when feature fusion is performed on the circulating tumor cell feature in the circulating tumor cell prediction map and the depth FISH feature map to input the feature into the decoder, it is considered that pixel-related feature information about the circulating tumor cells is provided between each pixel in the circulating tumor cell prediction map. Therefore, in the technical scheme of the application, after the cyclic tumor cell prediction graph is unfolded into the cell prediction one-dimensional pixel feature vector, the full-connection layer is used for encoding so as to extract the associated feature distribution information among the pixel values in the cyclic tumor cell prediction graph, thereby obtaining the cyclic tumor cell full-connection feature vector. And then, the depth FISH characteristic map is also encoded through a full-connection layer, so that the associated characteristic information among all characteristic values in the depth FISH characteristic map is extracted, and the FISH depth characteristic vector is obtained, so that the subsequent detection of the number of circulating tumor cells is facilitated. And then, fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fused feature vector, and carrying out decoding regression on the fused feature vector in a decoder to obtain the number of the circulating tumor cells contained in the FISH image. Specifically, here, after full-connection feature extraction is performed on the circulating tumor cell prediction graph, the full-connection feature extraction is performed on the circulating tumor cell prediction graph and the deep features of the FISH image are spliced and fused to obtain respective spliced features, and the number of prediction statistics is ensured to be larger than 0 through a layer of full-connection and ReLU activation function.
In particular, in the technical solution of the present application, when the fusion feature vector is obtained by fusing the circulating tumor cell full-connection feature vector and the FISH depth feature vector, it is considered that the circulating tumor cell full-connection feature vector expresses the context-associated image semantic feature of the FISH feature map along the channel dimension, and the FISH depth feature vector expresses the global-associated feature of the depth FISH feature map, so, in order to fully utilize the local-global-associated feature of the depth FISH feature map, the fusion feature vector is preferably obtained by directly concatenating the circulating tumor cell full-connection feature vector and the FISH depth feature vector. However, this also results in poor consistency between the individual eigenvalues of the resulting fused eigenvectors, affecting the convergence effect of its decoding regression through the decoder, reducing the training speed of the model.
Thus, in the solution of the present application, the fusion feature vector is represented, for example, asCarrying out tangential plane directional distance normalization of the characteristic manifold curved surface based on the neighborhood points, wherein the tangential plane directional distance normalization concretely comprises the following steps:
wherein the method comprises the steps ofAnd->Is a feature value set +.>Mean and standard deviation of (2), and- >Is the +.f of the optimized fusion feature vector>Characteristic values of the individual positions.
Here, the neighborhood point-based tangent plane directed distance normalization of the feature manifold surface may be performed by the fused feature vectorTo construct a statistical neighborhood based local linear tangent space for each feature value thereof, to orient the feature value by selecting the maximum geometric measure of the tangent vector within the local linear tangent space, and to base orientationNormalized expression of local non-European geometry of points on a manifold surface is performed by inner product distance expression of vectors, so that the fusion feature vector +_is improved by geometric correction of the manifold surface>The expression consistency of each characteristic value of the high-dimensional characteristic set is improved, so that the convergence effect of the optimized fusion characteristic vector through the decoding regression of the decoder can be improved, and the training speed of the model is improved. Thus, the number of the circulating tumor cells can be intelligently counted and detected, and further the analysis of the tumor cells is facilitated.
Fig. 1 is an application scenario diagram of a circulating tumor cell analyzer according to an embodiment of the present application. As shown in fig. 1, in this application scenario, first, FISH images (e.g., C as illustrated in fig. 1) acquired by a fluorescence microscope (e.g., M as illustrated in fig. 1) are acquired; the acquired FISH image is then input into a server (e.g., S as illustrated in fig. 1) deployed with a circulating tumor cell analysis algorithm, wherein the server is capable of processing the FISH image based on the circulating tumor cell analysis algorithm to generate a decoded value representative of the number of circulating tumor cells contained in the FISH image.
Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described in detail with reference to the accompanying drawings.
In one embodiment of the present application, fig. 2 is a block diagram of a circulating tumor cell analyzer according to an embodiment of the present application. As shown in fig. 2, a circulating tumor cell analyzer 100 according to an embodiment of the present application includes: a FISH image acquisition module 101 for acquiring a FISH image acquired by a fluorescence microscope; the image feature extraction module 102 is configured to pass the FISH image through a pyramid network-based image feature extractor to obtain a FISH feature map, where a feature map output by a last convolution layer of the pyramid network is the depth FISH feature map; a dynamic filtering module 103, configured to pass the FISH feature map through a dynamic filter based on a convolution kernel to obtain a filtered FISH feature map; a feature map expansion module 104, configured to expand each feature matrix of the filtered FISH feature map along the channel dimension into feature vectors to obtain a sequence of FISH feature vectors; the context Wen Yuyi associated encoding module 105 is configured to pass the sequence of FISH feature vectors through a context encoder based on a converter to obtain a plurality of FISH image global context semantic associated feature vectors;
The image semantic segmentation module 106 is configured to two-dimensionally arrange the global context semantic association feature vectors of the plurality of FISH images into a two-dimensional feature matrix, and then perform image semantic segmentation to obtain a circulating tumor cell prediction graph; a first full-connection module 107, configured to pass the circulating tumor cell prediction graph through a first full-connection layer to obtain a circulating tumor cell full-connection feature vector;
a second full connection module 108, configured to pass the depth FISH feature map through a second full connection layer to obtain a FISH depth feature vector; the feature fusion module 109 is configured to fuse the circulating tumor cell full-connection feature vector and the FISH depth feature vector to obtain a fused feature vector; the feature optimization module 110 is configured to normalize the directional distance of the tangent plane of the feature manifold curved surface based on the neighborhood point to obtain an optimized fusion feature vector; and a number detection module 111, configured to pass the optimized fusion feature vector through a decoder to obtain a decoded value, where the decoded value is used to represent the number of circulating tumor cells contained in the FISH image.
Specifically, in the embodiment of the present application, the FISH image acquisition module 101 is configured to acquire a FISH image acquired by a fluorescence microscope. As described above, the current fluorescent quantitative PCR technology needs to detect the product quantity of PCR reaction in a manual observation counting mode, the manual observation mode is inconvenient and high in cost, and has the defects of multiple uncertain factors, high requirements on personnel, large human error, difficult realization of clinical batch and accurate detection and standardized detection, and greatly limits the application of the fluorescent quantitative PCR technology in actual clinic. Thus, an optimized circulating tumor cell analyzer is desired.
It will be appreciated that FISH techniques can utilize fluorescent-labeled specific nucleic acid probes to hybridize to corresponding target DNA molecules or RNA molecules within cells, and by observing fluorescent signals under a fluorescent microscope or confocal laser scanner to determine the morphology and distribution of the stained cells or organelles after hybridization to the specific probes, or the localization of DNA regions or RNA molecules that bind fluorescent probes within chromosomes or other organelles, thereby facilitating the amount of products for subsequent PCR reactions.
Based on this, in the technical solution of the present application, it is desirable to realize the detection of the number of CTC cells by analyzing CTC FISH images acquired by a fluorescence microscope. However, since the characteristic information about CTC cells in the FISH image is a hidden characteristic with a small scale, that is, the proportion of the characteristic information in the occupied image is small, the fluorescence characteristic information is weak, and it is difficult to capture and extract the characteristic information in a traditional manner, so that the data detection accuracy of CTC cells is low. Therefore, in this process, it is difficult to mine implicit characteristic distribution information about CTC cells in the FISH image and perform CTC cell number detection based on the implicit characteristic distribution information of CTC cells.
In recent years, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, text signal processing, and the like. Deep learning and development of neural networks provide new solutions and schemes for mining implicit feature distribution information about CTC cells in the FISH image and for performing the number detection of CTC cells.
Specifically, in the technical solution of the present application, first, FISH images are acquired by a fluorescence microscope.
Specifically, in the embodiment of the present application, the image feature extraction module 102 is configured to pass the FISH image through a pyramid network-based image feature extractor to obtain a FISH feature map, where a feature map output by a last convolution layer of the pyramid network is the depth FISH feature map. Next, feature mining is performed on the FISH image using a convolutional neural network model having excellent performance in terms of implicit feature extraction, and in particular, focusing on shallow feature information such as appearance, detail, edges, contours, and positions of circulating tumor cells is more required in consideration of not only deep semantic feature information about circulating tumor cells in the FISH image when feature extraction of the FISH image is performed. The pyramid network mainly solves the multi-scale problem in target detection, and can simultaneously utilize the high resolution of low-layer features and the high semantic information of high-layer features to achieve a good effect by fusing the features of different layers.
Therefore, in the technical scheme of the application, the FISH image is encoded by an image feature extractor based on a pyramid network to obtain a FISH feature map. In particular, the encoder based on the pyramid network adopts the first to fifth convolution layers with different depths to perform feature mining on the FISH image respectively, so as to extract deep implicit features about circulating tumor cells in the FISH image, and simultaneously retain abundant feature information such as appearance, edge, contour, details, positions and the like of a shallow layer, thereby improving the detection accuracy when the number of circulating tumor cells is detected subsequently, wherein the feature map output by the last convolution layer of the pyramid network is the deep FISH feature map. It should be understood that the pyramid network mainly solves the multi-scale problem in target detection, and can independently detect on different feature layers by simply changing network connection under the condition of basically not increasing the calculation amount of the original model, thereby greatly improving the performance of detecting small targets of the circulating tumor cells.
Wherein, the image feature extraction module 102 is configured to: each layer of the pyramid network-based image feature extractor is used for respectively carrying out input data in forward transfer of the layer: carrying out convolution processing on the input data to obtain a convolution characteristic diagram; carrying out mean pooling treatment based on a feature matrix on the convolution feature map to obtain a pooled feature map; performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the pyramid network-based image feature extractor is the FISH feature map, and the input of the first layer of the pyramid network-based image feature extractor is the FISH image.
Specifically, in the embodiment of the present application, the dynamic filtering module 103 is configured to pass the FISH feature map through a dynamic filter based on a convolution kernel to obtain a filtered FISH feature map. In the technical scheme of the application, in order to improve the robustness and the anti-interference capability for detecting the number of the circulating tumor cells, the FISH feature map is filtered in a dynamic filter based on a convolution kernel to obtain a filtered FISH feature map.
Specifically, in the embodiment of the present application, the feature map expansion module 104 and the upper and lower Wen Yuyi association encoding module 105 are configured to expand each feature matrix of the filtered FISH feature map along a channel dimension into feature vectors to obtain a sequence of FISH feature vectors; and the sequence of the FISH feature vectors is used for passing through a context encoder based on a converter to obtain a plurality of FISH image global context semantic association feature vectors.
Then, the correlation of the characteristic distribution information about the circulating tumor cells due to the circulating tumor cells being distributed at the respective positions in the FISH image, that is, at the respective positions in the FISH image, is considered. Because of the inherent limitations of convolution operations, pure CNN methods have difficulty learning explicit global and remote semantic information interactions. Therefore, in the technical solution of the present application, in order to enable the full expression of the implicit feature distribution information of the circulating tumor cells, after each feature matrix of the filtered FISH feature map along the channel dimension is further expanded into feature vectors to obtain a sequence of FISH feature vectors, the sequence of FISH feature vectors is encoded in a context encoder based on a converter, so as to extract the context semantic association feature information based on the image global about the implicit features of the circulating tumor cells at each local position in the FISH image, thereby obtaining a plurality of FISH image global context semantic association feature vectors.
Fig. 3 is a block diagram of the upper and lower Wen Yuyi association code modules in the circulating tumor cell analyzer according to an embodiment of the present application, and as shown in fig. 3, the upper and lower Wen Yuyi association code modules 105 include: a one-dimensional arrangement unit 1051, configured to perform one-dimensional arrangement on the sequence of FISH feature vectors to obtain global FISH feature vectors; a self-attention unit 1052 for calculating the product between the global FISH feature vector and the transpose vector of each FISH feature vector in the sequence of FISH feature vectors to obtain a plurality of self-attention correlation matrices; a normalization unit 1053, configured to perform normalization processing on each of the plurality of self-attention correlation matrices to obtain a plurality of normalized self-attention correlation matrices; an activating unit 1054, configured to obtain a plurality of probability values from each normalized self-attention correlation matrix in the plurality of normalized self-attention correlation matrices by using a Softmax classification function; and a weighting unit 1055, configured to weight each FISH feature vector in the sequence of FISH feature vectors with each probability value in the plurality of probability values as a weight, so as to obtain the plurality of FISH image global context semantic association feature vectors.
The context encoder aims to mine for hidden patterns between contexts in the word sequence, optionally the encoder comprises: CNN (Convolutional Neural Network ), recurrent NN (RecursiveNeural Network, recurrent neural network), language Model (Language Model), and the like. The CNN-based method has a better extraction effect on local features, but has a poor effect on Long-Term Dependency (Long-Term Dependency) problems in sentences, so Bi-LSTM (Long Short-Term Memory) based encoders are widely used. The repetitive NN processes sentences as a tree structure rather than a sequence, has stronger representation capability in theory, but has the weaknesses of high sample marking difficulty, deep gradient disappearance, difficulty in parallel calculation and the like, so that the repetitive NN is less in practical application. The transducer has a network structure with wide application, has the characteristics of CNN and RNN, has a better extraction effect on global characteristics, and has a certain advantage in parallel calculation compared with RNN (RecurrentNeural Network ).
Specifically, in the embodiment of the present application, the image semantic segmentation module 106 is configured to perform image semantic segmentation after two-dimensionally arranging the feature vectors associated with global context semantics of the plurality of FISH images into a two-dimensional feature matrix to obtain a circulating tumor cell prediction graph. Further, the global context semantic association feature vectors of the multiple FISH images are two-dimensionally arranged into a two-dimensional feature matrix, and then image semantic segmentation is carried out, so that corresponding masking operation is carried out after the circulating tumor cell position area is detected, and a circulating tumor cell prediction graph is obtained.
Specifically, in the embodiment of the present application, the first full-connection module 107 and the second full-connection module 108 are configured to pass the circulating tumor cell prediction map through a first full-connection layer to obtain a circulating tumor cell full-connection feature vector; and the depth FISH feature map is used for obtaining a FISH depth feature vector through the second full connection layer. And determining the number of circulating tumor cells contained in the FISH image based on the circulating tumor cell prediction map. That is, specifically, after the circulating tumor cell prediction map is obtained, the circulating tumor cell prediction map and the deep semantic features of the tumor cells are input into a decoder in a combined manner, so as to obtain the number of circulating tumor cells contained in the FISH image. It should be understood that the deep FISH feature map has high-dimensional semantic feature information of tumor cells, and the integration of the feature can effectively count the number of circulating tumor cells.
More specifically, when feature fusion is performed on the circulating tumor cell feature in the circulating tumor cell prediction map and the depth FISH feature map to input the feature into the decoder, it is considered that pixel-related feature information about the circulating tumor cells is provided between each pixel in the circulating tumor cell prediction map. Therefore, in the technical scheme of the application, after the cyclic tumor cell prediction graph is unfolded into the cell prediction one-dimensional pixel feature vector, the full-connection layer is used for encoding so as to extract the associated feature distribution information among the pixel values in the cyclic tumor cell prediction graph, thereby obtaining the cyclic tumor cell full-connection feature vector. And then, the depth FISH characteristic map is also encoded through a full-connection layer, so that the associated characteristic information among all characteristic values in the depth FISH characteristic map is extracted, and the FISH depth characteristic vector is obtained, so that the subsequent detection of the number of circulating tumor cells is facilitated.
Fig. 4 is a block diagram of the first fully-connected module in the circulating tumor cell analyzer according to the embodiment of the present application, as shown in fig. 4, the first fully-connected module 107 includes: an image pixel expansion unit 1071, configured to expand the cyclic tumor cell prediction graph into a cell prediction one-dimensional pixel feature vector; and a full-connection association encoding unit 1072, configured to perform full-connection encoding on the cell prediction one-dimensional pixel feature vector by using the first full-connection layer to obtain the circulating tumor cell full-connection feature vector.
Specifically, in the embodiment of the present application, the feature fusion module 109 is configured to fuse the full-connection feature vector of the circulating tumor cell and the FISH depth feature vector to obtain a fused feature vector. And then, fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fused feature vector, and carrying out decoding regression on the fused feature vector in a decoder to obtain the number of the circulating tumor cells contained in the FISH image. Specifically, here, after full-connection feature extraction is performed on the circulating tumor cell prediction graph, the full-connection feature extraction is performed on the circulating tumor cell prediction graph and the deep features of the FISH image are spliced and fused to obtain respective spliced features, and the number of prediction statistics is ensured to be larger than 0 through a layer of full-connection and ReLU activation function.
Wherein, the feature fusion module 109 is configured to: fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fusion feature vector by using the following fusion formula; wherein, the fusion formula is:
wherein,for the fusion feature vector,/a. About.>For the circulating tumor cells, fully connected feature vectors, -/->For the FISH depth feature vector, +.>Representing addition by position +.>Is a weighting parameter for controlling the balance between the circulating tumor cell full junction feature vector and the FISH depth feature vector.
Specifically, in the embodiment of the present application, the feature optimization module 110 is configured to normalize the directional distance of the tangent plane of the feature manifold curved surface based on the neighborhood point to obtain the optimized fused feature vector. In particular, in the technical solution of the present application, when the fusion feature vector is obtained by fusing the circulating tumor cell full-connection feature vector and the FISH depth feature vector, it is considered that the circulating tumor cell full-connection feature vector expresses the context-associated image semantic feature of the FISH feature map along the channel dimension, and the FISH depth feature vector expresses the global-associated feature of the depth FISH feature map, so, in order to fully utilize the local-global-associated feature of the depth FISH feature map, the fusion feature vector is preferably obtained by directly concatenating the circulating tumor cell full-connection feature vector and the FISH depth feature vector. However, this also results in poor consistency between the individual eigenvalues of the resulting fused eigenvectors, affecting the convergence effect of its decoding regression through the decoder, reducing the training speed of the model.
Thus, in the solution of the present application, the fusion feature vector is represented, for example, asCarrying out tangential plane directional distance normalization of the characteristic manifold curved surface based on the neighborhood points, wherein the tangential plane directional distance normalization concretely comprises the following steps: carrying out directional distance normalization on the tangential plane of the feature manifold curved surface based on the neighborhood points on the fusion feature vector by using the following optimization formula to obtain the optimized fusion feature vector; wherein, the optimization formula is:
wherein,is the +.o of the fusion feature vector>Characteristic value of individual position->And->Is the mean and standard deviation of the respective sets of position feature values in the fusion feature vector, and +.>Is the +.f of the optimized fusion feature vector>Characteristic values of the individual positions.
Here, the neighborhood point-based tangent plane directed distance normalization of the feature manifold surface may be performed by the fused feature vectorTo construct a local linear cut space based on a statistical neighborhood for each of its feature values by averaging the mean and variance of the high-dimensional feature sets of (a) and by computing the mean and variance of the feature sets at the officeThe maximum geometric measure of the tangent vector is selected in the linear tangent space to carry out the vectorization on the characteristic value, and the normalized expression of the local non-European geometric property is carried out on the point on the manifold curved surface based on the inner product distance expression of the orientation vector, so that the fusion characteristic vector is promoted in a manifold curved surface geometric correction mode >The expression consistency of each characteristic value of the high-dimensional characteristic set is improved, so that the convergence effect of the optimized fusion characteristic vector through the decoding regression of the decoder can be improved, and the training speed of the model is improved. Thus, the number of the circulating tumor cells can be intelligently counted and detected, and further the analysis of the tumor cells is facilitated.
Specifically, in the embodiment of the present application, the number detection module 111 is configured to pass the optimized fusion feature vector through a decoder to obtain a decoded value, where the decoded value is used to represent the number of circulating tumor cells contained in the FISH image.
Wherein, the quantity detection module 111 is used for: performing decoding regression on the optimized fusion feature vector by using the decoder according to the following decoding formula to obtain the decoding value; wherein, the decoding formula is:wherein->Representing the optimized fusion feature vector, +.>Representing the decoded value->Representing a weight matrix, +.>Representing the bias vector +_>Representing a matrix multiplication.
In summary, a circulating tumor cell analyzer 100 according to an embodiment of the present application is illustrated that acquires FISH images acquired by a fluorescence microscope; and mining implicit characteristic distribution information about CTC cells in the FISH image by adopting an artificial intelligence technology based on deep learning, and detecting the number of the CTC cells based on the implicit characteristic distribution information of the CTC cells. Thus, the number of the circulating tumor cells can be intelligently counted and detected, and further the analysis of the tumor cells is facilitated.
In one embodiment of the present application, fig. 5 is a flow chart of a method of circulating tumor cell analysis according to an embodiment of the present application. As shown in fig. 5, a method for analyzing circulating tumor cells according to an embodiment of the present application includes: 201, acquiring a FISH image acquired by a fluorescence microscope; 202, passing the FISH image through an image feature extractor based on a pyramid network to obtain a FISH feature map, wherein the feature map output by the last convolution layer of the pyramid network is the depth FISH feature map; 203, passing the FISH feature map through a dynamic filter based on a convolution kernel to obtain a filtered FISH feature map; 204, expanding each feature matrix of the filtered FISH feature map along the channel dimension into feature vectors to obtain a sequence of FISH feature vectors; 205, passing the sequence of FISH feature vectors through a context encoder based on a converter to obtain a plurality of FISH image global context semantic association feature vectors; 206, performing image semantic segmentation after the global context semantic association feature vectors of the FISH images are two-dimensionally arranged into a two-dimensional feature matrix to obtain a circulating tumor cell prediction graph; 207, passing the circulating tumor cell prediction graph through a first full-connection layer to obtain a circulating tumor cell full-connection feature vector; 208, passing the depth FISH feature map through a second full connection layer to obtain a FISH depth feature vector; 209, fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fused feature vector; 210, carrying out tangential plane directional distance normalization on the fusion feature vector based on the neighborhood points on the feature manifold curved surface to obtain an optimized fusion feature vector; and, 211, passing the optimized fusion feature vector through a decoder to obtain a decoding value, wherein the decoding value is used for representing the number of circulating tumor cells contained in the FISH image.
Fig. 6 is a schematic diagram of a system architecture of a method of circulating tumor cell analysis according to an embodiment of the present application. As shown in fig. 6, in the system architecture of the circulating tumor cell analysis method, first, FISH images acquired by a fluorescence microscope are acquired; then, the FISH image passes through an image feature extractor based on a pyramid network to obtain a FISH feature map, wherein the feature map output by the last convolution layer of the pyramid network is the depth FISH feature map; then, the FISH characteristic diagram is passed through a dynamic filter based on convolution kernel to obtain a filtered FISH characteristic diagram; then, expanding each feature matrix of the filtered FISH feature map along the channel dimension into feature vectors to obtain a sequence of the FISH feature vectors; then, the sequence of the FISH feature vectors passes through a context encoder based on a converter to obtain a plurality of FISH image global context semantic association feature vectors; secondly, two-dimensionally arranging the global context semantic association feature vectors of the plurality of FISH images into a two-dimensional feature matrix, and then carrying out image semantic segmentation to obtain a circulating tumor cell prediction graph; then, the circulating tumor cell prediction graph passes through a first full-connection layer to obtain a circulating tumor cell full-connection feature vector; then, the depth FISH feature map passes through a second full-connection layer to obtain a FISH depth feature vector; then, fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fusion feature vector; then, carrying out tangential plane directional distance normalization on the fusion feature vector based on the neighborhood points on the feature manifold curved surface to obtain an optimized fusion feature vector; and finally, the optimized fusion characteristic vector is passed through a decoder to obtain a decoding value, wherein the decoding value is used for representing the number of circulating tumor cells contained in the FISH image.
In a specific example, in the above method for analyzing circulating tumor cells, the step of passing the FISH image through an image feature extractor based on a pyramid network to obtain a FISH feature map, where a feature map output by a last convolution layer of the pyramid network is the depth FISH feature map includes: each layer of the pyramid network-based image feature extractor is used for respectively carrying out input data in forward transfer of the layer: carrying out convolution processing on the input data to obtain a convolution characteristic diagram; carrying out mean pooling treatment based on a feature matrix on the convolution feature map to obtain a pooled feature map; performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the pyramid network-based image feature extractor is the FISH feature map, and the input of the first layer of the pyramid network-based image feature extractor is the FISH image.
In a specific example, in the above method for analyzing circulating tumor cells, passing the sequence of FISH feature vectors through a context encoder based on a converter to obtain a plurality of FISH image global context semantic association feature vectors includes: one-dimensional arrangement is carried out on the sequence of the FISH feature vectors so as to obtain global FISH feature vectors; calculating the product between the global FISH feature vector and the transpose vector of each FISH feature vector in the sequence of the FISH feature vectors to obtain a plurality of self-attention association matrixes; respectively carrying out standardization processing on each self-attention correlation matrix in the plurality of self-attention correlation matrices to obtain a plurality of standardized self-attention correlation matrices; obtaining a plurality of probability values by using a Softmax classification function through each normalized self-attention correlation matrix in the normalized self-attention correlation matrices; and weighting each FISH feature vector in the sequence of FISH feature vectors by taking each probability value in the plurality of probability values as a weight to obtain global context semantic association feature vectors of the plurality of FISH images.
In a specific example, in the above method for analyzing circulating tumor cells, passing the circulating tumor cell prediction map through a first full-junction layer to obtain a circulating tumor cell full-junction feature vector includes: expanding the circulating tumor cell prediction graph into a cell prediction one-dimensional pixel feature vector; and performing full-connection coding on the cell prediction one-dimensional pixel feature vector by using the first full-connection layer to obtain the circulating tumor cell full-connection feature vector.
In a specific example, in the above method for analyzing circulating tumor cells, fusing the circulating tumor cell full-junction feature vector and the FISH depth feature vector to obtain a fused feature vector includes: fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fusion feature vector by using the following fusion formula; wherein, the fusion formula is:
wherein,for the fusion feature vector,/a. About.>For the circulating tumor cells, fully connected feature vectors, -/->For the FISH depth feature vector, +.>Representing addition by position +.>Is a weighting parameter for controlling the balance between the circulating tumor cell full junction feature vector and the FISH depth feature vector.
In a specific example, in the above method for analyzing circulating tumor cells, performing directional distance normalization on the tangential plane of the feature manifold curved surface based on the neighborhood points on the fused feature vector to obtain an optimized fused feature vector, including: carrying out directional distance normalization on the tangential plane of the feature manifold curved surface based on the neighborhood points on the fusion feature vector by using the following optimization formula to obtain the optimized fusion feature vector; wherein, the optimization formula is:
wherein,is the +.o of the fusion feature vector>Characteristic value of individual position->And->Is the mean and standard deviation of the respective sets of position feature values in the fusion feature vector, and +.>Is the +.f of the optimized fusion feature vector>Characteristic values of the individual positions.
In a specific example, in the above method for analyzing circulating tumor cells, the optimizing the fusion feature vector is passed through a decoder to obtain a decoded value, where the decoded value is used to represent the number of circulating tumor cells contained in the FISH image, and the method includes: performing decoding regression on the optimized fusion feature vector by using the decoder according to the following decoding formula to obtain the decoding value; wherein, the decoding formula is: Wherein->Representing the optimized fusion feature vector, +.>Representing the decoded value->Representing a weight matrix, +.>Representing the bias vector +_>Representing a matrix multiplication.
It will be appreciated by those skilled in the art that the specific operation of the respective steps in the above-described method for analyzing circulating tumor cells has been described in detail in the above description of the circulating tumor cell analyzer with reference to fig. 1 to 4, and thus, repetitive descriptions thereof will be omitted.
The present application also provides a computer program product comprising instructions which, when executed, cause an apparatus to perform operations corresponding to the above-described methods.
In one embodiment of the present application, there is also provided a computer readable storage medium storing a computer program for executing the above-described method.
It should be appreciated that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the forms of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects may be utilized. Furthermore, the computer program product may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Methods, systems, and computer program products of embodiments of the present application are described in terms of flow diagrams and/or block diagrams. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The basic principles of the present application have been described above in connection with specific embodiments, however, it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are merely examples and not limiting, and these advantages, benefits, effects, etc. are not to be considered as necessarily possessed by the various embodiments of the present application. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, as the application is not intended to be limited to the details disclosed herein as such.
The block diagrams of the devices, apparatuses, devices, systems referred to in this application are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.
It is also noted that in the apparatus, devices and methods of the present application, the components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered as equivalent to the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the application to the form disclosed herein. Although a number of example aspects and embodiments have been discussed above, a person of ordinary skill in the art will recognize certain variations, modifications, alterations, additions, and subcombinations thereof.
Claims (6)
1. A circulating tumor cell analyzer, comprising:
the FISH image acquisition module is used for acquiring the FISH image acquired by the fluorescence microscope;
the image feature extraction module is used for enabling the FISH image to pass through an image feature extractor based on a pyramid network to obtain a depth FISH feature map, wherein the feature map output by the last convolution layer of the pyramid network is the depth FISH feature map;
the dynamic filtering module is used for enabling the depth FISH feature map to pass through a dynamic filter based on a convolution kernel so as to obtain a filtered FISH feature map;
the feature map expansion module is used for expanding each feature matrix of the filtered FISH feature map along the channel dimension into feature vectors to obtain a sequence of the FISH feature vectors;
the context Wen Yuyi associated coding module is used for enabling the sequence of the FISH feature vectors to pass through a context encoder based on a converter to obtain a plurality of FISH image global context semantic associated feature vectors;
The image semantic segmentation module is used for carrying out image semantic segmentation after the global context semantic association feature vectors of the plurality of FISH images are two-dimensionally arranged into a two-dimensional feature matrix so as to obtain a circulating tumor cell prediction graph;
the first full-connection module is used for enabling the circulating tumor cell prediction graph to pass through the first full-connection layer so as to obtain a circulating tumor cell full-connection feature vector;
the second full-connection module is used for enabling the depth FISH feature map to pass through a second full-connection layer to obtain a FISH depth feature vector;
the feature fusion module is used for fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fusion feature vector;
the feature optimization module is used for carrying out directional distance normalization on the tangential plane of the feature manifold curved surface based on the neighborhood points on the fusion feature vector to obtain an optimized fusion feature vector; and
the quantity detection module is used for enabling the optimized fusion characteristic vector to pass through a decoder to obtain a decoding value, wherein the decoding value is used for representing the quantity of circulating tumor cells contained in the FISH image;
wherein, the characteristic optimization module is used for: carrying out directional distance normalization on the tangential plane of the feature manifold curved surface based on the neighborhood points on the fusion feature vector by using the following optimization formula to obtain the optimized fusion feature vector;
Wherein, the optimization formula is:
，
wherein,is the +.o of the fusion feature vector>Characteristic value of individual position->And->Is the mean and standard deviation of the respective sets of position feature values in the fusion feature vector, and +.>Is the +.f of the optimized fusion feature vector>Characteristic values of the individual positions.
2. The circulating tumor cell analyzer of claim 1, wherein the image feature extraction module is configured to: each layer of the pyramid network-based image feature extractor is used for respectively carrying out input data in forward transfer of the layer:
carrying out convolution processing on the input data to obtain a convolution characteristic diagram;
carrying out mean pooling treatment based on a feature matrix on the convolution feature map to obtain a pooled feature map; and
non-linear activation is carried out on the pooled feature map so as to obtain an activated feature map;
wherein the output of the last layer of the pyramid network-based image feature extractor is the depth FISH feature map, and the input of the first layer of the pyramid network-based image feature extractor is the FISH image.
3. The circulating tumor cell analyzer of claim 2, wherein the upper and lower Wen Yuyi association coding modules comprise:
The one-dimensional arrangement unit is used for one-dimensionally arranging the sequence of the FISH feature vectors to obtain global FISH feature vectors;
a self-attention unit, configured to calculate a product between the global FISH feature vector and a transpose vector of each FISH feature vector in the sequence of FISH feature vectors to obtain a plurality of self-attention correlation matrices;
the normalization unit is used for respectively performing normalization processing on each self-attention correlation matrix in the plurality of self-attention correlation matrices to obtain a plurality of normalized self-attention correlation matrices;
the activation unit is used for enabling each normalized self-attention correlation matrix in the normalized self-attention correlation matrices to obtain a plurality of probability values through a Softmax classification function; and
and the weighting unit is used for weighting each FISH feature vector in the sequence of the FISH feature vectors by taking each probability value in the plurality of probability values as a weight so as to obtain global context semantic association feature vectors of the plurality of FISH images.
4. The circulating tumor cell analyzer of claim 3, wherein the first fully connected module comprises:
the image pixel unfolding unit is used for unfolding the circulating tumor cell prediction graph into a cell prediction one-dimensional pixel feature vector; and
And the full-connection association coding unit is used for carrying out full-connection coding on the cell prediction one-dimensional pixel feature vector by using the first full-connection layer so as to obtain the circulating tumor cell full-connection feature vector.
5. The circulating tumor cell analyzer of claim 4, wherein the feature fusion module is configured to: fusing the full-connection feature vector of the circulating tumor cells and the FISH depth feature vector to obtain a fusion feature vector by using the following fusion formula;
wherein, the fusion formula is:
，
wherein,for the fusion feature vector,/a. About.>For the circulating tumor cells, fully connected feature vectors, -/->For the FISH depth feature vector, +.>Representing addition by position +.>And->Is a weighting parameter for controlling the balance between the circulating tumor cell full junction feature vector and the FISH depth feature vector.
6. The circulating tumor cell analyzer of claim 5, wherein the number detection module is configured to: performing decoding regression on the optimized fusion feature vector by using the decoder according to the following decoding formula to obtain the decoding value;
wherein, the decoding formula is: Wherein->Representing the optimized fusion feature vector, +.>Representing the decoded value->Representing a weight matrix, +.>Representing the bias vector +_>Representing a matrix multiplication.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310671528.0A CN116434226B (en) | 2023-06-08 | 2023-06-08 | Circulating tumor cell analyzer |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310671528.0A CN116434226B (en) | 2023-06-08 | 2023-06-08 | Circulating tumor cell analyzer |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116434226A CN116434226A (en) | 2023-07-14 |
| CN116434226B true CN116434226B (en) | 2024-03-19 |
Family
ID=87085797
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310671528.0A Active CN116434226B (en) | 2023-06-08 | 2023-06-08 | Circulating tumor cell analyzer |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116434226B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117274270B (en) * | 2023-11-23 | 2024-01-26 | 吉林大学 | Digestive endoscope real-time auxiliary system and method based on artificial intelligence |
| CN117710166A (en) * | 2024-02-06 | 2024-03-15 | 吉林大学 | Teaching guidance detection system and method for pediatric neural nursing |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107748256A (en) * | 2017-10-11 | 2018-03-02 | 上海医盈网络科技有限公司 | A kind of liquid biopsy detection method of circulating tumor cell |
| CN109886273A (en) * | 2019-02-26 | 2019-06-14 | 四川大学华西医院 | A kind of CMR classification of image segmentation system |
| CN115398472A (en) * | 2019-12-23 | 2022-11-25 | 健肺生命人工智能公司 | System and method for accurate fluorescence in situ hybridization probe detection of microscopic blood cell images using machine learning design |
| CN115409844A (en) * | 2022-11-02 | 2022-11-29 | 杭州华得森生物技术有限公司 | Circulating tumor cell detection device and method thereof |
| CN115406815A (en) * | 2022-11-02 | 2022-11-29 | 杭州华得森生物技术有限公司 | Tumor cell detection equipment and method based on multi-source data fusion |
| CN115690109A (en) * | 2023-01-04 | 2023-02-03 | 杭州华得森生物技术有限公司 | Tumor cell detection device based on computational organisms and method thereof |
| CN115791640A (en) * | 2023-02-06 | 2023-03-14 | 杭州华得森生物技术有限公司 | Tumor cell detection device and method based on spectroscopic spectrum |
| CN116189179A (en) * | 2023-04-28 | 2023-05-30 | 北京航空航天大学杭州创新研究院 | Circulating tumor cell scanning analysis equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220343623A1 (en) * | 2019-11-28 | 2022-10-27 | Beijing Xiaoying Technology Co., Ltd. | Blood smear full-view intelligent analysis method, and blood cell segmentation model and recognition model construction method |
-
2023
- 2023-06-08 CN CN202310671528.0A patent/CN116434226B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107748256A (en) * | 2017-10-11 | 2018-03-02 | 上海医盈网络科技有限公司 | A kind of liquid biopsy detection method of circulating tumor cell |
| CN109886273A (en) * | 2019-02-26 | 2019-06-14 | 四川大学华西医院 | A kind of CMR classification of image segmentation system |
| CN115398472A (en) * | 2019-12-23 | 2022-11-25 | 健肺生命人工智能公司 | System and method for accurate fluorescence in situ hybridization probe detection of microscopic blood cell images using machine learning design |
| CN115409844A (en) * | 2022-11-02 | 2022-11-29 | 杭州华得森生物技术有限公司 | Circulating tumor cell detection device and method thereof |
| CN115406815A (en) * | 2022-11-02 | 2022-11-29 | 杭州华得森生物技术有限公司 | Tumor cell detection equipment and method based on multi-source data fusion |
| CN115690109A (en) * | 2023-01-04 | 2023-02-03 | 杭州华得森生物技术有限公司 | Tumor cell detection device based on computational organisms and method thereof |
| CN115791640A (en) * | 2023-02-06 | 2023-03-14 | 杭州华得森生物技术有限公司 | Tumor cell detection device and method based on spectroscopic spectrum |
| CN116189179A (en) * | 2023-04-28 | 2023-05-30 | 北京航空航天大学杭州创新研究院 | Circulating tumor cell scanning analysis equipment |
Non-Patent Citations (3)
| Title |
|---|
| SAU-Net: a universal deep network for cell counting;Yue Guo et al.;《The 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics》;20191231;299-306 * |
| 基于多尺度特征融合的细胞计数方法;张倩 等;《图学学报》;20230228;第44卷(第1期);41-49 * |
| 结合实例分割的多目标跟踪研究及系统实现;蒋欣欣;《中国优秀硕士学位论文全文数据库》;20220115;1-98 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116434226A (en) | 2023-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116434226B (en) | Circulating tumor cell analyzer | |
| EP3721406B1 (en) | Method of computing tumor spatial and inter-marker heterogeneity | |
| CN116189179B (en) | Circulating tumor cell scanning analysis equipment | |
| Picon et al. | Deep learning-based segmentation of multiple species of weeds and corn crop using synthetic and real image datasets | |
| CN115410050A (en) | Tumor cell detection equipment based on machine vision and method thereof | |
| CN115526834A (en) | Immunofluorescence image detection method and device, equipment and storage medium | |
| CN114445356A (en) | Multi-resolution-based full-field pathological section image tumor rapid positioning method | |
| Liu et al. | Self-supervised transformer-based pre-training method using latent semantic masking auto-encoder for pest and disease classification | |
| Li et al. | An automatic plant leaf stoma detection method based on YOLOv5 | |
| CN116287138B (en) | FISH-based cell detection system and method thereof | |
| CN113902669A (en) | Method and system for reading urine exfoliative cell fluid-based smear | |
| CN116842996A (en) | Space transcriptome method and device based on depth compressed sensing | |
| Guo et al. | Pathological Detection of Micro and Fuzzy Gastric Cancer Cells Based on Deep Learning. | |
| CN115035339A (en) | Cystoscope image classification method based on artificial intelligence | |
| CN116309543B (en) | Image-based circulating tumor cell detection equipment | |
| Silviya et al. | Deep Learning based Plant Leaf Disease Detection and Classification | |
| CN113012129A (en) | System and device for counting area positioning and marked nerve cells of brain slice image | |
| CN117393043B (en) | Thyroid papilloma BRAF gene mutation detection device | |
| Widneh et al. | Medicinal Plant Parts Identification and Classification using Deep Learning Based on Multi Label Categories | |
| CN116309595B (en) | CTC intelligent full-automatic detection integrated machine and method thereof | |
| Yan et al. | YOLOx model-based object detection for microalgal bioprocess | |
| Sivanandan et al. | Machine learning enabled pooled optical screening in human lung cancer cells | |
| Fida et al. | Leaf Image Processing and Detection: Computer-based Aid for Plant Identification and Classification | |
| Mohana et al. | Revolutionizing Lymphoma Diagnosis with Deep Learning and Natural Language Generation | |
| Shi et al. | Crop plant automatic detecting based on in-field images by lightweight DFU-Net model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |