CN112365980B - Brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method and system - Google Patents
Brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method and system Download PDFInfo
- Publication number
- CN112365980B CN112365980B CN202011279154.0A CN202011279154A CN112365980B CN 112365980 B CN112365980 B CN 112365980B CN 202011279154 A CN202011279154 A CN 202011279154A CN 112365980 B CN112365980 B CN 112365980B
- Authority
- CN
- China
- Prior art keywords
- brain tumor
- treatment
- target
- growth
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 208000003174 Brain Neoplasms Diseases 0.000 title claims abstract description 426
- 238000003745 diagnosis Methods 0.000 title claims abstract description 43
- 238000007794 visualization technique Methods 0.000 title claims abstract description 18
- 230000012010 growth Effects 0.000 claims abstract description 98
- 238000013135 deep learning Methods 0.000 claims abstract description 85
- 238000012800 visualization Methods 0.000 claims abstract description 68
- 238000000034 method Methods 0.000 claims abstract description 42
- 206010028980 Neoplasm Diseases 0.000 claims abstract description 41
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 41
- 238000007781 pre-processing Methods 0.000 claims abstract description 13
- 230000011218 segmentation Effects 0.000 claims abstract description 12
- 238000000605 extraction Methods 0.000 claims abstract description 10
- 230000004927 fusion Effects 0.000 claims abstract description 8
- 230000004614 tumor growth Effects 0.000 claims description 90
- 230000015654 memory Effects 0.000 claims description 41
- 238000002203 pretreatment Methods 0.000 claims description 35
- 238000012549 training Methods 0.000 claims description 35
- 239000011159 matrix material Substances 0.000 claims description 28
- 230000000007 visual effect Effects 0.000 claims description 26
- 239000013598 vector Substances 0.000 claims description 25
- 238000003384 imaging method Methods 0.000 claims description 17
- 238000003780 insertion Methods 0.000 claims description 13
- 230000037431 insertion Effects 0.000 claims description 13
- 230000006870 function Effects 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 10
- 238000012952 Resampling Methods 0.000 claims description 8
- 230000008485 antagonism Effects 0.000 claims description 8
- 210000004556 brain Anatomy 0.000 claims description 8
- 238000000586 desensitisation Methods 0.000 claims description 8
- 210000003625 skull Anatomy 0.000 claims description 8
- 238000012163 sequencing technique Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000004140 cleaning Methods 0.000 claims description 4
- 230000006403 short-term memory Effects 0.000 claims description 3
- 238000005406 washing Methods 0.000 claims description 2
- 230000007787 long-term memory Effects 0.000 claims 1
- 238000002595 magnetic resonance imaging Methods 0.000 description 125
- 230000004913 activation Effects 0.000 description 6
- 108090000623 proteins and genes Proteins 0.000 description 6
- 238000012360 testing method Methods 0.000 description 5
- 102100033793 ALK tyrosine kinase receptor Human genes 0.000 description 4
- 230000006835 compression Effects 0.000 description 4
- 238000007906 compression Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 102000012011 Isocitrate Dehydrogenase Human genes 0.000 description 3
- 108010075869 Isocitrate Dehydrogenase Proteins 0.000 description 3
- 102100032543 Phosphatidylinositol 3,4,5-trisphosphate 3-phosphatase and dual-specificity protein phosphatase PTEN Human genes 0.000 description 3
- 102100023931 Transcriptional regulator ATRX Human genes 0.000 description 3
- 238000012217 deletion Methods 0.000 description 3
- 238000009792 diffusion process Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 230000005284 excitation Effects 0.000 description 3
- 101710168331 ALK tyrosine kinase receptor Proteins 0.000 description 2
- 102000001301 EGF receptor Human genes 0.000 description 2
- 108060006698 EGF receptor Proteins 0.000 description 2
- 108010011536 PTEN Phosphohydrolase Proteins 0.000 description 2
- 108010017842 Telomerase Proteins 0.000 description 2
- 102100032938 Telomerase reverse transcriptase Human genes 0.000 description 2
- 108700042462 X-linked Nuclear Proteins 0.000 description 2
- 238000002512 chemotherapy Methods 0.000 description 2
- 238000003759 clinical diagnosis Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- JXSJBGJIGXNWCI-UHFFFAOYSA-N diethyl 2-[(dimethoxyphosphorothioyl)thio]succinate Chemical compound CCOC(=O)CC(SP(=S)(OC)OC)C(=O)OCC JXSJBGJIGXNWCI-UHFFFAOYSA-N 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 2
- 102000052116 epidermal growth factor receptor activity proteins Human genes 0.000 description 2
- 108700015053 epidermal growth factor receptor activity proteins Proteins 0.000 description 2
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 2
- 230000002401 inhibitory effect Effects 0.000 description 2
- YOHYSYJDKVYCJI-UHFFFAOYSA-N n-[3-[[6-[3-(trifluoromethyl)anilino]pyrimidin-4-yl]amino]phenyl]cyclopropanecarboxamide Chemical compound FC(F)(F)C1=CC=CC(NC=2N=CN=C(NC=3C=C(NC(=O)C4CC4)C=CC=3)C=2)=C1 YOHYSYJDKVYCJI-UHFFFAOYSA-N 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 230000007170 pathology Effects 0.000 description 2
- 230000010412 perfusion Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000002560 therapeutic procedure Methods 0.000 description 2
- XUHRVZXFBWDCFB-QRTDKPMLSA-N (3R)-4-[[(3S,6S,9S,12R,15S,18R,21R,24R,27R,28R)-12-(3-amino-3-oxopropyl)-6-[(2S)-butan-2-yl]-3-(2-carboxyethyl)-18-(hydroxymethyl)-28-methyl-9,15,21,24-tetrakis(2-methylpropyl)-2,5,8,11,14,17,20,23,26-nonaoxo-1-oxa-4,7,10,13,16,19,22,25-octazacyclooctacos-27-yl]amino]-3-[[(2R)-2-[[(3S)-3-hydroxydecanoyl]amino]-4-methylpentanoyl]amino]-4-oxobutanoic acid Chemical compound CCCCCCC[C@H](O)CC(=O)N[C@H](CC(C)C)C(=O)N[C@H](CC(O)=O)C(=O)N[C@@H]1[C@@H](C)OC(=O)[C@H](CCC(O)=O)NC(=O)[C@@H](NC(=O)[C@H](CC(C)C)NC(=O)[C@@H](CCC(N)=O)NC(=O)[C@H](CC(C)C)NC(=O)[C@@H](CO)NC(=O)[C@@H](CC(C)C)NC(=O)[C@@H](CC(C)C)NC1=O)[C@@H](C)CC XUHRVZXFBWDCFB-QRTDKPMLSA-N 0.000 description 1
- 101150020330 ATRX gene Proteins 0.000 description 1
- 201000002434 Alpha-thalassemia-X-linked intellectual disability syndrome Diseases 0.000 description 1
- 101000904868 Homo sapiens Transcriptional regulator ATRX Proteins 0.000 description 1
- 208000036626 Mental retardation Diseases 0.000 description 1
- 102100025825 Methylated-DNA-protein-cysteine methyltransferase Human genes 0.000 description 1
- 101710132081 Phosphatidylinositol 3,4,5-trisphosphate 3-phosphatase and dual-specificity protein phosphatase PTEN Proteins 0.000 description 1
- 102000004160 Phosphoric Monoester Hydrolases Human genes 0.000 description 1
- 108090000608 Phosphoric Monoester Hydrolases Proteins 0.000 description 1
- 102100024547 Tensin-1 Human genes 0.000 description 1
- 108010088950 Tensins Proteins 0.000 description 1
- 102000001742 Tumor Suppressor Proteins Human genes 0.000 description 1
- 108010040002 Tumor Suppressor Proteins Proteins 0.000 description 1
- 201000006288 alpha thalassemia Diseases 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 238000001574 biopsy Methods 0.000 description 1
- 210000005013 brain tissue Anatomy 0.000 description 1
- 210000004027 cell Anatomy 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004195 computer-aided diagnosis Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002224 dissection Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 208000029824 high grade glioma Diseases 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 208000030173 low grade glioma Diseases 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 201000011614 malignant glioma Diseases 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 108040008770 methylated-DNA-[protein]-cysteine S-methyltransferase activity proteins Proteins 0.000 description 1
- 230000011987 methylation Effects 0.000 description 1
- 238000007069 methylation reaction Methods 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 210000004129 prosencephalon Anatomy 0.000 description 1
- 238000011158 quantitative evaluation Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000004579 scanning voltage microscopy Methods 0.000 description 1
- 208000011580 syndromic disease Diseases 0.000 description 1
- 238000002626 targeted therapy Methods 0.000 description 1
- 210000001519 tissue Anatomy 0.000 description 1
- 239000000225 tumor suppressor protein Substances 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
- G06T7/41—Analysis of texture based on statistical description of texture
- G06T7/45—Analysis of texture based on statistical description of texture using co-occurrence matrix computation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10088—Magnetic resonance imaging [MRI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30016—Brain
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Public Health (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Primary Health Care (AREA)
- Radiology & Medical Imaging (AREA)
- Geometry (AREA)
- Probability & Statistics with Applications (AREA)
- Epidemiology (AREA)
- Pathology (AREA)
- Quality & Reliability (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Processing (AREA)
- Magnetic Resonance Imaging Apparatus (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method and system, comprising the following steps: acquiring and preprocessing brain tumor multi-target multi-mode MRI data paired before and after treatment; tumor region segmentation is carried out on the brain tumor multi-target multi-mode MRI data paired before and after pretreatment and treatment through a 3DU-net convolutional neural network to obtainAndwill beAndobtaining a growth characteristic label L= { L by an image histology method 1 ,l 2 ,l 3 ,...,l n -a }; will beAndfeature extraction is carried out through a multichannel convolutional neural network, and SE fusion operation is carried out, so that deep learning features are obtainedAndwill beInputting the prediction model to obtain a brain tumor multi-target growth prediction labelWill beAndinputting the trained prospective treatment visualization model to obtain a final brain tumor region of interest growth evolution image, and inserting the brain tumor region of interest growth evolution image into a non-brain tumor region I background In the middle, the brain tumor prospective is completedTreatment visualization tasks; the invention has better clinical practicability.
Description
Technical Field
The invention relates to the technical field of medical image processing, in particular to a brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method and system.
Background
Brain tumors are common tumors of the human body, and central neurologists diagnose the disease condition of a patient through information such as multi-mode magnetic resonance imaging (Magnetic Resonance Imaging, MRI) of the brain of the patient. At present, the gold standard for accurately diagnosing brain tumor is still histopathological examination and gene detection, but tumor puncture biopsy is invasive operation, has high operation risk and can not accurately reflect the internal heterogeneity of tumor tissue; in the medical digitization era, the computer-aided diagnosis (Computer Assisted Diagnosis, CAD) technology can combine the effective information of the cross fusion of brain multi-mode MRI images (including T1 weighted imaging (T1), T1 weighted enhanced imaging (T1C), T2 weighted imaging (T2), water-inhibiting imaging (Flair), magnetic resonance perfusion imaging (perfusion weighted imaging, PWI) and apparent diffusion coefficient (Apparent Diffusion Coefficient, ADC), pathology, molecular genes and the like, and construct a multi-target molecular image intelligent noninvasive auxiliary diagnosis and treatment system of brain tumors, thereby having important significance for clinical diagnosis and accurate treatment of brain tumors.
In recent years, the digital informatization of the medical industry promotes the rapid development of CAD technology, and the CAD technology can improve diagnosis and treatment efficiency by segmenting, classifying, predicting and customizing personalized accurate treatment schemes for medical images; because the convolutional neural network (Convolutional Neural Network, CNN) can learn and capture the characteristics of different layers, a prediction model is built by searching a certain relation among the data interiors so as to realize the mapping of input into output (label or predicted value), and the convolutional neural network is widely applied to various classification tasks and obtains better results; with the growing importance of gene molecules in tumor diagnosis, researchers have performed classification-aided diagnosis on individual genotypes by CNN models and achieved satisfactory results. Next, some researchers performed a classification predictive study on genes such as isocitrate dehydrogenase (isocitrate dehydrogenase mutant/wildtype status, IDH-variant/wildtype), 1p/19q combined Deletion (1 p/19q Co-Deletion), epidermal growth factor receptor (epidermal growth factor receptor, EGFR), phosphatase and tensin homologs (phosphatase and tensin homolog, PTEN), telomerase reverse transcriptase (telomerase reverse transcriptase, TERT), oncostatin (tumor suppressor protein p, p53TP 53), alpha thalassemia/mental retardation syndrome X linkage (Alpha thalassemia/mental retardation syndrome X-linked, ATRX) and anaplastic lymphoma kinase (Anaplastic Lymphoma kinase, ALK), but only at an accuracy of 83% [1]. The auxiliary diagnosis methods are relatively fixed in the specific gene types and cannot completely meet the clinical diagnosis and treatment requirements; due to the limitation of brain tumor multi-mode MRI multi-task data, no research on multi-task auxiliary diagnosis based on brain tumor multi-mode MRI is seen at present. Some simple logistic regression, SVM and neural network pages are used for auxiliary diagnosis of 4-5 genes, and the highest accuracy is 72% [2].
Although these models have certain promotion effects on brain tumor assistance, they have certain disadvantages: (a) The problems of insufficient data and the like cause less classification auxiliary diagnosis research based on multiple disease types, and the classification result is poor; (b) lack of a multi-target therapy assessment model; (c) Compared with the current brain tumor CAD system, most auxiliary diagnosis systems only pay attention to auxiliary diagnosis, while auxiliary treatment systems only carry out retrospective digital quantitative evaluation, and prospective visual chemotherapy effect evaluation cannot be carried out before treatment.
[1]Zhou H,Chang K,Bai HX,et al.Machine learning reveals multimodal MRI patterns predictive of isocitrate dehydrogenase and 1p/19q status in diffuse low-and high-grade gliomas[J].Journal of Neuro-Oncology,2019, 142(2):299-307.
[2]Korfiatis P,Kline T L,Lachance D H,et al.Residual Deep Convolutional Neural Network Predicts MGMT Methylation Statusp[J].Journal of Digital Imaging,2017,30:622-628.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method and system.
The invention provides a brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method, which comprises the following steps:
step M1: acquiring pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data, and preprocessing the pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data to obtain unified pre-treatment standardPost-paired brain tumor multi-target multi-mode MRI data I original And I later ；
Step M2: pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data I original And I later Tumor region segmentation is carried out through a 3DU-net convolutional neural network to obtain tumor interested regions matched before and after treatmentAnd->
Step M3: tumor regions of interest paired before and after treatmentAnd->Obtaining the growth characteristic label L= { L of paired brain tumor before and after treatment by an image histology method 1 ,l 2 ,l 3 ,...,l n }；
Step M4: tumor regions of interest paired before and after treatmentAnd->Feature extraction is carried out through a multichannel convolutional neural network to obtain a feature map, SE fusion operation is carried out on the feature map, and finally multi-mode MRI image deep learning features of brain tumor regions of interest before and after treatment are obtained>And->
Step M5: deep learning features using multi-modality MRI images of brain tumor regions of interest prior to treatmentAnd the corresponding growth feature tag l= { L 1 ,l 2 ,l 3 ,...,l n Constructing and training a long-period memory network based on multi-task learning, training the long-period memory network of the multi-task learning by a method of dynamically sequencing a predicted tag sequence, and obtaining a brain tumor multi-target growth prediction model after training; deep learning feature of multi-mode MRI image of brain tumor region of interest before treatment>Inputting the brain tumor multi-target growth prediction model to obtain a brain tumor multi-target growth prediction label
Step M6: establishing a prospective treatment visualization model of brain tumor growth evolution based on CNN network generating countermeasure strategy, and performing multi-mode MRI image deep learning feature on brain tumor ROI before treatmentAnd predictive brain tumor growth signature +.>Inputting a trained prospective treatment visual model to obtain a final brain tumor region-of-interest growth evolution image, and inserting the brain tumor region-of-interest growth evolution image into the pre-treatment brain tumor multi-target multi-mode MRI data I by using the prospective treatment visual model original Non-brain tumor region I of (C) background Obtaining a final brain tumor multi-mode MRI image, and completing a brain tumor prospective treatment visualization task;
the prospective treatment visualization model comprises a text encoder module, a tumor growth prediction generation visualization module and a tumor focus insertion module; the GAN network is used for predicting the generation of tumor growth images, and the GAN network is used for predicting the brain tumor MRI images in the corresponding treatment time through the existing multi-mode brain MRI images of the patient, so that the prospective treatment result visualization is completed;
the tumor growth prediction generation visualization module comprises a generator G and a discriminator D for generating an image true and false classifier RF 。
Preferably, the preprocessing in the step M1 includes performing data processing including desensitization, cleaning, resampling and skull peeling on the pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data, so as to obtain pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data with uniform resolution and same gray scale distribution.
Preferably, the step M3 includes:
step M3.1: acquiring brain tumor multi-target multi-mode MRI data I paired before and after treatment by imaging method original And I later Is a group of imaging features;
step M3.2: obtaining a growth characteristic label through image histology characteristic calculation;
the image histology characteristics comprise brain tumor target point type, brain tumor volume, intensity mean value, intensity standard deviation, gray level co-occurrence matrix and entropy;
the growth characteristic label comprises a brain tumor target point type, brain tumor volume change, intensity mean value and intensity standard deviation, gray level co-occurrence matrix and entropy.
Preferably, the step M4 includes:
step M4.1: multi-target multi-modality MRI data I of brain tumor using pre-and post-treatment pairing after pretreatment original And I later Training a multichannel convolutional neural network with a preset number of convolutional-active layer modules;
step M4.2: tumor region of interest paired before and after treatment using multichannel convolutional neural network And->Extracting multi-channel convolution feature map, and obtaining ++through concat operation from the feature map>And->
Step M4.3: for the obtainedAnd->SE operation is carried out, and finally, the multi-mode MRI image deep learning characteristics of the brain tumor interested areas before and after treatment are obtained>And->
Preferably, the step M5 includes:
step M5.1: the preset long-period memory network is connected in parallel to construct a long-period memory network based on multi-task learning;
step M5.2: multi-mode MRI image deep learning feature of brain tumor region of interest before treatment through fully connected layerInitializing to obtain->
Step M5.3: will beGrowth characteristic tag l= { L 1 ,l 2 ,l 3 ,...,l n Inputting preset start and end marks into a long-period memory network; at time step t, the prediction output from the previous time step t-1 is +.>Converting the word embedding matrix into feature vector, and obtaining +.>Feature vector and->Inputting a long-period memory network in a time step t, and obtaining a predictive tag +.>
Step M5.4: predicting a plurality of long-period and short-period memory networks to obtain a plurality of preset prediction vectors pt to form a prediction matrix p= [ p ] 1 ,p 2 ,...,p n ]A new label is predicted at each time step based on a long-period memory network of multitask learning until the loss alignment loss function converges, a trained brain tumor multi-target growth prediction model is obtained, and a predicted brain tumor growth characteristic label is obtained according to the trained brain tumor multi-target growth prediction model
Preferably, the step M6 includes:
step M6.1: establishing a prospective treatment visualization model of brain tumor growth evolution based on a CNN network generating an countermeasure strategy;
step M6.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAnd->And brain tumor growth characteristic label l= { L 1 ,l 2 ,l 3 ,...,l n Inputting into a prospective treatment visualization model, generating a generator G of an countermeasure network through training, and carrying out multi-mode MRI image deep learning characteristics of a brain tumor region of interest before treatment through the generator G of the countermeasure network +.>Deep learning feature for generating multi-mode MRI images of brain tumor region of interest after treatmentSyndrome of->And obtaining a predicted multi-mode MRI image I of the brain tumor region of interest after treatment through an up-sampling network of a generator G G ；
Step M6.3: generating an image discriminator D RF Multi-modality MRI image I of a region of interest of a brain tumor after treatment predicted by contrast G Deep learning features of (a)And real treated brain tumor ROI multi-mode MRI image deep learning characteristicsFinishing antagonism learning, and finally obtaining a prospective treatment visual model of brain tumor growth evolution;
step M6.4: deep learning features of multi-mode MRI images of brain tumor interested areas before clinical treatment And predictive brain tumor growth signature +.>Inputting a prospective treatment visual model for finally obtaining the growth evolution of the brain tumor to obtain a plurality of different brain tumor growth evolution images, and selecting a brain tumor ROI growth evolution image meeting preset requirements according to the different brain tumor growth images;
step M6.5: inserting the obtained brain tumor ROI growth evolution image meeting the preset requirements into I by a poisson image editing method original Non-brain tumor region I of (C) background And obtaining a final brain tumor multi-mode MRI image to complete the brain tumor prospective treatment visualization task.
Preferably, the step M6.2 comprises:
step M6.2.1: text editor final feature F 0 Comprising the following steps:
where z is the noise vector sampled from a normal distribution in general,brain tumor predictive growth characteristic tag extracted from LSTM network>Features;
step M6.2.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAnd->And text editor final feature F 0 After passing the concat operation, as input of the tumor growth prediction generator G +.>Generating a predictive treated region of interest multi-modality MRI image I of a brain tumor by a predictive generator G G 。
Preferably, the step M6.5 includes:
Step M6.5.1: binarizing the obtained multiple different brain tumor growth evolution images to obtain brain tumor mask I G_mask ；
Step M6.5.2: brain tumor multi-target multi-mode MRI data I of pairing before and after treatment original And I later Each of the modality images and I G_mask Performing positioning and operation to obtain a background area non-brain tumor area I of brain tumor multi-target multi-mode MRI data background ；
Step M6.5.3: inserting a target image into a corresponding modality source image I background In (1), setting the source image gradient asMinimizing source image gradient +.>Thereby obtaining the target image inserted into the corresponding modal source image I background And (3) completing focus insertion according to the corresponding expected image f.
The invention provides a brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization system, which comprises the following components:
module M1: acquiring pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data and preprocessing the pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data to obtain unified standard pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data I original And I later ；
Module M2: pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data I original And I later Tumor region segmentation is carried out through a 3DU-net convolutional neural network to obtain tumor interested regions matched before and after treatment And->
Module M3: tumor regions of interest paired before and after treatmentAnd->Obtaining the growth characteristic label L= { L of paired brain tumor before and after treatment by an image histology method 1 ,l 2 ,l 3 ,...,l n }；
Module M4: tumor regions of interest paired before and after treatmentAnd->Feature extraction is carried out through a multichannel convolutional neural network to obtain a feature map, SE fusion operation is carried out on the feature map, and finally multi-mode MRI image deep learning features of brain tumor regions of interest before and after treatment are obtained>And->
Module M5: deep learning features using multi-modality MRI images of brain tumor regions of interest prior to treatmentAnd the corresponding growth feature tag l= { L 1 ,l 2 ,l 3 ,...,l n Constructing and training a long-period memory network based on multi-task learning, training the long-period memory network of the multi-task learning by a method of dynamically sequencing a predicted tag sequence, and obtaining a brain tumor multi-target growth prediction model after training; deep learning feature of multi-mode MRI image of brain tumor region of interest before treatment>Inputting the brain tumor multi-target growth prediction model to obtain a brain tumor multi-target growth prediction label
Module M6: establishing a prospective treatment visualization model of brain tumor growth evolution based on CNN network generating countermeasure strategy, and performing multi-mode MRI image deep learning feature on brain tumor ROI before treatment And predictive brain tumor growth signature +.>Inputting the trained prospective treatment visualization model to obtain a final brain tumor region of interest growth evolution image, and obtaining the brain tumor region of interest growth evolution imageInserting and treating forebrain tumor multi-target multi-mode MRI data I by using prospective treatment visualization model original Non-brain tumor region I of (C) background Obtaining a final brain tumor multi-mode MRI image, and completing a brain tumor prospective treatment visualization task;
the prospective treatment visualization model comprises a text encoder module, a tumor growth prediction generation visualization module and a tumor focus insertion module; the GAN network is used for predicting the generation of tumor growth images, and the GAN network is used for predicting the brain tumor MRI images in the corresponding treatment time through the existing multi-mode brain MRI images of the patient, so that the prospective treatment result visualization is completed;
the tumor growth prediction generation visualization module comprises a generator G and a discriminator D for generating an image true and false classifier RF 。
Preferably, the preprocessing in the module M1 includes performing data processing including desensitization, cleaning, resampling and skull peeling on the pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data to obtain pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data with uniform resolution and same gray scale distribution;
The module M3 includes:
module M3.1: acquiring brain tumor multi-target multi-mode MRI data I paired before and after treatment by imaging method original And I later Is a group of imaging features;
module M3.2: obtaining a growth characteristic label through image histology characteristic calculation;
the image histology characteristics comprise brain tumor target point type, brain tumor volume, intensity mean value, intensity standard deviation, gray level co-occurrence matrix and entropy;
the growth characteristic label comprises a brain tumor target point type, brain tumor volume change, intensity mean value, intensity standard deviation, gray level co-occurrence matrix and entropy;
the module M4 includes:
module M4.1: multi-target multi-modality MRI data I of brain tumor using pre-and post-treatment pairing after pretreatment original And I later TrainingA multi-channel convolutional neural network having a predetermined number of convolutional-active layer modules;
module M4.2: tumor region of interest paired before and after treatment using multichannel convolutional neural networkAndextracting multi-channel convolution feature map, and obtaining ++through concat operation from the feature map>And->
Module M4.3: for the obtainedAnd->SE operation is carried out, and finally, the multi-mode MRI image deep learning characteristics of the brain tumor interested areas before and after treatment are obtained >And->
The module M5 includes:
module M5.1: the preset long-period memory network is connected in parallel to construct a long-period memory network based on multi-task learning;
module M5.2: multi-mode MRI image deep learning feature of brain tumor region of interest before treatment through fully connected layerInitializing to obtain->
ModuleM5.3: will beGrowth characteristic tag l= { L 1 ,l 2 ,l 3 ,...,l n Inputting preset start and end marks into a long-period memory network; at time step t, the prediction output from the previous time step t-1 is +.>Converting the word embedding matrix into feature vector, and obtaining +.>Feature vector and->Inputting a long-period memory network in a time step t, and obtaining a predictive tag +.>
Module M5.4: predicting a plurality of long-period and short-period memory networks to obtain a plurality of preset prediction vectors pt to form a prediction matrix p= [ p ] 1 ,p 2 ,...,p n ]A new label is predicted at each time step based on a long-period memory network of multitask learning until the loss alignment loss function converges, a trained brain tumor multi-target growth prediction model is obtained, and a predicted brain tumor growth characteristic label is obtained according to the trained brain tumor multi-target growth prediction model
The module M6 includes:
module M6.1: establishing a prospective treatment visualization model of brain tumor growth evolution based on a CNN network generating an countermeasure strategy;
Module M6.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAnd->And brain tumor growth characteristic label l= { L 1 ,l 2 ,l 3 ,...,l n Inputting into a prospective treatment visualization model, generating a generator G of an countermeasure network through training, and carrying out multi-mode MRI image deep learning characteristics of a brain tumor region of interest before treatment through the generator G of the countermeasure network +.>Generating multi-mode MRI image deep learning characteristics of brain tumor region of interest after treatment>And obtaining a predicted multi-mode MRI image I of the brain tumor region of interest after treatment through an up-sampling network of a generator G G ；
Module M6.3: generating an image discriminator D RF Multi-modality MRI image I of a region of interest of a brain tumor after treatment predicted by contrast G Deep learning features of (a)And real treated brain tumor ROI multi-mode MRI image deep learning characteristicsFinishing antagonism learning, and finally obtaining a prospective treatment visual model of brain tumor growth evolution;
module M6.4: deep learning features of multi-mode MRI images of brain tumor interested areas before clinical treatmentAnd predictive brain tumor growth signature +.>Inputting a prospective treatment visual model for finally obtaining the growth evolution of the brain tumor to obtain a plurality of different brain tumor growth evolution images, and selecting a brain tumor ROI growth evolution image meeting preset requirements according to the different brain tumor growth images;
Module M6.5: inserting the obtained brain tumor ROI growth evolution image meeting the preset requirements into I by a poisson image editing method original Non-brain tumor region I of (C) background Obtaining a final brain tumor multi-mode MRI image, and completing a brain tumor prospective treatment visualization task;
the module M6.2 comprises:
module M6.2.1: text editor final feature F 0 Comprising the following steps:
where z is the noise vector sampled from a normal distribution in general,brain tumor predictive growth characteristic tag extracted from LSTM network>Features;
module M6.2.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAnd->And text editor final feature F 0 After passing the concat operation, as input of the tumor growth prediction generator G +.>Generating a predictive treated region of interest multi-modality MRI image I of a brain tumor by a predictive generator G G ；
The module M6.5 comprises:
module M6.5.1: binarizing the obtained multiple different brain tumor growth evolution images to obtain brain tumor mask I G_mask ；
Module M6.5.2: pairing before and after treatmentBrain tumor multi-target multi-mode MRI data I original And I later Each of the modality images and I G_mask Performing positioning and operation to obtain a background area non-brain tumor area I of brain tumor multi-target multi-mode MRI data background ；
Module M6.5.3: inserting a target image into a corresponding modality source image I background In (1), setting the source image gradient asMinimizing source image gradient +.>Thereby obtaining the target image inserted into the corresponding modal source image I background And (3) completing focus insertion according to the corresponding expected image f.
Compared with the prior art, the invention has the following beneficial effects:
1. the invention solves the problem that the existing brain tumor auxiliary diagnosis and treatment model cannot carry out prospective visual chemotherapy effect evaluation before treatment by utilizing a deep learning method through extracting the deep learning characteristics of the multi-mode MRI images of the brain tumor ROI, a brain tumor multi-target growth prediction model and a prospective treatment visualization model of brain tumor growth evolution; compared with other brain tumor auxiliary diagnosis and treatment models, the brain tumor evolution visualization method changes the communication mode of doctors and patients, and can provide more visual curative effect evaluation results for doctors and patients by selecting more accurate brain tumor evolution visualization images, thereby having better clinical practicability.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
FIG. 1 is a flow chart of a method for brain tumor multi-target assisted diagnosis and prospective treatment evolution visualization.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present invention.
Example 1
In order to construct a prospective visual model of brain tumor multi-target auxiliary diagnosis and brain tumor therapy growth evolution, which can meet clinical application requirements, the invention provides a brain tumor multi-target auxiliary diagnosis and prospective therapy evolution visual method, so as to overcome the defects of the existing brain tumor auxiliary diagnosis and treatment technology.
The invention provides a brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method, which comprises the following steps:
step M1: acquiring pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data and preprocessing the pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data to obtain unified standard pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data I original And I later ；
Step M2: pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data I original And I later Tumor region segmentation is carried out through a 3DU-net convolutional neural network to obtain tumor interested regions matched before and after treatmentAnd->
Step M3: tumor regions of interest paired before and after treatmentAnd->The method for obtaining the treatment front and back by image groupIs matched with the growth characteristic label L= { L of brain tumor 1 ,l 2 ,l 3 ,...,l n }；
Step M4: tumor regions of interest paired before and after treatmentAnd->Feature extraction is carried out through a multichannel convolutional neural network to obtain a feature map, SE fusion operation is carried out on the feature map, and finally multi-mode MRI image deep learning features of brain tumor regions of interest before and after treatment are obtained>And->
Step M5: deep learning features using multi-modality MRI images of brain tumor regions of interest prior to treatmentAnd the corresponding growth feature tag l= { L 1 ,l 2 ,l 3 ,...,l n Constructing and training a long-period memory network based on multi-task learning, training the long-period memory network of the multi-task learning by a method of dynamically sequencing a predicted tag sequence, and obtaining a brain tumor multi-target growth prediction model after training; deep learning feature of multi-mode MRI image of brain tumor region of interest before treatment >Inputting the brain tumor multi-target growth prediction model to obtain a brain tumor multi-target growth prediction label
Step M6: establishing a prospective treatment visualization model of brain tumor growth evolution based on CNN network generating countermeasure strategy, and performing multi-mode MRI image depth study on brain tumor ROI before treatmentCharacteristics of learningAnd predictive brain tumor growth signature +.>Inputting a trained prospective treatment visual model to obtain a final brain tumor region-of-interest growth evolution image, and inserting the brain tumor region-of-interest growth evolution image into the pre-treatment brain tumor multi-target multi-mode MRI data I by using the prospective treatment visual model original Non-brain tumor region I of (C) background Obtaining a final brain tumor multi-mode MRI image, and completing a brain tumor prospective treatment visualization task;
the prospective treatment visualization model comprises a text encoder module, a tumor growth prediction generation visualization module and a tumor focus insertion module; the GAN network is used for predicting the generation of tumor growth images, and the GAN network is used for predicting the brain tumor MRI images in the corresponding treatment time through the existing multi-mode brain MRI images of the patient, so that the prospective treatment result visualization is completed;
the tumor growth prediction generation visualization module comprises a generator G and a discriminator D for generating an image true and false classifier RF 。
Specifically, the preprocessing in the step M1 includes performing data processing including desensitization, cleaning, resampling and skull peeling on the pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data, so as to obtain pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data with uniform resolution and same gray scale distribution.
Specifically, the step M3 includes:
step M3.1: acquiring brain tumor multi-target multi-mode MRI data I paired before and after treatment by imaging method original And I later Is a group of imaging features;
step M3.2: obtaining a growth characteristic label through image histology characteristic calculation;
the image histology characteristics comprise brain tumor target point type, brain tumor volume, intensity mean value, intensity standard deviation, gray level co-occurrence matrix and entropy;
the growth characteristic label comprises a brain tumor target point type, brain tumor volume change, intensity mean value and intensity standard deviation, gray level co-occurrence matrix and entropy.
Specifically, the step M4 includes:
step M4.1: multi-target multi-modality MRI data I of brain tumor using pre-and post-treatment pairing after pretreatment original And I later Training a multichannel convolutional neural network with a preset number of convolutional-active layer modules;
step M4.2: tumor region of interest paired before and after treatment using multichannel convolutional neural network And->Extracting multi-channel convolution feature map, and obtaining ++through concat operation from the feature map>And->
Step M4.3: for the obtainedAnd->SE operation is carried out, and finally, the multi-mode MRI image deep learning characteristics of the brain tumor interested areas before and after treatment are obtained>And->
Specifically, the step M5 includes:
step M5.1: the preset long-period memory network is connected in parallel to construct a long-period memory network based on multi-task learning;
step M5.2: multi-mode MRI image deep learning feature of brain tumor region of interest before treatment through fully connected layerInitializing to obtain->
Step M5.3: will beGrowth characteristic tag l= { L 1 ,l 2 ,l 3 ,...,l n Inputting preset start and end marks into a long-period memory network; at time step t, the prediction output from the previous time step t-1 is +.>Converting the word embedding matrix into feature vector, and obtaining +.>Feature vector and->Inputting a long-period memory network in a time step t, and obtaining a predictive tag +.>
Step M5.4: predicting a plurality of long-period and short-period memory networks to obtain a plurality of preset prediction vectors pt to form a prediction matrix p= [ p ] 1 ,p 2 ,...,p n ]A new label is predicted at each time step based on a long-period memory network of multitask learning until the loss alignment loss function converges, a trained brain tumor multi-target growth prediction model is obtained, and a predicted brain tumor growth characteristic label is obtained according to the trained brain tumor multi-target growth prediction model
Specifically, the step M6 includes:
step M6.1: establishing a prospective treatment visualization model of brain tumor growth evolution based on a CNN network generating an countermeasure strategy;
step M6.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAnd->And brain tumor growth characteristic label l= { L 1 ,l 2 ,l 3 ,...,l n Inputting into a prospective treatment visualization model, generating a generator G of an countermeasure network through training, and carrying out multi-mode MRI image deep learning characteristics of a brain tumor region of interest before treatment through the generator G of the countermeasure network +.>Generating multi-mode MRI image deep learning characteristics of brain tumor region of interest after treatment>And obtaining a predicted multi-mode MRI image I of the brain tumor region of interest after treatment through an up-sampling network of a generator G G ；
Step M6.3: generating an image discriminator D RF Multi-modality MRI image I of a region of interest of a brain tumor after treatment predicted by contrast G Deep learning features of (a)And real treated brain tumor ROI multi-mode MRI image deep learning characteristicsFinishing antagonism learning, and finally obtaining a prospective treatment visual model of brain tumor growth evolution;
step M6.4: multimodality MRI image depth of brain tumor region of interest before clinical treatment Characteristics of learningAnd predictive brain tumor growth signature +.>Inputting a prospective treatment visual model for finally obtaining the growth evolution of the brain tumor to obtain a plurality of different brain tumor growth evolution images, and selecting a brain tumor ROI growth evolution image meeting preset requirements according to the different brain tumor growth images;
step M6.5: inserting the obtained brain tumor ROI growth evolution image meeting the preset requirements into I by a poisson image editing method original Non-brain tumor region I of (C) background And obtaining a final brain tumor multi-mode MRI image to complete the brain tumor prospective treatment visualization task.
Specifically, the step M6.2 includes:
step M6.2.1: text editor final feature F 0 Comprising the following steps:
where z is the noise vector sampled from a normal distribution in general,brain tumor predictive growth characteristic tag extracted from LSTM network>Features;
step M6.2.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAnd->And text editor final feature F 0 After concat operation, the tumor growth prediction generator is usedInput of G->Generating a predictive treated region of interest multi-modality MRI image I of a brain tumor by a predictive generator G G 。
Specifically, the step M6.5 includes:
step M6.5.1: binarizing the obtained multiple different brain tumor growth evolution images to obtain brain tumor mask I G_mask ；
Step M6.5.2: brain tumor multi-target multi-mode MRI data I of pairing before and after treatment original And I later Each of the modality images and I G_mask Performing positioning and operation to obtain a background area non-brain tumor area I of brain tumor multi-target multi-mode MRI data background ；
Step M6.5.3: inserting a target image into a corresponding modality source image I background In (1), setting the source image gradient asMinimizing source image gradient +.>Thereby obtaining the target image inserted into the corresponding modal source image I background And (3) completing focus insertion according to the corresponding expected image f.
Example 2
Example 2 is a modification of example 1
Fig. 1 shows a brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method, which comprises the following steps:
step (1): preprocessing brain tumor multi-target multi-mode MRI data (comprising T1, T1C, T2, flair, PWI and ADC); according to WHO's latest recommendations, clinical guidelines, and pathology data, a doctor demarcates multiple molecular genetic categories of brain tumor multimodal MRI data: IDH-mutant/holdtype, 1p/19qCo-Deletion, EGFR, PTEN), TERT, p53TP53, ATRX, ALK, etc.; then, by data desensitization, resampling and The pretreatment methods such as skull peeling and the like obtain brain tumor multi-target multi-mode MRI data paired before and after treatment with uniform resolution and approximately same gray level distribution, which are respectively recorded as I original And I later And each modality. Nii.gz data size is 256 x 16.
Step (2): extracting the characteristics of the brain tumor ROI multi-mode MRI image deep learning (DeepLearning, DL); first, I obtained in step (1) is used original And I later Tumor region of interest (region ofinterest, ROI, size 256X 4) was segmented through 3DU-net networkAnd->Then use +.>And->Training 4-channel convolutional neural network with 5 convolutional-active layer modules (the modules are composed of 3 convolutional layers with the convolution kernel size of 3×3 step sizes of 1, a ReLU activation function and a maxpool layer with the step sizes of 2), performing multi-channel convolutional feature map extraction on ROI to obtain 512 path feature maps with the size of 16×16 for each channel, performing squeze-and-Excitation (SE) operation on the obtained 4-channel feature maps, focusing on channel features and spatial features with the largest information amount, inhibiting unimportant features, and obtaining final pre-treatment and post-treatment ROI multi-mode MRI image deep learning features of brain tumors>And->The sizes are 512 multiplied by 16 multiplied by 4; use- >And->The growth characteristic labels of paired brain tumors before and after treatment are obtained by an image histology method (6 growth characteristic labels are set in the invention: brain tumor target point type, brain tumor volume change (such as increase or decrease), intensity mean value and intensity standard deviation, gray level co-occurrence matrix and entropy), and are recorded as L= { L 1 ,l 2 ,…,l 6 }。
Step (3): a brain tumor multi-target growth prediction model; in the training stage, utilizing the pre-treatment brain tumor ROI multi-mode MRI image deep learning characteristics obtained in the step (2)And corresponding growth characteristic labels L, constructing and training a long-short-period memory network (Long Short Term Memory networks, LSTM) based on multi-task learning, training the LSTM by a method of dynamically sequencing predicted label sequences, and obtaining a final brain tumor multi-target growth prediction model; in the test stage, the pre-treatment brain tumor ROI multi-mode MRI image deep learning feature obtained in the step (2) is +.>As the input of a brain tumor multi-target growth prediction model, outputting brain tumor target type, brain tumor volume change (such as enlargement/reduction), intensity change (such as mean value and standard deviation) and texture change (such as gray level symbiotic matrix and entropy) brain tumor prediction growth characteristic label through the tumor growth prediction model >
Step (4): a prospective treatment visualization model of brain tumor growth evolution; the invention establishes a prospective treatment visual model of brain tumor growth evolution based on CNN network generating countermeasure strategy; in the training stage, the model uses the multi-mode MRI image deep learning characteristics of the brain tumor ROI before and after treatment obtained in the step (2)And->And brain tumor growth characteristic label l= { L 1 ,l 2 ,…,l 6 By training the generator G for generating the antagonism network from the pre-treatment brain tumor ROI multi-mode MRI image deep learning characteristics +.>Generating multi-modal MRI image deep learning features of a treated brain tumor ROI>And obtaining predicted post-treatment brain tumor ROI multi-modal MRI image I through an up-sampling network of a generator G G Generating an image discriminator D RF By comparison of I G Deep learning feature->And real post-treatment brain tumor ROI multi-mode MRI image deep learning feature +.>Finishing antagonism learning, and finally obtaining a prospective treatment visual model of brain tumor growth evolution; in the test stage, the pre-clinical brain tumor ROI multi-mode MRI image deep learning characteristic obtained in the step (2) is subjected to deep learning>And (3) a predicted brain tumor growth characteristic tag +.>As input, 5 different brain tumor growth evolution images are output through a trained prospective treatment visualization model of brain tumor growth evolution; then, the doctor manually selects the most suitable image as the final brain tumor ROI growth evolution image; finally, the predicted brain tumor ROI growth evolution image is inserted into I by a poisson image editing method original Non-brain tumor region I of (C) background Obtaining final multi-mode MRI image of brain tumor to finishVisual task of brain tumor prospective treatment.
A brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method shown in fig. 1, wherein the preprocessing method in the step (1) comprises the following steps of
Data desensitization and washing; carrying out data deformation on sensitive information in original brain tumor multi-mode MRI data acquired by a hospital according to a desensitization rule;
resampling data; the pixel size, the granularity of the thickness is different for different scanning planes, the invention resamples all samples to 256×256×16 size with fixed isomorphic resolution from all data sets;
skull dissection and data preservation; according to the invention, after resampling, 256×256×16 data are subjected to skull peeling operation by an Fsslingteller_BetBuainextraction method, and non-brain tissues are removed; and uniformly storing each mode as data in a format of nii.gz with the size of 256 multiplied by 16.
The brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method shown in fig. 1, wherein the brain tumor ROI multi-mode MRI image deep learning feature extraction in step (2) comprises:
(1) brain tumor ROI region segmentation; the invention adopts a 3DU-net segmentation network to segment the I obtained in the step (1) original And I later Tumor segmentation was performed to obtain pre-and post-treatment paired brain tumor regions of interest (ROI, size 256X 4)And->According to the invention, brain tumor multi-mode MRI data with a segmentation Mask are randomly divided into a training set (0.8), a verification set (0.1) and a test set (0.1), the data are sent into a 3DU-net segmentation network, and ROI segmentation is completed through network training and network testing. />
(2) Deep image feature extraction of brain tumor multi-mode MRI; using the product obtained in step (2) (1)And->Training a 4-channel convolutional neural network with 5 convolutional-active layer modules (the modules are composed of 3 convolutional layers with the convolution kernel size of 3×3 step sizes of 1, a ReLU activation function and a maxpool layer with the step sizes of 2), performing multi-channel convolutional feature map extraction on the ROI to obtain 512 path feature maps with the 16×16 size of each channel, and obtaining the feature map of the 4 channels through the concat operation>And->The sizes are 16×16×512×4.
Then, a group is givenOr->The method is subjected to the operation of Squeeze-and-Excitation (SE), the channel characteristics and the spatial characteristics with the largest information quantity are focused, the unimportant characteristics are restrained, and the SE fusion operation comprises:
compression operation (Squeeze)
In the compression operation, the characteristic diagram is Wherein z is i,j ∈ R 16 ×16×512×4 I epsilon 1,2, …, 512, j epsilon 1,2,3,4, and performing global average pooling operation (global average poolilng) to obtain a weight matrix W with a size of 1×1×512×4 s Compression feature Z S The method comprises the following steps:
excitation operation (expression)
In the excitation operation, in the compression operation, the characteristic map is shown asWherein z is i,j ∈R 16×16×512×4 I epsilon 1,2, …, 512, j epsilon 1,2,3,4, first performing full convolution operation (FC) of 2048×1024 neurons and Sigmoid activation layer to obtain first activation feature weight W C ' size 1 x 2048 x 1024; then, the full convolution operation (FC) and Sigmoid activation layer of 2048 neurons are performed to obtain the second activation characteristic acquisition channel weight W C (1×1×512×4), the excitation characteristics are:
finally obtaining the deep learning characteristics of the multi-mode MRI images of the brain tumor ROI:
in the same way, can also obtain
(3) Acquisition by image histologyAnd->Is characterized by 6 images: the brain tumor target type, the brain tumor volume, the intensity mean value, the intensity standard deviation, the gray level co-occurrence matrix and the entropy are calculated and compared to obtain 6 growth characteristic labels: brain tumor target type, brain tumor volume change (such as increasing or decreasing), intensity mean and intensity standard deviation, gray level co-occurrence matrix and entropy, which are marked as L= { L 1 ,l 2 ,…,l 6 }。
The brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method shown in fig. 1, wherein in the step (3), a brain tumor multi-target growth prediction model is shown; in order to generate a growth characteristic label of brain tumor, a long-short-period memory network (Long Short Term Memory networks, LSTM) is introduced into a multi-target growth prediction model of brain tumor, and the growth characteristic label of brain tumor is predicted by adopting a method of dynamically sequencing predicted label sequences; the method comprises the following steps:
the invention further provides the method obtained in step (3) of step (2)And the 6 growth characteristic tags l= { L obtained in step (2) (4) 1 ,l 2 ,…,l 6 -and start and end marks as inputs; will be +.>Initializing to LSTM input->At time t, to control the forward propagation of LSTM predictions, the formula is as follows:
h t ＝o t ⊙tanh(c t )
wherein c t And h t Is a model cell and hidden state, f t And o t Is the input at time t, W, U and b are the network weights and errors to learn, and σ and tanh are sigmoid and tanh functions.
The invention is at timeAt step t, the prediction output from the previous time step t-1 is usedEmbedding as input E is a word embedding matrix for embedding the tag +.>Converted into a vector, and the predicted pt for the current time step t is calculated by:
The invention uses the prediction of 6 LSTM to obtain 6 prediction vectors pt, which form a prediction matrix P= [ P ] 1 ,…,p 6 ,]The LSTM model predicts a new label at each time step until an end signal is generated, obtaining a predicted brain tumor growth signature.
According to the brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method disclosed by the invention, in the brain tumor multi-target growth prediction model in the step (3), the prediction of a correct label and a network is carried out before the loss is calculated
Alignment, minimum loss alignment loss functionThe definition is as follows:
at time step T, comparing the prediction result with corresponding labels in the same step of the gold standard sequence, calculating a matrix T to minimize the summation cross entropy loss, and solving the predicted brain tumor growth characteristic labels through a Hungarian algorithm
The method for multi-target auxiliary diagnosis and prospective treatment evolution visualization of brain tumor shown in fig. 1, wherein the prospective treatment visualization model of brain tumor growth evolution in step (4) comprises:
a text encoder module, the module comprising:
the text encoder adopts a pre-trained bidirectional LSTM network, and can extract semantic vectors from the predictive brain tumor growth feature labels obtained in the step (3); where z is typically the noise vector sampled from a standard normal distribution, Is sentence characteristic extracted by LSTM network, final characteristic F of text encoder 0 Enhanced by z and conditions +.>The composition is expressed as:
/>
a tumor growth prediction generation visualization module, the module comprising: a generator G, a generator image true and false discriminator D RF ；
In the training stage, the model uses the multi-mode MRI image deep learning characteristics of the brain tumor ROI before and after treatment obtained in the step (2)And->And a brain tumor growth characteristic tag l= { L 1 ,l 2 ,…,l 6 As input, by training the generator G for generating the countermeasure network>Generating predicted post-treatment brain tumor ROI multi-modality MRI image deep learning features>And pass through the generator GSampling network to obtain predicted multi-mode MRI image I of brain tumor ROI after treatment G Generating an image discriminator D RF By comparison->Deep learning feature->And real post-treatment brain tumor ROI multi-mode MRI image deep learning feature +.>Finishing antagonism learning, and finally obtaining a prospective treatment visual model of brain tumor growth evolution;
in the test stage, the pre-clinical brain tumor ROI multi-mode MRI image deep learning characteristics obtained in the step (2) are characterizedAnd (3) a predicted brain tumor growth characteristic tag +.>As input, 5 different brain tumor growth evolution images are output through a trained prospective treatment visualization model of brain tumor growth evolution; then the doctor manually selects the most suitable image as the final brain tumor ROI growth evolution image I G 。
The tumor growth prediction generates a visualization module loss function comprising:
discriminator D RF Is defined as:
the loss function of generator G is defined as:
a tumor lesion insertion module, comprising:
firstly, selecting the I from the Chinese medicine in the step (1) of the step (4) G Binarization is carried out to obtain a brain tumor Mask expressed as I G_mask The method comprises the steps of carrying out a first treatment on the surface of the Then, each mode image in the brain tumor multi-target multi-mode MRI data obtained in the step (1) is combined with I G_mask Performing positioning and operation to obtain background area I of brain tumor multi-target multi-mode MRI data background (i.e., non-brain tumor areas); finally, I is edited according to the Poisson image editing method G I inserted into corresponding modality background And obtaining a predicted brain tumor multi-mode MRI image to complete the brain tumor prospective treatment visualization task.
The method for multi-target auxiliary diagnosis and prospective treatment evolution visualization of brain tumor shown in fig. 1, wherein the poisson editing in the step (3) in the step (4) comprises the following steps:
the invention uses the target image I G Insertion into corresponding modality source image I background In which we willDefined as the lateral domain, the closed subset of D is denoted as Ω and the boundary is +.>The desired result after mixing is denoted as f. Poisson editing is essentially a diffusion process, and the present invention finds f by solving the following minimization problem:
Wherein,representing the gradient operator. In order to solve the actual image insertion problem, the present invention introduces the guiding field v into the minimization problem and sets it to +.>To solve the minimization problem:
/>
to achieve image editing, the present invention solves the unique solution of the following Poisson equation with Dirichlet boundary conditions:
wherein the method comprises the steps ofIs a divergence operator.
The present invention obtains vector v using the following equation:
finally discretizing through a pixel grid of the digital image, and then applying an iteration method such as a Jacobi method or Gauss-Seidel iteration with continuous overstretch to solve the minimization problem.
Finally, it should be noted that the examples are disclosed for the purpose of aiding in the further understanding of the present invention, but those skilled in the art will appreciate that: various alternatives and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the disclosed embodiments, but rather the scope of the invention is defined by the appended claims.
Those skilled in the art will appreciate that the systems, apparatus, and their respective modules provided herein may be implemented entirely by logic programming of method steps such that the systems, apparatus, and their respective modules are implemented as logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc., in addition to the systems, apparatus, and their respective modules being implemented as pure computer readable program code. Therefore, the system, the apparatus, and the respective modules thereof provided by the present invention may be regarded as one hardware component, and the modules included therein for implementing various programs may also be regarded as structures within the hardware component; modules for implementing various functions may also be regarded as being either software programs for implementing the methods or structures within hardware components.
The foregoing describes specific embodiments of the present invention. It is to be understood that the invention is not limited to the particular embodiments described above, and that various changes or modifications may be made by those skilled in the art within the scope of the appended claims without affecting the spirit of the invention. The embodiments of the present application and features in the embodiments may be combined with each other arbitrarily without conflict.
Claims (10)
1. A brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization system, comprising:
module M1: acquiring pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data and preprocessing the pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data to obtain unified standard pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data I original And I later ；
Module M2: pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data I original And I later Tumor region segmentation is carried out through a 3DU-net convolutional neural network to obtain tumor interested regions matched before and after treatmentAnd->
Module M3: tumor regions of interest paired before and after treatmentAnd->By image group learning methodObtaining growth characteristic labels L= { L of paired brain tumors before and after treatment 1 ，l 2 ，l 3 ，...，l n }；
Module M4: tumor regions of interest paired before and after treatmentAnd->Feature extraction is carried out through a multichannel convolutional neural network to obtain a feature map, SE fusion operation is carried out on the feature map, and finally multi-mode MRI image deep learning features of brain tumor regions of interest before and after treatment are obtained>And->
Module M5: deep learning features using multi-modality MRI images of brain tumor regions of interest prior to treatmentAnd the corresponding growth feature tag l= { L 1 ，l 2 ，l 3 ，...，l n Constructing and training a long-period memory network based on multi-task learning, training the long-period memory network of the multi-task learning by a method of dynamically sequencing a predicted tag sequence, and obtaining a brain tumor multi-target growth prediction model after training; deep learning features for multi-modal MRI images of brain tumor regions of interest prior to treatmentInputting the brain tumor multi-target growth prediction model to obtain a brain tumor multi-target growth prediction label
Module M6: establishing a prospective treatment visualization model of brain tumor growth evolution based on CNN network generating countermeasure strategy, and performing multi-mode MRI image deep learning feature on brain tumor ROI before treatmentAnd predictive brain tumor growth signature +.>Inputting a trained prospective treatment visual model to obtain a final brain tumor region-of-interest growth evolution image, and inserting the brain tumor region-of-interest growth evolution image into the pre-treatment brain tumor multi-target multi-mode MRI data I by using the prospective treatment visual model original Non-brain tumor region I of (C) background Obtaining a final brain tumor multi-mode MRI image, and completing a brain tumor prospective treatment visualization task;
the prospective treatment visualization model comprises a text encoder module, a tumor growth prediction generation visualization module and a tumor focus insertion module; the GAN network is used for predicting the generation of tumor growth images, and the GAN network is used for predicting the brain tumor MRI images in the corresponding treatment time through the existing multi-mode brain MRI images of the patient, so that the prospective treatment result visualization is completed;
the tumor growth prediction generation visualization module comprises a generator G and a discriminator D for generating an image true and false classifier RF ；
The module M6 includes:
module M6.1: establishing a prospective treatment visualization model of brain tumor growth evolution based on a CNN network generating an countermeasure strategy;
module M6.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAndand brain tumor growth characteristic label l= { L 1 ，l 2 ，l 3 ，...，l n Inputting into a prospective treatment visualization model, generating a generator G of an countermeasure network through training, and carrying out multi-mode MRI image deep learning characteristics of a brain tumor region of interest before treatment through the generator G of the countermeasure network +. >Generating multi-mode MRI image deep learning characteristics of brain tumor region of interest after treatment>And obtaining a predicted multi-mode MRI image I of the brain tumor region of interest after treatment through an up-sampling network of a generator G G ；
Module M6.3: generating an image discriminator D RF Multi-modality MRI image I of a region of interest of a brain tumor after treatment predicted by contrast G Deep learning features of (a)And real treated brain tumor ROI multi-mode MRI image deep learning characteristicsFinishing antagonism learning, and finally obtaining a prospective treatment visual model of brain tumor growth evolution;
module M6.4: deep learning features of multi-mode MRI images of brain tumor interested areas before clinical treatmentAnd predictive brain tumor growth signature +.>Inputting to finally obtain the growth and evolution of brain tumorA prospective treatment visual model is used for obtaining a plurality of different brain tumor growth evolution images, and a brain tumor ROI growth evolution image meeting preset requirements is selected according to the different brain tumor growth images;
module M6.5: inserting the obtained brain tumor ROI growth evolution image meeting the preset requirements into I by a poisson image editing method original Non-brain tumor region I of (C) background Obtaining a final brain tumor multi-mode MRI image, and completing a brain tumor prospective treatment visualization task;
The module M6.2 comprises:
module M6.2.1: text editor final feature F 0 Comprising the following steps:
where z is the noise vector sampled from a normal distribution in general,brain tumor predictive growth characteristic tag extracted from LSTM network>Features;
module M6.2.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAnd->And text editor final feature F 0 By concat operation, as input to the tumor growth prediction generator GGeneration of a predictive treated brain tumor by a predictive generator GMulti-modality MRI image I of a tumor region of interest G ；
The module M6.5 comprises:
module M6.5.1: binarizing the obtained multiple different brain tumor growth evolution images to obtain brain tumor mask I G_mask ；
Module M6.5.2: brain tumor multi-target multi-mode MRI data I of pairing before and after treatment original And I later Each of the modality images and I G_mask Performing positioning and operation to obtain a background area non-brain tumor area I of brain tumor multi-target multi-mode MRI data background ；
Module M6.5.3: inserting a target image into a corresponding modality source image I background In (1), setting the source image gradient asMinimizing source image gradient +.>Thereby obtaining the target image inserted into the corresponding modal source image I background And (3) completing focus insertion according to the corresponding expected image f.
2. The brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization system according to claim 1, wherein the preprocessing in the module M1 comprises performing data processing including desensitization, cleaning, resampling and skull peeling on the brain tumor multi-target multi-mode MRI data paired before and after treatment to obtain the brain tumor multi-target multi-mode MRI data paired before and after treatment with uniform resolution and same gray scale distribution;
the module M3 includes:
module M3.1: acquiring brain tumor multi-target multi-mode MRI data I paired before and after treatment by imaging method original And I later Is a group of imaging features;
module M3.2: obtaining a growth characteristic label through image histology characteristic calculation;
the image histology characteristics comprise brain tumor target point type, brain tumor volume, intensity mean value, intensity standard deviation, gray level co-occurrence matrix and entropy;
the growth characteristic label comprises a brain tumor target point type, brain tumor volume change, intensity mean value, intensity standard deviation, gray level co-occurrence matrix and entropy;
the module M4 includes:
module M4.1: multi-target multi-modality MRI data I of brain tumor using pre-and post-treatment pairing after pretreatment original And I later Training a multichannel convolutional neural network with a preset number of convolutional-active layer modules;
module M4.2: tumor region of interest paired before and after treatment using multichannel convolutional neural networkAndextracting multi-channel convolution feature map, and obtaining ++through concat operation from the feature map>And->
Module M4.3: for the obtainedAnd->SE operation is carried out, and finally, the multi-mode MRI image deep learning characteristics of the brain tumor interested areas before and after treatment are obtained>And->
The module M5 includes:
module M5.1: the preset long-period memory network is connected in parallel to construct a long-period memory network based on multi-task learning;
module M5.2: multi-mode MRI image deep learning feature of brain tumor region of interest before treatment through fully connected layerInitializing to obtain->
Module M5.3: will beGrowth characteristic tag l= { L l ，l 2 ，l 3 ，...，l n Inputting preset start and end marks into a long-period memory network; at time step t, the prediction output from the previous time step t-1 is +.>Converting the word embedding matrix into feature vector, and obtaining +.>Feature vector and->Inputting a long-period memory network in a time step t, and obtaining a predictive tag +.>
Module M5.4: predicting a preset long-period and short-period memory network to obtain a preset prediction Vector pt, which constitutes the prediction matrix p= [ p ] 1 ，p 2 ，...，p n ]A new label is predicted at each time step based on a long-period memory network of multitask learning until the loss alignment loss function converges, a trained brain tumor multi-target growth prediction model is obtained, and a predicted brain tumor growth characteristic label is obtained according to the trained brain tumor multi-target growth prediction model
3. A brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method based on the brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization system as claimed in claim 1, characterized by comprising:
step M1: acquiring pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data and preprocessing the pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data to obtain unified standard pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data I original And I later ；
Step M2: pre-treatment and post-treatment paired brain tumor multi-target multi-mode MRI data I original And I later Tumor region segmentation is carried out through a 3DU-net convolutional neural network to obtain tumor interested regions matched before and after treatmentAnd->
Step M3: tumor regions of interest paired before and after treatment And->Obtaining the growth characteristic label L= { L of paired brain tumor before and after treatment by an image histology method 1 ，l 2 ，l 3 ，...，l n }；
Step M4: tumor regions of interest paired before and after treatmentAnd->Feature extraction is carried out through a multichannel convolutional neural network to obtain a feature map, SE fusion operation is carried out on the feature map, and finally multi-mode MRI image deep learning features of brain tumor regions of interest before and after treatment are obtained>And->
Step M5: deep learning features using multi-modality MRI images of brain tumor regions of interest prior to treatmentAnd the corresponding growth feature tag l= { L 1 ，l 2 ，l 3 ，...，l n Constructing and training a long-period memory network based on multi-task learning, training the long-period memory network of the multi-task learning by a method of dynamically sequencing a predicted tag sequence, and obtaining a brain tumor multi-target growth prediction model after training; deep learning features for multi-modal MRI images of brain tumor regions of interest prior to treatmentInputting the brain tumor multi-target growth prediction model to obtain a brain tumor multi-target growth prediction label
Step M6: establishing a prospective treatment visualization model of brain tumor growth evolution based on CNN network generating countermeasure strategy, and performing multi-mode MRI image deep learning feature on brain tumor ROI before treatment And predictive brain tumor growth signature +.>Inputting a trained prospective treatment visual model to obtain a final brain tumor region-of-interest growth evolution image, and inserting the brain tumor region-of-interest growth evolution image into the pre-treatment brain tumor multi-target multi-mode MRI data I by using the prospective treatment visual model original Non-brain tumor region I of (C) background Obtaining a final brain tumor multi-mode MRI image, and completing a brain tumor prospective treatment visualization task;
the prospective treatment visualization model comprises a text encoder module, a tumor growth prediction generation visualization module and a tumor focus insertion module; the GAN network is used for predicting the generation of tumor growth images, and the GAN network is used for predicting the brain tumor MRI images in the corresponding treatment time through the existing multi-mode brain MRI images of the patient, so that the prospective treatment result visualization is completed;
the tumor growth prediction generation visualization module comprises a generator G and a discriminator D for generating an image true and false classifier RF 。
4. The method for assisting diagnosis and prospective treatment evolution visualization of brain tumor multi-target according to claim 3, wherein the preprocessing in the step M1 comprises performing data processing including desensitization, washing, resampling and skull peeling on the brain tumor multi-target multi-mode MRI data paired before and after treatment, so as to obtain the brain tumor multi-target multi-mode MRI data paired before and after treatment with uniform resolution and same gray scale distribution.
5. The brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method according to claim 3, wherein the step M3 comprises:
step M3.1: acquiring brain tumor multi-target multi-mode MRI data I paired before and after treatment by imaging method original And I 1ater Is a group of imaging features;
step M3.2: obtaining a growth characteristic label through image histology characteristic calculation;
the image histology characteristics comprise brain tumor target point type, brain tumor volume, intensity mean value, intensity standard deviation, gray level co-occurrence matrix and entropy;
the growth characteristic label comprises a brain tumor target point type, brain tumor volume change, intensity mean value and intensity standard deviation, gray level co-occurrence matrix and entropy.
6. The brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method according to claim 3, wherein the step M4 comprises:
step M4.1: multi-target multi-modality MRI data I of brain tumor using pre-and post-treatment pairing after pretreatment original And I later Training a multichannel convolutional neural network with a preset number of convolutional-active layer modules;
step M4.2: tumor region of interest paired before and after treatment using multichannel convolutional neural network Andextracting multi-channel convolution feature map, and obtaining ++through concat operation from the feature map>And->
Step M4.3: for the obtainedAnd->SE operation is carried out, and finally, the multi-mode MRI image deep learning characteristics of the brain tumor interested areas before and after treatment are obtained>And->
7. The method for multi-target assisted diagnosis and prospective treatment evolution visualization of brain tumor according to claim 3, wherein the step M5 comprises:
step M5.1: the preset long-period memory network is connected in parallel to construct a long-period memory network based on multi-task learning;
step M5.2: multi-mode MRI image deep learning feature of brain tumor region of interest before treatment through fully connected layerInitializing to obtain->
Step M5.3: will beGrowth characteristic tag l= { L 1 ，l 2 ，l 3 ，...，l n Input of preset start and end marks into long and short term memory networkThe method comprises the steps of carrying out a first treatment on the surface of the At time step t, the prediction output from the previous time step t-1 is +.>Converting the word embedding matrix into feature vector, and obtaining +.>Feature vector and->Inputting a long-period memory network in a time step t, and obtaining a predictive tag +.>
Step M5.4: predicting a plurality of long-period and short-period memory networks to obtain a plurality of preset prediction vectors pt to form a prediction matrix p= [ p ] 1 ，p 2 ，...，p n ]A new label is predicted at each time step based on a long-period memory network of multitask learning until the loss alignment loss function converges, a trained brain tumor multi-target growth prediction model is obtained, and a predicted brain tumor growth characteristic label is obtained according to the trained brain tumor multi-target growth prediction model
8. The method for multi-target assisted diagnosis and prospective treatment evolution visualization of brain tumor according to claim 3, wherein the step M6 comprises:
step M6.1: establishing a prospective treatment visualization model of brain tumor growth evolution based on a CNN network generating an countermeasure strategy;
step M6.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAndand brain tumor growth characteristic label l= { L 1 ，l 2 ，l 3 ，...，l n Inputting into a prospective treatment visualization model, generating a generator G of an countermeasure network through training, and carrying out multi-mode MRI image deep learning characteristics of a brain tumor region of interest before treatment through the generator G of the countermeasure network +.>Generating multi-mode MRI image deep learning characteristics of brain tumor region of interest after treatment>And obtaining a predicted multi-mode MRI image I of the brain tumor region of interest after treatment through an up-sampling network of a generator G G ；
Step M6.3: generating an image discriminator D RF Multi-modality MRI image I of a region of interest of a brain tumor after treatment predicted by contrast G Deep learning features of (a)And real treated brain tumor ROI multi-mode MRI image deep learning characteristicsFinishing antagonism learning, and finally obtaining a prospective treatment visual model of brain tumor growth evolution;
step M6.4: deep learning features of multi-mode MRI images of brain tumor interested areas before clinical treatmentAnd predictive brain tumor growth signature +.>Inputting a prospective treatment visual model for finally obtaining the growth evolution of the brain tumor to obtain a plurality of different brain tumor growth evolution images, and selecting a brain tumor ROI growth evolution image meeting preset requirements according to the different brain tumor growth images;
step M6.5: inserting the obtained brain tumor ROI growth evolution image meeting the preset requirements into I by a poisson image editing method original Non-brain tumor region I of (C) background And obtaining a final brain tumor multi-mode MRI image to complete the brain tumor prospective treatment visualization task.
9. The brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method according to claim 8, wherein the step M6.2 comprises:
Step M6.2.1: text editor final feature F 0 Comprising the following steps:
where z is the noise vector sampled from a normal distribution in general,brain tumor predictive growth characteristic tag extracted from LSTM network>Features;
step M6.2.2: multi-mode MRI image deep learning features for brain tumor region of interest before and after treatmentAnd->And text editor final feature F 0 By concat operation, as input to the tumor growth prediction generator GGenerating a predictive treated region of interest multi-modality MRI image I of a brain tumor by a predictive generator G G 。
10. The brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method according to claim 8, wherein the step M6.5 comprises:
step M6.5.1: binarizing the obtained multiple different brain tumor growth evolution images to obtain brain tumor mask I G_mask ；
Step M6.5.2: brain tumor multi-target multi-mode MRI data I of pairing before and after treatment original And I later Each of the modality images and I G_mask Performing positioning and operation to obtain a background area non-brain tumor area I of brain tumor multi-target multi-mode MRI data background ；
Step No. 6.5.3: inserting a target image into a corresponding modality source image I background In (1), setting the source image gradient as Minimizing source image gradient +.>Thereby obtaining the target image inserted into the corresponding modal source image I background And (3) completing focus insertion according to the corresponding expected image f.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011279154.0A CN112365980B (en) | 2020-11-16 | 2020-11-16 | Brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011279154.0A CN112365980B (en) | 2020-11-16 | 2020-11-16 | Brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112365980A CN112365980A (en) | 2021-02-12 |
| CN112365980B true CN112365980B (en) | 2024-03-01 |
Family
ID=74515035
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011279154.0A Active CN112365980B (en) | 2020-11-16 | 2020-11-16 | Brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112365980B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112767420B (en) * | 2021-02-26 | 2021-11-23 | 中国人民解放军总医院 | Nuclear magnetic image segmentation method, device, equipment and medium based on artificial intelligence |
| WO2022221991A1 (en) * | 2021-04-19 | 2022-10-27 | 深圳市深光粟科技有限公司 | Image data processing method and apparatus, computer, and storage medium |
| CN113571203B (en) * | 2021-07-19 | 2024-01-26 | 复旦大学附属华山医院 | Multi-center federal learning-based brain tumor prognosis survival prediction method and system |
| CN113628325B (en) * | 2021-08-10 | 2024-03-26 | 海盐县南北湖医学人工智能研究院 | Model building method for small organ tumor evolution and computer readable storage medium |
| CN113717930A (en) * | 2021-09-07 | 2021-11-30 | 复旦大学附属华山医院 | Cranial carotid interlayer specific induced pluripotent stem cell line carrying FBN1 mutation |
| CN114927216A (en) * | 2022-04-29 | 2022-08-19 | 中南大学湘雅医院 | Method and system for predicting treatment effect of PD-1 of melanoma patient based on artificial intelligence |
| CN115861303B (en) * | 2023-02-16 | 2023-04-28 | 四川大学 | EGFR gene mutation detection method and system based on lung CT image |
| CN118429734A (en) * | 2024-07-05 | 2024-08-02 | 之江实验室 | Magnetic resonance data classification system |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106296699A (en) * | 2016-08-16 | 2017-01-04 | 电子科技大学 | Cerebral tumor dividing method based on deep neural network and multi-modal MRI image |
| CN109242860A (en) * | 2018-08-21 | 2019-01-18 | 电子科技大学 | Based on the brain tumor image partition method that deep learning and weight space are integrated |
| CN110136828A (en) * | 2019-05-16 | 2019-08-16 | 杭州健培科技有限公司 | A method of medical image multitask auxiliary diagnosis is realized based on deep learning |
| CN110265141A (en) * | 2019-05-13 | 2019-09-20 | 上海大学 | A kind of liver neoplasm CT images computer aided diagnosing method |
| EP3576020A1 (en) * | 2018-05-30 | 2019-12-04 | Siemens Healthcare GmbH | Methods for generating synthetic training data and for training deep learning algorithms for tumor lesion characterization, method and system for tumor lesion characterization, computer program and electronically readable storage medium |
| WO2020028382A1 (en) * | 2018-07-30 | 2020-02-06 | Memorial Sloan Kettering Cancer Center | Multi-modal, multi-resolution deep learning neural networks for segmentation, outcomes prediction and longitudinal response monitoring to immunotherapy and radiotherapy |
| CN111210909A (en) * | 2020-01-13 | 2020-05-29 | 青岛大学附属医院 | Deep neural network-based rectal cancer T stage automatic diagnosis system and construction method thereof |
| CN111445946A (en) * | 2020-03-26 | 2020-07-24 | 北京易康医疗科技有限公司 | Calculation method for calculating lung cancer genotyping by using PET/CT (positron emission tomography/computed tomography) images |
| CN111584073A (en) * | 2020-05-13 | 2020-08-25 | 山东大学 | Artificial intelligence fusion multi-modal information-based diagnosis model for constructing multiple pathological types of benign and malignant pulmonary nodules |
| CN111599464A (en) * | 2020-05-13 | 2020-08-28 | 吉林大学第一医院 | Novel multi-modal fusion auxiliary diagnosis method based on rectal cancer imaging omics research |
| WO2020190821A1 (en) * | 2019-03-15 | 2020-09-24 | Genentech, Inc. | Deep convolutional neural networks for tumor segmentation with positron emission tomography |
| KR20200114228A (en) * | 2019-03-28 | 2020-10-07 | 한국과학기술원 | Method and system for predicting isocitrate dehydrogenase (idh) mutation using recurrent neural network |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10282588B2 (en) * | 2016-06-09 | 2019-05-07 | Siemens Healthcare Gmbh | Image-based tumor phenotyping with machine learning from synthetic data |
| US11730387B2 (en) * | 2018-11-02 | 2023-08-22 | University Of Central Florida Research Foundation, Inc. | Method for detection and diagnosis of lung and pancreatic cancers from imaging scans |
-
2020
- 2020-11-16 CN CN202011279154.0A patent/CN112365980B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106296699A (en) * | 2016-08-16 | 2017-01-04 | 电子科技大学 | Cerebral tumor dividing method based on deep neural network and multi-modal MRI image |
| EP3576020A1 (en) * | 2018-05-30 | 2019-12-04 | Siemens Healthcare GmbH | Methods for generating synthetic training data and for training deep learning algorithms for tumor lesion characterization, method and system for tumor lesion characterization, computer program and electronically readable storage medium |
| WO2020028382A1 (en) * | 2018-07-30 | 2020-02-06 | Memorial Sloan Kettering Cancer Center | Multi-modal, multi-resolution deep learning neural networks for segmentation, outcomes prediction and longitudinal response monitoring to immunotherapy and radiotherapy |
| CN109242860A (en) * | 2018-08-21 | 2019-01-18 | 电子科技大学 | Based on the brain tumor image partition method that deep learning and weight space are integrated |
| WO2020190821A1 (en) * | 2019-03-15 | 2020-09-24 | Genentech, Inc. | Deep convolutional neural networks for tumor segmentation with positron emission tomography |
| KR20200114228A (en) * | 2019-03-28 | 2020-10-07 | 한국과학기술원 | Method and system for predicting isocitrate dehydrogenase (idh) mutation using recurrent neural network |
| CN110265141A (en) * | 2019-05-13 | 2019-09-20 | 上海大学 | A kind of liver neoplasm CT images computer aided diagnosing method |
| CN110136828A (en) * | 2019-05-16 | 2019-08-16 | 杭州健培科技有限公司 | A method of medical image multitask auxiliary diagnosis is realized based on deep learning |
| CN111210909A (en) * | 2020-01-13 | 2020-05-29 | 青岛大学附属医院 | Deep neural network-based rectal cancer T stage automatic diagnosis system and construction method thereof |
| CN111445946A (en) * | 2020-03-26 | 2020-07-24 | 北京易康医疗科技有限公司 | Calculation method for calculating lung cancer genotyping by using PET/CT (positron emission tomography/computed tomography) images |
| CN111584073A (en) * | 2020-05-13 | 2020-08-25 | 山东大学 | Artificial intelligence fusion multi-modal information-based diagnosis model for constructing multiple pathological types of benign and malignant pulmonary nodules |
| CN111599464A (en) * | 2020-05-13 | 2020-08-28 | 吉林大学第一医院 | Novel multi-modal fusion auxiliary diagnosis method based on rectal cancer imaging omics research |
Non-Patent Citations (3)
| Title |
|---|
| Liu,X.BTSC-TNAS: A neural architecture search-based transformer for brain tumor segmentation and classification.COMPUTERIZED MEDICAL IMAGING AND GRAPHICS.2023,110. * |
| 基于BP神经网络的脑肿瘤MRI图像分割;王锦程;郁芸;杨坤;胡新华;;生物医学工程研究(第04期);76-79 * |
| 王锦程 ; 郁芸 ; 杨坤 ; 胡新华 ; .基于BP神经网络的脑肿瘤MRI图像分割.生物医学工程研究.2016,(第04期),76-79. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112365980A (en) | 2021-02-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112365980B (en) | Brain tumor multi-target auxiliary diagnosis and prospective treatment evolution visualization method and system | |
| Zaharchuk et al. | Deep learning in neuroradiology | |
| CN111488914B (en) | Alzheimer disease classification and prediction system based on multitask learning | |
| US20230109108A1 (en) | Estimating uncertainty in predictions generated by machine learning models | |
| Liu et al. | An enhanced multi-modal brain graph network for classifying neuropsychiatric disorders | |
| CN117218453B (en) | Incomplete multi-mode medical image learning method | |
| Angkoso et al. | Multiplane Convolutional Neural Network (Mp-CNN) for Alzheimer’s Disease Classification. | |
| CN111709485A (en) | Medical image processing method and device and computer equipment | |
| Vu et al. | Deep convolutional neural networks for automatic segmentation of thoracic organs‐at‐risk in radiation oncology–use of non‐domain transfer learning | |
| CN115190999A (en) | Classifying data outside of a distribution using contrast loss | |
| Liu et al. | AHU-MultiNet: Adaptive loss balancing based on homoscedastic uncertainty in multi-task medical image segmentation network | |
| Toikkanen et al. | ReSGAN: Intracranial hemorrhage segmentation with residuals of synthetic brain CT scans | |
| CN118212501A (en) | Method, system, equipment and medium for detecting multiple focus of lung cancer | |
| Murmu et al. | A novel Gateaux derivatives with efficient DCNN-Resunet method for segmenting multi-class brain tumor | |
| Hu et al. | Automatic detection of melanins and sebums from skin images using a generative adversarial network | |
| CN116759076A (en) | Unsupervised disease diagnosis method and system based on medical image | |
| Padma et al. | Texture feature based analysis of segmenting soft tissues from brain CT images using BAM-type artificial neural network | |
| Tasnadi et al. | Structure preserving adversarial generation of labeled training samples for single-cell segmentation | |
| CN112633405A (en) | Model training method, medical image analysis device, medical image analysis equipment and medical image analysis medium | |
| Ramakrishnan et al. | Automated lung cancer nodule detection | |
| Li et al. | CAGAN: Classifier‐augmented generative adversarial networks for weakly‐supervised COVID‐19 lung lesion localisation | |
| CN118230074B (en) | Knowledge-aided migration learning breast cancer molecular subtype identification method and system | |
| Jesus et al. | Spatial normalization of MRI brain studies using a U-Net based neural network | |
| CN117078703B (en) | CT image segmentation method and system based on MRI guidance | |
| CN113850826B (en) | Image segmentation-based heart image processing method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |