WO2023194090A1 - Apprentissage à instances multiples prenant en compte des agrégations de voisinage - Google Patents
Apprentissage à instances multiples prenant en compte des agrégations de voisinage Download PDFInfo
- Publication number
- WO2023194090A1 WO2023194090A1 PCT/EP2023/057120 EP2023057120W WO2023194090A1 WO 2023194090 A1 WO2023194090 A1 WO 2023194090A1 EP 2023057120 W EP2023057120 W EP 2023057120W WO 2023194090 A1 WO2023194090 A1 WO 2023194090A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- feature vector
- patches
- patch
- neighborhoods
- machine learning
- Prior art date
Links
- 238000004220 aggregation Methods 0.000 title description 14
- 230000002776 aggregation Effects 0.000 title description 14
- 238000010801 machine learning Methods 0.000 claims abstract description 140
- 238000012549 training Methods 0.000 claims abstract description 107
- 238000000034 method Methods 0.000 claims abstract description 66
- 206010028980 Neoplasm Diseases 0.000 claims abstract description 59
- 206010064571 Gene mutation Diseases 0.000 claims abstract description 15
- 238000001514 detection method Methods 0.000 claims abstract description 6
- 238000012512 characterization method Methods 0.000 claims abstract description 5
- 239000013598 vector Substances 0.000 claims description 240
- -1 MET Proteins 0.000 claims description 24
- 108090000623 proteins and genes Proteins 0.000 claims description 21
- 230000015654 memory Effects 0.000 claims description 16
- 102100025064 Cellular tumor antigen p53 Human genes 0.000 claims description 14
- 230000007246 mechanism Effects 0.000 claims description 12
- 102100030708 GTPase KRas Human genes 0.000 claims description 11
- WZUVPPKBWHMQCE-UHFFFAOYSA-N Haematoxylin Chemical compound C12=CC(O)=C(O)C=C2CC2(O)C1C1=CC=C(O)C(O)=C1OC2 WZUVPPKBWHMQCE-UHFFFAOYSA-N 0.000 claims description 10
- 101000721661 Homo sapiens Cellular tumor antigen p53 Proteins 0.000 claims description 10
- 101000584612 Homo sapiens GTPase KRas Proteins 0.000 claims description 10
- 101000596894 Homo sapiens High affinity nerve growth factor receptor Proteins 0.000 claims description 9
- 102000009508 Cyclin-Dependent Kinase Inhibitor p16 Human genes 0.000 claims description 8
- 108010009392 Cyclin-Dependent Kinase Inhibitor p16 Proteins 0.000 claims description 8
- 102100035108 High affinity nerve growth factor receptor Human genes 0.000 claims description 8
- 101000579425 Homo sapiens Proto-oncogene tyrosine-protein kinase receptor Ret Proteins 0.000 claims description 8
- 101000984753 Homo sapiens Serine/threonine-protein kinase B-raf Proteins 0.000 claims description 8
- 102100028286 Proto-oncogene tyrosine-protein kinase receptor Ret Human genes 0.000 claims description 8
- 102100027103 Serine/threonine-protein kinase B-raf Human genes 0.000 claims description 8
- 102000052116 epidermal growth factor receptor activity proteins Human genes 0.000 claims description 8
- 108700015053 epidermal growth factor receptor activity proteins Proteins 0.000 claims description 8
- YOHYSYJDKVYCJI-UHFFFAOYSA-N n-[3-[[6-[3-(trifluoromethyl)anilino]pyrimidin-4-yl]amino]phenyl]cyclopropanecarboxamide Chemical compound FC(F)(F)C1=CC=CC(NC=2N=CN=C(NC=3C=C(NC(=O)C4CC4)C=CC=3)C=2)=C1 YOHYSYJDKVYCJI-UHFFFAOYSA-N 0.000 claims description 8
- 108010011536 PTEN Phosphohydrolase Proteins 0.000 claims description 7
- 102000014160 PTEN Phosphohydrolase Human genes 0.000 claims description 7
- 108010058546 Cyclin D1 Proteins 0.000 claims description 6
- 102100024165 G1/S-specific cyclin-D1 Human genes 0.000 claims description 6
- 101001030211 Homo sapiens Myc proto-oncogene protein Proteins 0.000 claims description 6
- 101000850794 Homo sapiens Tropomyosin alpha-3 chain Proteins 0.000 claims description 6
- 102100038895 Myc proto-oncogene protein Human genes 0.000 claims description 6
- 102100029166 NT-3 growth factor receptor Human genes 0.000 claims description 6
- 101001012157 Homo sapiens Receptor tyrosine-protein kinase erbB-2 Proteins 0.000 claims description 5
- 102100030086 Receptor tyrosine-protein kinase erbB-2 Human genes 0.000 claims description 5
- YQGOJNYOYNNSMM-UHFFFAOYSA-N eosin Chemical compound [Na+].OC(=O)C1=CC=CC=C1C1=C2C=C(Br)C(=O)C(Br)=C2OC2=C(Br)C(O)=C(Br)C=C21 YQGOJNYOYNNSMM-UHFFFAOYSA-N 0.000 claims description 5
- 102100033793 ALK tyrosine kinase receptor Human genes 0.000 claims description 4
- 102100021569 Apoptosis regulator Bcl-2 Human genes 0.000 claims description 4
- 108091012583 BCL2 Proteins 0.000 claims description 4
- 102100035080 BDNF/NT-3 growth factors receptor Human genes 0.000 claims description 4
- 108700020463 BRCA1 Proteins 0.000 claims description 4
- 102000036365 BRCA1 Human genes 0.000 claims description 4
- 101150072950 BRCA1 gene Proteins 0.000 claims description 4
- 102000052609 BRCA2 Human genes 0.000 claims description 4
- 108700020462 BRCA2 Proteins 0.000 claims description 4
- 101150008921 Brca2 gene Proteins 0.000 claims description 4
- 102100028914 Catenin beta-1 Human genes 0.000 claims description 4
- ZEOWTGPWHLSLOG-UHFFFAOYSA-N Cc1ccc(cc1-c1ccc2c(n[nH]c2c1)-c1cnn(c1)C1CC1)C(=O)Nc1cccc(c1)C(F)(F)F Chemical compound Cc1ccc(cc1-c1ccc2c(n[nH]c2c1)-c1cnn(c1)C1CC1)C(=O)Nc1cccc(c1)C(F)(F)F ZEOWTGPWHLSLOG-UHFFFAOYSA-N 0.000 claims description 4
- 102100021147 DNA mismatch repair protein Msh6 Human genes 0.000 claims description 4
- 101001003194 Eleusine coracana Alpha-amylase/trypsin inhibitor Proteins 0.000 claims description 4
- 102100023593 Fibroblast growth factor receptor 1 Human genes 0.000 claims description 4
- 101710182386 Fibroblast growth factor receptor 1 Proteins 0.000 claims description 4
- 102100029974 GTPase HRas Human genes 0.000 claims description 4
- 101000779641 Homo sapiens ALK tyrosine kinase receptor Proteins 0.000 claims description 4
- 101000596896 Homo sapiens BDNF/NT-3 growth factors receptor Proteins 0.000 claims description 4
- 101000916173 Homo sapiens Catenin beta-1 Proteins 0.000 claims description 4
- 101000968658 Homo sapiens DNA mismatch repair protein Msh6 Proteins 0.000 claims description 4
- 101000584633 Homo sapiens GTPase HRas Proteins 0.000 claims description 4
- 101000578287 Homo sapiens Non-POU domain-containing octamer-binding protein Proteins 0.000 claims description 4
- 101000686031 Homo sapiens Proto-oncogene tyrosine-protein kinase ROS Proteins 0.000 claims description 4
- 102100025725 Mothers against decapentaplegic homolog 4 Human genes 0.000 claims description 4
- 101710143112 Mothers against decapentaplegic homolog 4 Proteins 0.000 claims description 4
- 102100028102 Non-POU domain-containing octamer-binding protein Human genes 0.000 claims description 4
- 102100023347 Proto-oncogene tyrosine-protein kinase ROS Human genes 0.000 claims description 4
- 230000035772 mutation Effects 0.000 claims description 4
- 108010064892 trkC Receptor Proteins 0.000 claims description 4
- 102100022014 Angiopoietin-1 receptor Human genes 0.000 claims description 3
- 101000753291 Homo sapiens Angiopoietin-1 receptor Proteins 0.000 claims description 3
- 101000974343 Homo sapiens Nuclear receptor coactivator 4 Proteins 0.000 claims description 3
- 102100039137 Insulin receptor-related protein Human genes 0.000 claims description 3
- 102100022927 Nuclear receptor coactivator 4 Human genes 0.000 claims description 3
- 102100029981 Receptor tyrosine-protein kinase erbB-4 Human genes 0.000 claims description 3
- 101710100963 Receptor tyrosine-protein kinase erbB-4 Proteins 0.000 claims description 3
- 102100033080 Tropomyosin alpha-3 chain Human genes 0.000 claims description 3
- 102100033179 Vascular endothelial growth factor receptor 3 Human genes 0.000 claims description 3
- 108010054372 insulin receptor-related receptor Proteins 0.000 claims description 3
- QYAPHLRPFNSDNH-MRFRVZCGSA-N (4s,4as,5as,6s,12ar)-7-chloro-4-(dimethylamino)-1,6,10,11,12a-pentahydroxy-6-methyl-3,12-dioxo-4,4a,5,5a-tetrahydrotetracene-2-carboxamide;hydrochloride Chemical compound Cl.C1=CC(Cl)=C2[C@](O)(C)[C@H]3C[C@H]4[C@H](N(C)C)C(=O)C(C(N)=O)=C(O)[C@@]4(O)C(=O)C3=C(O)C2=C1O QYAPHLRPFNSDNH-MRFRVZCGSA-N 0.000 claims description 2
- 102100033822 A-kinase anchor protein 10, mitochondrial Human genes 0.000 claims description 2
- 102100040084 A-kinase anchor protein 9 Human genes 0.000 claims description 2
- 102000000872 ATM Human genes 0.000 claims description 2
- 102100035886 Adenine DNA glycosylase Human genes 0.000 claims description 2
- 102100034540 Adenomatous polyposis coli protein Human genes 0.000 claims description 2
- 101710119858 Alpha-1-acid glycoprotein Proteins 0.000 claims description 2
- 101100243447 Arabidopsis thaliana PER53 gene Proteins 0.000 claims description 2
- 108010004586 Ataxia Telangiectasia Mutated Proteins Proteins 0.000 claims description 2
- 102100035683 Axin-2 Human genes 0.000 claims description 2
- 102100025423 Bone morphogenetic protein receptor type-1A Human genes 0.000 claims description 2
- 102100033849 CCHC-type zinc finger nucleic acid binding protein Human genes 0.000 claims description 2
- 101710116319 CCHC-type zinc finger nucleic acid binding protein Proteins 0.000 claims description 2
- 102000000905 Cadherin Human genes 0.000 claims description 2
- 108050007957 Cadherin Proteins 0.000 claims description 2
- 102100024155 Cadherin-11 Human genes 0.000 claims description 2
- 102100026127 Clathrin heavy chain 1 Human genes 0.000 claims description 2
- 102100031048 Coiled-coil domain-containing protein 6 Human genes 0.000 claims description 2
- 102100027442 Collagen alpha-1(XII) chain Human genes 0.000 claims description 2
- 102100033775 Collagen alpha-5(IV) chain Human genes 0.000 claims description 2
- 102000016736 Cyclin Human genes 0.000 claims description 2
- 108050006400 Cyclin Proteins 0.000 claims description 2
- 108010016788 Cyclin-Dependent Kinase Inhibitor p21 Proteins 0.000 claims description 2
- 102000000577 Cyclin-Dependent Kinase Inhibitor p27 Human genes 0.000 claims description 2
- 108010016777 Cyclin-Dependent Kinase Inhibitor p27 Proteins 0.000 claims description 2
- 102100033270 Cyclin-dependent kinase inhibitor 1 Human genes 0.000 claims description 2
- 102100034157 DNA mismatch repair protein Msh2 Human genes 0.000 claims description 2
- 102100033587 DNA topoisomerase 2-alpha Human genes 0.000 claims description 2
- 108050002772 E3 ubiquitin-protein ligase Mdm2 Proteins 0.000 claims description 2
- 102000012199 E3 ubiquitin-protein ligase Mdm2 Human genes 0.000 claims description 2
- 101150076616 EPHA2 gene Proteins 0.000 claims description 2
- 101150016325 EPHA3 gene Proteins 0.000 claims description 2
- 101150097734 EPHB2 gene Proteins 0.000 claims description 2
- 102100027100 Echinoderm microtubule-associated protein-like 4 Human genes 0.000 claims description 2
- 108010055211 EphA1 Receptor Proteins 0.000 claims description 2
- 108010055323 EphB4 Receptor Proteins 0.000 claims description 2
- 101150078651 Epha4 gene Proteins 0.000 claims description 2
- 101150025643 Epha5 gene Proteins 0.000 claims description 2
- 102100030322 Ephrin type-A receptor 1 Human genes 0.000 claims description 2
- 102100030340 Ephrin type-A receptor 2 Human genes 0.000 claims description 2
- 102100030324 Ephrin type-A receptor 3 Human genes 0.000 claims description 2
- 102100021616 Ephrin type-A receptor 4 Human genes 0.000 claims description 2
- 102100021605 Ephrin type-A receptor 5 Human genes 0.000 claims description 2
- 102100021604 Ephrin type-A receptor 6 Human genes 0.000 claims description 2
- 102100021606 Ephrin type-A receptor 7 Human genes 0.000 claims description 2
- 102100021601 Ephrin type-A receptor 8 Human genes 0.000 claims description 2
- 102100030779 Ephrin type-B receptor 1 Human genes 0.000 claims description 2
- 102100031968 Ephrin type-B receptor 2 Human genes 0.000 claims description 2
- 102100031982 Ephrin type-B receptor 3 Human genes 0.000 claims description 2
- 102100031983 Ephrin type-B receptor 4 Human genes 0.000 claims description 2
- 241000402754 Erythranthe moschata Species 0.000 claims description 2
- 102100023600 Fibroblast growth factor receptor 2 Human genes 0.000 claims description 2
- 101710182389 Fibroblast growth factor receptor 2 Proteins 0.000 claims description 2
- 102100027842 Fibroblast growth factor receptor 3 Human genes 0.000 claims description 2
- 101710182396 Fibroblast growth factor receptor 3 Proteins 0.000 claims description 2
- 102100027844 Fibroblast growth factor receptor 4 Human genes 0.000 claims description 2
- 102100039788 GTPase NRas Human genes 0.000 claims description 2
- 102100022951 Gamma-secretase subunit APH-1A Human genes 0.000 claims description 2
- 102100030943 Glutathione S-transferase P Human genes 0.000 claims description 2
- 102100022057 Hepatocyte nuclear factor 1-alpha Human genes 0.000 claims description 2
- 102100038970 Histone-lysine N-methyltransferase EZH2 Human genes 0.000 claims description 2
- 102100028092 Homeobox protein Nkx-3.1 Human genes 0.000 claims description 2
- 101000779365 Homo sapiens A-kinase anchor protein 10, mitochondrial Proteins 0.000 claims description 2
- 101000890598 Homo sapiens A-kinase anchor protein 9 Proteins 0.000 claims description 2
- 101001000351 Homo sapiens Adenine DNA glycosylase Proteins 0.000 claims description 2
- 101000924577 Homo sapiens Adenomatous polyposis coli protein Proteins 0.000 claims description 2
- 101000874569 Homo sapiens Axin-2 Proteins 0.000 claims description 2
- 101000934638 Homo sapiens Bone morphogenetic protein receptor type-1A Proteins 0.000 claims description 2
- 101000762236 Homo sapiens Cadherin-11 Proteins 0.000 claims description 2
- 101000912851 Homo sapiens Clathrin heavy chain 1 Proteins 0.000 claims description 2
- 101000777370 Homo sapiens Coiled-coil domain-containing protein 6 Proteins 0.000 claims description 2
- 101000861874 Homo sapiens Collagen alpha-1(XII) chain Proteins 0.000 claims description 2
- 101000710886 Homo sapiens Collagen alpha-5(IV) chain Proteins 0.000 claims description 2
- 101001134036 Homo sapiens DNA mismatch repair protein Msh2 Proteins 0.000 claims description 2
- 101001057929 Homo sapiens Echinoderm microtubule-associated protein-like 4 Proteins 0.000 claims description 2
- 101000898696 Homo sapiens Ephrin type-A receptor 6 Proteins 0.000 claims description 2
- 101000898708 Homo sapiens Ephrin type-A receptor 7 Proteins 0.000 claims description 2
- 101000898676 Homo sapiens Ephrin type-A receptor 8 Proteins 0.000 claims description 2
- 101001064150 Homo sapiens Ephrin type-B receptor 1 Proteins 0.000 claims description 2
- 101001064458 Homo sapiens Ephrin type-B receptor 3 Proteins 0.000 claims description 2
- 101000917134 Homo sapiens Fibroblast growth factor receptor 4 Proteins 0.000 claims description 2
- 101000744505 Homo sapiens GTPase NRas Proteins 0.000 claims description 2
- 101001002170 Homo sapiens Glutamine amidotransferase-like class 1 domain-containing protein 3, mitochondrial Proteins 0.000 claims description 2
- 101001010139 Homo sapiens Glutathione S-transferase P Proteins 0.000 claims description 2
- 101001045751 Homo sapiens Hepatocyte nuclear factor 1-alpha Proteins 0.000 claims description 2
- 101000882127 Homo sapiens Histone-lysine N-methyltransferase EZH2 Proteins 0.000 claims description 2
- 101000578249 Homo sapiens Homeobox protein Nkx-3.1 Proteins 0.000 claims description 2
- 101000852815 Homo sapiens Insulin receptor Proteins 0.000 claims description 2
- 101001034652 Homo sapiens Insulin-like growth factor 1 receptor Proteins 0.000 claims description 2
- 101000917826 Homo sapiens Low affinity immunoglobulin gamma Fc region receptor II-a Proteins 0.000 claims description 2
- 101001106413 Homo sapiens Macrophage-stimulating protein receptor Proteins 0.000 claims description 2
- 101000582631 Homo sapiens Menin Proteins 0.000 claims description 2
- 101000979001 Homo sapiens Methionine aminopeptidase 2 Proteins 0.000 claims description 2
- 101000969087 Homo sapiens Microtubule-associated protein 2 Proteins 0.000 claims description 2
- 101001122114 Homo sapiens NUT family member 1 Proteins 0.000 claims description 2
- 101000801664 Homo sapiens Nucleoprotein TPR Proteins 0.000 claims description 2
- 101000605639 Homo sapiens Phosphatidylinositol 4,5-bisphosphate 3-kinase catalytic subunit alpha isoform Proteins 0.000 claims description 2
- 101001091365 Homo sapiens Plasma kallikrein Proteins 0.000 claims description 2
- 101000611614 Homo sapiens Proline-rich protein PRCC Proteins 0.000 claims description 2
- 101000605534 Homo sapiens Prostate-specific antigen Proteins 0.000 claims description 2
- 101000800847 Homo sapiens Protein TFG Proteins 0.000 claims description 2
- 101000798015 Homo sapiens RAC-beta serine/threonine-protein kinase Proteins 0.000 claims description 2
- 101000932478 Homo sapiens Receptor-type tyrosine-protein kinase FLT3 Proteins 0.000 claims description 2
- 101000728860 Homo sapiens Ribonuclease T2 Proteins 0.000 claims description 2
- 101000628562 Homo sapiens Serine/threonine-protein kinase STK11 Proteins 0.000 claims description 2
- 101000951145 Homo sapiens Succinate dehydrogenase [ubiquinone] cytochrome b small subunit, mitochondrial Proteins 0.000 claims description 2
- 101000728490 Homo sapiens Tether containing UBX domain for GLUT4 Proteins 0.000 claims description 2
- 101000795185 Homo sapiens Thyroid hormone receptor-associated protein 3 Proteins 0.000 claims description 2
- 101000837845 Homo sapiens Transcription factor E3 Proteins 0.000 claims description 2
- 101000837841 Homo sapiens Transcription factor EB Proteins 0.000 claims description 2
- 101000813738 Homo sapiens Transcription factor ETV6 Proteins 0.000 claims description 2
- 101000819074 Homo sapiens Transcription factor GATA-4 Proteins 0.000 claims description 2
- 101000843556 Homo sapiens Transcription factor HES-1 Proteins 0.000 claims description 2
- 101000807561 Homo sapiens Tyrosine-protein kinase receptor UFO Proteins 0.000 claims description 2
- 101000643895 Homo sapiens Ubiquitin carboxyl-terminal hydrolase 6 Proteins 0.000 claims description 2
- 101000808011 Homo sapiens Vascular endothelial growth factor A Proteins 0.000 claims description 2
- 102100036721 Insulin receptor Human genes 0.000 claims description 2
- 102100039688 Insulin-like growth factor 1 receptor Human genes 0.000 claims description 2
- 102100029204 Low affinity immunoglobulin gamma Fc region receptor II-a Human genes 0.000 claims description 2
- 229910015837 MSH2 Inorganic materials 0.000 claims description 2
- 101150078127 MUSK gene Proteins 0.000 claims description 2
- 102100021435 Macrophage-stimulating protein receptor Human genes 0.000 claims description 2
- 102100030550 Menin Human genes 0.000 claims description 2
- 102100023174 Methionine aminopeptidase 2 Human genes 0.000 claims description 2
- 108010074346 Mismatch Repair Endonuclease PMS2 Proteins 0.000 claims description 2
- 102000008071 Mismatch Repair Endonuclease PMS2 Human genes 0.000 claims description 2
- 102100025751 Mothers against decapentaplegic homolog 2 Human genes 0.000 claims description 2
- 101710143123 Mothers against decapentaplegic homolog 2 Proteins 0.000 claims description 2
- 102100038168 Muscle, skeletal receptor tyrosine-protein kinase Human genes 0.000 claims description 2
- 102000013609 MutL Protein Homolog 1 Human genes 0.000 claims description 2
- 108010026664 MutL Protein Homolog 1 Proteins 0.000 claims description 2
- 101150117329 NTRK3 gene Proteins 0.000 claims description 2
- 102100027086 NUT family member 1 Human genes 0.000 claims description 2
- 102000007530 Neurofibromin 1 Human genes 0.000 claims description 2
- 108010085793 Neurofibromin 1 Proteins 0.000 claims description 2
- 102100033615 Nucleoprotein TPR Human genes 0.000 claims description 2
- 102100026747 Osteomodulin Human genes 0.000 claims description 2
- 101150038994 PDGFRA gene Proteins 0.000 claims description 2
- 102100038332 Phosphatidylinositol 4,5-bisphosphate 3-kinase catalytic subunit alpha isoform Human genes 0.000 claims description 2
- 102100034869 Plasma kallikrein Human genes 0.000 claims description 2
- 102100040829 Proline-rich protein PRCC Human genes 0.000 claims description 2
- 102100033661 Protein TFG Human genes 0.000 claims description 2
- 102100032315 RAC-beta serine/threonine-protein kinase Human genes 0.000 claims description 2
- 102100029986 Receptor tyrosine-protein kinase erbB-3 Human genes 0.000 claims description 2
- 101710100969 Receptor tyrosine-protein kinase erbB-3 Proteins 0.000 claims description 2
- 102100020718 Receptor-type tyrosine-protein kinase FLT3 Human genes 0.000 claims description 2
- 102100029683 Ribonuclease T2 Human genes 0.000 claims description 2
- 101000744436 Saccharomyces cerevisiae (strain ATCC 204508 / S288c) Trans-acting factor D Proteins 0.000 claims description 2
- 102100026715 Serine/threonine-protein kinase STK11 Human genes 0.000 claims description 2
- 101710201479 Splicing factor, proline- and glutamine-rich Proteins 0.000 claims description 2
- 102100038014 Succinate dehydrogenase [ubiquinone] cytochrome b small subunit, mitochondrial Human genes 0.000 claims description 2
- 101150080074 TP53 gene Proteins 0.000 claims description 2
- 102100029773 Tether containing UBX domain for GLUT4 Human genes 0.000 claims description 2
- 102100029689 Thyroid hormone receptor-associated protein 3 Human genes 0.000 claims description 2
- 102100028507 Transcription factor E3 Human genes 0.000 claims description 2
- 102100028502 Transcription factor EB Human genes 0.000 claims description 2
- 102100039580 Transcription factor ETV6 Human genes 0.000 claims description 2
- 102100021380 Transcription factor GATA-4 Human genes 0.000 claims description 2
- 102100030798 Transcription factor HES-1 Human genes 0.000 claims description 2
- 108010078814 Tumor Suppressor Protein p53 Proteins 0.000 claims description 2
- 108010046308 Type II DNA Topoisomerases Proteins 0.000 claims description 2
- 102100037236 Tyrosine-protein kinase receptor UFO Human genes 0.000 claims description 2
- 102100021015 Ubiquitin carboxyl-terminal hydrolase 6 Human genes 0.000 claims description 2
- 108010053100 Vascular Endothelial Growth Factor Receptor-3 Proteins 0.000 claims description 2
- 102100039037 Vascular endothelial growth factor A Human genes 0.000 claims description 2
- 102100025093 Zinc fingers and homeoboxes protein 2 Human genes 0.000 claims description 2
- 101000851018 Homo sapiens Vascular endothelial growth factor receptor 1 Proteins 0.000 claims 1
- 108091007767 MALAT1 Proteins 0.000 claims 1
- 102000005650 Notch Receptors Human genes 0.000 claims 1
- 108010070047 Notch Receptors Proteins 0.000 claims 1
- 102100033178 Vascular endothelial growth factor receptor 1 Human genes 0.000 claims 1
- 101150048834 braF gene Proteins 0.000 claims 1
- 238000004590 computer program Methods 0.000 abstract description 8
- 210000001519 tissue Anatomy 0.000 description 47
- 238000012545 processing Methods 0.000 description 31
- 230000004927 fusion Effects 0.000 description 25
- 201000011510 cancer Diseases 0.000 description 21
- 230000006870 function Effects 0.000 description 20
- 230000008569 process Effects 0.000 description 20
- 238000003860 storage Methods 0.000 description 15
- 231100000590 oncogenic Toxicity 0.000 description 13
- 230000002246 oncogenic effect Effects 0.000 description 13
- 238000000605 extraction Methods 0.000 description 10
- 230000007170 pathology Effects 0.000 description 10
- 238000011282 treatment Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 230000004077 genetic alteration Effects 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 5
- 210000004027 cell Anatomy 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000002372 labelling Methods 0.000 description 5
- 210000002569 neuron Anatomy 0.000 description 5
- 210000000056 organ Anatomy 0.000 description 5
- 238000011160 research Methods 0.000 description 5
- 238000002560 therapeutic procedure Methods 0.000 description 5
- 210000001685 thyroid gland Anatomy 0.000 description 5
- 238000013459 approach Methods 0.000 description 4
- 238000001574 biopsy Methods 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 4
- 239000000975 dye Substances 0.000 description 4
- 231100000118 genetic alteration Toxicity 0.000 description 4
- 238000002595 magnetic resonance imaging Methods 0.000 description 4
- 238000007481 next generation sequencing Methods 0.000 description 4
- 206010058467 Lung neoplasm malignant Diseases 0.000 description 3
- 230000004931 aggregating effect Effects 0.000 description 3
- 230000004075 alteration Effects 0.000 description 3
- 238000002591 computed tomography Methods 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 3
- 201000010099 disease Diseases 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 229940079593 drug Drugs 0.000 description 3
- 239000003814 drug Substances 0.000 description 3
- 238000003364 immunohistochemistry Methods 0.000 description 3
- 239000003112 inhibitor Substances 0.000 description 3
- 210000002364 input neuron Anatomy 0.000 description 3
- 210000003734 kidney Anatomy 0.000 description 3
- 201000005202 lung cancer Diseases 0.000 description 3
- 208000020816 lung neoplasm Diseases 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 208000024770 Thyroid neoplasm Diseases 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000003556 assay Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 210000001185 bone marrow Anatomy 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000001839 endoscopy Methods 0.000 description 2
- 210000002950 fibroblast Anatomy 0.000 description 2
- 238000012252 genetic analysis Methods 0.000 description 2
- 238000011835 investigation Methods 0.000 description 2
- 210000004185 liver Anatomy 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000013188 needle biopsy Methods 0.000 description 2
- 238000012015 optical character recognition Methods 0.000 description 2
- 238000012014 optical coherence tomography Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 210000004205 output neuron Anatomy 0.000 description 2
- 102000004169 proteins and genes Human genes 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 210000003491 skin Anatomy 0.000 description 2
- 238000001356 surgical procedure Methods 0.000 description 2
- 201000002510 thyroid cancer Diseases 0.000 description 2
- 238000002604 ultrasonography Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 208000010507 Adenocarcinoma of Lung Diseases 0.000 description 1
- 206010052747 Adenocarcinoma pancreas Diseases 0.000 description 1
- 108091005625 BRD4 Proteins 0.000 description 1
- 208000018084 Bone neoplasm Diseases 0.000 description 1
- 206010006187 Breast cancer Diseases 0.000 description 1
- 208000026310 Breast neoplasm Diseases 0.000 description 1
- 102100029895 Bromodomain-containing protein 4 Human genes 0.000 description 1
- 241000282472 Canis lupus familiaris Species 0.000 description 1
- 201000009030 Carcinoma Diseases 0.000 description 1
- 208000017897 Carcinoma of esophagus Diseases 0.000 description 1
- 241000282693 Cercopithecidae Species 0.000 description 1
- 208000030808 Clear cell renal carcinoma Diseases 0.000 description 1
- 102100033601 Collagen alpha-1(I) chain Human genes 0.000 description 1
- 206010009944 Colon cancer Diseases 0.000 description 1
- 208000001333 Colorectal Neoplasms Diseases 0.000 description 1
- 101001004391 Drosophila melanogaster Protein jim lovell Proteins 0.000 description 1
- 102000001301 EGF receptor Human genes 0.000 description 1
- 102100036725 Epithelial discoidin domain-containing receptor 1 Human genes 0.000 description 1
- 101710131668 Epithelial discoidin domain-containing receptor 1 Proteins 0.000 description 1
- 101710113436 GTPase KRas Proteins 0.000 description 1
- 101000851181 Homo sapiens Epidermal growth factor receptor Proteins 0.000 description 1
- 101000916644 Homo sapiens Macrophage colony-stimulating factor 1 receptor Proteins 0.000 description 1
- 101001126417 Homo sapiens Platelet-derived growth factor receptor alpha Proteins 0.000 description 1
- 101000851030 Homo sapiens Vascular endothelial growth factor receptor 3 Proteins 0.000 description 1
- 208000008839 Kidney Neoplasms Diseases 0.000 description 1
- 102100028198 Macrophage colony-stimulating factor 1 receptor Human genes 0.000 description 1
- 241000124008 Mammalia Species 0.000 description 1
- 206010027476 Metastases Diseases 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 241000699670 Mus sp. Species 0.000 description 1
- 102000001759 Notch1 Receptor Human genes 0.000 description 1
- 108010029755 Notch1 Receptor Proteins 0.000 description 1
- 206010030155 Oesophageal carcinoma Diseases 0.000 description 1
- 241000283973 Oryctolagus cuniculus Species 0.000 description 1
- 206010033128 Ovarian cancer Diseases 0.000 description 1
- 206010061535 Ovarian neoplasm Diseases 0.000 description 1
- 108010051742 Platelet-Derived Growth Factor beta Receptor Proteins 0.000 description 1
- 102100030485 Platelet-derived growth factor receptor alpha Human genes 0.000 description 1
- 102100026547 Platelet-derived growth factor receptor beta Human genes 0.000 description 1
- 206010060862 Prostate cancer Diseases 0.000 description 1
- 208000000236 Prostatic Neoplasms Diseases 0.000 description 1
- 206010038019 Rectal adenocarcinoma Diseases 0.000 description 1
- 206010038389 Renal cancer Diseases 0.000 description 1
- 208000034254 Squamous cell carcinoma of the cervix uteri Diseases 0.000 description 1
- 208000033781 Thyroid carcinoma Diseases 0.000 description 1
- 102100022356 Tyrosine-protein kinase Mer Human genes 0.000 description 1
- 208000020990 adrenal cortex carcinoma Diseases 0.000 description 1
- 208000007128 adrenocortical carcinoma Diseases 0.000 description 1
- 108010029483 alpha 1 Chain Collagen Type I Proteins 0.000 description 1
- 230000008236 biological pathway Effects 0.000 description 1
- 230000031018 biological processes and functions Effects 0.000 description 1
- 239000000090 biomarker Substances 0.000 description 1
- 206010005084 bladder transitional cell carcinoma Diseases 0.000 description 1
- 201000001528 bladder urothelial carcinoma Diseases 0.000 description 1
- 238000004820 blood count Methods 0.000 description 1
- 230000036772 blood pressure Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 210000000481 breast Anatomy 0.000 description 1
- 230000035565 breathing frequency Effects 0.000 description 1
- 108010018804 c-Mer Tyrosine Kinase Proteins 0.000 description 1
- 231100000504 carcinogenesis Toxicity 0.000 description 1
- 208000002458 carcinoid tumor Diseases 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 201000006612 cervical squamous cell carcinoma Diseases 0.000 description 1
- 210000003679 cervix uteri Anatomy 0.000 description 1
- 208000006990 cholangiocarcinoma Diseases 0.000 description 1
- 206010073251 clear cell renal cell carcinoma Diseases 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 210000001072 colon Anatomy 0.000 description 1
- 201000010897 colon adenocarcinoma Diseases 0.000 description 1
- 208000029742 colonic neoplasm Diseases 0.000 description 1
- 208000030381 cutaneous melanoma Diseases 0.000 description 1
- 238000013481 data capture Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 210000004207 dermis Anatomy 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 230000003467 diminishing effect Effects 0.000 description 1
- 238000002091 elastography Methods 0.000 description 1
- 201000003683 endocervical adenocarcinoma Diseases 0.000 description 1
- 238000001861 endoscopic biopsy Methods 0.000 description 1
- 210000002615 epidermis Anatomy 0.000 description 1
- 201000005619 esophageal carcinoma Diseases 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000013534 fluorescein angiography Methods 0.000 description 1
- 238000002594 fluoroscopy Methods 0.000 description 1
- 210000000232 gallbladder Anatomy 0.000 description 1
- 201000006585 gastric adenocarcinoma Diseases 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 206010073071 hepatocellular carcinoma Diseases 0.000 description 1
- 231100000844 hepatocellular carcinoma Toxicity 0.000 description 1
- 230000002962 histologic effect Effects 0.000 description 1
- 238000013275 image-guided biopsy Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007901 in situ hybridization Methods 0.000 description 1
- 208000024312 invasive carcinoma Diseases 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 201000010982 kidney cancer Diseases 0.000 description 1
- 238000002357 laparoscopic surgery Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 201000007270 liver cancer Diseases 0.000 description 1
- 208000014018 liver neoplasm Diseases 0.000 description 1
- 210000004072 lung Anatomy 0.000 description 1
- 201000005249 lung adenocarcinoma Diseases 0.000 description 1
- 201000005243 lung squamous cell carcinoma Diseases 0.000 description 1
- 230000009401 metastasis Effects 0.000 description 1
- 238000001000 micrograph Methods 0.000 description 1
- 238000000386 microscopy Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 208000002154 non-small cell lung carcinoma Diseases 0.000 description 1
- 201000010302 ovarian serous cystadenocarcinoma Diseases 0.000 description 1
- 210000001672 ovary Anatomy 0.000 description 1
- 238000006213 oxygenation reaction Methods 0.000 description 1
- 210000000496 pancreas Anatomy 0.000 description 1
- 201000002094 pancreatic adenocarcinoma Diseases 0.000 description 1
- 230000001575 pathological effect Effects 0.000 description 1
- 210000003899 penis Anatomy 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000035479 physiological effects, processes and functions Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000002600 positron emission tomography Methods 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 210000002307 prostate Anatomy 0.000 description 1
- 201000005825 prostate adenocarcinoma Diseases 0.000 description 1
- 238000007388 punch biopsy Methods 0.000 description 1
- 238000002601 radiography Methods 0.000 description 1
- 210000000664 rectum Anatomy 0.000 description 1
- 201000001281 rectum adenocarcinoma Diseases 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000002271 resection Methods 0.000 description 1
- 210000004706 scrotum Anatomy 0.000 description 1
- 201000003708 skin melanoma Diseases 0.000 description 1
- 210000000813 small intestine Anatomy 0.000 description 1
- 210000000952 spleen Anatomy 0.000 description 1
- 238000010186 staining Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000004083 survival effect Effects 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000008685 targeting Effects 0.000 description 1
- 238000001931 thermography Methods 0.000 description 1
- 210000003813 thumb Anatomy 0.000 description 1
- 210000001541 thymus gland Anatomy 0.000 description 1
- 208000013077 thyroid gland carcinoma Diseases 0.000 description 1
- 238000003325 tomography Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 208000029729 tumor suppressor gene on chromosome 11 Diseases 0.000 description 1
- 210000000626 ureter Anatomy 0.000 description 1
- 210000003708 urethra Anatomy 0.000 description 1
- 210000003932 urinary bladder Anatomy 0.000 description 1
- 201000003701 uterine corpus endometrial carcinoma Diseases 0.000 description 1
- 210000004291 uterus Anatomy 0.000 description 1
- 230000036642 wellbeing Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F18/2155—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/09—Supervised learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
- G06V20/698—Matching; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30024—Cell structures in vitro; Tissue sections in vitro
Definitions
- Systems, methods, and computer programs disclosed herein relate to training a machine learning model to classify images, preferably medical images, using multiple instance learning techniques.
- the machine learning model can be trained and the trained machine learning model can be used for various purposes, in particular for the detection, identification and/or characterization of tumor types and/or gene mutations in tissues.
- Digital pathology is an image-based information environment enabled by computer technology that allows for the management of information generated from a digital slide. Scanning converts tissues on glass slides into digital whole-slide images for assessment, sharing, and analysis.
- Artificial intelligence methods can be used to analyze the digital images, e.g., for diagnostic performance improvement, discovery purposes, patient selection, treatment effects monitoring, etc.
- WO2020229152A1 discloses a method of identifying signs indicative of an NTRK oncogenic fusion within patient data comprising a histopathological image of tumor tissue using a deep neural network;
- C.-L. Chen et al. disclose a wholeslide training approach to pathological classification of lung cancer types using deep learning (https://doi.org/10.1038/s41467-021-21467-y).
- Image patches are typically rectangular regions with dimensions ranging from 32 x 32 pixels to 10000 x 10000 pixels.
- training data In order to train a machine learning model, training data is required.
- This training data usually includes images of tissue that is known to be tumor tissue or healthy tissue. The type of tumor present in each case may also be known. This data is also called labeled data and the labeling is usually done by experts.
- Patch-level labeling by expert pathologists is very time consuming. In specific clinical scenarios such as NTRK oncogenic fusion detection, a patch-level labeling might even be impossible as even the pathology experts may not be able to identify any decisive pattern.
- ground truth labeling in most cases is done at the level of the w-hole- slide images rather than at the level of individual patches. Labelled whole-slide images are weakly annotated data that can be analyzed using multiple instance learning techniques.
- multiple instance learning involves a set of bags that are labeled such that each bag consists of many unlabeled instances, i.e., the instances in the bags have no label information.
- the goal of multiple instance learning can be to train a classifier that assigns labels to test bags or assigns labels to the unlabeled instances in the bags.
- Each wdiole-slide image can be considered as a bag that contains many instances of patches.
- Multiple instance learning uses training sets that consist of bags w here each bag contains several instances that are either positive or negative examples for the class of interest, but only bag-level labels are given, and the instance-level labels are unknown during training.
- A. V. Konstantinov et al. propose to base a classification not on bags of individual patches, but to take into account adjacent patches (arXiv:2112.06071vl [cs.LG]).
- adjacent patches arXiv:2112.06071vl [cs.LG]
- the approach of the present disclosure is to perform classification not only on the basis of one or more selected patches and its/their nearest neighbors, but also to consider neighbors from more distant regions whose features are aggregated at different hierarchy levels, wherein the more distant the neighbors from the one or more selected patches, the higher the aggregation.
- the present disclosure provides a computer-implemented multiple instance learning method fortraining a model for classifying images, the method comprising: providing a machine learning model, wherein the machine learning model is configured to assign at least one patch of an image to one of at least two classes, training the machine learning model on a plurality of training images, each training image of the plurality of training images being assigned to one of the at least two classes, the training comprising: o receiving a training image, o generating a plurality of patches from the training image, o selecting at least one patch from the plurality of patches, o generating a feature vector on the basis of the selected patch, o defining a sequence of neighborhoods to the selected patch, wherein neighborhoods with increasing rank in the sequence of neighborhoods have an increasing distance to the selected patch and contain increasingly more patches, o selecting a number of neighboring patches in the neighborhoods and generating a feature vector for each selected neighboring patch, o generating a joint feature vector on the basis of the feature vector of the selected patch and the feature vectors of the selected
- the present disclosure provides a computer system comprising: a processor; and a memory storing an application program configured to perform, when executed by the processor, an operation, the operation comprising: providing a machine learning model, wherein the machine learning model is configured to assign at least one patch of an image to one of at least two classes, training the machine learning model on a plurality of training images, each training image of the plurality 7 of training images being assigned to one of the at least two classes, the training comprising: o receiving a training image, o generating a plurality of patches from the training image, o selecting at least one patch from the plurality of patches, o generating a feature vector on the basis of the selected patch, o defining a sequence of neighborhoods to the selected patch, wherein neighborhoods with increasing rank in the sequence of neighborhoods have an increasing distance to the selected patch and contain increasingly more patches, o selecting a number of neighboring patches in the neighborhoods and generating a feature vector for each selected neighboring patch, o generating a joint feature vector on the basis of the feature vector of
- the present disclosure provides a non-transitory computer readable medium having stored thereon software instructions that, when executed by a processor of a computer system, cause the computer system to execute the following steps: providing a machine learning model, wherein the machine learning model is configured to assign at least one patch of an image to one of at least two classes, training the machine learning model on a plurality of training images, each training image of the plurality 7 of training images being assigned to one of the at least two classes, the training comprising: o receiving a training image, o generating a plurality of patches from the training image, o selecting at least one patch from the plurality of patches, o generating a feature vector on the basis of the selected patch, o defining a sequence of neighborhoods to the selected patch, wherein neighborhoods with increasing rank in the sequence of neighborhoods have an increasing distance to the selected patch and contain increasingly more patches, o selecting a number of neighboring patches in the neighborhoods and generating a feature vector for each selected neighboring patch, o generating a joint feature vector on the basis of the
- the present disclosure relates to the use of a trained machine learning model for the detection, identification, and/or characterization of tumor types and/or gene mutations in tissues
- training of the machine learning model comprises: o receiving a training image, the training image being assigned to one of at least two classes, wherein at least one class comprises images of tumor tissue and/or tissue in which a gene mutation is present, o generating a plurality of patches from the training image, o selecting at least one patch from the plurality of patches, o generating a feature vector on the basis of the selected patch, o defining a sequence of neighborhoods to the selected patch, wherein neighborhoods with increasing rank in the sequence of neighborhoods have an increasing distance to the selected patch and contain increasingly more patches, o selecting a number of neighboring patches in the neighborhoods and generating a feature vector for each selected neighboring patch, o generating a joint feature vector on the basis of the feature vector of the selected patch and the feature vectors of the selected neighboring patches, wherein the feature vectors of the selected neighboring patches are aggregated
- the present disclosure provides a computer-implemented method for classifying an image, the method comprising: providing a machine learning model, wherein the machine learning model is configured to assign at least one patch of an image to one of at least two classes, training the machine learning model on a plurality of training images, each training image of the plurality of training images being assigned to one of the at least two classes, the training comprising: o receiving a training image, o generating a plurality of patches from the training image, o selecting at least one patch from the plurality of patches, o generating a feature vector on the basis of the selected patch, o defining a sequence of neighborhoods to the selected patch, wherein neighborhoods with increasing rank in the sequence of neighborhoods have an increasing distance to the selected patch and contain increasingly more patches, o selecting a number of neighboring patches in the neighborhoods and generating a feature vector for each selected neighboring patch, o generating a joint feature vector on the basis of the feature vector of the selected patch and the feature vectors of the selected neighboring patches, wherein the feature vector
- the term “one” or similar language is used.
- the terms “has”, “have”, “having”, or tlie like are intended to be open-ended terms.
- the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. Further, the phrase “based on” may mean “in response to” and be indicative of a condition for automatically triggering a specified operation of an electronic device (e.g., a controller, a processor, a computing device, etc.) as appropriately referred to herein.
- the present disclosure provides means for training a machine learning model and using the trained model for prediction purposes.
- Such a “machine learning model”, as used herein, may be understood as a computer implemented data processing architecture.
- Hie machine learning model can receive input data and provide output data based on that input data and on parameters of the machine learning model.
- the machine learning model can learn a relation between input data and output data through training. In training, parameters of the machine learning model may be adjusted in order to provide a desired output for a given input.
- the process of training a machine learning model involves providing a machine learning algorithm (that is the learning algorithm) with training data to learn from.
- the term “trained machine learning model” refers to the model artifact that is created by the training process.
- the training data must contain the correct answer, which is referred to as the target.
- the learning algorithm finds patterns in the training data that map input data to the target, and it outputs a trained machine learning model that captures these patterns.
- training data are inputted into the machine learning model and the machine learning model generates an output.
- the output is compared with the (known) target.
- Parameters of the machine learning model are modified in order to reduce the deviations between the output and the (known) target to a (defined) minimum.
- a loss function can be used for training, where the loss function can quantify the deviations between the output and the target.
- the loss function may be chosen in such a way that it rewards a wanted relation between output and target and/or penalizes an unwanted relation between an output and a target.
- Such a relation can be, e.g., a similarity, or a dissimilarity, or another relation.
- a loss function can be used to calculate a loss value for a given pair of output and target.
- the aim of the training process can be to modify (adjust) parameters of the machine learning model in order to reduce the loss value to a (defined) minimum.
- the machine learning model of the present disclosure is trained to assign a patch or a number of patches to one of at least two classes.
- a patch is one part of an image.
- the patch together with further patches, forms an image.
- the number of patches forming the image is usually more than 100, preferably more than 1000.
- image means a data structure that represents a spatial distribution of a physical signal.
- the spatial distribution may be of any dimension, for example 2D, 3D or 4D.
- the spatial distribution may be of any shape, for example forming a grid and thereby defining pixels, the grid being possibly irregular or regular.
- the physical signal may be any signal, for example color, level of gray, depth, surface or volume occupancy, such that the image may be a 2D or 3D RGB/grayscale/depth image, or a 3D surface/volume occupancy model.
- the invention is described herein mainly on the basis of two-dimensional images comprising a rectangular array of pixels. However, this is not to be understood as limiting the invention to such images. Those skilled in machine learning based on image data will know how to apply the invention to image data comprising more dimensions and/or being in a different format.
- the image is a medical image.
- a medical image is a visual representation of the human body or a part thereof or a visual representation of the body of an animal or a part thereof. Medical images can be used, e.g., for diagnostic and/or treatment purposes.
- Techniques for generating medical images include X-ray radiography, computerized tomography, fluoroscopy, magnetic resonance imaging, ultrasonography, endoscopy, elastography, tactile imaging, thermography, microscopy, positron emission tomography and others.
- Examples of medical images include CT (computer tomography) scans, X-ray images, MRI (magnetic resonance imaging) scans, fluorescein angiography images, OCT (optical coherence tomography) scans, histopathological images, ultrasound images.
- CT computer tomography
- X-ray images X-ray images
- MRI magnetic resonance imaging
- fluorescein angiography images fluorescein angiography images
- OCT optical coherence tomography
- the image is a whole slide histopathological image of tissue of a human body.
- the histopathological image is an image of a stained tissue sample.
- One or more dyes can be used to create the stained image. Usual dyes are hematoxylin and/or eosin.
- the image is labelled, i.e., the image is assigned to one of at least two classes; the label provides information about the class to which the image is assigned.
- the labelling can be done, e.g., by (medical) experts. For example, in histopathology, it is usually known what the tissue is that is depicted in a histopathological image. It is possible, for example, to take a genetic analysis of a tissue sample and identify gene mutations. A slide and a slide image can then be generated from the tissue sample. The slide image can then be labeled accordingly with the information obtained from the gene analysis, for example.
- the class can indicate whether the tissue shown in the image has a certain property or does not have the certain property’.
- the class can indicate whether the subject from which the tissue depicted in the image originates has a particular disease or does not have the particular disease.
- each part (patch) can be assigned to the class to which the image is assigned.
- the model is trained on the basis of a plurality of patches of a plurality of training images to assign a given patch or a number of given patches to the correct class.
- plurality means an integer greater than 1, usually greater than 10, preferably greater than 100.
- a number of patches can be generated from each training image.
- each training image is divided into the same number of patches.
- all patches have the same size and shape.
- the patches have a square or rectangular shape.
- the resolution of a patch is usually in the range of 32 pixels x 32 pixels to 10000 pixels x 10000 pixels, preferably in the range of 128 pixels x 128 pixel to 4096 pixels x 4096 pixels.
- a number of patches is selected.
- the number of selected patches can be, e.g., 1 to 5000.
- Preferably the number of selected patches is in the range from 10 to 500.
- the selection can be random or by rules. Rules for selecting patches can be found in the prior art on multiple instance learning.
- a feature vector is generated on the basis of the selected patch.
- a feature vector is an /w-dimensional vector of numerical features that represent an object, wherein m is an integer greater than 0.
- Many algorithms in machine learning require a numerical representation of objects since such representations facilitate processing and statistical analysis.
- the feature values might correspond to the pixels/voxels of the patch.
- the term “feature vector” shall also include single values, matrices, tensors, and the like.
- the generation of a feature vector is often accompanied by a dimension reduction in order to reduce an object (a patch) to those features that are important for the classification. Which features these are, the machine learning model learns during training.
- a feature extraction unit can be used to generate the feature vectors.
- the feature extraction unit can be an artificial neural network, for example. Examples of artificial neural networks which can be used for feature extraction are disclosed in: WO2020229152A1; J. Hoehne et al. Detecting genetic alterations in BRAF and NTRK as oncogenic drivers in digital pathology images: towards model generalization within and across multiple thyroid cohorts. Proceedings of Machine Learning Research 156, 2021, pages 1-12; A. V. Konstantinov et al. -. Multi-Attention Multiple Instance Learning, arXiv:2112.0607 Ivl [cs.LG], A feature vector is also referred to as “embedding” in the publications mentioned.
- Fig. 9 shows schematically by way of example a feature extraction unit/network.
- the feature extraction unit/network FEU comprises an input layer IL, a number n of hidden layers HLi to HL n and an output layer.
- the input neurons of the input layer IL serve to receive a patch P.
- the output neurons serve to output a feature vector FV (patch embedding).
- connection lines having a connection weight The neurons of the input layer IL and the hidden layer HLi are connected by connection lines having a connection weight, and the neurons of the hidden layer HL n and the output layer are also connected by connection lines with a connection weight. Similarly, the neurons of the hidden layers are connected to the neurons of neighboring hidden layers in a predetermined manner (not shown in Fig. 9).
- the connection weights can be learned through training.
- one or more feature extraction units used for feature extraction are or comprise a convolutional neural network (CNN).
- CNN convolutional neural network
- a CNN is a class of deep neural networks that comprises an input layer with input neurons, an output layer with at least one output neuron, as well as multiple hidden layers between the input layer and the output layer.
- the hidden layers of a CNN typically comprise filters (convolutional layer) and aggregation layers (pooling layer) which are repeated alternately and, at the end, one layer or multiple layers of completely connected neurons (dense/fully connected layer(s)).
- a sequence of neighborhoods to the selected patch is selected.
- the sequence of neighborhoods consists of a number n of neighborhoods: Nl, N2, . . . , No, wherein n is an integer greater than 1.
- the sequence of neighborhoods is a sequence in the mathematical sense.
- the number n of neighborhoods is usually in the range from 2 to 10, more preferably in the range from 3 to 8.
- neighborhoods in neighborhoods with increasing rank in the sequence of neighborhoods have an increasing distance to the selected patch.
- neighborhoods with increasing rank in the sequence of neighborhoods contain increasingly more patches.
- neighborhood N 1 is closest to the selected patch and neighborhood Nw is farthest from the selected patch; neighborhood N 1 includes the least number of patches and neighborhood Nw includes the most patches.
- Tire distance of a neighborhood to a selected patch can be, for example, an arithmetic distance averaged over all patches of the neighborhood.
- the neighborhoods extend around the selected patch.
- Fig. 1 shows schematically four examples ((a), (b), (c) and (d)) of sequences of neighborhoods in an image.
- the patches form a square grid in which each patch can be assigned an x-coordinate and a y-coordinate that determine its location within the grid.
- Patch P is a selected patch within the image I.
- Each neighborhood comprises a number of patches, each patch referred to as neighboring patch.
- the neighboring patches in neighborhood N 1 are also referred to herein as first-order neighbors
- the neighboring patches in neighborhood N2 are also referred to as second-order neighbors
- the neighboring patched in neighborhood N3 are also referred to as third-order neighbors and so on.
- neighboring patches in a neighborhood Nz are called z th -order neighbors, wherein z is an index that indicates the rank of the neighborhood in the sequence of the neighborhoods and can take a number from 1 to n.
- each selected patch with coordinates x, y has exactly eight first-order neighbors with the following coordinates:
- the brackets have been placed for better readability.
- the neighborhoods Nl, N2, N3, N4 form frames around the selected patch with a width of one patch, each frame enclosing the selected patch at its center.
- the neighborhoods have a greater distance to the selected patch with increasing rank in the order of the neighborhoods: N4 is further away from the selected patch than N3; N3 is further away from the selected patch than N2; N2 is further away from the selected patch than Nl .
- Nl consists of 8 patches: N2 consists of 16 patches: N3 consists of 24 patches; N4 consists of 32 patches.
- the number of patches in a neighborhood Rz is 8z, wherein z is an index that indicates the rank of the neighborhood in the sequence of the neighborhoods and can take a number from 1 to n.
- neighborhood N2 comprises neighborhood Nl
- neighborhood N3 comprises neighborhood N2
- neighborhood N4 comprises neighborhood N3.
- Fig. 1 (a) four neighborhoods are drawn. However, the sequence of neighborhoods can be continued up to the number n.
- the neighborhoods form frames around the selected patch with an increasing width.
- the width of the frame forming neighborhood Nl is 1 patch; the width of the frame forming neighborhood N2 is 2 patches; the width of the frame forming neighborhood N3 is 3 patches; the width of the frame forming neighborhood N4 is 4 patches.
- four neighborhoods are drawn. However, the sequence of neighborhoods can be continued up to the number n.
- the width of a frame forming the neighborhood Nz is z, wherein z is an index that indicates the rank of the neighborhood in the sequence of the neighborhoods and can take a number from 1 to n.
- neighborhood N2 comprises neighborhood Nl
- neighborhood N3 comprises neighborhood N2
- neighborhood N4 comprises neighborhood N3.
- the neighborhoods form frames around the selected patch with an increasing width.
- the width of the frame forming neighborhood Nl is 1 patch; the width of the frame forming neighborhood N2 is 2 patches; the width of the frame forming neighborhood N3 is 4 patches.
- the sequence of neighborhoods can be continued up to the number n.
- the width of a frame forming a neighborhood doubles from one neighborhood to the next in the sequence of neighborhoods.
- neighborhood N2 comprises neighborhood Nl
- neighborhood N3 comprises neighborhood N2.
- Fig. 1(d) shows a preferred embodiment of a sequence of neighborhoods.
- the neighborhoods form frames around the selected patch, each neighborhood enclosing the selected patch at its center.
- the frames of the neighborhoods have an increasing width.
- the width of the frame forming neighborhood N 1 is 1 patch; the width of the frame forming neighborhood N2 is 3 patches; the width of the frame forming neighborhood N3 is 9 patches.
- the sequence of neighborhoods can be continued up to the number n.
- the width of a frame forming a neighborhood triples from one neighborhood to the next in the sequence of neighborhoods.
- the width of the frame forming neighborhood Nz equals 3 ( ) , wherein i is an index that indicates the rank of the neighborhood in the sequence of the neighborhoods and can take a number from 1 to n.
- neighborhood N2 comprises neighborhood Nl
- neighborhood N3 comprises neighborhood N2.
- the neighborhoods shown in Fig. 1 are point-symmetric with respect to their center in which the selected patch is located; they each have the same extent in the x- and y-directions; the neighborhoods do not overlap; the neighborhoods are directly adjacent to each other. These are preferred properties of neighborhoods. However, it is also possible that the neighborhoods are not symmetrical, that they have different extents in x- and y- directions, that they overlap, and/or that they are spaced apart.
- Fig. 1 It is possible that the neighborhoods are defined differently than shown in Fig. 1. For example, it is possible to define the neighborhoods based on other distance metrics. An example is shown in Fig. 2.
- Fig. 2 shows an image I divided into a plurality of patches.
- a patch P is selected which has the coordinates x, y.
- the Manhattan distance is given for neighboring patches. More precisely, the Manhattan distance is given for those neighboring patches that are located at a maximum Manhattan distance of 8 patches from the selected patch P.
- a neighborhood can be formed by patches that have a defined Manhattan distance from the selected patch P.
- a first neighborhood with first-order neighbors can be formed by patches with a Manhattan distance of one;
- a second neighborhood with second-order neighbors can be formed by patches with a Manhattan distance of two;
- a third neighborhood with third-order neighbors can be formed by patches with a Manhattan distance of three, and so on.
- neighbors with different Manhattan distances may be combined into one neighborhood. For example, neighbors with a Manhattan distance of 1 or 2 may form a first neighborhood, neighbors with a Manhattan distance of 3 or 4 may form a second neighborhood, neighbors with a Manhattan distance of 5 or 6 may form a third neighborhood, and so on.
- the mean Manhattan distance can also increase with increasing order.
- neighbors with a Manhattan distance of 1 may form a first neighborhood, neighbors with a Manhattan distance of 2 or 3 a second neighborhood, neighbors with a Manhattan distance of 4, 5 or 6 a third neighborhood, and so on.
- the neighborhoods can also overlap.
- neighbors with a maximum Manhattan distance of 2 can form a first neighborhood
- neighbors with a maximum Manhattan distance of 4 can form a second neighborhood
- neighbors with a maximum Manhattan distance of 8 can form a third neighborhood, and so on.
- the present disclosure is not limited to specific definitions of neighborhoods. It's just that feature vectors of neighbors in a more distant neighborhood are aggregated to a higher degree than feature vectors of neighbors in a closer neighborhood to the selected patch. This aggregation is now described in more detail.
- each selected patch For each selected patch, a number of neighboring patches in the neighborhoods are selected. A preferred selection process is described below.
- a feature vector is generated. The same feature extraction unit that was used to generate the feature vector of the selected patch can be used to generate the feature vectors of the selected neighboring patches.
- each feature vector representing a selected neighboring patch has the same dimension.
- each feature vector representing a selected neighboring patch has the same dimension as the feature vector representing the selected patch.
- the joint feature vector forms the basis for the classification.
- the generation of the joint feature vector can be done in an iterative process, wherein feature vectors of selected neighboring patches are first aggregated before being fused with the feature vector of the selected patch.
- aggregation can take place sequentially at different hierarchy levels, with each hierarchy level corresponding to a neighborhood.
- hierarchy level means that a feature vector of a neighboring patch is aggregated with more feature vectors of other neighboring patches, the farther the neighborhood where the neighboring patch is located is from the selected patch.
- the distance of a neighborhood to a selected patch can be, for example, an arithmetic distance averaged over all patches of the neighborhood.
- the aggregation process is shown schematically in the form of an example in Fig. 3.
- the neighborhoods are defined as shown in Fig. 1 (d).
- Fig. 3 (a) shows a section of an image divided into a number of patches.
- a patch P with the coordinates x, y is selected.
- a first neighborhood N 1 extends.
- Neighborhood N 1 consists of eight first-order neighbors.
- the next step is to select a number of first-order neighboring patches in neighborhood Nl. Up to eight neighboring patches can be selected in neighborhood Nl . Multiple selection strategies are applicable, including a random selection of neighboring patches. The number of neighboring patches selected can also be random.
- Fig. 3 (b) shows that two of the eight neighboring patches in neighborhood N 1 have been selected: P’ and P”.
- a feature vector is generated (not shown in Fig. 3 (b)).
- the feature vectors of the selected neighboring patches are then aggregated into a neighborhood feature vector representing neighborhood N 1.
- Aggregation means that a single feature vector is generated from a number of feature vectors (at least two). In other words, a number of feature vectors are combined into one feature vector.
- the feature vectors are not simply appended (concatenated) to each other but are subjected to mathematical operations that produces a feature vector whose dimension is less than the sum of the dimensions of the feature vectors from which it is computed.
- the feature vectors can be added or multiplied element by element, or arithmetic averages can be formed element by element.
- the resulting neighborhood feature vector has the same dimension as any of the feature vectors from which it was generated.
- the neighborhood feature vector is a numerical representation of the neighborhood N 1.
- an attention mechanism is used to aggregate the feature vectors of the selected neighboring patches.
- atention is a technique that mimics cognitive atention. The effect enhances some parts of the input data while diminishing other parts - the thought being that the machine learning model should devote more focus to that small but important part of the data. Learning which part of the data is more important than others is trained in the training phase.
- an atention mechanism can be used as described in: A. V. Konstantinov et al. '. Multi-Attention Multiple Instance Learning, arXiv:21 12.06071vl [cs.LG] and/or in M. Use et al. '. Attention-based Deep Multiple Instance Learning, arXiv: 1802.04712v4 [cs.LG].
- an attention mechanism weights are applied to the feature vectors when they are combined into a single feature vector, and the machine learning model learns the weights during training, and thus learns what influence the different neighbors (represented by their respective feature vectors) have on the classification result.
- the neighborhood feature vector representing neighborhood N 1 can be fused with the feature vector of the selected patch P, e.g., using concatenation.
- the resulting feature vector is a joint representation of the selected patch and its nearest environment (its nearest neighborhood Nl).
- the selected patch together with neighborhood Nl is referred to herein as a first-order aggregate.
- Hie feature vector representing such first-order aggregate is referred to herein as first-order-aggregate feature vector.
- Fig. 3 (c) shows a first-order aggregate Al.
- the first-order aggregate Al contains the selected patch P and its neighborhood N 1 (see also Figs. 3 (a) and 3 (b)).
- the first order aggregate Al can be considered as a patch which is larger than the selected patch P (Al is eight times the size of patch P).
- the aggregate Al like the selected patch P, has a first neighborhood. This first neighborhood corresponds to the neighborhood N2 of the selected patch P.
- the process of selecting neighboring patches, generating feature vectors for neighboring patches, aggregating feature vectors of neighboring patches, and fusing the aggregated feature vectors with the feature vector of Al can be repeated as described with respect to Fig. 3 (a) and (b), except that the operations are performed at the next level of the hierarchy. It is like zooming out of the image and repeating the operations at a different level. Instead of patches, operations are performed based on aggregates of patches. This process of zooming out and repeating the step at the next hierarchy level can occur multiple times. Preferably, this process is repeated 2 to 8 times. It is now described how the process is carried out on the next hierarchy level.
- the neighborhood N2 extends around the aggregate Al (see Fig. 3 (c)).
- the neighborhood N2 has eight times the size of aggregate Al.
- the neighborhood N2 can be divided into eight neighboring first-order aggregates, all of which have the size of Al. This is depicted in Fig. 3 (d).
- a number of neighboring first-order aggregates is selected. Multiple selection strategies are applicable, including a random selection of aggregates. The number of aggregates selected can also be random.
- Fig. 3 (e) it is shown that two neighboring first-order aggregates Al’ and Al” have been selected. Now, for each selected first-order aggregate, a feature vector representing such aggregate needs to be generated. This can be done as described with respect to Fig. 3 (a), (b) and (c), and it is shown in Fig. 3 (f) and (g).
- two second-order neighboring patches Q and R have been selected.
- the selected second-order neighboring patches are located at specific locations within neighborhood N2: they are each located in a center of a first-order aggregate.
- second-order neighboring patches in neighborhood N2 which are located in a center of a first-order aggregate.
- the coordinates of these specific neighboring patches are: (x-3, y+3); (x, y+3); (x+3, y+3); (x-3, y); (x+3, y); (x-3, y- 3); (x, y-3); (x+3, y-3).
- the brackets have been placed for beter readability.
- a feature vector is generated for each selected second-order neighboring patch Q and R.
- the same feature extraction unit that was used to generate the feature vector of the selected patch and the feature vectors of the first-order neighbors can be used to generate the feature vectors of the selected second- order neighbors.
- these feature vectors of the selected second-order neighbors are aggregated with feature vectors of further neighboring patches (Q’, Q” and R’, R”, respectively) to generate feature vectors of the first-order aggregates Al’ and Al”.
- Each of the feature vectors representing the neighborhoods N 1 ’ and N F ’ can then be fused with the respective feature vector of the selected second-order neighbor the neighborhood includes. This results two first-order-aggregate feature vectors.
- the respective first-order aggregates AF and Al” are depicted in Fig. 3 (i).
- the feature vectors of the first-order aggregates Al, AF and Al” can then be fused into a second -order-aggregate feature vector representing the second-order aggregate A2 (see Fig. 3 (k)) which consists of the first-order aggregate Al and the neighborhood N2 (see Fig. 2 (j)).
- a feature vector representing the neighborhood N2 from the feature vectors representing the aggregates AF and A” is generated.
- An attention mechanism (as described with respect to Fig. 2 (b)) can be used for aggregating the feature vectors representing AF and Al” in a feature vector representing neighborhood N2.
- the feature vector representing neighborhood N2 can then be concatenated with the feature vector representing aggregate Al to generate a feature vector representing aggregate A2.
- Figs. 3 (1) to 3 (n) show a larger section of the image shown in Figs. 3 (a) to 3 (k).
- Neighborhood N3 extends around the aggregate A2.
- the Neighborhood N3 has eight times the size of aggregate A2.
- the neighborhood N3 can be divided into eight second- order aggregates, all of which have the size of A2 (see Fig. 3 (m)).
- Fig. 3 (m) represents the next (third) hierarchy level on which the process can be continued.
- Fig. 3 (d) represents the second hierarchy level and Fig. 3 (a) the first hierarchy level.
- a number of second-order aggregates neighbored to the second-order aggregate A2 can be selected. Multiple selection strategies are applicable, including a random selection of aggregates. The number of aggregates selected can also be random. For example, the process can be continued by selecting the second-order aggregates A2’ and A2” (see Fig. 3 (m)). In order to generate feature vectors representing the second-order aggregates A2’ and A2”, the process as described for second -order aggregate A can be applied to the second-order aggregates A2’ and A2”. The process can be started from the center patches of aggregates A2’ and A2”.
- center patches of an aggregate Az are characterized by the follow ing coordinates (in relation to the coordinates x, y of the selected patch): (x - 3 (,-1) , y + 3 (, ) ); (x, y + 3 ⁇ i ) ); (x + 3 (, ) , y + 3 (, ) ); (x - 3 o-i), y). ( x + 3 O-D, y). ( x _ 3 o-i), y _ 3 o-i))- ( x _ y _ 3 o-i))- ( x + 3 o- 1 ). y - 3 ⁇ ' )). wherein i is an index that can take a number from 1 to n.
- Fig. 3 (n) shows how 7 the process can be continued.
- Feature vectors can be generated for the selected center patches and these can be aggregated at various hierarchy levels until a number of second-order aggregates are obtained, which can then be aggregated into a third-order aggregate, and so on and so forth. This continues until an aggregate An is formed that is equal in size to the neighborhood Nn plus the aggregate A(w-l). Whenever feature vectors representing aggregates are aggregated to generate a feature vector representing a higher order neighborhood, an attention mechanism can be used.
- Fig. 3 (n) shows all patched, neighboring patches, first-order aggregates, and second-order aggregates which have been selected in the example depicted in Figs. 3 (a) to 3 (n), and from which feature vectors have been generated.
- Fig. 4 shows how a feature vector representing the third-order aggregates A3 (see Fig. 3 (n)) can be generated.
- the third-order-aggregate feature vector representing the third-order aggregate A3 can be a concatenation of feature vectors representing the selected patch P, the first neighborhood Nl, the second neighborhood N2, and the third neighborhood N3.
- the feature vector representing the first neighborhood Nl can be generated from the feature vectors representing patches PF and Pl” by aggregation using an attention mechanism.
- the feature vector representing the second neighborhood N2 can be generating from a feature vectors representing first-order aggregates AT and Al”.
- a feature vector representing first-order aggregate Al’ can be a concatenation of a feature vector representing neighborhood N F and a feature vector representing patch Q.
- a feature vector representing first-order aggregate Al” can be a concatenation of a feature vector representing neighborhood N 1 ” and a feature vector representing patch R.
- a feature vector representing neighborhood N F can be generated from feature vectors of patches Q’ and Q” by aggregation using an attention mechanism.
- a feature vector representing neighborhood Nl can be generated from feature vectors of patches R’ and R” by aggregation using an attention mechanism.
- the feature vector representing neighborhood N3 can also be generated, aggregating even more feature vectors from even more patches than in the case of N2 and Nl.
- This is not shown in Fig. 4.
- the feature vectors of two patches (P’, P”) are included in the generation of a feature vector representing neighborhood Nl
- the feature vectors of six patches Q, Q’, Q”, R, R’, R
- the feature vectors of 18 patches are included in the generation of a feature vector representing neighborhood N3.
- the farther a neighborhood is from the selected patch (P) the more patches are considered to generate a feature vector representing that neighborhood, and the greater the aggregation of feature vectors.
- the feature vector representing the aggregate Aw (A3 in the case of the example shown in Figs. 3 and 4) can be the joint feature vector of the selected patch and the selected neighboring patches. It includes information about the selected patch as well as information about neighboring patches in different neighborhoods at different distances from the selected patch.
- each selected patch there is a feature vector representing an aggregate that includes the selected patch and its environment.
- These feature vectors can be combined into a joint feature vector, for example by concatenation.
- the joint feature vector can be used for classification. Therefore, in a further step, the joint feature vector is fed to a classifier.
- the classifier is configured to assign the joint feature vector to one of the at least two classes.
- the classifier is part of machine learning model.
- the classifier can be, e.g., an artificial neural network. Examples of preferred classifiers can be found, e.g., in J. Hoehne et al. -. Detecting genetic alterations in BRAF and NTRK as oncogenic drivers in digital pathology images: towards model generalization within and across multiple thyroid cohorts, Proceedings of Machine Learning Research 156, 2021, pages 1-12; M. Y. Lu et al. -.
- Al-hased pathology predicts origins for cancers of unknown primary. Nature, 594(7861): 106-110, 2021; M. Use et al.: Attention-based deep multiple instance learning, International conference on machine learning, pages 2127-2136, PMLR, 2018; M. Use et al. -. Deep multiple instance learning for digital histopathology, Handbook of Medical Image Computing and Computer Assisted Intervention, pages 521-546. Elsevier, 2020.
- the classification result provided by the classifier can be analyzed.
- the class to which the joint feature vector is assigned can be compared with the class to which the training image is assigned. If the classes match, the classification is correct. If the classes do not match, the classification performed by the machine learning model is wrong, and parameters of the machine learning model need to be modified.
- a loss function can be used for quantifying the deviation between the output and the target. Examples of loss functions can be found, e.g., in J. Hoehne et al. : Detecting genetic alterations in BRAF and NTRK as oncogenic drivers in digital pathology images: towards model generalization within and across multiple thyroid cohorts, Proceedings of Machine Learning Research 156, 2021, pages 1-12; A. V. Konstantinov et al. -. Multi-Attention Multiple Instance Learning, arXiv:2112.06071vl [cs.LG]; M. Use et al. -. Attention-based Deep Multiple Instance Learning, arXiv: 1802.04712v4 [cs.LG]). The model is trained on a large number of images until the model reaches a defined accuracy in prediction.
- Fig. 5 shows schematically the architecture of the machine learning model of the present disclosure.
- the machine learning model receives a (training) image I and outputs a class C.
- the machine learning model comprises a (neighboring) patch selection unit (PSU) which is configured to select patches and neighboring patches as described herein.
- PSU patch selection unit
- FEU feature extraction unit
- Feature vectors (FVs) of neighboring patches are aggregated using an attention mechanism (AM), ad means attention distribution and ao means attention output.
- Feature vectors (FVs) are aggregated on different hierarchy levels.
- a joint feature vector (JFV) is generated comprising the feature vector of the at least one selected patch and feature vectors of aggregated patches at different hierarchy levels.
- the joint feature vector (JFV) is fed into the classifier (CF) and the classifier (CF) is configured to output the classification result (class output C).
- unit as used in this disclosure is not intended to imply that there is necessarily a separate unit performing the functions described. Rather, the term is intended to be understood to mean that computation means are present which perform the appropriate functions. These computation means are typically one or more processors configured to perform corresponding operations. Details are described below with reference to Fig. 6.
- the trained machine learning model can be stored in a data storage, transmitted to another computer system, or used to classify one or more new images.
- new means that the corresponding image was not used during training.
- the machine learning model can be trained to perform various tasks. Accordingly, a trained machine learning model can be used for various purposes.
- the machine leamin model of the present disclosure is trained and the trained machine learning model is uses to detect, identify, and/or characterize tumor types and/or gene mutations in tissues.
- the machine learning model can be trained and the trained machine learning model can be used to recognize a specific gene mutation and/or a specific tumor type, or to recognize multiple gene mutations and/or multiple tumor types.
- the machine learning model can be trained and the trained machine learning model can be used to characterize the type or types of cancer a patient or subject has.
- the machine learning model can be trained and the trained machine learning model can be used to select one or more effective therapies for the patient.
- the machine learning model can be trained and the trained machine learning model can be used to determine how a patient is responding over time to a treatment and, if necessary, to select a new therapy or therapies for the patient as necessary.
- the machine learning model can be trained and the trained machine learning model can be used to determine whether a patient should be included or excluded from participating in a clinical trial.
- the machine learning model can be trained and the trained machine learning model can be used to classify images of tumor tissue in one or more of the following classes: inflamed, non-inflamed, vascularized, non-vascularized, fibroblast-enriched, non-fibroblast-enriched (such classes are defined, e.g., in EP3639169A1).
- the machine learning model can be trained and the trained machine learning model can be used to identify 7 differentially expressed genes in a sample from a subject (e.g., a patient) having a cancer (e.g., a tumor).
- the machine learning model can be trained and the trained machine learning model can be used to identify genes that are mutated in a sample from a subject having a cancer (e.g., a tumor).
- the machine learning model can be trained and the trained machine learning model can be used to identify a cancer (e.g., a tumor) as a specific subtype of cancer selected.
- a cancer e.g., a tumor
- Such uses may be useful for clinical purposes including, for example, selecting a treatment, monitoring cancer progression, assessing the efficacy of a treatment against a cancer, evaluating suitability of a patient for participating in a clinical trial, or determining a course of treatment for a subject (e.g., a patient).
- the trained machine learning model may also be used for non-clinical purposes including (as a nonlimiting example) research purposes such as, e.g., studying the mechanism of cancer development and/or biological pathways and/or biological processes involved in cancer, and developing new therapies for cancer based on such studies.
- the machine learning model of the present disclosure is trained based on images and it generates predictions based on images.
- the images usually show the tissue of one or more subjects.
- the images can be created from tissue samples of a subject.
- the subject is usually a human, but may also be any mammal, including mice, rabbits, dogs, and monkeys.
- the tissue sample may be any sample from a subject known or suspected of having cancerous cells or pre-cancerous cells.
- the tissue sample may be from any source in the subject's body including, but not limited to, skin (including portions of the epidermis, dermis, and/or hypodermis), bone, bone marrow, brain, thymus, spleen, small intestine, appendix, colon, rectum, liver, gall bladder, pancreas, kidney, lung, ureter, bladder, urethra, uterus, ovary 7 , cervix, scrotum, penis, prostate.
- skin including portions of the epidermis, dermis, and/or hypodermis
- bone including portions of the epidermis, dermis, and/or hypodermis
- brain including portions of the epidermis, dermis, and/or hypodermis
- thymus spleen
- small intestine appendix
- colon colon
- rectum liver
- gall bladder pancreas
- kidney lung
- ureter bladder
- urethra
- the tissue sample may be a piece of tissue, or some or all of an organ.
- the tissue sample may be a cancerous tissue or organ or a tissue or organ suspected of having one or more cancerous cells.
- the tissue sample may be from a healthy (e.g. non-cancerous) tissue or organ.
- the tissue sample may include both healthy and cancerous cells and/or tissue.
- one sample has been taken from a subject for analysis. In some embodiments, more than one (e.g., 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, or more) samples may have been taken from a subject for analysis.
- one sample from a subject will be analyzed.
- more than one (e.g., 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, or more) samples may be analyzed. If more than one sample from a subject is analyzed, the samples may have been procured at the same time (e.g., more than one sample may be taken in the same procedure), or the samples may have been taken at different times (e.g., during a different procedure including a procedure 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 days; 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 weeks; 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 months, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 years, or 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 decades after a first procedure).
- a second or subsequent sample may be taken or obtained from the same region (e.g., from the same tumor or area of tissue) or a different region (including, e.g. a different tumor).
- a second or subsequent sample may be taken or obtained from the subject after one or more treatments and may be taken from the same region or a different region.
- the second or subsequent sample may be useful in determining whether the cancer in each sample has different characteristics (e.g., in the case of samples taken from two physically separate tumors in a patient) or whether the cancer has responded to one or more treatments (e.g., in the case of two or more samples from the same tumor prior to and subsequent to a treatment).
- the sample described herein may have been obtained from the subject using any known technique.
- the sample may have been obtained from a surgical procedure (e.g., laparoscopic surgery 7 , microscopically controlled surgery, or endoscopy), bone marrow biopsy, punch biopsy, endoscopic biopsy, or needle biopsy (e.g., a fine-needle aspiration, core needle biopsy, vacuum- assisted biopsy, or image-guided biopsy).
- a surgical procedure e.g., laparoscopic surgery 7 , microscopically controlled surgery, or endoscopy
- bone marrow biopsy e.g., punch biopsy, endoscopic biopsy, or needle biopsy (e.g., a fine-needle aspiration, core needle biopsy, vacuum- assisted biopsy, or image-guided biopsy).
- Detection, identification, and/or characterization of tumor types may be applied to any cancer and any tumor.
- Exemplary 7 cancers include, but are not limited to, adrenocortical carcinoma, bladder urothelial carcinoma, breast invasive carcinoma, cervical squamous cell carcinoma, endocervical adenocarcinoma, colon adenocarcinoma, esophageal carcinoma, kidney renal clear cell carcinoma, kidney renal papillary 7 cell carcinoma, liver hepatocellular carcinoma, lung adenocarcinoma, lung squamous cell carcinoma, ovarian serous cystadenocarcinoma, pancreatic adenocarcinoma, prostate adenocarcinoma, rectal adenocarcinoma, skin cutaneous melanoma, stomach adenocarcinoma, thyroid carcinoma, uterine corpus endometrial carcinoma, and cholangiocarcinoma.
- the machine learning model can be trained and the trained machine learning model can be used to detect, identify and/or characterize gene mutations in tissue samples.
- genes related to proliferation of cancer or response rates of molecular target drugs include HERZ, TOP2A, HER3, EGFR, P53, and MET.
- tyrosine kinase related genes include ALK, FLT3, AXL, FLT4 (VEGFR3, DDR1, FMS(CSFIR), DDR2, EGFR(ERBB 1), HER4(ERBB4), EML4- ALK, IGF1R, EPHA1, INSR, EPHA2, IRR(INSRR), EPHA3, KIT, EPHA4, LTK, EPHA5, MER(MERTK), EPHA6, MET, EPHA7, MUSK, EPHA8, NPM1-ALK, EPHB1, PDGFRa(PDGFRA), EPHB2, PDGFR0(PDGFRB)EPHB3, RET, EPHB4, RON(MSTIR), FGFR1, ROS(ROSl), FGFR2, TIE2(TEK), FGFR3, TRKA(NTRK1), FGFR4, TRKB(
- breast cancer related genes include ATM, BRCA1, BRCA2, BRCA3, CCND1, E-Cadherin, ERBB2, ETV6, FGFR1, HRAS, KRAS, NRAS, NTRK3, p53, and PTEN.
- genes related to carcinoid tumors include BCL2, BRD4, CCND1, CDKN1A, CDKN2A, CTNNB1, HES1, MAP2, MEN1, NF1, NOTCH1, NUT, RAF, SDHD, and VEGFA.
- colorectal cancer related genes include APC, MSH6, AXIN2, MYH, BMPR1A, p53, DCC, PMS2, KRAS2 (or Ki-ras), PTEN, MLH1, SMAD4, MSH2, STK11, and MSH6.
- lung cancer related genes include ALK, PTEN, CCND1, RASSF1A, CDKN2A, RBI, EGFR, RET, EML4, ROS1, KRAS2, TP53, and MYC.
- liver cancer related genes include Axinl, MALATl, b-catenin, p!6 INK4A, c-ERBB-2, p53, CTNNB1, RBI, Cyclin DI, SMAD2, EGFR, SMAD4, IGFR2, TCF1, and KRAS.
- kidney cancer related genes include Alpha, PRCC, ASPSCR1, PSF, CLTC, TFE3, p54nrb/NONO, and TFEB.
- thyroid cancer related genes include AKAP10, NTRK1, AKAP9, RET, BRAF, TFG, ELEI, TPM3, H4/D10S170, and TPR.
- Examples of ovarian cancer related genes include AKT2, MDM2, BCL2, MYC, BRCA1, NCOA4, CDKN2A, p53, ERBB2, PIK3CA, GATA4, RB, HRAS, RET, KRAS, and RNASET2.
- Examples of prostate cancer related genes include AR, KLK3, BRCA2, MYC, CDKN1B, NKX3.1, EZH2, p53, GSTP1, and PTEN.
- Examples of bone tumor related genes include CDH11, COL12A1, CNBP, OMD, COL1A1, THRAP3, COL4A5, and USP6.
- the machine learning model is trained and used for classification of tissue types on the basis of whole slide images.
- the machine learning model is trained and used for identification of gene mutations, such as BRAF mutations and/or NTRK fusions, as described in WO2020229152A1 and/or J. Hoehne et al. -. Detecting genetic alterations in BRAF and NTRK as oncogenic drivers in digital pathology images: towards model generalization within and across multiple thyroid cohorts, Proceedings of Machine Learning Research 156, 2021, pages 1-12, the contents of which are incorporated by reference in their entirety into this specification.
- the machine learning model can be trained to detect signs of the presence of oncogenic drivers in patient tissue images stained with hematoxylin and eosin.
- Penault-Llorca et al. describe a testing algorithm for identification of patients with TRK fusion cancer (see J. Clin. Pathol., 2019, 72, 460-467).
- the algorithm comprises immunohistochemistry 7 (IHC) studies, fluorescence in situ hybridization (FISH) and next-generation sequencing.
- Immunohistochemistry 7 provides a routine method to detect protein expression of NTRK genes. However, performing immunohistochemistry 7 requires additional tissue section(s) and time to proceed and interprete (following hematoxylin and eosin initial staining based on which tumor diagnosis is performed), skills and the correlation between protein expression and gene fusion status is not trivial. Interpretation of IHC results requires the skills of a trained and certified medical professional pathologist.
- Next-generation sequencing provides a precise method to detect NTRK gene fusions.
- performing gene analyses for each patient is expensive, tissue consuming (not always feasible when available tissue specimen is minimal, as in diagnostic biopsies), not universally available in various geographic locations or diagnostic laboratories/healthcare institutions and, due to the low incidence of NTRK oncogenic fusions, inefficient.
- CT scans computed tomography
- MRI magnetic resonance imaging
- Additional studies may also be considered, such as other forms of medical imaging (CT scans, MRI, etc.) that can be co-assessed using Al to generate multimodal biomarkers/characteristics for diagnostic purposes.
- the machine learning model of the present disclosure can, e.g., be used to a) detect NTRK fusion events in one or more indications, b) detect NTRK fusion events in other indications than in those being trained on (i.e., an algorithm trained on thyroid data sets is useful in lung cancer data sets), c) detect NTRK fusion events involving other TRK family members (i.e., an algorithm trained on NTRK1, NTRK3 fusions is useful to predict also NTRK2 fusions), d) detect NTRK fusion events involving other fusion partners (i.e., an algorithm trained on LMNA- fusion data sets is useful also in TPM3 -fusion data sets), e) discover novel fusion partners (i.e., an algorithm trained on known fusion events might predict a fusion in a new data set wdiich is then confirmed via molecular assay to involve a not yet described fusion partner of a NTRK family member), f) catalyze the diagnostic ww
- NTRK-fusion inhibitors g) identify 7 specific genetic aberrations based on histological specimen can additionally be used to confirm/exclude or re-label certain tumor diagnosis, in cases the presence or absence of this/these alterations(s) is pathognomonic of specific tumors.
- Identification of specific genetic aberrations based on histological specimen can additionally be used to confirm/exclude or re-label certain tumor diagnosis, in cases the presence or absence of this/these alterations(s) is pathognomonic of specific tumors.
- Histopathological images used fortraining and prediction of the machine learning model can be obtained from patients by biopsy or surgical resection specimens.
- a histopathological image is a microscopic image of tumor tissue of a human patient.
- the histopathological image is a whole-slide image.
- the histopathological image is an image of a stained tumor tissue sample.
- One or more dyes can be used to create the stained images.
- Preferred dyes are hematoxylin and/or eosin.
- the machine learning model can also be configured to generate a probability value, the probability value indicating the probability of a patient suffering from cancer, e.g., caused by an NTRK oncogenic fusion.
- the probability 7 value can be outputted to a user and/or stored in a database.
- the probability value can be a real number in the range from 0 to 1, whereas a probability value of 0 usually means that it is impossible that the cancer is caused by an NTRK oncogenic fusion, and a probability value of 1 usually means that there is no doubt that the cancer is caused by an NTRK oncogenic fusion.
- the probability value can also be expressed by a percentage.
- the probability value is compared with a predefined threshold value. In the event the probability value is lower than the threshold value, the probability that the patient suffers from cancer caused by an NTRK oncogenic fusion is low; treating the patient with a Trk inhibitor is not indicated; further investigations are required in order to determine the cause of cancer. In the event the probability value equals the threshold value or is greater than the threshold value, it is reasonable to assume that the cancer is caused by an NTRK oncogenic fusion; the treatment of the patient with a Trk inhibitor can be indicated; further investigations to verify the assumption can be initiated (e.g., performing a genetic analysis of the tumor tissue).
- the threshold value can be a value between 0.5 and 0.99999999999, e.g. 0.8 (80%) or 0.81 (81 %) or 0.82 (82% ) or 0.83 (83%) or 0.84 (84%) or 0.85 (85%) or 0.86 (86%) or 0.87 (87 %) or 0.88 (88%) or 0.89 (89%) or 0.9 (90%) or 0.91 (91 %) or 0.92 (92% ) or 0.93 (93%) or 0.94 (94%) or 0.95 (95%) or 0.96 (96%) or 0.97 (97 %) or 0.98 (98%) or 0.99 (99%) or any other value (percentage).
- the threshold value can be determined by a medical expert.
- Additional patient data can also be included in the classification.
- Additional patient data can be, e.g., anatomic or physiology data of the patient, such as information about patient's height and weight, gender, age, vital parameters (such as blood pressure, breathing frequency and heart rate), tumor grades, ICD-9 classification, oxygenation of tumor, degree of metastasis of tumor, blood count value tumor indicator value like PA value, information about the tissue the histopathological image is created from (e.g. tissue type, organ), further symptoms, medical history 7 etc.
- the pathology report of the histopathological images can be used for classification, using text mining approaches.
- a next generation sequencing raw 7 data set which does not cover the TRK genes’ sequences can be used for classification.
- non-transitory is used herein to exclude transitory, propagating signals or waves, but to otherwise include any volatile or non-volatile computer memory' technology suitable to the application.
- the term “computer” should be broadly construed to cover any kind of electronic device with data processing capabilities, including, by way of non-limiting example, personal computers, servers, embedded cores, computing system, communication devices, processors (e.g., digital signal processor (DSP)), microcontrollers, field programmable gate array (FPGA), application specific integrated circuit (ASIC), etc.) and other electronic computing devices.
- processors e.g., digital signal processor (DSP)
- microcontrollers e.g., field programmable gate array (FPGA), application specific integrated circuit (ASIC), etc.
- ASIC application specific integrated circuit
- processor as used above is intended to include any type of computation or manipulation or transformation of data represented as physical, e.g., electronic, phenomena which may occur or reside e.g., within registers and/or memories of at least one computer or processor.
- the tenn processor includes a single processing unit or a plurality of distributed or remote such units.
- Fig. 6 illustrates a computer system (1) according to some example implementations of the present disclosure in more detail.
- the computer may include one or more of each of a number of components such as, for example, processing unit (20) connected to a memory' (50) (e.g., storage device).
- processing unit (20) connected to a memory' (50) (e.g., storage device).
- memory' e.g., storage device
- the processing unit (20) may be composed of one or more processors alone or in combination with one or more memories.
- the processing unit is generally any piece of computer hardware that is capable of processing information such as, for example, data, computer programs and/or other suitable electronic information.
- the processing unit is composed of a collection of electronic circuits some of which may be packaged as an integrated circuit or multiple interconnected integrated circuits (an integrated circuit at times more commonly referred to as a “chip”).
- the processing unit may be configured to execute computer programs, which may be stored onboard the processing unit or otherwise stored in the memory (50) of the same or another computer.
- the processing unit (20) may be a number of processors, a multi -core processor or some other type of processor, depending on the particular implementation. Further, the processing unit may be implemented using a number of heterogeneous processor systems in which a main processor is present with one or more secondary' processors on a single chip. As another illustrative example, the processing unit may be a symmetric multi-processor system containing multiple processors of the same type. In yet another example, the processing unit may be embodied as or otherwise include one or more ASICs, FPGAs or the like. Thus, although the processing unit may be capable of executing a computer program to perform one or more functions, the processing unit of various examples may be capable of performing one or more functions without the aid of a computer program. In either instance, the processing unit may be appropriately programmed to perform functions or operations according to example implementations of the present disclosure.
- the memory' (50) is generally any piece of computer hardware that is capable of storing information such as, for example, data, computer programs (e.g., computer-readable program code (60)) and/or other suitable information either on a temporary' basis and/or a permanent basis.
- the memory' may include volatile and/or non-volatile memory', and may be fixed or removable. Examples of suitable memory' include random access memory' (RAM), read-only memory-’ (ROM), a hard drive, a flash memory-’, a thumb drive, a removable computer diskette, an optical disk, a magnetic tape or some combination of the above.
- Optical disks may include compact disk - read only memory' (CD-ROM), compact disk - read/yvrite (CD-R/W), DVD, Blu-ray disk or the like.
- the memory' may be referred to as a computer-readable storage medium.
- the computer-readable storage medium is a non-transitory' device capable of storing information, and is distinguishable from computer-readable transmission media such as electronic transitory' signals capable of carrying information from one location to another.
- Computer-readable medium as described herein may generally refer to a computer-readable storage medium or computer-readable transmission medium.
- the processing unit (20) may also be connected to one or more interfaces for displaying, transmitting and/or receiving information.
- the interfaces may include one or more communications interfaces and/or one or more user interfaces.
- the communications interface(s) may be configured to transmit and/or receive information, such as to and/or from other computer(s), network(s), database(s) or the like.
- the communications interface may be configured to transmit and/or receive information by physical (wired) and/or wireless communications links.
- the communications interface(s) may include interface(s) (41) to connect to a network, such as using technologies such as cellular telephone, Wi-Fi, satellite, cable, digital subscriber line (DSL), fiber optics and the like.
- the communications interface(s) may include one or more short-range communications interfaces (42) configured to connect devices using short-range communications technologies such as NFC, RFID, Bluetooth, Bluetooth LE, ZigBee, infrared (e.g., IrDA) or the like.
- short-range communications technologies such as NFC, RFID, Bluetooth, Bluetooth LE, ZigBee, infrared (e.g., IrDA) or the like.
- the user interfaces may include a display (30).
- the display may be configured to present or otherwise display information to a user, suitable examples of which include a liquid cry stal display (LCD), lightemitting diode display (LED), plasma display panel (PDP) or the like.
- the user input interface(s) (11) may be wired or wireless, and may be configured to receive information from a user into the computer system (1), such as for processing, storage and/or display. Suitable examples of user input interfaces include a microphone, image or video capture device, keyboard or keypad, joystick, touch -sensitive surface (separate from or integrated into a touchscreen) or the like.
- the user interfaces may include automatic identification and data capture (AIDC) technology (12) for machine-readable information. This may include barcode, radio frequency identification (RFID), magnetic stripes, optical character recognition (OCR), integrated circuit card (ICC), and the like.
- the user interfaces may further include one or more interfaces for communicating with peripherals such as printers and the like.
- program code instructions may be stored in memory, and executed by processing unit that is thereby programmed, to implement functions of the systems, subsystems, tools and their respective elements described herein.
- any suitable program code instructions may be loaded onto a computer or other programmable apparatus from a computer-readable storage medium to produce a particular machine, such that the particular machine becomes a means for implementing the functions specified herein.
- These program code instructions may also be stored in a computer-readable storage medium that can direct a computer, processing unit or other programmable apparatus to function in a particular manner to thereby generate a particular machine or particular article of manufacture.
- the instructions stored in the computer-readable storage medium may produce an article of manufacture, where the article of manufacture becomes a means for implementing functions described herein.
- the program code instructions may be retrieved from a computer-readable storage medium and loaded into a computer, processing unit or other programmable apparatus to configure the computer, processing unit or other programmable apparatus to execute operations to be performed on or by the computer, processing unit or other programmable apparatus.
- Retrieval, loading and execution of the program code instructions may be performed sequentially such that one instruction is retrieved, loaded and executed at a time. In some example implementations, retrieval, loading and/or execution may be performed in parallel such that multiple instructions are retrieved, loaded, and/or executed together. Execution of the program code instructions may produce a computer-implemented process such that the instructions executed by the computer, processing circuitry or other programmable apparatus provide operations for implementing functions described herein.
- a computer system (1) may include processing unit (20) and a computer-readable storage medium or memory (50) coupled to the processing circuitry 7 , where the processing circuitry 7 is configured to execute computer-readable program code (60) stored in the memory.
- processing circuitry 7 is configured to execute computer-readable program code (60) stored in the memory.
- computer-readable program code 60
- Fig. 7 shows a preferred method of training the machine learning model of the present disclosure in the form of a flowchart. The method (100) comprises the steps:
- (110) providing a machine learning model, wherein the machine learning model is configured to assign at least one patch of an image to one of at least two classes,
- Fig. 8 shows a preferred method of using the trained machine learning model for classifying an image in the form of a flowchart.
- Hie computer-implemented method (200) comprises the steps:
- (210) providing a trained machine learning model, wherein the trained machine learning model is configured and trained in a multiple instance training method as described herein to assign an image to one of at least tw o classes,
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- Quality & Reliability (AREA)
- Multimedia (AREA)
- Radiology & Medical Imaging (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Medical Informatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Image Analysis (AREA)
Abstract
Des systèmes, des procédés et des programmes informatiques de l'invention se rapportent à l'apprentissage d'un modèle d'apprentissage automatique pour classifier des images, de préférence des images médicales, à l'aide de multiples techniques d'apprentissage d'instances. Le modèle d'apprentissage automatique peut être entraîné et le modèle d'apprentissage automatique entraîné peut être utilisé à diverses fins, en particulier pour la détection, l'identification et/ou la caractérisation de types de tumeur et/ou de mutations géniques dans des tissus.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202263329127P | 2022-04-08 | 2022-04-08 | |
US63/329,127 | 2022-04-08 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023194090A1 true WO2023194090A1 (fr) | 2023-10-12 |
Family
ID=85800288
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/EP2023/057120 WO2023194090A1 (fr) | 2022-04-08 | 2023-03-21 | Apprentissage à instances multiples prenant en compte des agrégations de voisinage |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2023194090A1 (fr) |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3639169A1 (fr) | 2017-06-13 | 2020-04-22 | BostonGene, Corporation | Systèmes et procédés de génération, de visualisation et classification de profils fonctionnels moléculaires |
WO2020229152A1 (fr) | 2019-05-10 | 2020-11-19 | Bayer Consumer Care Ag | Identification de signes candidats indiquant une fusion oncogénique ntrk |
-
2023
- 2023-03-21 WO PCT/EP2023/057120 patent/WO2023194090A1/fr unknown
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3639169A1 (fr) | 2017-06-13 | 2020-04-22 | BostonGene, Corporation | Systèmes et procédés de génération, de visualisation et classification de profils fonctionnels moléculaires |
WO2020229152A1 (fr) | 2019-05-10 | 2020-11-19 | Bayer Consumer Care Ag | Identification de signes candidats indiquant une fusion oncogénique ntrk |
Non-Patent Citations (19)
Title |
---|
A. F. FRANGI ET AL.: "Medical Image Computing and Computer Assisted Intervention - MICCAI 2018", INTERNATIONAL CONFERENCE GRANADA, 2018 |
A. V. KONSTANTINOV ET AL.: "Multi-Attention Multiple Instance Learning", ARXIV:2112.0607 |
A. V. KONSTANTINOV ET AL.: "Multi-Attention Multiple Instance Learning", ARXIV:2112.06071 |
A. V. KONSTANTINOV ET AL.: "Multi-Attention Multiple Instance Learning", ARXIV:2112.06071VL |
F. PENAULT-LLORCA ET AL., J. CLIN. PATHOL., vol. 72, 2019, pages 460 - 467 |
G.A. TSIHRINTZISL.C. JAIN: "Learning and Analytics in Intelligent Systems", vol. 18, 2020, SPRINGER NATURE, article "Machine Learning Paradigms: Advances in Deep Learning-based Technological Applications" |
J. HOEHNE ET AL.: "Detecting genetic alterations in BRAF and NTRK as oncogenic drivers in digital pathology images: towards model generalization within and across multiple thyroid cohorts", PROCEEDINGS OF MACHINE LEARNING RESEARCH, vol. 156, 2021, pages 1 - 12 |
J. HOEHNE: " Detecting genetic alterations in BRAF and NTRK as oncogenic drivers in digital pathology images: towards model generalization within and across multiple thyroid cohorts", PROCEEDINGS OF MACHINE LEARNING RESEARCH, vol. 156, 2021, pages 1 - 12 |
K. GRZEGORCZYK: "Vector representations of text data in deep learning", DOCTORAL DISSERTATION, 2018 |
KONSTANTINOV ANDREI V. ET AL: "Multi-attention multiple instance learning", NEURAL COMPUTING AND APPLICATIONS, 11 December 2021 (2021-12-11), pages 1 - 23, XP093051531, Retrieved from the Internet <URL:https://arxiv.org/pdf/2112.06071.pdf> [retrieved on 20230602], DOI: 10.1007/s00521-022-07259-5 * |
L.C. JUNQUEIRA ET AL.: "Histologie", 2001, SPRINGER |
M. ILSE ET AL.: "Attention-based Deep Multiple Instance Learning", ARXIV: 1802.04712V4 |
M. ILSE ET AL.: "Attention-based Deep Multiple Instance Learning", ARXIV:1802.04712V4 |
M. ILSE ET AL.: "Attention-based deep multiple instance learning", INTERNATIONAL CONFERENCE ON MACHINE LEARNING, 2018, pages 2127 - 2136 |
M. USE ET AL.: "Handbook of Medical Image Computing and Computer Assisted Intervention", 2020, ELSEVIER, article "Deep multiple instance learning for digital histopathology", pages: 521 - 546 |
M. Y. LU ET AL.: "Al-based pathology predicts origins for cancers of unknown primary", NATURE, vol. 594, no. 7861, 2021, pages 106 - 110, XP037471046, DOI: 10.1038/s41586-021-03512-4 |
N. COUDRAY ET AL.: "Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning", NATURE MEDICINE, vol. 24, 2018, pages 1559 - 1567, XP036608997, DOI: 10.1038/s41591-018-0177-5 |
S. K. SUVARNA ET AL.: "Bancroft's Theory and Practice of Histological Techniques", 2019, ELSEVIER |
SUDHARSHAN P J ET AL: "Multiple instance learning for histopathological breast cancer image classification", EXPERT SYSTEMS WITH APPLICATIONS, ELSEVIER, AMSTERDAM, NL, vol. 117, 24 September 2018 (2018-09-24), pages 103 - 111, XP085513817, ISSN: 0957-4174, DOI: 10.1016/J.ESWA.2018.09.049 * |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Mahmood et al. | Deep adversarial training for multi-organ nuclei segmentation in histopathology images | |
Madabhushi et al. | Image analysis and machine learning in digital pathology: Challenges and opportunities | |
CN111275130B (zh) | 基于多模态的深度学习预测方法、系统、介质及设备 | |
US8831327B2 (en) | Systems and methods for tissue classification using attributes of a biomarker enhanced tissue network (BETN) | |
Zhang et al. | Artificial intelligence-assisted esophageal cancer management: Now and future | |
Linmans et al. | Predictive uncertainty estimation for out-of-distribution detection in digital pathology | |
Bhattacharya et al. | A review of artificial intelligence in prostate cancer detection on imaging | |
Liang et al. | Development of artificial intelligence technology in diagnosis, treatment, and prognosis of colorectal cancer | |
JP2024522266A (ja) | 病理スライド画像からの遺伝子改変予測のための方法及びシステム | |
Bevilacqua | Three-dimensional virtual colonoscopy for automatic polyps detection by artificial neural network approach: New tests on an enlarged cohort of polyps | |
Gulati et al. | Artificial intelligence in luminal endoscopy | |
Janse et al. | Early esophageal cancer detection using RF classifiers | |
Courot et al. | Automatic cervical lymphadenopathy segmentation from CT data using deep learning | |
Liu et al. | Extendable and explainable deep learning for pan-cancer radiogenomics research | |
Corradini et al. | Challenges in the use of artificial intelligence for prostate cancer diagnosis from multiparametric imaging data | |
Yu et al. | Artificial intelligence in gastric cancer: A translational narrative review | |
Busby et al. | Applications of artificial intelligence in prostate cancer histopathology | |
Tian et al. | Radiomics and its clinical application: artificial intelligence and medical big data | |
Vaiyapuri et al. | Design of Metaheuristic Optimization‐Based Vascular Segmentation Techniques for Photoacoustic Images | |
Tenali et al. | Oral Cancer Detection using Deep Learning Techniques | |
Modi et al. | Role of artificial intelligence in detecting colonic polyps during intestinal endoscopy | |
WO2023194090A1 (fr) | Apprentissage à instances multiples prenant en compte des agrégations de voisinage | |
WO2023208663A1 (fr) | Apprentissage à instances multiples basé sur des intégrations régionales | |
Muis et al. | Cnn-based approach for enhancing brain tumor image classification accuracy | |
WO2023213623A1 (fr) | Stratégie d'échantillonnage dynamique pour apprentissage à instances multiples |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23714647 Country of ref document: EP Kind code of ref document: A1 |