IES87151B2 - A method to determine a degree of abnormality, a respective computer readable medium and a distributed cancer analysis system - Google Patents
A method to determine a degree of abnormality, a respective computer readable medium and a distributed cancer analysis systemInfo
- Publication number
- IES87151B2 IES87151B2 IES87151B2 IE S87151 B2 IES87151 B2 IE S87151B2 IE S87151 B2 IES87151 B2 IE S87151B2
- Authority
- IE
- Ireland
- Prior art keywords
- image
- degree
- value
- tne
- abnormality
- Prior art date
Links
- 230000005856 abnormality Effects 0.000 title claims description 55
- 238000000034 method Methods 0.000 title claims description 45
- 206010028980 Neoplasm Diseases 0.000 title claims description 21
- 201000011510 cancer Diseases 0.000 title claims description 21
- 238000004458 analytical method Methods 0.000 title claims description 15
- 238000013528 artificial neural network Methods 0.000 claims description 52
- 238000012549 training Methods 0.000 claims description 37
- 241000282414 Homo sapiens Species 0.000 claims description 16
- 210000004027 cell Anatomy 0.000 claims description 14
- 230000011218 segmentation Effects 0.000 claims description 9
- 210000005260 human cell Anatomy 0.000 claims description 5
- 238000010200 validation analysis Methods 0.000 description 38
- 238000012360 testing method Methods 0.000 description 27
- 230000001537 neural effect Effects 0.000 description 17
- 230000006870 function Effects 0.000 description 12
- 230000008569 process Effects 0.000 description 10
- 238000012216 screening Methods 0.000 description 10
- 210000001519 tissue Anatomy 0.000 description 9
- 230000008901 benefit Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 206010008342 Cervix carcinoma Diseases 0.000 description 6
- 208000006105 Uterine Cervical Neoplasms Diseases 0.000 description 6
- 201000010881 cervical cancer Diseases 0.000 description 6
- 238000012552 review Methods 0.000 description 6
- 230000034994 death Effects 0.000 description 5
- 231100000517 death Toxicity 0.000 description 5
- 230000001419 dependent effect Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000010186 staining Methods 0.000 description 5
- 238000001574 biopsy Methods 0.000 description 4
- 230000002380 cytological effect Effects 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 4
- 230000002068 genetic effect Effects 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 108090000623 proteins and genes Proteins 0.000 description 4
- 238000013515 script Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 241000282326 Felis catus Species 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 3
- 239000000090 biomarker Substances 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000002405 diagnostic procedure Methods 0.000 description 3
- 201000010099 disease Diseases 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012797 qualification Methods 0.000 description 3
- XEEYBQQBJWHFJM-UHFFFAOYSA-N Iron Chemical compound [Fe] XEEYBQQBJWHFJM-UHFFFAOYSA-N 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 210000002919 epithelial cell Anatomy 0.000 description 2
- 210000004907 gland Anatomy 0.000 description 2
- 230000000877 morphologic effect Effects 0.000 description 2
- 238000009595 pap smear Methods 0.000 description 2
- 230000001575 pathological effect Effects 0.000 description 2
- 210000002307 prostate Anatomy 0.000 description 2
- 238000001303 quality assessment method Methods 0.000 description 2
- 210000001685 thyroid gland Anatomy 0.000 description 2
- 241000008371 Alseis Species 0.000 description 1
- 241000370685 Arge Species 0.000 description 1
- 241001133287 Artocarpus hirsutus Species 0.000 description 1
- 101100167365 Caenorhabditis elegans cha-1 gene Proteins 0.000 description 1
- 101100494773 Caenorhabditis elegans ctl-2 gene Proteins 0.000 description 1
- 101100181929 Caenorhabditis elegans lin-3 gene Proteins 0.000 description 1
- 101100112369 Fasciola hepatica Cat-1 gene Proteins 0.000 description 1
- 241000963438 Gaussia <copepod> Species 0.000 description 1
- 241000282412 Homo Species 0.000 description 1
- 241000845077 Iare Species 0.000 description 1
- 206010058467 Lung neoplasm malignant Diseases 0.000 description 1
- 101100345589 Mus musculus Mical1 gene Proteins 0.000 description 1
- 101100005271 Neurospora crassa (strain ATCC 24698 / 74-OR23-1A / CBS 708.71 / DSM 1257 / FGSC 987) cat-1 gene Proteins 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 208000008784 apnea Diseases 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 210000000481 breast Anatomy 0.000 description 1
- 150000001768 cations Chemical class 0.000 description 1
- 210000003169 central nervous system Anatomy 0.000 description 1
- GPUADMRJQVPIAS-QCVDVZFFSA-M cerivastatin sodium Chemical compound [Na+].COCC1=C(C(C)C)N=C(C(C)C)C(\C=C\[C@@H](O)C[C@@H](O)CC([O-])=O)=C1C1=CC=C(F)C=C1 GPUADMRJQVPIAS-QCVDVZFFSA-M 0.000 description 1
- 210000003679 cervix uteri Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013502 data validation Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 210000005095 gastrointestinal system Anatomy 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 239000010931 gold Substances 0.000 description 1
- 229910052737 gold Inorganic materials 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 235000012907 honey Nutrition 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 229910052742 iron Inorganic materials 0.000 description 1
- 238000003064 k means clustering Methods 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 201000005202 lung cancer Diseases 0.000 description 1
- 208000020816 lung neoplasm Diseases 0.000 description 1
- 210000001165 lymph node Anatomy 0.000 description 1
- 230000036210 malignancy Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000010339 medical test Methods 0.000 description 1
- 238000000386 microscopy Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 239000002674 ointment Substances 0.000 description 1
- 230000007170 pathology Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 210000002345 respiratory system Anatomy 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 210000004872 soft tissue Anatomy 0.000 description 1
- 235000013547 stew Nutrition 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- BGRJTUBHPOOWDU-UHFFFAOYSA-N sulpiride Chemical compound CCN1CCCC1CNC(=O)C1=CC(S(N)(=O)=O)=CC=C1OC BGRJTUBHPOOWDU-UHFFFAOYSA-N 0.000 description 1
- 210000001635 urinary tract Anatomy 0.000 description 1
- 230000003245 working effect Effects 0.000 description 1
Description
A method to determine a geggqg gr abnormality, a lgmgglm ggmpmg; mafia fig medium and a distributed cancer analysis system The present applicatior is directed towards a method to determine a degree of abnormality, a respective computer readable medium and a distributed cancer analysis system.
Cancer screening programs rely on consistent and early detection of cancer lesions through a trusted expert. If cancer is detected early erough, it can be treated locally and thus the health risk for the patient can usually be efficiently and effectively avoided. In many cases. cancer screening involves taking biopsies, the biopsies being small tissue samalcs of potentially cancerous regions. Such biopsies are usually taken during a routine medical checkup or upon special indications following preceding medical tests. Biopsies. as any other pathological tissue specimen, are evaluated after their preparation on a glass sllce through an expert. usually a Board certified pathologist. The pathologist has been trained Inside the ocrnmunity of available experts to a certain deg'ee anc is solely responsible for the diagnostics. Often, pathologists request a second opinion from colleagues. The patient himself is usually not aware and does not know anything about the diagnostic process of the tissue specimen. neve' meets and doesn't know anything about the person or his qualification making this important medical decision for him or her. As a result, the patient is the end depe1dent on a rather non- transparent process. This process results In a limited trust of the relevart stakeholders in the overall system. This feeling can be objectively SUDP0l'lied through many studies, which have shown considerable variations of the quality of medical decisions in cance' diagnostics. Moreover, there is no gold standarc In the evaluation of pathological samples. leading to a big variation ‘n the di39"0StlC results. Thus, there is a need for an objective screening method that yields comparable results and which is transparent for the participating parties.
Cervical cancer is one of the leading causes of cancer death among worren. world-Nice. Cervical cancer is the fourth most frequently occurring malignancy in women and resulm in an estimated 530,000 new cases annually with 270,000 deaths. In addition, approximately 85 % of the worldwide deaths from cervical cancer occur In underdeveloped or developing courtries, and the death rate is 18 times higher in low income and middle-income countries compared with wealthy countries. 72'] Recognition that cervlca neoolasa Jegins as an intraepit‘1elia' change, which Jsuaily takes many years to develop into an invasive disease, led to the use of cervical exfolativc cytology (0.9. with the brush) in screening. In this way. detected cervical intraepithclial neoplasla may be treated early to prevent the development of cervical cancer. Cervical cance' screening is recommended for all sexually active women worldwide.
Currently. such screening is based mainly on the morphological and cytological examination of cytosrnears of the cervix uteri, what is called the PAP test, which is ‘name on the bas s of gynecological routine examinations at regula' intesvals for sexually active women. Unfortunately. up to 30-50 % of the results of the PAP test are false negative, rendering PAP testing not a satisfying solution. l-loreover, PA? testing has not been efficiently implemented in many low income and middle Income countries. Thus. over 85 % of global cance' cases and related deaths occur In these countries.
Genetic testing through, for example, sequencing tecnnology is a further diagnostic oossibility, which is upcoming for nea'ly all cancer types. Still, genetic testing has an insufficient test specificity. Genetic testing alone. however. is insufficient.
In this line, to suppo"t the genetic HPV testing in cervical cancer screening, biomarker-based immtllnosgytolbgy plays a critical role. For examnie, tests are used for cell samples using p16 and l liquid-based cytology. These tests show orornising results in orimary sc'eer-ing and as a triage of HPV positive for women. Unfortunate y, the good accuracy results of these tests have so far only been reached in an optimal setting with specifically trained numan experts. As a result, the accuracy also depends heavily on the human expert. samples of each cancer type show a very specific collection of morphological features. Pathologists are trained often l‘o° at least a decade before tnev become exoe'ts in usually just a few specific fields. In eacn field, they are trained to recognize specific patterns indicative of specific disease forms 0' also different cancer grades (in cervical cancer. e.g. cenaica lntracpithclial neopiasia grades asa quantified l:hl'OJl_Z!1 n.lmbe's 0-3). This is even more comolicated when histological or cytological bioniarkers are used which highlight specific spatial patterns.
Unfortunately, despite the fact that standard protoco s for such tests are usually specified and predetermined, there is further considerable variability introduced between laboratories. For example, unavoidabe intedaboratory differences In the diemical behavior of the substances used for preparing and performing the actual sample staining leads to variability.
The most important factor, which introduces variation in overall diagnostics, is the liurnari expert ilseil’. Humans are very good at visual interpretation of spatlal patterns and are often able to reproduce decisions with high accuracy because they tend to stick to the decislon patterns establlshed in their owr brains. This, however, leads to high inter-observer variation in a larger group of human experts.
In conclusion. when oonslderlng all regional differences and also takes the variability of the underlying experimental processes In the laboratories into account as well as the different experts. it is clear that the comblnation of all these factors in the diagnostic process will lead to a large variability in the diagnostic quality.
Even more, biomarker-based immunocytology-based tests, sucn as the Roche- Ventana Clhltec test are provided with an extensive documentation. Even in its very general workflow, it recuires the expert to follow very oomplex visual screening procedures. Morec-ver, very complex decisions need to be made, which are hardly possible to follow In an objective fashion, even for an expert. There are specific criteria given for dual stained cervical epithelial cells comprising the spatial location of p16 and l In a specific appearance. so that for example a red ruclcus must be wlthln the same microscopy plane as a brown stain. The expert then has to decide if the staining intensity is “weak” or “strong” to correctly apply the test. This is very difficult. Furthermore, KI6? signal Intensity (red) may be uniformly stained with the nucleus containing a speckled or granular staining pattern and rec staining of nucleoli while negative cells comprise cervical epithelial cells stain with only one blue counterstain, or only brown andfor nuclear stain or only with a red nuclear stain. This very general set of criteria is then specified with exarriples of all different kinds of patterns of these types on over 70 pages. A general specific procedure is described for the overall slide requesting a cytologist or pathologist to systematically scan the full slide at mac or 20:: times magnification for cance' events in a snakelike fashion. one speclflc problem also addressed in the above- mentloned documentation is the handling of cell clusters, whicn may appear and have dedicated further special protocols describing their evaluation.
All those examples show that the interpretation of cytological tests like the Cmtcc DlUS U’-‘St i5 Vet? Complicated. very hard to standard ze ard reproduce. Accordingly, It is clear tnat inter-observer variation is the rnost critical challenge for such tests.
In conclusion, tne analysis of cytological or histological 3iODSi8S in cancer screening is based on the visual intersretation of Images as described above.
Visual interp'etation can be acconnpiisred usirg artif.cial neural networks, for example. In recent years. deeri learning-based a'ti|‘icial neural netvvor' gained attraction in ‘nanny fields and accomplisned ve'y good accuracy in recognizing patterns. in general, artificial neural networ in genera have been used in cl nical decision supacrt systenis since the early days of computing. Except for cytology evaluation, artificial neural networks have been used tor lung cancer screening in radiology images. Stating in the 1990's. tne concept of neural networks in cytopatnology was suggested. Generally, artificial neural networks when properly trained have the ability to tolerate ambiguous and noisy data. Artificial neural networks were p'oposed to be used alorgside other traditional algorithmic processing technics for the development of systems useful in quantitative patnology. The vast majority of neural network applications are for cervical pathology, which this invention further extends.
The majority of he nroposed neural network aopllcatlons are related to breast and thyroid cytoaathology as well as cytopathniogy of the urinary tract. Neural networks have been applied to a smaller degree in cytopathoiogy of the gastrointestinal system and to a lesser degree in effusion cytopathology. Still. there a'e cytopathology sub disciplines that have not yet used neural networks, especially cvtonatho ogy of the lymph node, respiratory system, soft tissues, none and skin, live’ ard pa ncreas. central nervous system and the eye. among others.
Additionally, the presently available algorithnis use no contextual information and nearly exclusively refrain iron cell nuclear features. Usually. noise which is disturbing the diagnosis (for example cooloid presence in the thyroid cytopathology) is not considered. The available algorithms heavily depend on both staining and cell characteristics of each tissue type and anatomical site.
Finally, adding new knowledge to neural networks is a great proolem, as robustness and classification may be unde'mined oy overfitting of the training data.
It is therefore an object or the present invention to provide a method and a system to detect cancerous cells to a sufficient degree or certainty ‘vioreover. It is an L91) object of the present invention to provide a method and system that Is able to process whole sllde images in an automatic way. Additionally, it is an object or the Dtcscnt invention to reduce the variability of test results. Even more, it is an Object of the Dresent invention to reduce the costs and processing time of cell testing. In addition, it is an object of the present invention to make the process of diagnostics transparent to the patient.
The objects of the present invention are solved by the stbje-ct matter of claims 1, 12 and 13.
In particular. the object of the present Invention ls solved by a method to determine a degree of abnormality, the method comprising the following shops: a) receiving a whole sllde image. the whole sllde image deplctlng at least a portion of a cell, in particular a human cell; b) classifying at least one image tile of the whole slide image using a neural network to determine a local abnormality degree value associated W th at least one Image tlle, the local abnormatity deg*ee value indicatlng a likelihood that the associated at least one portion depicts at least a part of a cancerous cell; and c} determining a degree of abnorrralitv for the whole slide image based on the local abnormality degree value for the at least one image tile.
A core aspect of the present invention is the fact that a whole slide image may be automatically processed using a neural network to determine a degree of abnor.-nality. Thus, no separate feature detectors, for example corner detectors, are necessary. Thus. the present inventior provides end-to-end learning system, relying on neJral network techrology. Moreover, each layer in a neural network resembles hard criteria that are used to evaluate the image tile, i.e. each layer functions as an elaborate teature detector. Thus, uslng a neural network has the further advantage that comparable results are achlcvoo. Additionally, a further advantage lies in the segmentation of the whole slide image into at least one image tile. Thus, each image tile may be processed in parallel, resulting in better processing speeds. The present invention also allows for processing each step of the above-mentioned method at different locations. For example, the whole slide image may be generated at a first location, and the classifying and determining may be conducted at a second location. Thus, a global application of the described method is easily achieved. As a result, the cost-per-test drops significantly, In the context of this application, the term "‘local'° may be interpreted as relating to a specific area on the whole slide image, to particular characteristics of the whole slide image. e.g. color channels or further characterisitcs or the whole slide image.
In one embodiment, the method may comprise segmenting the whole slide image into a plurality of image tiles. the size of each image tile being equal, in particular using a K means clustering algorithm.
The method can thus also include a segmenting steo. wherein the whole slide image is segmented into a plurality of image tiles. Preferably, the size of the image tiles is equal, for example a square. In one embodiment, the size of the image tiles is dependent on the Implementation of the neural network. For example. the neural network can be implemented as a cowolutional neural network. wherein the size of a tile is dependent on a kernel definition. Consequently. very efficient irnpiementations of a neural network are possible.
In one embodiment, the degree of abnormality for the whole slide image may be indicated by a function, in particular a max-function, a statistical aggregation of the local abnormality degree value (15, a_j, 519, 719, 719’, ?19"} or an average function dependent on the local abnormality degree value (15, a_], 519, ?19, 719’, 719").
Using a function to determine the degree of abnormality has the advantage that the appropriate method may be used to determine the the degree of abnormality.
Thus, in some cases using a max-function that is adapted to determine the maxlmlum value of the local abnormality degree values may yield good results. In other cases it may be more appropriate to compute the average of the local abnomaiity degree values. In one embodiment. the function is cependcnt on the disease andror type of cancer to be detected.
In one embodiment, the size of each image tile may be within an interval of 32x32 pixel to 1000x1000 pixel, 200 :4 200 pixel to 700 x ?00 pixel or larger than 1000 x 1000 pixel.
In one emoodiment, the neural network is implemented as a corwolutional neural network. he neural network comprising: at least fifty layers. at udat twenty zwullng layers. - at east foarty cswalutianal fay-ers. - at east twenty - at least one fu y connected ayer, ~ a softmax-cIassI‘e- layer an:l:'0t - ncu':>ns using log t andlor logistic funct ons as activating funztisns.
In one em Jotl"nent, the East |aye° 3|‘ the neara networ'< may be nnade of a ssftmax-class fe- layer wltn two classes.
As a :onse:l.rence of tne aaaye described ennbozl nert. a dig ya'iab ty of nenral netw:>r< innplementat ons are Jassib e. Irnzizrtantly. using a deeper network, i.e. n1v‘e layers. often increases accuracy of the classification 'es.rlts when comained with -esaectlve pooling, subsampllng and kernel def nltlons. For examp e, networks na-aing more tnan fifty 'ayers. referably more than eignty Iaye*s, are able to detect and generalize 'no'e ea3:>°ate s.ib strJcl:.rres at an ‘mage tile. In one areferretl embodiment. a soft-rnax class i e’ |aye' 5 used as a last ayer. nd cat ng tn: final result 0‘ tne class llcatlon p-occss. Thus. a discrete set of classes, nreferenly two. can 32 used as :: assificaticm resu t data.
In one embed nnent, the nnetnod may cemprise t-alnlng tne neu-al nets-to-k wltn t-am -lg data stzrr-e:l in a icuowledge ease. the training data including a p .ua"ty of up es. eacn tuple inzlicating an 'nage tile, a training abnornnality value and a IikeIin:>o:l value. l'v tnain a nearal netw:>'<. Jsually a 'arge num:ie' of tra ning samples are needed.
For the p'2sent invention, tnese samales incl Jde data concerning tne imaget e, a t'ain‘ng abnarnnality ya J3 an:l!3r a ' may zomarise a degree of cance-'. for example a Gleason ya L-e or Gleason score, which IS a va Je between 1 and S. In the case of a Gleason ya .12, a score of 1 indicates tnat a cance':>us p':>state :: ose'y resennb'es normal prostate tissue.
It sco°e or 5 nzl cates that tne tissue does not have any 0' only a few recognzable glands. The vaIJcs eetw-eon 1 and 5 are gradations of tne grades 1 and S. Tne Iikelihaod value Ind cates a confidence in tne training aanornnality value. Trus. a very nigh likelihood value Indicates that the assignment of the training abn:>'ma. ty value to tne respect ye Image tile is most like y two. The nnage tile may ac stored as a pointe* or a nk in the tuple, so that not tne entire 11age tile needs to be stored In the knswledge base. Tnls reduces the necessary data to st:>'e in the knowleslge base Itself and Improve; Ieo<-up speed In case the we Itse I Is not needed. The tra ‘ling ab'Io'maIi:y va‘ue may be re *ese'Ited as a roaring po':t[ -0.- 3’! integer value. T19 likelihood value ‘nay be stared as a float ng ooint.
In one e'nbod ‘newt, the 11et1od may comprise: Recely-=19 an update whoie slide knage to update he know edgebase; Segmenthg the uddate w'1o'e slide inage into a p ura ty of image tiles: Determining a training azno-malty degree ya .12 to 22:31 image ti e of at least one sunset of the oIu'ality of image t'les, 1 particuIa' by a hdman expe't; Updating the nowledgeaase with he s.tb$et:>I't1e plaralty of Image tiles and the assoc ated t-aln fig aonorhallty degree values If I: is determined that addlng the sunset of the olu-allty of Image tiles and the assoc ated tra ding abnormality degree values inproves the a:cu'a:y of the neu'aI network when trained with he updated noycledgebase.
The afo*emcnt oned embodiment a‘ was the eff:cient uodating of the .-znowledge Jase to rain the neural netavork. That is, the knowledge Jase as only undated it it is clete'mined that the update actually 11p°;>ves the et:r.'urat;y of he prr:'ti=<'.r|utr r:on:l.;:te:l by ['15 hi-:U':1| rretsvurk. Tnls e'nbud ‘ne'1tth.rs prevents .ll'l"lt:£CSS3l'9’ <'I:)w|a.°‘.‘|’gt: base dpdates. wh ch do not actually enhance the ca oabil ties of the drocess.
In one ernbod near, the uddating of the Conpdting a p'edZcted ab1o'ma Zty deg'ee va .te and a1 associated likelihood value (or each of he olu'a|il;y of Image I Ies using the ‘ieura netwo-'l<; Dete'm “ring a priority ya J8 based on the oredlcted aonormallty degree value and the assoc ated Ilkelinc-od value for each of the plurality of image tiles: and Date-m nlng the subset of the image tiles oased on the determined priority values In one embodiment. the priority value Is comoutee by using a 2-dimensional Driorlliv mapping funztlon p(d,1) -> [(1.1], calculated as D[.2.i} :=9-'9'-EL "a" aelng the abncvmalzty value and “I” be 1;; tne likelihood value. Using th s :lrio’itv value, tne subset o‘ nnage tlles nay ee determined using a tnreshold va ue, wnerein only image t es are added to the subset tnat have a p'i3r ty value greate' than the tfreshold value. As a 'esul‘:. the above-nnentioned embod nnent orovldes a Jrese ection of innaget es to be :onside°ed Jr; the exnert. Thus, the number or nnage tiles to be -eviet-red and to ca sent to the 5 reduced.
Consequently. tne annount of data to be sent to the ea<:>e'ts is red used.
In one emoodlment, deternnlnsng tnat adding tne subset of the Image tiles and the associated traln ng aanornnalltv degree values Innarovcs the azctracy of tne ncunal 'l2tw:u'k wnen t-alned w th the updated cno.-:'edge base may comp-'lse: - Jpdating a validation database wtn the sunset ol the image tiles and the associated training abnzrma ty degree vi: .125, the v;-:li'Jat on data Jase n dart cu ar Including at least a subset of the (now-ledge base: - Training the neura network using the validat on dataoase; - Jslng fh:-e lit-mml netw:.>'l< trained on the validat on dataaase to p'edict the data in an independent validat on c:>n:>rt to com nute a first accuracy value; - Jslng the neural netwo-k trained on the l m the ndedendent validation :ono't to compute a second accuracy value: - Cornparing tne first and the second accu-acy values to determhe wnetner add ng tne subset of tne Imaget es and the assoziatcd abnormality degree 93 ues imp’)-«es tne accuracy of the neura networ-t when trained witn the updated knowledgedase.
Tnus, a total of th'ec databases ‘nay be used for the inventive metnod. Tne knowledge Jase may capture the kn:>tr.'|edge that presently procuces a neural network that results 1 the nest a:cu°acy. The valdatlon database may eapttre the data of the knowledge dase plus additional data of experts tnat ‘nay lead to better a::cu'acy results for the orediclior using the neural |1e|ZWO'K. Tne Independent validation cohort stores g’OL'1dt'ut.’i data, wni-.-h may he used to test tne accuracy of a neural netw:>'< t'aine:l on the knowledge base and a .'ie.ira ‘I:-'el’Wor|( rxalnezl on tne validation base. 35' tnis, t.-to ne.ira: netw:>r< implenientat ons may be compa-ed and the nest ‘nay be selected for l‘u'ther p':>cess‘ng.
In one ennbod nent. tne method niav comar se sto'ing the training abnornia lty degree va ..e for eazn of at lea st a subset of the mage tiles determined 3-; a human expen ‘n a bloc< of a blockchain.
Storing tne knowledge of a nunnan exaert ‘n a bl:i:-tchain a lows Jsing a dstr outed data st'uctJ'e that reco':ls transactions. tnportantl-,-. when st:>°ing the t'ain ng abnormality degree value and an indication of the image tiles in the t-lockznain. it is nignly transoarent to tne aatient and otner d:>et:>rs{e: used to train tne neu°a| net.vo'lt that was used to classify nis o' he’ tissue saniole. as o °es.iIt, the nve‘i:lve method p°:>uides a transparent solution, where n the natlent. other experts and :locto's always knows wnat know edge contributed to tne assessment of tne patients tissue samples.
In one enibod nient, the Jlockcnain indicates tne image t les, in part cu a° us ng a Ink andfor a pointer, the assoc ated t':alri rig aariurmility degree values and the exaert dete*mining the abncrma by deg 'ee value at the tra ning data sto'ed in the wle:lge base.
In one enn30d'n1c'il:, each block of tne blo:l nasn value of the neacle* of a p°evious b ock and a rasn value of the root node of a Vle'kle tree. tne Mer<'e tree ndicating all data sto'ed ‘n the knowledge Jase.
The blockcha n the*efo'e is a cha'n of :iIo:c=ts are "nkezt to each othe' via pointer wnizn may Je hash vaiues. Adding a new assessment of an exaert. i.e. a new rain 13 abnormality degree value and the associated iniage tile, to the -cnowledgc base ':3p'csents a transaction, wh°ch s recorded by the olocltchain. For th 5, each b'o:< stores a link to an associated i'riage tile as well as tne associated wnale slide innage and in pa.'ticula' the exaert that niade the assessment oi a tra'nli'ig abnormality degree value. 5 nce a blockcna it usually is a .nu:i|i:: ledger of vansaction, using a lin< to nd cate tne associated image tiles has tne benefit tlnat .inauth:>'I2ed users cannot access tne image data used to train a 'ieu'el retwort but mere y can access tne essessriients assoz; atezl with tne irnayes.
Thus. tne Image ties are orotectea’ from Jlla|.ll."l0l'iZ8d access.
V‘-' . 1'1 ::'1;: o111boalrner1:. 2: ‘award unction may oe associated with eazn human I.-2!’);-,'r[ :ont'ib.iting assessments to the olocicchazn, In oartcular using an dentif cation n.1mbe'. tne va ue of the reward fun::t'on in pa'ti:;Jlar aemg depe'|de'|t on the numbe° of :ont°ibutiors made o1/the associated human expert.
Using a reward fun:t'on nas tne benefit tnat exoerts can be "1ot'vated to :ont'ibJte data that may be Lsed for trainlrg tne neural netwzrk and th.1s imp'ove the accu'a:v of tne neu*aI ne:wo*k. Tne rewa':I function co.1I:l, fa’ exainole, trigge' the oavment of an a11o.int of ‘honey in some :.ir'encv. for exa 11pe oitcoln or any otne° ctrrency. to tne human expert as a rewa°:l fo' the contr out on.
Moreover. he value to be oa d to the expe't could vary deoend 13 on the numoer of :;ont'ib.iti;1'1s :1 may added by tne p-3rt|c.1|a° 31.11101 expert. F:.1° exam ole, the reward funet'or1 co.1l:1 be mode ed using a falling exponentia function. Thus, a high va'.ie, for exa'noIe 10. co.il:l be the initial rewa'd fa’ the 1"':ont.'ib.1tion, and a low value. fo' example 1. could be the reward fo' the 10"‘ sontr oution. Th 5 ma-(cs first-time contributions more attractive to’ an exoort and orovents food -19 of the bo:k:na n .vitn bogus and low-qualitv data.
In ore embodiment, the blo:.
In one embodiment. tne value otthe 'ev-rard tunct on may tu'the° be dependent on the numoer of contr out ons at the associated nu'nan exf.>e't that 'n'J'ove tne accuracy va'.1e of e neura °1etwor< trained at least pa't1a Iy on the assessments of the assoc atezl nu nan e1 validation coho't.
It 3 at |u°lher ’:1'JV;3l|['¢lgl.-3 if the reward funztloli ls '1°.>t Lin 3» depenoento-1 the n.unbe- or ;;ont°ib.lt|:)ns bot also on the qua ty of eaen 20nt’lbJIZi0'l. Tnus. '1 case the cunt-lbution n1pro'.res the oredicfon accuracy of a news! network tra'ned tvitn the cont‘ out'oc over the pea ction accuracy of a neural network without the so-it-il>.ni:n. a nIghe- -award Is warranted. Tnls ensures that numan experts a'e motivated to contribute a hign n.im:1cr of nign quality assessments. *vlo'eove', flooding of the b ozkchain witn low quality assessments is prevented.
In one embodiment. tne rewa':l function may be implemented as a smart :ont'a:t on the olockchain.
In one embodiment. tne boctchain may ‘rip ement a scrioting language, wne'ein the smart contract may oe imolenentett as a script in the s:;'ipt ng language associated with a trarscation or the blocltchain, wherein the script may be executed when a new block that comprises the script, is added to the blockchaln.
C0fl5€€lUE"'“Y. a smart contract can be used to Implement the reward lunction.
This provides a fully automatic mechanism to add new data to the blockchaln and reward the human experts providing the new data. Thus, a very efficient implementation of such a scherre is provided.
In conclusion, using a blockchain to store the assessments of the human expert gives rise to elaborate feedback loops to improve the accuracy of the neural network.
In one embodiment, each block of the blocl-(chain may f.irther comprise meta-data, he meta-data comprising an indication of the geographic location of the human expert, the qualification of the human expert, the years of experience of the human expert andior an association the human expert Is part of.
In one embodiment, the method may further‘ comprise selecting a: least a subset of the data stored in the blockchain as the validation database.
In one embodiment, the subset of the data stored in the blockchain may be selected based on the meta-data stored in the blockthain. with the above-mentioned embodiments, it is possible to filter the data used for raining the neural network based on the meta-data. For example, it may be necessary due to regulatory restrictions, to train the neural network only on data contributed by experts that practice In a specific country. The above-mentioned embodiments provide an easy way to aocommodate for such requirerrents.
In one embodiment, the method further comprises storing an indication of the neural network, e.g. a hash value of the neural network. in a second blockchaln.
The indication may comprise a further lndlcatlon to all training data used to train toe neural network.
In the scope of this patent application, blockchain may refer to any digital ledger technology, such as blockchain. tangle or hashgraph. with the above-describec embodiment. an Immutable version history of the neural networks is provided. Thus, it is highly transparent to each stakeholder which data Is used =n what version o‘ the neural network.
In particular the problem is also solved by a computer readable medium storing instructions that when executed by at least one processor cause the at least one processor to implement a method according to any of the aforementioned embodiments.
The advantages of the above-mentioned solution are similar or equal to the method.
In particular, the problem is also solved by a distributed cancer analysls system, comprising the following components: - a segmentation entity adapted to receive a whole slide image, the whole slide image depicting at least a portion of a cell, in particular a human cell; - a computation entlty adapted to compute a degree of abnormality for the whole slide image using a neural network.
In one embodiment, the distributed Cancer analysis system may comprise a knowlcdgetasc comprising training data, wherein the computation entity is further adapted to train the neural network using the training data.
In one embodlment, the computation entity may be adapted to compute the degree of abnormality for the wriole slide image using a function, in particular a max-function, a statistical aggregation of local abnormality degree values associated with the at feast one images tilc andjor an average function dependent on local abnormality degree values associated with the at least one image tile.
In one embodiment, the distributed cancer analysis system may comprise a communication interface adapted to distribute at least one subset of a plurality -of image tiles to at least one expert, wherein the segmentation entity is further adapted to segment the whole slide image into the plurality of images tiles.
In one embodiment, the distributed cancer analysis system may comprise a priorization entity adapted to determine a subset of the image tiles to be transmitted to the at least one expert by the corrmunlcatlon Interface, wherein the griorizatidn entity may fu-the he Bdailtétd tu dHtZ!t'llllIt-! t’Ir-2 !s'.l')!s'E'.[ at candizlate image tiles based on a :o'n:iuted p'i:>r§ty va J1‘ to’ each image t"e. in one e'n3odi11e'It, the dist'ib.ited cause‘ analysis system may com arise a testing Qflt or adapted to: - rece ve vaidatia1trai1ing data of a validation datadase. tne validation training data comer si1g at least one val datim set, t1e validation data 3358 in dart cuIa' Including at least a subset of the -mowledge oase. detemined by he at least one expert: - training the 'ie.:ra 1etw3r< using the received va'idat on data: - using the 1euraI networt trained or the va datiow data to Jredict the data In an Independent validation cohort to compute a f.rst accnracy value; - using the neural '1etwor< trained 3*. the <'1:>w.edgebase to arcdict the data in an independent val dation zohort to compute a r 'st ac:.rrac-( va'ue: - :oni:Jar'1y he first and the Satzorid ;::.'cu'aey va .s=:$ to dete'rn|ne w’Iet'ie' adding the subset of the 'nage tiles and associated training asnormality degree values 'n3'ove3 the accuracy of tfie neu'a| net-ro'k s-men t'a|rIe:J w tn the Jpdated know edgcbase.
In one e'n:iodi11e'it, the distribited canoe‘ ana ysis system further tamer ses a Jlockchain adapted to sto'e the training aI>1o'me' ty degree value fo- each of at east 3 sabset of he Image tl es ‘I a bloc-C, he training abnormality ilcgrcc va Jc deterzn -nee oy a human expert.
In one emod 11e1t,t"ie blocicchain may be adapted to indicate the image tiles, the associated training aonormality degree vaI.ies and the exserts determining the rain '19 abnormal ty degree values of the trading data stored '1 a knnvredge Jase.
In one etnoodinnent, each blo:< of the b.o:kcha 1 may c:>'np'ise a header, the neader comprising a hash veiue of he header of a Jrev uus vlock and a hash value or a roul, node of 4 M;-rkle I.-ce, the ‘4c'kI2 tree nd cat 1; a data stored In Me is The Jenefits and advantages of the aforr.-rrrc--rtrorred distributed ::a1t;er analysis system are equal or similar to the advantages of the abs-re-mcwtiowed metqod to deternlne a degree 3f ab'ro'ma ty.
M an a'ter1ati'Ie enbod ‘nefit. me deterrrrlrrarrg step of the nretwd may dete'mi're a value ind cative of an aanormality far t’ll'.‘ whole slide mag: based on Ioca values. indioat we of .o:el ab‘ro"na ity va Jes, fr the at zeast one irrage tile. £1 one emaodiment. the value i'rd cativc of an abno*mality may Ind cate the area of abnormal glands. 1 particular in a prostate, andfor a tissue area T12 alternative metrod t'rerel:>° allo.-rs tor the ide‘rtiI‘i:ati:>'r of the area t'l3t 5 affected oy canoe-:>.rs tissue. e.g. 53% of twe whole slrde -mage.
Fu'the' enbodiments are ind’cated by the :lepe'rde'rtolai11s. la the following. em 3odi*nents of the went on are described with respect to the figures. wherein Fig. 1 shows a flow diagram oi‘ the 'nethod to determine a degree of abnormality; Fig. 2 shows a schematic oi’ a system to cleterrnine a degree of abnrlrrrreiity: Fig. 3 illustrates a flow diagram showing the different phases of a method to improve the accuracy of the used neural network; Fig. 4 illustrates a flow diagram showing the first phase of the method of Fig. 3: Fig. 5 illustrates a flow diagram showing the second phase of the method of Fig, 3; Fig. 6 illustrates a flow diagram showing the third phase of the method of Fig. 3; Fig. 7 illustrates a flow diagram showing the fourth phase of the method of Fig. 3; ‘i9. 8 illustrates a schematic of a system to imziro-we the azcu-acy of tne neu-al network. =ig. 9 illustrates a schemat'c of a convolut‘ona neura networ-t, wnicn :an be used with tnis invention; and =Ig. 10 Illustrates a schemat c of a bIo:< cha 1. ‘i9. 1. shows a flow diagram of a method to determine a aarormslity degree value 13. In a first stew. a whole slide image 11 is pmcessecl d.ir 1-,; a seg mentation chase 12. Tne whee slide mage 11 dep as a azrtion of a human cell that may be cancerous. Vlo*eover, the whole slide image 11 may sénorr a cell that has seen t'eated t-ritn biomarkers. for exampe of the CINTEC test. Tnus. certain regions in tne wnole slide “rage apnea‘ :9 ored ind cating certa'n cnemice 'eactions.
The w'i;>e 3 ‘tie image 11 is t'i°-en seg'ne'iled ‘ii the segrrietitaliori uhase 12 into a old 'allty of Image t es 13. Eacn image t'le nep'esents a s:>*tl:>n of the whole 3 lde mage. Tnus. tne plurality of image tiles 13 togetner form the whole side image 11. P°el'era sly. the nage tiles 13 are of a same size. In the p'esent embodiment. tne image t es 13 are each of a size of 30x33 p are s. in othe- embodiments. othe- t e sizes are possib e, to examole tne sizes of a 100x130 nixels, 2UUx2UU pixels ur 1303x1030 Jixels. The whole slide image 11 3 usually of a ye'y nign resolution. for example :o'na-'is ‘I3 mare than 15 m llion p xe'.<=. in a next step. the image tiles 13 are p°:i:essed in a sredict on phase 14. whe'ein a o:.al aonor-iality deg°ee value 15 s computed for each mage tile. in the resent embed 'rie°it, this local abnormality degree value 15 Is computed in a curwulutiurial neural netwzzr-<. Imoartanty, each image ti.e 13 may be drocessed ‘n oaraélel. Tn: aret teet.ire of the neural rctworv. wi be exple ned in detail «tn 'es3e:t to Fig. 9. -raving determined a local aonormality degree value 15 fa’ eacn image ti e 13, a degree of asinorma ty 1? is comnuted dur '1; an era J3|IiO'l pnase 16. Th.is, the a'.in:ir1ia ty degree value 17 s eased on the plurality of mage tiles 13. in tne aresent cmb:>d"nent, the degree of abnormality 17 is merely tne maximum value of the different local abnormality degree va .ies 15 to‘ each image t le. Th 5 is d.ie to the fact tnat i‘o- a human cell to be cance-:>.ns, It may be s.iff.clent that a single dart of the numan cc snows a canecrots eharactefistic.
F g. 2 shews a respect ye system. ":Iplume'1[‘l':g (‘la method or ’-"g. I. Tne system 23 of F g. 2 zom arises an image pracessing ent ty 33, wnicn zomar ses a segmentation entity 22 as well as a ::om:Iui:ati:>n entity 24. The contputat on entity 24 i5 C0mmJfliC3tiVC'$' counled :0 a database 25. sta"ng training data 26. white is used by he eomautatien ent ty 24 to t'ain a neJra| netwo-ic. The nenra network 5 adapted to determine aanorna ity degree values. For exemale, a wno e slide Image 21 is read by he segmentation entity 22. The segmentat on ent ty 22 is adapted to segment the whole slide image 21 into 3 plure ty of‘11age ties 23. The alu°ality of image tiles 23 may sue stored as a set, or as an away. Otnei data structures are possible as well, fa’ exam ale a hash man or a s:>*te:i list. Tne eomoutatian cnt ty 24 is iu't1er adapted to p°o:;esst1e p-arelty of image tiles 23 to :ieternine a result 27. Le. a degree of aanwmallty. =ar tnis, the c:>‘noutet.on entity 24 uses a neu*aI network to nrocess each of tnc tiles of thc ;>lu'ality of t See 23. the degree of aanarma ty 2? is in the end 'et:rne::I oy the 11392 p'a:ess ng entity 33 to the user.
In tne e'nbo:lIment shown In = g. 2. the «mowledge base 25 may ne stored remotely from tne image arocessing ertity 30. Th .15, the comnnunication Ink aetween the knowledge base 25 ard he ':.o'n::utatiar ent ty 24 hey be inplemented as an Internet connectort. Other tyaes of netwwks :-are pusslb e as well, for cxamale intsanets. The whole s de image 21 may be obta ned in any laozratory aro.m:i tne wor d. .-.e. the and e slide image 21 ‘nay also be c:>mm.ini:ated via an nte'net sonnezt on to the '5e;|11.=.‘l'it§llZiU'l unit 22. '-' g. 3 illustrates the different phases of a nietnod to improve tne ac:.iracy of the 'net1ed stow‘: ir Fig. 1. As is evident f'am Fig. 3. the method L°;1mpr'ses a aredietor ahuse 130, :1 p-'i0‘lZatlun phase 203. a decision making phase 300 and an iinpruvenient uhase 4:10. The details of each alias: wi now be described with °esp2ct to Figs. 4-7. ‘*9. 4 illustrates the :l fre'ent sub-phases oi‘ the pred!ct'on ahase 130. In a first aartztiuning ahase 1L0. tne wnoe slide nnage w is segmented .nto a ;>lu'a|ity of «wage tiles t_l to t_n. In the next si:e:i, a ne.ira2 network is .Jse:l to arediet aredicted aanornality values a,j atd likelihood values |_i for each image tile L].
The p-edicted values may be stored togctncr with a :i:>inte' to the image tile t_]' as a t.iple. Conseqaently. an a'ray of t.Jples may ae gene'ated .‘.lJi' in; he abnormality aredi::tIor' ahase 120. -'-or L*x3‘Ylplt-2. l”lt: ‘it-cu‘.-1| rretmark rrray aredizl: a tuple [a 1, l_1] = [4, 33%}. Thus.
U9 fl0U"3| NCTNOVK "39 Dredlcted 3 G eason grade of 4 and a likelihood of 33 % for he irnaget e t_1. The sane p’ co:l.lre s ‘epeatecl for ea;:1 mage tile, resulting in an array wih the size Jeing equal to the rumbcr of image files t_l to t_n oene'ated dar 1;: he partit uniwg pwase 110.
A complete I st is gene°ated '1 the Ist :'eatio'n pnase 130, .vhere :1 all p';~:l cred abnormality and likelihood values are grouped togeher i1to 3 st L_w along at h a winter to he ' 'esaecti¢e image tiles. -lg. '2 details the workings of he p’iD'l£3tlO'l phase. In a 3: Ofltlz ‘lg phase the I at ._w is processed. Ea::1 entry in this list, i.e. a single p'ed‘cted ab1:>'ma'ty val.re along wlh the (e 100:! -raise as well as a pointer to he associated l11age |.l|e is srocessed using a 2-d 112f|Sl:)"l3l p"or§ty ‘nap f.rnctie'1. Freacamsle, a functon with he comautatim definition , 2t33+t:D2 t) 1: Thus. fo' the asove-ment onerl tupe (4, 30 -Yo] he resutant nrioritv value 5 r;nmp.iterl 9.; follows: «OR 4 0.3 «:02 = 0.65 The same pcocess s reoeated for eazh tusle stored 21 gw. T1e results are grouped together 1 a list of ill’ oritv lusles S w. where'°1 each taple was the |‘:.>'m ILJ} 3.)} -.1} lJIl-..i«l.J}l- In a we:-rt cand date elim nation pwase 230. he list S_w is filtered, filter 19 oat all t.ip'es w th 3 low p'i3r ty. For example, all entries in the list S_w are e nlnatcd which ‘rave in rlority of lower han 2J.5.Tl1c -cs.rltant l.st C is then diSt’llJl.Il2:l to exaerts C during a ‘.‘llSt'lbJtl0’l phase 243. The experts E may se ruman exoerts 1 he fleld. fo' exanpe sathologists. The human experts E may oe ocated arou 1:! he wor d and thus dist'ihuti:>n during the dist'ib.ition pfiase 240 may be done .r5ing ad ht‘-:°net ur uny other electron c ccmmudleatlon moo 1'3. Imssrtant y, oaly he list 3 s distcibuted to the ex9e'ts and net he ertire list S_w. '3 g. 5 Indicates he dccis on making D1358 300. During a review shase 310 each exoert reviews the data stored ‘I the candidate list C. The expe'ts use tl*eir knowledge and experience to make a decision on the respective image tiles, providing a training abnormality value and a likelihood. For example, the expert may simply agree to the predictions produced by the neural network. In another case, the expert may correct the prediction rrado by the neural network to a different one. The expert may be supported curing this process by a graphical user interface, wherein the expert may easily review each image tile and make a decision on the training abnormality degree and likelihood value. The determination of the training abnormality is conducted by the expert during the abnormality determination phase 320. In the next phase, the different training abnormality values a_vr are combined during a knowledge base extension phase 330 to generate a validation database VD. Consequently, the valication tzase VD comprises the data of the original knowledge base 25 used to train the neural network used to predict the predicted abnormality and likelihood values for each of the image tiles.
Fig. 7 shows that the val‘dation base V0 is further processed during an accuracy comparison phase 410. During the accuracy comparison phase 410, a neural network is trained on the data stored in the validation base VD. Then, the neural network is used to predict abnormality values stored in an independent validation cohort. The independent validation oohort stores abnormality values. image tiles and likelihood values. Thus, the independent validation cohort comprises ground truth data, which can be used to compare the results of a neural network trained on he validation database VD and the original knowledge base 25.
Consequently, a first accuracy value is computed for the independent validation cohort using the neural network trained on the knowledge base 25. A second accuracy value is then computed for the neural network trained on the validation database VD. Finally. the first and the second accuracy values may be compared, indicating, which training data leads to better prediction results on the independent validation cohort. The accuracy may be computed as t."uel-'osin'ves -l- Z7'uei\'eyatives tree? oszcrne: -l- crue.'J9gorivss + fem-Pasi:.':'es + ,r’alsei'i-'c_o ntlres riccur-:ztt_s' = Consequently, in the determination phase 420, it may be determined, which tralning data leads to better results. If the validation database VD leads to a better accuracy value, the method proceeds with the yes-branch, continuing with the network replacement phase 430. In the network replacement phase 430, the knowledge base 25 Is replaced by the validation database VD. Also, the neural 08tW0Vl< U580 to C0m;3ute the nredlctlons in phase 120 is replaced ay the neural network trained on the validation database VD. If the determination phase 420 finds that the knowledge base 25 leads to tetter results than the neural network trained on the validation database. the no-branch is used and the process ends.
Fig. 8 shows a system 500, which is adapted to implement he methods of Figs. 3- 7. Fig. 8 shows a distributed cancer analysis system 500 {DCASL comprising a validation database $15, an independent validation cohort S16 and a knowledge base 517. The knowledge base 5!? is communscatively coupled to a computation entity $12 or an analysis system 510. ‘rite knowledge base 517 stores training data 509, which may be used by the computation entity 512 to train a neural network to classify image tiles into loal cr predicted abnormality degree and likelihood pairs 503. A segmentatior entity S11 is adapted to receive a whole slide image 501 and generate a set of Image tiles 502, which are sent to the oomputation entity 512.
A prlorization entity S13 determines priority values, as already explained above, for the entries in the list oi‘ abnormality degree and likelihood pairs 503 as determined by the computation entity 512.
Based on the priority values, the pricrization entity 513 determines a list of candidate image tiles 504 by comparing the computed priority values with a threshold value, e.g. 0.7 or 0.5. The list of candidate image tiles S04 is sent. to a communication Interface 514. which Is communicativcly connected to three exports E, E’, E". The experts E, E’. E" are located outside of the analysis system 510 and may be located around the world. Each expert E, E’, E" processes the received list of candidate image tiles 504 to produce respective validation sets 505. 505', 605”.
That is, each expert E, E’, E" validates the predicted abnormality degree values for the image tiles or changes the values ard thus creates training abnormality degree values. Having reviewed the list of cand'date image tiles 504, the experts E, E‘, E" send the results of the review process as validation sets 505, 505', 505" to the validation database 515. The validation database 515 comprises the training data 509 of the knowledge base SL7 with the additional data obtained by the experts E, E’, E”.
The validation database 515 sends validation training data SD6 to a testing entity 518 which is comprised in the analysis system 510. Using the validation training data 506,. the testing entity S18 trains a neural network and uses the trained neural network to predict the data In an independent validation cohort $16. The independent validation cohort 516 also sends its validation data 508 to the testing entity 518. The testing entity S18 is further adapted to compute a first accuracy value is‘ the neu'aI fle[t'lOl(lZf'33'1:d on the va datlon [am -1; data 505, The comauted accuracy value is then compared to an accuracy value of the neural notwor< trained on the t'alnIng data 509 cf the knowledge Jase 517. In case tne accuracy value achieved tnc neu'a| netwo'k tra ned on tine validation tra ning data 505 is greater tnan tne accuracy value of the neura network t'ainc:l on the tra nlrg data 509 of tne knowledge base, tne ya dation training data 536 replaces the data in l:l‘e knowledge case 512’. As», the ne.ira' netw:>'< t'aine:J on the va|i:lat'on training data 506 's then used by the connputat on entlty 512 to process imaget es.
Fig. 9 Is a scnennatlc of a ::orwo=utIona neu'a| network 603, mien can be used to imolennent the invent yo nlctnod and system. The scnematlcs of Flg. 9 shows an in out 11393 tile 601, wnicn may 3e processed by tne neu*aI networt 63.’). As the neural netwzrlt 503 is a conyolutional neural netv.°or<. a p ura’ty of used to p-ocess tne In:>..-t Image the 631. In part cula°. each kernel scans eye’ the p are s of [hue npul; Image til: 601 in a sequent al manner, l".>° cxamule (run: the top left to tne bottom right in a line-by-line fash on. A pa'a neter. the so-called st: de, indicates by now many p xes eacn kernel is mzwed in eve'y 'nove.
M:-re:>ye'. a kernel. Thus, deaending on the size of the input image tile $01, tne l the stride. the size of the feature mans 633 In the f Wt r.'.nnv:ilutiona| layer It: :|eter°°1lm-ad. Em,°n feature map 23213 rt-‘ep°:-'eSe°1ts a featu re detectur. Fm exarnplu, ‘:1 Wet featu-e map may be adaotecl to detect c:>rne's. Consequently, the 'esuItant feature nnap 503 is a map ol corne°s detected in tne input image t .e 581. A second feature nnap ‘nay indicate edges.
In the next Iaye* of the convolutional reual network 503, scasannaling 604 gene'ates four second feature naps 605. In th 5 layer, the fe':itu'e naps 503 at the p'ayi:>us layer are subsamp ed in o'der to generate a n1o*e oompact representation 3f the mnut nnage tile b0l. This s -n part cuIa' useful to reduce the size of no convo utiona neu*a| network in orcer to 'nc'ease tra ning and prediction sseed.
From tne four second feature naps 505 erward, another convolution 506 generates a greats’ alurality of third featu'e maps 587 in he sanne fashion as :lesc'i::ed before. F'om the generated tnird teature rraps 50? the outpct s in nut of a leedforwa-d neu'a| netwcrk, which s fully connected and connp-lses two aye's 608 and 639 in the descr zed ennbcdirnent |'ni>orfH'it|y. the ast layer 509 oi the 'ie.iral rei:wor< 630 is a so cal ed soft iriax ‘W’-‘I’. V-"i0’~'3ifl I'll? "till" lfiiage tile I531 is t;|ass‘f‘ecl into me of many classes.
Each layer in he convolut one‘ }eU’3| networ< 639 is built from a great number of neurons, i.e. act-vation function. having weig1ts. Depending on the weignt and va'Jc of an inout. no output of he neuros is activated or left deactivated.
Possible activat on f.inctio'is include to’ examale a ogit, arc tan. or Gaussias functions. The t'a ning of the neural networt 630 s CD’i‘.‘lJCl:Od using the sack oropagation algor t'irr and us'1g t.'ain':ig data to dete'n' 1e t'ie wei; 1ts associated with the activation finctions. ‘-la 17 :l ffe'ent architectures of cowo JtlO13l neural 1etwo'ks are possible to npiemewt the invewtlve aspects of he present aap icatiow. Po‘ exam ole. tne tumer of riumer of ayers can so varied. Possible arch tectures include the VGG-net, RES- net, gene'al acvers al networks, google Lehlet with inceotion nio:l.iles.
The trai1i'ig of ['18 convolutbnal neu'a| netwtrk may be conducted In a coud sew ce sun that he CD‘!1pJl3[lO"l is spread across nult ple machines, Jsing sarallelism to increase he training soeeds.
Pg. 10 shows a scnenatc of a blo:l<:‘ia n 703. which ‘nay oe used to store the info"nati:>r Jsed to t'ain t'iB ne.i'al netwo'k 603 andfor also a l:'ained neural 'l2lZW:>’k. In general, a b'o: are linked to ea:;1 Dt‘l:-3|’. Eacw ret:°.>':l r;-:p°ese'its u t'ai1sactIoi\ I‘! the tilockcfialri. 1': the uresr.-nt ease. adding data to the <1ow'e:lge base by an expert E. E’. E" represents a tra'isactio'i and thus he fact that the data is added to the know edge "Jase can be °e:.orued using the b|o::kt;"i'u n /00. in this way, it ‘s tra'i$:iare‘it to t"ie :>:ztIe'it, w’iic1 data was used in t'ain"i9 the ‘1€.'U'3| network 603.
F9. 10 shows three bo:-ts 710. ?l()', 710". 'e:ording at least tnree transactions, fie. addng three t'a 1;ng ab'io'ma"ty degree values fo'the mage tiles. Each block can also contain more transactions out for the sea: of simpll: tr, the following deser 3tl0"l is limited to the ease of a single t'ansa::t on.
For exaimle. block 710’ comprises a header 711' and a data olock 715'. The header 711' comprises a wash value ?12', sto'..'ig the hasn value of the ‘leader oi‘ the orevious 3locZ< 710. As a result, the block that precedes t1e blocs 710' ir the blockzriain 700 :5 u"|l'.[J(,'|y islentiflaole. ‘do-cover, he-ade' 711' comprises as Mr.-rklc root value ?l.3’. The Merkle root value 713’ is the hash value of the root node of a Merkle tree. The Merltle tree may be used to identify all training abnormality values used to train a neural network.
The data block ?1S' comprises a link to a data block 716’, which Is stored outside of the blocltchain 700. Also, the data block 715’ stores a link to an image tile 71?’, to which the abnormality degree value 719' is associated to. In addition, the data b|o<:< 715’ stores the training abnormality degree value 719’ as well as a link to the whole slide image 718' or’ which the image tile 721 is a part of. Even more, the data block 715’ stores a link to the expert E, E’, E" that evaluated the linked image tile of the linked whole slide image and determined the abnormality degree value stored in the block ?10'.
Furthermore, the blocltchain 700 can be configured to only allow expert E, E’, E” to insert data into the block chain 700, which received a proper education and which have the necessary qualifications. This can be implemented using scripting mechanisms of the blockchafn technology. The scripts may detine conditions that need to be fulfilled before an entry may be added to the blockchain. These mechanisms are also known as smart contracts.
It is worth pointing out that blockchain technology coes not rely on a single central server out is a distributed data structure shared among all peers at a peer- to-peer network. Data is added to the blockchaln using a trust mechanism, which is well-known In the art. As a result. all peers In the network usually accept the current longest chain as the most trustworthy. Moreover, the content of the blockchain 700 is publicly available and thus any user may identify all records that are stored in the blockchain 700. This allows a userrpatientfdc-ctor to review all the experts that have contributed to the blockchaln 700 and their 'ev|c.-ws of image tiles stored in the blockchain 700. This provides an unprecedented degree of transpa'ency in the diagnostic process. method to determine abnormality value whole slide image 12 Segmentation phase image tiles H [S [6 I7 2 1 22 23 24 I03 103 IU4 . 123 130 203 201 210 280 230 240 300 310 320 330 410 430 440 S00 S31 53?. 533 5arediction phase local abnormalltv degree value for each tile evaluation phase degree of abnormality cancer analysis system whole slide image segmentatiow ertity set of Image tlles OOl1‘lpJtal.'iO.'l entity mowledgebase raining data result image processing entity orediction phase aairs of abwormality and likelihood list of pairs L_w _oartitioni1g phase al:~nor1iaIll:y p-ediction pwase Llst creatlon arlorlzation phase pairs of priorities and Prioritize sorting phase candidate elimination Jhase i:listrlb.itlo'i phase decisiuii |lIdki'|9 pliase review phase abnormality determination phase KB extension phase improvement phase accuracy compa*iso1 phase Determination phase Network rezilacement phase Lea are nctwo*k DCAS wriole slide Image set of image tiles list of abnormality cegree and likelihood pairs llst of candidate image tllcs 505, 505 507 508 509 513 511. 512 S13 S14 515 516 517 518 519 500 $01 502 603 504 605 605 50? I308 609 ‘/13. ?11, 712, 713. 714. 715, 715, 71/. 713, 719. 72:1, 721, 722 ‘J05’. 5115" 7131710" 7112 711" 7122 712" 713C 713" 714§ 714" 715fl 715" 7153 716" 31/’. III" 7l3fl'?18" 713C 719" ?23fl 720" 7212 721" vs: Iaatlwr st-at va dation training data 1:Jepe1:1e1t validation data validation data t'aini'1g data analysis system segmentation ant ty computation ent ty o*io'.2atlon entity communication interface Validatiow Database Indeaendewt velldat on cohort testing enttv degree or abno'maIity convolut onal Veural Networ< Input image t It: first corwo JUSII elght fl°3t featue naps subsampling roar second reatu'.=: maps secmd convolution th ‘:1 featare ‘news feed fa-wa'd lave 3 fu V cornccted ayer output lave‘ Jiockcha ‘I Oluck weader was‘: value of orevious ‘reader Vlerk e '30: ‘W10 hash Data slack ink to data alock link to image tile ink to wno e sl de inage abnormality degree data blo:< mage ti'e whole slide 'naga S__w aw VB expert whole slide image sequence of image tiles fikefihood abnormality degree priority List of abnormal events for whole slide image List subset of S w list of experts assignments final abnormality degree value Validation database
Claims (3)
1. A
2. Claims method to determine a degree of aonorma ity, the metiotl comprising the following steps:
3. ) receiving a whole slide inage (11, w, 722), the whole slide image [11, w, ?22) depicting at least a portio1 of a cell, in particular a htman cell; classiryi-lg at least one image tile (13, not, ?21, ?21', 721"} or the whole slide image (11, w, 7'22) using a neural network (600) to determine a local abnormality deg'ee va ue (15, a_j, 519, 719, 719', 719”; associated wit: the at least one irrage tile (13, 601, 72:, 7'21‘, 721"}, he local abnocrnality degree value (15, a_j, $19, 719, 719', 719”) indicating a likelihood that the associated at least one segment depicts at least a pa *t of a cancerous ce ; and determining a degree of abnorma ity (1 7) for the mole s idc image (11, W, 722) based on he local abnormality degree value [15, a_j, 519, 71.9, 719', 7'19”) for he at least one image ti e (13, 601, 721, ?21’, ?2l”). . The method of claim 1, characterized by lzecelvlng an update whole sllde Image (11, 21, 501, ?22, w) to update a knowledgebase (25, 50?); Segmenting the update whole slide image [11, 21, 501, 722, w} into a plurality of image tiles (13, 601, 721, 221’, 721"); Determlnlng a tralnlng abnollnality degree value (a w) for each Image tile of at least one subset of the plurality of image tiles ((2), in partictlar by a human expert (E); Updating the knowledgeaase (25, S0?) with the subset of the plurality of Image tiles (C) and the associated training abnormality degree values (a_w] if it is determined that adding the subset of the plurality of image tiles (C) and the associated training abnormality degree values (a_w] improves the accuracy of the neural network when trained with the updated knowledgebase (25. S07). . The 'net'lod of clai'n 2, characterized i1 that updating the know edgebase (25, 507) further comprises: Computing a predicted abnornlzlity degree value (a_j) and an associated likelihood value (I J) for each of tie p urallty of Image tiles {13, 601, 721, 721’, 721”) using the neural network; Determining a priority value (p_j) based on the predicted ahnnrmalitv degree value (a_i] and the associated likelihood value (l_j) for each of the pluralit}'ol'ime9e tiles (13,. 601, ?21, 721', 721"); Determining the subset of tie Image tiles ((3) based on the determined priority (p_]') values. . A computer readable medium storing instructions that when executed hy at least one processor cause the at least one processor to implement a method according to any of the preceding claims. . A distributed cancer analysis system (500), comprising the following corrponents: a segmentation entity {S11} adapted to receive a wltole slide image (501), the whole slide image (501) depicting at least a portion of a cell, In particular a human cell; a computation entity (S12) adapted to compute a degree of abnorma it; (5 19] for the whole slide image (501) using a neural network [60D].
Family
ID=
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11954593B2 (en) | Method to determine a degree of abnormality, a respective computer readable medium and a distributed cancer analysis system | |
| Tellez et al. | Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks | |
| JP7504116B2 (en) | Machine learning using distance-based similarity labels | |
| Bridge et al. | Fully-automated analysis of body composition from CT in cancer patients using convolutional neural networks | |
| Poudel et al. | Recurrent fully convolutional neural networks for multi-slice MRI cardiac segmentation | |
| US9489562B2 (en) | Image processing method and apparatus | |
| JP2022527145A (en) | Multiple instance Lana for predictive organizational pattern identification | |
| WO2019245597A1 (en) | Method and system for improving cancer detection using deep learning | |
| US20210216745A1 (en) | Cell Detection Studio: a system for the development of Deep Learning Neural Networks Algorithms for cell detection and quantification from Whole Slide Images | |
| Ryalat et al. | Harris hawks optimization for COVID-19 diagnosis based on multi-threshold image segmentation | |
| Sornapudi et al. | Comparing deep learning models for multi-cell classification in liquid-based cervical cytology image | |
| Ray et al. | Cauchy with whale optimizer based eagle strategy for multi-level color hematology image segmentation | |
| Han et al. | Automatic recognition of 3D GGO CT imaging signs through the fusion of hybrid resampling and layer-wise fine-tuning CNNs | |
| Luo et al. | Deep angular embedding and feature correlation attention for breast MRI cancer analysis | |
| Bouilhol et al. | DeepSpot: A deep neural network for RNA spot enhancement in single-molecule fluorescence in-situ hybridization microscopy images | |
| Anand et al. | Optimized Swarm Enabled Deep learning technique for bone tumor detection using Histopathological Image | |
| Cao et al. | Automatic segmentation of pathological glomerular basement membrane in transmission electron microscopy images with random forest stacks | |
| Shrestha et al. | A deep learning based convolution neural network-DCNN approach to detect brain tumor | |
| Khan et al. | Sequential segmentation of the left atrium and atrial scars using a multi-scale weight sharing network and boundary-based processing | |
| Qin et al. | Intelligent objective osteon segmentation based on deep learning | |
| Roszkowiak et al. | Clustered nuclei splitting based on recurrent distance transform in digital pathology images | |
| Keller et al. | Quantitative assessment and classification of tissue-based biomarker expression with color content analysis | |
| Abinaya et al. | Accurate Liver Fibrosis Detection Through Hybrid MRMR-BiLSTM-CNN Architecture with Histogram Equalization and Optimization | |
| Abdulkareem et al. | Quantification of epicardial adipose tissue volume and attenuation for cardiac CT scans using deep learning in a single multi-task framework | |
| IE87151Y1 (en) | A method to determine a degree of abnormality, a respective computer readable medium and a distributed cancer analysis system |