WO2021198279A1

WO2021198279A1 - Methods and devices for virtual scoring of tissue samples

Info

Publication number: WO2021198279A1
Application number: PCT/EP2021/058335
Authority: WO
Inventors: Alexander Freytag; Christian KUNGEL
Original assignee: Carl Zeiss Ag
Priority date: 2020-03-30
Filing date: 2021-03-30
Publication date: 2021-10-07

Abstract

Methods for obtaining scores based on imaging data of tissue samples using machine-learning logics, training methods and corresponding devices are provided. With various devices and methods disclosed, automatic scoring of tissue samples becomes possible.

Description

Methods and devices for virtual scoring of tissue samples

TECHNICAL FIELD

The present application relates to methods and devices for virtual scoring of tissue samples.

BACKGROUND

Histological examination of tissue samples taken from patients remains the gold standard for diagnosis and evaluation in many areas of today’s clinical practice. Most notably, for diagnosing cancer, tissue samples of patients are taken, which are then examined. Various scoring systems exist for evaluation of the tissue sample, a final score then indicating a probability of a respective decease like cancer. As an example, the Gleason grading system is widely used as a prognostic predictor for patients with prostate cancer since the 1960s.

In conventional workflows, tissue samples are prepared and laboratories apply stains to them to produce stained samples. The stained samples are then manually or semi- automatically screened by a highly trained pathologist. This screening may for example include manually counting mitosis cells. Scores are assigned to the samples based on different scoring schemes, and the scores are then combined to form a basis for a final decision, for example final diagnosis or grading and treatment strategy associated therewith.

This conventional workflow presents various problems. The conventional workflow comes with a lot of “process noise” for example induced by the preparation of sample slides (cutting tissue, preparing, staining) as well as by the evaluation. For example, the manual examination of stained tissue samples is error prone for various reasons. Histopathology images based on such tissue samples have a high resolution (high spatial volume), they are recorded for multiple types of staining (high type volume) and, every day, a multitude of cases needs to be inspected (high task frequency). These factors increase the likelihood for errors. Furthermore, diagnosis performed by different pathologists may vary at least slightly regarding the results. Thus, there is both an intraobserver variability (The same pathologist does not always produce the same results in a given situation) and interobserver variability (different observers do not always produce the same results in a given situation).

A further example for a scoring scheme is the Nottingham histology score (NHS) for breast cancer, where a score from 1 to 3 is assigned to the tissue samples in different categories, for example the presence of glandular/tubular structures, nuclear pleomorphism and mitotic count.

Various approaches have been made to automatize the scoring. For example,

EP 2 973397 B1 relates to a tissue object-based machine learning system for automatic scoring of digital whole slides, which, however, is limited to specimen stained with immunohistochemical (IHC) assay. Another approach for IHC stained sample is disclosed in EP 3588382 A1. Other approaches to scoring are disclosed in

Heinemann, F., Birk, G. & Stierstorfer, B. Deep learning enables pathologist-like scoring of NASH models. Sci Rep 9, 18454 (2019). https://doi.org/10.1038/s41598-019-54904-6

Deep learning enables automated scoring of liver fibrosis stages. (2018) Yang Yu1, Jiahao Wang4, Chan Way Ng2,5,6, Yukun Ma2,6, Shupei Mo1, Eliza Li Shan Fong, Jiangwa Xing, Ziwei Song, Yufei Xie, Ke Si, Aileen Wee, Roy E. Welsch, Peter T. C. So & Hanry Yu

Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Gabriele Campanella, Matthew G. Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J. Busam, Edi Brogi, Victor E. Reuter, David S. Klimstra and Thomas J. Fuchs. Nat Med, vol. 25, 8, p. 1301- 1309, 8/2019

Terabyte-scale Deep Multiple Instance Learning for Classification and Localization in Pathology. Gabriele Campanella, Vitor Werneck Krauss Silva and Thomas J. Fuchs. arXiv: 1805.06983 [cs], 2018-05-17 Clinically applicable deep learning for diagnosis and referral in retinal disease Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera-Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O'Donoghue, Daniel Visentin, George van den Driessche, Balaji Lakshminarayanan, Clemens Meyer, Faith Mackinder, Simon Bouton, Kareem Ayoub, Reena Chopra, Dominic King, Alan Karthikesalingam, Cian OHughes, Rosalind Raine, Julian Hughes, Dawn A Sim, Catherine Egan, Adnan Tufail,

Hugh Montgomery, Demis Hassabis, Geraint Rees, Trevor Back, Peng T. Khaw, Mustafa Suleyman, Julien Cornebise, Pearse A. Keane, Olaf Ronneberger.

Furthermore, various approaches to virtual staining are known for example from US 9786050 B2 or US 2019 / 0 188446 A1. These documents describe approaches where a tissue sample is not stained in a lab, but a virtual staining, i.e. a staining in a computer graphic, is applied, which then may be evaluated.

It is an object to provide improved methods and devices related to virtual scoring of tissue samples.

SUMMARY

A method as defined in claim 1 or 22 and a device as defined in claim 21 or 31 are provided. The dependent claims define further embodiments as well as corresponding computer programs, storage mediums or data carrier signals.

According to an embodiment, a method of obtaining a score indicating a disease from a tissue sample is provided, comprising: obtaining imaging data of the tissue sample, processing the imaging data in at least one machine-learning logic, the at least one machine-learning logic being configured to output at least one score based on the imaging data provided.

The term machine-learning logic refers to an entity that may be trained by training data to be able to perform certain tasks. A machine-learning logic may for example be based on neural networks like deep neural networks, general adversarial networks, convolutional neural networks or support vector machines. As used herein, in addition to such a part that can be trained bay training data, optionally a machine-learning logic may comprise a post-processing portion which further processes an output. Machine learning logics are implemented on electrical devices like computers. All references to such electrical devices and the functionality provided by each are not intended to be limited to encompassing only what is illustrated and described herein. While particular labels may be assigned to such electrical devices disclosed, such labels are not intended to limit the scope of operation for the electrical devices. Such electrical devices may be combined with each other and/or separated in any manner based on the particular type of electrical implementation that is desired. It is recognized that an electrical device disclosed herein or usable for implementing techniques discussed herein may include any number of microcontrollers, machine-learning- specific hardware, e.g., a graphics processor unit (GPU) and/or a tensor processing unit (TPU), integrated circuits, memory devices (e.g. FLASH, random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), or other suitable variants thereof), and software which co-act with one another to perform operation(s) disclosed herein. In addition, any one or more of the electrical devices may be configured to execute a set of program code that is embodied in a non- transitory computer readable medium programmed to perform any number of the functions as disclosed.

Imaging data of the tissue sample, as used herein, refers to any kind of data, in particular digital imaging data, representing the tissue sample or parts thereof. For example, depending on the image modality, the dimensionality of the imaging data of the tissue sample may vary. The imaging data may be two-dimensional (2-D), one dimensional (1-D) or even three-dimensional (3-D). If more than one image modality is used for obtaining imaging data, a part of the imaging data may be two-dimensional and another of the imaging data may be one-dimensional or three-dimensional. For instance, microscopy imaging may provide imaging data that includes images having spatial resolution, i.e. , including multiple pixels. Scanning through the tissue sample with a confocal microscope may provide imaging data comprising three-dimensional voxels. Spectroscopy of the tissue sample may result in imaging data providing spectral information of the whole tissue sample without spatial resolution. In another embodiment, spectroscopy of the tissue sample may result in imaging data providing spectral information for several positions of the tissue sample which results in imaging data comprising spatial resolution but being sparsely sampled. A hyperspectral scanner may be used for acquiring images of the tissue samples in one or more spectral bands. However, the spectral bands are not limited to spectral bands in the visible spectrum but may also comprise spectral bands in the ultraviolet, and infrared range. The image modalities for acquiring digital imaging data of tissue samples may also comprise a Raman analysis of the tissue samples. In particular, the imaging modalities may comprise some simulated Raman scattering (SRS) analysis of the tissue samples, coherent anti-stokes Raman scattering (CARS) analysis of the tissue samples, surface enhanced Raman scattering (SERS) analysis of the tissue samples. In further embodiments, the image modalities may also comprise fluorescence lifetime imaging microscopy (FLIM) analysis of the tissue samples. The image modalities may also comprise a phase sensitive analysis of the tissue samples. Yet a further example would be transmitted-light or reflected-light microscopy, e.g., for observing cells. Imaging modalities may, as a general rule, imaging tissue in-vivo or ex-vivo. An endoscope may be used to acquire images in-vivo, e.g., a confocal microscope or using endoscopic optical coherence tomography (e.g., scanned or full-field). A confocal fluorescence scanner could be used. Endoscopic two-photon microscopy would be a further imaging modality. A surgical microscope may be used; the surgical microscope may, itself provide for multiple imaging modalities, e.g., microscopic images or fluorescence images, e.g., in specific spectral bands or combinations of two or more wavelengths, or even hyperspectral images.

A tissue sample may be a tissue section or a tissue slice of a tissue probe, in particular a tissue sample as used in histopathology. In this case, tissue samples may be thin sections of a wax block comprising an embedded processed sample. However, the term tissue sample may also refer to tissue having been processed differently or not having been processed at all. For example, tissue sample may refer to a part of tissue observed in vivo and/or tissue excised from a human, an animal or a plant, wherein the observed tissue sample has been further processed ex vivo, e.g., prepared using a frozen section method. The term tissue sample may also refer to a cell, which cell can be of procaryotic or eucaryotic origin, a plurality of procaryotic and/or eucaryotic cells such as an array of single cells, a plurality of adjacent cells such as a cell colony or a cell culture, a complex sample such as a biofilm or a microbiome that contains a mixture of different procaryotic and/or eucaryotic cell species and/or an organoid. Beside histopathology, such tissue samples may be subjected to methods as discussed herein for example in the field of life science or medicine.

The imaging data may be imaging data of an unlabeled tissue sample or of a tissue sample provided with a simple stain. An unlabeled tissue sample , also referred to as unstained tissue sample, refers to a tissue sample where processing steps necessary to prepare the sample for imaging, like biopsy to take the sample from a living being, processing steps to prevent decay, embedding e.g. in a wax block and slicing have been performed, but no staining. A simple stain is a stain which may be provided quickly and at low costs. An example are stains like a so-called H&E (hematoxylin and eosin) stain, which takes less than an hour, e.g. about 45mins, and costs typically less than 10 US$ in March 2020 including laboratory costs. Applying simple stains is also commonly referred to as routine staining in histopathology, whereas applying other stains is commonly referred to as special staining. In other examples, the imaging data may be obtained in vivo, for example by imaging skin, or other body parts. The imaging may be done using a microscope to obtain imaging data on a microscopic level. Radiation may be used during capturing of imaging data for example to excite autofluorescence.

The at least one machine-learning logic may be configured to provide at least two scores of different scoring systems based on the imaging data. By employing different scoring schemes, essentially a full analysis of a tissue sample may be read out.

Staining with different scoring schemes enables essentially a full analysis of tissues with the tissue sample.

A score in the narrower sense relates to a number indicating the absence, presence or likelihood of a disease, for example presence of cancer or a specific type thereof, as for the scores explained in the introductory portion. In a broader sense, the term score in the sense of this application may also apply to other numbers quantifying properties of the tissue sample examined. Examples for such scores in the broader sense include a number of predefined objects in the imaging data, areas or are ratios of predefined objects in the imaging data, a mean size of predefined objects in the imaging data, a mean distance between predefined objects, a mean intensity of predefined objects and the like. Objects may be disease markers, cells, parts of cells, specific cells (e.g. tumor cells) or the like. More examples will be given further below.

Such scores conventionally are obtained based on a specific treatment of tissue or tissue samples, for example chemical staining of tissue specific to a score to be obtained or providing tissue with fluorescence markers and examining fluorescence images. In contrast, in embodiments a “rawer” form of imaging data may be used, for example imaging data of unstained tissue or imaging data of tissue samples showing only autofluorescence without the use of fluorescence markers, or phase-contrast images without fluorescence.

The at least one machine-learning logic may comprise separate machine-learning logics for at least some of the at least two scores.

Alternatively or additionally, the at least one machine-learning logic may comprise a common machine-learning logic having different output layers or blocks for at least some of the at least two scores.

“Additionally” here means that in some embodiments for some scores separate machine-learning logics may be provided, whereas for other scores a common machine-learning logic may be provided.

Obtaining the imaging data may comprise tiling the imaging data, and processing the imaging data may comprise processing tiled imaging data, wherein the method further comprises obtaining the at least one score based on the processed tiled imaging data. For example, processing the tiled imaging data may result in tile-based scores for the regions of the tissues sample included in the respective tiles. These tile-based scores may then be aggregated to obtain one continuous metric, which then may be mapped to a discrete scoring metric to provide the scores. The aggregating and mapping may be done by a post-processing portion of the at least one machine-learning logic, which may be implemented by simple computer code for aggregating (e.g. counting) and then mapping, and receiving data from an output layer or block of the machine-learning logic. The at least one machine-learning logic may be configured to obtain the at least one score based on the imaging data in an end-to-end approach without intermediate images.

The at least one machine-learning logic may also comprise a first machine-learning logic configured to generate a virtually stained image based on the imaging data, and at least one second machine-learning logic configured to provide the at least one score. Virtual staining refers to a process of modifying colors or intensities of an image to simulate a staining process conventionally performed by other means. This includes simulating the chemical staining conventionally used in histopathology, as explained in the introductory portion. Staining in a broader sense may comprise modifying mole cules of any one of the different types of tissue sample mentioned above. The modifi cation may lead to fluorescence under a certain illumination (e.g., an illumination under ultra-violet (UV) light), by introduction of fluorescence markers. For example, staining may include modifying genetic material of the tissue sample. Stained tissue samples may comprise transfected cells. Transfection may refer to a process of deliberately in troducing naked or purified nucleic acids into eukaryotic cells. It may also refer to other methods and cell types. It may also refer to non-viral DNA transfer in bacteria and non animal eukaryotic cells, including plant cells. Therefore, also in the sense of the present invention also such tissue samples are to be regarded as stained tissues samples.

Another example for virtual staining would pertain to virtual fluorescence staining. For example, in life-science applications, images of cells - e.g., arranged as live or fixated cells in a multi-well plate or another suitable container - are acquired using transmitted- light microscopy. Also, a reflected light microscope may be used, e.g., in an endoscope or as a surgical microscope. It is then possible to selectively stain certain cell organelles, e.g., nucleus, ribosomes, the endoplasmic reticulum, the golgi apparatus, chloroplasts, or the mitochondria. A fluorophore (or fluorochrome, similarly to a chromophore) is a fluorescent chemical compound that can re-emit light upon light excitation. Fluorophores can be used to provide a fluorescence chemical stain. By using different fluorophores, different chemical stains can be achieved. For example, a Hoechst stain would be a fluorescent dye that can be used to stain DNA. Other fluorophores include 5-aminolevulinic acid (5-ALA), fluorszine, and Indocyanine green (ICG) that can even be used in-vivo. Fluorescence can be selectively excited by using light in respective wavelengths; the fluorophores then emit light at another wavelength. Respective fluorescence microscopes use respective light sources. It has been observed that illumination using light to excite fluorescence can harm the sample; this is avoided when providing Fluorescence-like images through virtual staining. The virtual fluorescence staining mimics the fluorescence chemical staining, without exposing the tissue to respective excitation light.

Modifying genetic material of the tissue sample in this way may make the genetic mate rial observable using a certain image modality. For example, the genetic material may be rendered fluorescent. In some examples, modifying genetic material of the tissue sample may cause the tissue sample to produce molecules being observable using a certain image modality. For example, modifying genetic material of the tissue sample may induce the production of fluorescent proteins by the tissue sample.

Providing a virtually stained image for a respective scoring allows easier control or verification by a pathologist of the obtained score. Furthermore, this may make transparent what the scoring machine-learning logic was sensitive to. Additionally or alternatively, the virtually stained images may be archived. Overall, this may improve acceptance of the method by pathologists. In other words, with the virtually stained image an image is provided which essentially looks as if the tissue sample were subjected to the actual staining process like chemical staining, providing with fluorescence markers and subjecting to corresponding illumination etc.

The at least one second machine-learning logic may be configured to provide the score based on the virtually stained image.

The first machine-learning logic may comprise a plurality of layers or blocks, and the at least one second machine-learning logic may be configured to provide the at least one score based on a representation of the imaging data in one of the plurality of layers or blocks.

Generally, layers of a machine-learning logic and more particular a neural network are entities including a plurality of nodes operating at a same depth. Layers typically include an input layer receiving data (e.g. image date) provided to the machine-learning logic and an output layer outputting data (e.g. a virtually stained image or a score). One or more so-called hidden layers may be provided between input layer and output layer. One or more layers may be included in or form a block. Within a block, layers may process data in sequence or in parallel in the layers. Between layers or blocks, a spatial contraction or a spatial expansion may occur. I.e., the x-y-resolution of respective representations of the imaging data or an output image may be decreased (spatial contraction) from layer to layer (block to block)or increased from layer to layer (block to block) (spatial expansion). In a U-net implementation e.g. of a deep neural network, one or more encoder branches having respective layers or blocks where spatial contraction occurs are linked to one or more decoder branches where spatial expansion occurs via a bottleneck. Layers may be selected from the group including: convolutional layers, activation function layers (e.g., ReLU (rectified linear unit), Sigmoid, tanh, Maxout, ELU (Exponential Linear Unit), SeLU (scaled exponential linear unit), Softmax and so on), downsampling layers, upsampling layers, normalization layers (e.g., batch normalization, instance normalization, group normalization, channel normalization, etc.), dropout layers, etc.. Thus, each layer defines a respective mathematical operation.

For example, encoder branches can be built from encoder blocks followed by downsampler blocks. Downsampler blocks may be implemented by using max-pooling, average-pooling, or strided convolution. Upsampler blocks may be implemented by using transposed-convolution, nearest neighbor interpolation, or bilinear interpolation. We also found it helpful to follow them by convolution with activations.

Decoder branches can be built from upsampler blocks followed by decoder blocks. For upsampler blocks, it is possible to apply transposed-convolution, nearest neighbor interpolation, or bilinear interpolation. Especially for the latter two, it has been found that placing several convolution layers thereafter is highly valuable.

More generally, an example encoder or decoder block includes convolutional layers with activation layers and followed by normalization layers. Alternatively, each encoder or decoder block may include more complex blocks, e.g., inception blocks (see, e.g., Szegedy, Christian, et al. "lnception-v4, inception-resnet and the impact of residual connections on learning." Thirty-first AAAI conference on artificial intelligence. 2017), DenseBlocks, RefineBlocks, or having multiple operations in parallel (e.g., convolution and strided convolution) or having multiple operations after each other (e.g., three convolution with activation and then followed by normalization before going to downsampling), etc..

The one layer of the plurality of layers may be such a bottleneck where the representation has a maximum spatial contraction among the plurality of layers. Performing the virtual scoring based on a spatial contraction of the tissue samples may reduce the amount of data to be processed for the virtual scoring. In particular, at the bottleneck the highest spatial contraction is present.

The at least one second machine-learning logic may be configured to provide regions of interest for the virtually stained images. Regions of interests may be certain kinds of cells relevant for scoring, cells with certain properties relevant for scoring etc. A post processing portion may then determine the scores based on the regions of interest, for example based on counting the regions of interest. Additionally, an image with marked regions of interest based on the virtually stained image may be provided. Providing marked regions of interest of the virtually stained image further may enhance control by a pathologist.

According to another embodiment, a device including at least one machine-learning logic is provided, wherein the device is configured to perform any of the above methods.

According to another embodiment, a method of training at least one machine-learning logic is provided, the method comprising: providing imaging data of tissue samples, providing reference scores for the imaging data, and training the at least one machine-learning logic based on the imaging data and the reference scores.

In some embodiments the imaging data may be imaging data without staining or other modification (e.g. fluorescence image with fluorescence markers). The reference score may be obtained based on reference imaging data, which represents the tissue with modifications like staining or fluorescence of fluorescence markers. The method may further comprise providing reference stained images associated with the imaging data, wherein the training is further based on the reference stained images.

The method may further comprise providing reference images with an indication of regions of interest associated with the stained images, wherein the training is further based on the marked reference images.

The method may be adapted for training the at least one machine-learning logic of the above-mentioned device.

As mentioned above, the imaging data may be based on unstained tissue, tissue provided with a simple stain, autofluorescent tissue without fluorescence markers etc, while conventionally the respective score is based on specifically stained tissue samples, tissue sample provided with fluorescence markers etc. Nevertheless, for obtaining the reference scores, such specifically stained tissue samples, tissue sample provided with fluorescence markers etc. may be used. In other words, the reference scores may be obtained based on tissue samples treated in a manner conventionally used for obtaining the respective score.

A computer program may also be provided, comprising a program code, which, when executed on one or more processors, causes execution of any of the above methods.

A tangible storage medium (e.g. CD, DVD, memory card, memory stick, hard disk drive, solid state disk) storing the above computer program or a data carrier signal carrying the above computer program may also be provided.

Devices corresponding to the above methods, e.g. correspondingly programmed computers or other processing devices, are also provided.

The above summary is merely a brief overview over some embodiments and is not to be construed as limiting in any way. BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments will be described in the following referring to the attached drawings, wherein:

Fig. 1 is a diagram of a workflow and device according to an embodiment,

Fig. 2 is a flowchart illustrating a training method according to some embodiments,

Fig. 3 is a flowchart illustrating a scoring method according to some embodiments,

Fig. 4 is a flowchart illustrating a scoring method according to some further embodiments, and

Figs. 5A, 5B and 6 to 9 are diagrams for illustrating devices and methods according to various embodiments.

DETAILED DESCRIPTION

In the following, various embodiments will be discussed referring to the attached drawings. These embodiments serve illustrative purposes only and are not to be construed as limiting in any way.

Features from different embodiments may be combined to form further embodiments. Variations or modifications described with respect to one of the embodiments may also be applicable to other embodiments.

Fig. 1 illustrates a workflow and a device according to an embodiment. This workflow and device is an example for an application for example in histopathology.

As shown in Fig. 1, tissue 2102 may be obtained from a living creature 2101 by surgery, biopsy or autopsy. After some processing steps to remove water and to prevent decay, said tissue 2102 may be embedded in a wax block 2103. From said block 2103, a plurality of slices 2104 may be obtained for further analysis. One slice of said plurality of slices 2104 may also be called a tissue sample 2005.

Before analyzing the tissue sample 2005, a chemical stain may be applied to the tissue sample 2005 to obtain a chemically stained tissue sample 2006. Said chemical stain may be a simple stain like an H&E stain. In some embodiments, the tissue sample 2005 may also be directly analyzed as an unlabeled tissue sample. A chemically stained tissue sample 2006 may facilitate the analysis. In particular, chemical stains may reveal cellular components, which are very difficult to observe in the unstained tissue sample 2005. Moreover, chemical stains may provide an increased contrast.

Traditionally, the tissue sample 2005 or 2006 is analyzed by an expert using a bright field microscope 2107, who then provides one or more scores.

According to some embodiments, image acquisition systems 2108 configured for acquiring digital imaging data of the tissue sample 2105 or the chemically stained tissue sample 2106 using one or more image modalities are used. Image modalities may comprise images of the tissue sample in one or more specific spectral bands, in particular, spectral bands in the ultra violet, visible and/or infrared range. Image modalities may also comprise a Raman analysis of the tissue samples, in particular a stimulated Raman scattering (SRS) analysis of the tissue sample, a coherent anti- Stokes Raman scattering, CARS, analysis of the tissue sample, a surface enhanced Raman scattering, SERS, analysis of the tissue sample. Further, the image modalities may comprise a fluorescence analysis of the tissue sample, in particular, fluorescence lifetime imaging microscopy. FLIM, analysis of the tissue sample. The image modality may prescribe a phase sensitive acquisition of the digital imaging data. The image modality may also prescribe a polarization sensitive acquisition of the digital imaging data.

In other embodiments, an in-vivo tissue may be used, and the image acquisition system may obtain images in vivo, for example by capturing images of tissue with a camera or other image modality. The digital imaging data 2109 may be processed in a device 2110 according to an embodiment. Device 2110 may be a computer. Device 2110 may comprise memory 2111 for (temporarily) storing the digital imaging data 2109 and a processor 2112 for processing the digital imaging data 2109. Device 2110 may process the digital imaging data 2109 to provide one or more scores 2113 which may be displayed on a display 2114 to be analyzed. Additionally, images of the tissue sample may be displayed.. Device 2110 may comprise different types of trained or untrained machine-learning logic, e.g. as explained further below, for analyzing the tissue sample 2105 or the chemically stained tissue sample 2106. The image acquisition system 2108 may be used for providing training data for said machine-learning logic.

The device of Fig. 1 as explained above comprises a machine-learning logic, which may for example comprise one or more neural networks like deep neural networks, convolutional neural networks (CNN) or general adversarial networks (GAN). First, a general training method will be explained referring to Fig. 2. Then, general methods using the trained machine-learning logic will be discussed referring to Figs. 3 and 4. Following this, specific implementation examples and their training will be discussed referring to Figs. 5A, 5B and 6 to 9.

Fig 2 is a flowchart illustrating a training method according to some embodiments. At 1201, the method of Fig. 2 comprises providing training data. The training data comprises input images of tissue slides and scores for the input images, also referred to as reference scores. In some embodiments, scores may be provided for a single input image according to different scoring systems, e.g. for breast cancer grading, e.g., presence of glandular/tubular structures, nuclear pleomorphism and mitotic count. Example scores for histopathology applications may include one or more of a Ki67 score, a Her2 score, an ER score or a PR score. For example, to evaluate breast cancer, these four scores may be evaluated in combination to determine presence and type of breast cancer. In other embodiments, a single output score for a single input image is provided. A score, in this respect, generally refers to a quantity enabling a diagnosis and in particular to a numerical value indicating a likelihood of a specific disease. Such scores, as already mentioned in the introductory portion, are essentially standardized and in part have been applied for a long time by pathologists. For the training data, the scores are provided by pathologists examining tissue samples corresponding to the input images.

In some embodiments, as will be illustrated further in the following, the input images may be stained input images, either based on images of tissue samples stained in a lab or virtually stained input images where the staining process is performed virtually by a machine-learning logic, for example by approaches as mentioned in the background portion or as described in co-pending applications bearing official filing numbers: PCT/EP2021/058270, PCT/EP2021/058283, PCT/EP2021/058277,

PCT/EP2021/058272 and PCT/EP2021/058273. Optionally, the training data also comprises marked images, where compared to the input images regions of interest, for example cells exhibiting certain properties like mitosis, or a certain shape, are marked.

Even if the input images are unstained images, for obtaining one or more scores for the input images for training purposes stained images may be used. For example, as input images for training images of unstained tissue samples or tissue samples provided with a simple stain may be used. For obtaining the reference scores, the tissue samples may be stained in a lab, or virtual staining may be provided to the images, to obtain stained images as conventionally used for obtaining the relevant scores. The stained images are than analyzed to obtain the reference score.

In applications other than histopathology, similar approaches may be used. Here, the input images may be images without a modification conventionally used to obtain the respective score in the broader sense as defined above. For example, in life science or medical applications in some cases conventionally fluorescence markers are used, for example green fluorescent protein (GFP) which is incorporated into DNA of a cell and causes the cell to produce fluroescent dye. Then, fluorescence images are obtained by exciting the fluroescence dye. In such cases, in some embodiments the input images may be obtained based on tissue samples without such fluorescence markers, for example transmitted light images, autofluorescence images (i,e, fluorescence images showing only the “natural” fluroescence of the tissue sample) or phase-contrast images, i.e. images different from thos conventionally used for determining the respective score. For obtaining the reference score, then tissues samples treated conventionally, e.g. with fluorescence markers, may be used. For the following further explanation, the image based on which the reference score is obtained is referred to as reference image. It should be noted that in other embodiments, the input image may be treated in the same way as in conventional approaches, e.g. with the use of staining, fluroescence markers and the like. In this case, for training purposes the input image may be the reference image.

Apart from the histopathological scores mentioned above, scores in the broader sense may include other quantified information about the tissue sample. This quantified information may relate to predefined objects, which in conventional approaches are made visible or at least more easily identifiable by measures like staining or fluorescence markers.

Examples for such scores related to objects may include:

- number of predefined objects in the input image

- area or area ratios of predefined objects in the input image

- mean size, median or other property of predefined objects in the input image

- mean distance between predefined objects

- mean intensity (e.g. fluorescence intensity) of predefined objects

In histopathology applications, the predefined objects may be disease markers like Kl- 67-stained cells, ER-stained cells or PR-stained cells. The score may than be the area ratio between stained cells and the whole tissue.

In medical applications, the object may be a tumor, which may be made visible conventionally by staining or fluorescence markers. The score in this case may be the area of the tumor or an area ratio between the tumor and non-tumorous tissue.

In life science applications, the score may describe a state of cells, cell cultures, or organoids. The predefined objects in this case may be cells or parts thereof (e.g. nucleus, cytoplasm etc.), possibly with certain markers or susceptible to markers or combinations thereof in conventional approaches, possible in a certain cell stadium. The score in this case may be a number of cells, an area of cells, a mean distance between cells or a staining intensity of cells in case of conventional staining.

More than one score may be assigned to a simple input image for training. Several approaches may be used to obtain a reference score from a reference image. Generally, for obtaining the reference score, the predefined objects are identified in the image. Then, the respective reference score is determined. This process of finding the reference score is also referred to as annotation.

In a first approach, the annotation may be performed manually, i.e. by a human being. For example, based on a reference image (e.g. stained tissue image or fluorescence image), the number of cells may be counted manually, or cell sizes may be estimated/measured manually, and the result may be used as reference score.

In a second approach, the annotation may be performed by a human being with the aid of a computer. For example a fluorescence image may be segmented to foreground (fluroescent part) and background (non-fluorescent part), or a stained image may be segmented into a foreground (stained part) and background (non-stained part) by a segmentation algorithm on a computer, and based on this segmentation the human being may determine the reference score. In another example, a human being may perform the above segmentation, and the areas of foreground, background or a ratio therebetween may be determined automatically by a computer as the score.

In a third approach, the annotation may be performed automatically. For example the segmentation into foreground and background may be performed by a segmentation algorithm. Segmentation algorithms usable are fore example described in Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015 or Tao, Andrew, Karan Sapra, and Bryan Catanzaro. "Hierarchical multi-scale attention for semantic segmentation." arXiv preprint arXiv:2005.10821 (2020). These approaches relate to so called semantic segmentation, where the image is transformed to a pixel map and for each pixel a decision is made if the pixel belongs to the foreground or background. In some cases, an istance segmentation may be performed, as described for example in He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017). Mask r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 2961-2969) or Mohan, Rohit, and Abhinav Valada. "Efficientps: Efficient panoptic segmentation." International Journal of Computer Vision (2021): 1-29.

Then areas may be determined by a further algorithm. For example, a connected component analysis may be performed on the semantically segmented image, resulting in spatially connected areas, so called connected components, that may be counted or otherwise analyzed. Connected component analysis is for example described in “Digital Image Processing (3rd Edition)” R. Gonzales and R. Woods Chapter 9 or in the Wikipedia article “Connected component labeling” as of March 25, 2021. .In other embodiments, an intance segmentation may be performed, as described for example in He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017). Mask r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 2961-2969) or Mohan, Rohit, and Abhinav Valada. "Efficientps: Efficient panoptic segmentation." International Journal of Computer Vision (2021): 1-29. This is somewhat a combination of semantic segmentation and connected components that transforms the image to a pixel map and assigns the pixels to different foreground objects (e.g. areas) or to the background.

At 1202, the method comprises training the machine-learning logic with the training data. The training may employ for example any conventional training methods for convolutional neural networks and/or general adversarial networks, depending on the type of machine-learning logic used. The thus trained machine-learning logic is then able to produce output scores based on input images (possibly stained input images) and to optionally produce also marked images. Examples for these cases and specifics on training will be discussed later using the specific examples of Figs. 5A, 5B and 6 to 9.

Such a trained machine-learning logic may then be used for automatic scoring of tissue sample. The corresponding method is illustrated in Fig. 3.

At 1301 in Fig. 3, the method comprises providing imaging data of tissue sample. At 1302, the method comprises processing the imaging data by a trained machine learning logic, in particular trained as explained above with reference to Fig. 2. As already mentioned above, examples for such machine-learning logics will be discussed further below. At 1303, the method comprises outputting one or more scores, for example two or more scores, as a result of the processing at 1302.

A complete imaging data for a typical tissue sample may be large. For example, typical resolutions used for such images are in the order of 40000-40000 pixels, or 1.6 Gpixel or even larger. Processing such an image as a whole requires a high amount of computing power. To be able to perform techniques discussed herein also with reduced computing power, tiling of images may be used. An embodiment of the method using such tilings is shown in Fig. 4.

At 1401, the method comprises partitioning an image of a tissue sample into tiles. The tiles may for example be square tiles or rectangular tiles. The tiles may be overlapping or non-overlapping tiles. The size of the tiles and hence the number of tiles the image is partitioned into may depend on the available computing power. The size of the resulting tiles is chosen such that the tiles may be processed by a machine-learning logic trained as discussed above.

An example tile size may be about 2000-2000 pixels, but is not limited thereto.

At 1402, the method comprises providing the tiles, for example one after the other, to a trained machine-learning logic, and at 1403, the method comprises obtaining corresponding outputs from the machine-learning logic. The actions at 1402 and 1403 essentially correspond to the actions at 1302 and 1303 in Fig. 3, the image processing now occurring tile by tile.

At 1404, the method comprises combining the outputs. The combined outputs may then for example give a certain probability in percent for a certain disease like a certain kind of cancer. At 1405, the combined outputs are then mapped to some final score according to some scoring scheme, for example certain percentages ranges may be mapped to certain grades, according to one or more conventional scoring schemes.

Next, various kinds of machine-learning logics, their training and their operation will be discussed referring to Figs. 5A, 5B and 6 to 9. In Figs. 5A, 5B and 6 to 9, in order to avoid repetitions, like elements are designated with the same reference numerals and will not be described repeatedly. It should be noted that each of the embodiments of Figs. 5A, 5B and 6 to 9 discussed in the following may use tiling as explained with reference to Fig. 4, and this will not be discussed again specifically for these embodiments. These machine-learning logics will be discussed using histopathology scores and corresponding virtually stained images as examples. However, these machine-learning logics are also applicable to other types of scores and images as discussed above.

First, an end-to-end approach will be discussed where based on an imaging data input an output score is directly produced.

Figs. 5A and 5B show an example embodiment where separate machine-learning logics are used to produce different scores. In Fig. 5A, a machine-learning logic 1502A is trained and used to produce a Ki67 score 1504A based on tissue sample images 1501, and in Fig. 5B, a machine-learning logic 1502B produces a Her2 score 1504B based on images 1501. Correspondingly, machine-learning logic 1502A is trained with images and their associated Ki67 scores (determined for example by a conventional process including staining and evaluation by a pathologist), and machine-learning logic 1502B is trained using tissue sample images and their corresponding Her2 scores. It should be noted that Ki67 score and Her2 score are merely two examples, and all scores that are conventionally used may also be used in embodiments. Machine learning logics 1502A and 1502B may be implemented on the same physical device, for example as shown in Fig. 1, such that with a single device a plurality of different scores may be obtained.

Machine-learning logic 1502A comprises a plurality of layers 1503A_1 to 1503A_4, and machine-learning logic 1502B comprises a plurality of layers 1503B_1 to 1503B_4. In this and the following embodiments, it is to be understood that while reference is made to layers, these layers may be part of blocks, and therefore the respective machine learning logic may also comprise a plurality of blocks as explained above. While four layers are shown for each of machine-learning logics 1502A and 1502B, other numbers of layers may also be used. Also, in some embodiments, for different scores different numbers of layers may be used. The respective first layers 1503A_1, 1503B_1 in the example of Figs. 5A and 5B serves as input layer, and a respective last layer (1503A_4 and 1503B_4 in the embodiment of Figs. 5A and 5B) serves as output layer. From one layer to the next, a spatial contraction of the data input to the respective layer (tissue sample images 1501 for the respective input layer, data from the respective preceding layers for the remaining layers) may be provided.

In some embodiments, machine-learning logic 1502A and 1502B each may be implemented as a convolutional neural network (CNN).

In an alternative embodiment, which is shown in Fig. 6, a single machine-learning logic 1601 is provided which is configured to output different scores for an input image provided.

In the example of Fig. 6, machine-learning logic 1601 receives tissue sample images 1501 and outputs Ki67 score 1504A and Her2 score 1504B. As already explained with reference to Figs. 5A and 5B, these scores are merely examples, and other scores may be used as well. Machine-learning logic 1601 comprises a plurality of layers 1602_1 to 1602_3, 1603A and 1603B. Layers 1602_1 serve as input layers. Layers 1602_1 to 1602_3 are common both for providing Ki67 score 1504A and Her2 score 1504B. However, in the embodiment of Fig. 6, machine-learning logic 1601 includes separate output layers 1603A, 1603B to output Ki67 score 1504A and Her2 score 1504B, respectively. In case more scores are to be generated, correspondingly more output layers may be provided. Also machine-learning logic 1601 may be implemented using a convolutional neural network, and the layer may comprise spatial contraction as explained with reference to Figs. 5A and 5B.

Figs. 5A, 5B and 6 illustrate a so-called end-to-end approach, where input tissue sample images are directly processed to output a respective score. In other embodiments, by a first machine-learning logic a virtually stained image is provided, and this virtually stained image is then processed by a second machine-learning logic to obtain an output score. Examples for such approaches will be discussed referring to Figs. 7 to 9. Fig. 7 shows an embodiment including a first machine-learning logic 1701 and a second machine-learning logic 1703. Machine-learning logic 1701, 1703 may be implemented in the same device, for example the device of Fig. 1.

Machine-learning logic 1701 receives tissue sample images 1501 as input and outputs corresponding virtually stained images 1702. Virtually stained images are images where a “coloring” corresponding to a conventional staining process in a lab is applied. This virtual staining process by machine-learning logic 1701 may be performed in any conventional manner or as described in more detail in co-pending application bearing official filing number DE 102020 108745.4. In particular, in the example of Fig. 7, machine-learning logic 1701 is implemented similarly to the one described in the above-mentioned co-pending application and comprises a plurality of layers 1704_1 to 1704_7. Layer 1704_1 is an input layer, and layer 1704_7 is an output layer. Layers 1704_1 to 1704_4 form an encoder branch providing spatial contraction of representatives of imaging data 1501, and layers 1704_4 to 1704_7 provide a decoder branch providing spatial expansion of the respective representations to provide images 1702. Layer 1704_4, where the maximum contraction is present and which may be seen as being the last layer of the encoder branch or the first layer of the decoder branch, is also referred to as bottleneck. Spatial contraction and spatial expansion implemented by the encoder branch and the decoder branch, respectively, means that the x-y-resolution of respective representations of the imaging data 1501 and the output images 1702 may be decreased (increased) from layer to layer along the one or more encoder branches (decoder branches). At the same time, feature channels can increase and decrease along the encoder branch and decoder branch, respectively. The bottleneck may also comprise a plurality of layers. Machine-learning logic 1701 is trained with images of tissue sample and correspondingly stained images (images of tissue samples stained by a corresponding laboratory).

Virtually stained images 1702 are provided to second machine-learning logic 1703, which outputs a corresponding score, in the example of Fig. 7 Ki67 score 1504. For training, apart from having virtually stained images 1702 as input data instead of tissue sample images 1501, machine-learning logic 1703 essentially may be implemented as machine-learning logic 1502A, 1502B or 1601 discussed above. For training, virtually stained images are provided together with a corresponding score attributed to the virtually stained image by a pathologist. Machine-learning logic 1703 comprises a plurality of layers 1705_1 to 1705_4, where the number of four layers again serves only as an example, layer 1705_1 is an input layer and layer 1705_4 is an output layer. Multiple machine-learning logics 1703 may be provided to output different scores (similar to separate machine-learning logics 1502A and 1502B of Figs. 5A and 5B), or a single machine-learning logic with multiple output layers for different scores may be provided (similar to machine-learning logic 1601 of Fig. 6). The layers provide a stepwise spatial contraction of a representation of virtually stained image 1702.

It should be noted that in the embodiment of Fig. 7, first and second machine learning logics 1701, 1703 may be trained separately from each other, but may also be trained jointly by providing input images, stained images and associated scores as training data. This also applies to the below embodiments of Figs. 8 and 9.

In other embodiments, a machine learning logic may determine regions of interest, based on which the score is determined. Examples for regions of interest, include cells having certain properties, which then lead to a score. Additionally, a virtually stained image where the regions of interest are marked may be output. A corresponding embodiment is shown in Fig. 8.

Similar to Fig. 7, the embodiment of Fig. 8 comprises a machine-learning logic 1701 to provide virtually stained image 1702 from input images 1501. Virtually stained image 1702 is then provided to a machine-learning logic 1804 which provides regions of interest, for example a number of regions of interest having a certain property or indications of positions of such regions of interest. In some embodiments, a virtually stained image 1802 with marked regions of interest 1803 may output to enable control by a pathologist. By a post-processing portion 1805, which may be based on program code , a score like Ki67 score 1504A is provided based on the regions of interest, for example based on a number of regions of interest and mapping them to a score. This post-processing may be performed using a corresponding computer code for counting the marked regions and mapping the result to a score. In other embodiments, post processing 1805 may be based on image 1802 and may count markers 1803 by image processing techniques. In other embodiments, this post-processing may be performed manually by a pathologist or the like. Machine-learning logic 1804 may comprise a plurality of layers, two layers 1801_1 and 1801_2, in the example of Fig. 8. For training, machine-learning logic 1804 may be implemented based on a convolutional neural network. For training machine-learning logic 1804, stained images and stained images with marked regions of interest are provided, in some implementations together with a scoring for the marked images. The marking may be done by a trained pathologist.

Instead of using virtually stained images as an output, also some other representations of the images may be used as an input for a machine-learning logic providing a score. An example is shown in Fig. 9. Here, machine-learning logic 1701 is provided as in the embodiment of Figs. 7 and 8. An additional machine-learning logic branch 1902 is provided comprising layers 1901_1 and 1901_2 as input and output layers, respectively. Also another number of layers may be provided. Input layer 1901_1 receives the spatially compressed representation of layer 1704_4 as an input. In other embodiments, also a representation from any other layer of machine-learning logic 1701 may be used.

For training, the embodiment of Fig. 9, tissue sample images, corresponding virtually stained images and scores attributed to them are provided. Machine-learning logic 1701 is trained as previously. Moreover, for each training image, the respective compressed representation at layer 1704_4 together with the respective score is provided to branch 1902 for training. Several branches 1902 may be provided for different scores, or branch 1902 may be provided with different output layers 1901_2 for different scores, as explained with reference to Figs. 5A, 5B and 6.

Several machine-learning logics 1804 may be provided to provide different markings 1803 and corresponding different sores, or different output layers 1801_2 may be provided for different markings and different scores, similar to what has been explained for Figs. 5A, 5B and 6.

It should be noted that input image 1701 need not be a single image, but in embodiments a plurality of images from a certain tissue sample, for example from different layers, may be provided to obtain a final score.

Some embodiments may be defined by the following examples: Example 1. A method of obtaining a score indicating a disease from a tissue sample comprising: obtaining imaging data of the tissue sample, processing the imaging data in at least one machine-learning logic, the at least one machine-learning logic being configured to output at least one score based on the imaging data provided.

Example 2. The method of example 1 , wherein the imaging data is imaging data of an unlabeled tissue sample or of a tissue sample provided with a simple stain.

Example 3. The method of example 1 or 2, wherein the at least one machine learning logic is configured to provide at least two scores of different scoring systems based on the imaging data.

Example 4. The method of example 3, wherein the at least one machine-learning logic comprises separate machine-learning logics for at least some of the at least two scores.

Example 5. The method of example 3 or 4, wherein the at least one machine learning logic comprises a common machine-learning logic having different output layers or blocks for at least some of the at least two scores.

Example 6. The method of any one of examples 1 to 5, wherein obtaining the imaging data comprises tiling the imaging data, and wherein processing the imaging data comprises processing the tiled imaging data, wherein the method further comprises obtaining the at least one score based on the processed tiled imaging data.

Example 7. The method of any one of examples 1 to 6, wherein the at least one machine-learning logic is configured to obtain the at least one score based on the imaging data in an end-to-end approach without intermediate images.

Example 8. The method of any one of examples 1 to 7, wherein the at least one machine-learning logic comprises a first machine-learning logic configured to generate a virtually stained image based on the imaging data, and at least one second machine learning logic configured to provide the at least one score.

Example 9. The method of example 8, wherein the at least one second machine learning logic is configured to provide the score based on the virtually stained image.

Example 10. The method of example 8, wherein the first machine-learning logic comprises a plurality of layers or blocks, and wherein the at least one second machine learning logic is configured to provide the at least one score based on a representation of the imaging data in one of the plurality of layers or blocks.

Example 11. The method of example 10, wherein the one layer or block of the plurality of layers or blocks is a bottleneck where the representation has a maximum spatial contraction among the plurality of layers or blocks.

Example 12. The method of any one of examples 8 to 11 , wherein the at least one second machine-learning logic is configured to provide regions of interest based on the virtually stained image, wherein the method further comprises providing the at least one score based on the regions of interest.

Example 13. The method of example 12, wherein the machine-learning logic is further configured to provide an image with the regions of interest marked therein based on the virtually stained image.

Example 14. A device including at least one machine-learning logic, wherein the device is configured to perform the method of any one of examples 1 to 13.

Example 15. A method of training at least one machine-learning logic, the method comprising: providing imaging data of tissue samples, providing reference scores for the imaging data, and training the at least one machine-learning logic based on the imaging data and the reference scores. Example 16. The method of example 15, wherein the imaging data is imaging data of an unlabeled tissue sample or of a tissue sample provided with a simple stain.

Example 17. The method of example 15 or 16, further comprising providing reference stained images associated with the imaging data, wherein the training is further based on the stained images.

Example 18. The method of example 17, further comprising providing marked reference images with an indication of regions of interest associated with the stained images, wherein the training is further based on the indications of the regions of interest.

Example 19. The method of any one of examples 15 to 18, wherein the method is adapted for training the at least one machine-learning logic of the device of example 12.

Example 20. A device including at least one machine learning logic, wherein the device is configured to perform the method of any one of examples 15 to 19.

Example 21. A computer program, comprising a program code, which, when executed on one or more processors, causes execution of the method of any one of examples 1 to 13 or 15 to 19.

Example 22. A tangible storage medium storing the computer program of example 21.

Example 23. A data carrier signal carrying the program of example 21.

Example 24. A device of obtaining a score indicating a disease from a tissue sample comprising: an input for receiving imaging data of the tissue sample, at least one machine-learning logic for processing the imaging data, the at least one machine-learning logic being configured to output at least one score based on the imaging data provided. Example 25. The device of example 24, wherein the imaging data is imaging data of an unlabeled tissue sample or of a tissue sample provided with a simple stain.

Example 26. The device of example 24 or 25, wherein the at least one machine learning logic is configured to provide at least two scores of different scoring systems based on the imaging data.

Example 27. The device of example 26, wherein the at least one machine-learning logic comprises separate machine-learning logics for at least some of the at least two scores.

Example 28. The device of example 26 or 27, wherein the at least one machine learning logic comprises a common machine-learning logic having different output layers or blocks for at least some of the at least two scores.

Example 29. The device of any one of examples 24 to 28, further being configured to tile the imaging data, and wherein processing the imaging data comprises processing tiled imaging data, wherein the method further comprises obtaining the at least one score based on the processed tiled imaging data.

Example 30. The device of any one of examples 24 to 29, wherein the at least one machine-learning logic is configured to obtain the at least one score based on the imaging data in an end-to-end approach without intermediate images.

Example 31. The device of any one of examples 24 to 30, wherein the at least one machine-learning logic comprises a first machine-learning logic configured to generate a virtually stained image based on the imaging data, and at least one second machine learning logic configured to provide the at least one score.

Example 32. The device of example 31 , wherein the at least one second machine learning logic is configured to provide the score based on the virtually stained image.

Example 33. The device of example 31 , wherein the first machine-learning logic comprises a plurality of layers or blocks, and wherein the at least one second machine- learning logic is configured to provide the at least one score based on a representation of the imaging data in one of the plurality of layers or blocks.

Example 34. The device of example 33, wherein the one layer or block of the plurality of layers or blocks is a bottleneck where the representation has a maximum spatial contraction among the plurality of layers or blocks.

Example 35. The device of any one of examples 31 to 34, wherein the at least one second machine-learning logic is configured to provide regions of interest based on the virtually stained image and comprises a post-processing portion configured to provide the at least one score based on the regions of interest.

Example 36. The device of example 35, wherein the at least one second machine learning logic is further configured to provide an image with the regions of interest marked based on the virtually stained image.

Example 37. A device of training at least one machine-learning logic, the device comprising: an input for receiving imaging data of tissue samples and reference scores for the imaging data, the device being configured for training the at least one machine-learning logic based on the imaging data and the reference scores.

Example 38. The device of example 37, wherein the imaging data is imaging data of an unlabeled tissue sample or of a tissue sample provided with a simple stain.

Example 39. The device of example 37 or 38, the input being further configured to receive reference stained images associated with the imaging data, wherein the training is further based on the stained images.

Example 40. The device of example 39, the input being configured to receive marked reference images with indications of regions of interest associated with the stained images, wherein the training is further based on the indications of the regions of interest. As can be seen, various possibilities exist for virtual scoring, and therefore the embodiments above are not be construed as limiting in any way.

Claims

1. A method of obtaining a score (1504A, 1504B; 2113) indicating a disease from a tissue sample comprising: obtaining imaging data (1501; 2109) of the tissue sample, processing the imaging data (1501; 2109) in at least one machine-learning logic (1502A, 1502B; 1601; 1701, 1703; 1804; 1902), the at least one machine-learning logic (1502A, 1502B; 1601; 1701, 1703; 1804; 1902) being configured to output at least one score (1504A, 1504B; 2113) based on the imaging data (1501; 2109) provided.

2. The method of claim 1 , wherein the tissue sample includes an in vivo tissue.

3. The method of claim 1 , wherein the tissue sample includes a cell culture.

4. The method of claim 1 , wherein the tissue sample includes a histopathological tissue sample.

5. The method of any one of claims 1 to 4, wherein the imaging data is one of phase contrast imaging data of the tissue sample or autofluorescence imaging data of the tissue sample without fluorescence markers.

6. The method of any one of claims 1 to 4, wherein the imaging data (1501; 2109) is imaging data of an unlabeled tissue sample (2105) or of a tissue sample(2106) provided with a simple stain.

7. The method of any one of claims 1 to 6, wherein the at least one score includes one or more scores selected from the group consisting of: a number of predefined objects in the tissue sample, an area of predefined objects in the tissue sample, area ratios between predefined objects in the tissue sample, mean sizes or a median of sizes of predefined objects in the tissue sample, mean distances between predefined objects in the tissue sample, and mean intensities of predefined objects in the tissue sample when applying a staining or fluorescence marker.

8. The method of claim 7, wherein at least some of the predefined objects are selected from the group consisting of: cells stained when applying a predefined stain to the tissue sample, cells not stained when applying the predefined stain to the tissue sample tumor cells, non-tumor cells, cells, or cell components.

9. The method of any one of claims 1 to 8, wherein the at least one score includes a histopathological score.

10. The method of any one of claims 1 to 9, wherein the at least one machine learning logic (1502 A, 1502B; 1601; 1701, 1703; 1804; 1902) is configured to provide at least two scores (1504A, 1504B; 2113) of different scoring systems based on the imaging data (1501; 2109).

11. The method of claim 10, wherein the at least one machine-learning logic (1502A, 1502B; 1601; 1701, 1703; 1804; 1902) comprises separate machine-learning logics (1502A, 1502B) for at least some of the at least two scores (1504A, 1504B; 2113).

12. The method of claim 10 or 11, wherein the at least one machine-learning logic (1502A, 1502B; 1601; 1701, 1703; 1804; 1902) comprises a common machine-learning logic (1601) having different output layers (1603A, 1603B) or blocks for at least some of the at least two scores (1504A, 1504B; 2113).

13. The method of any one of claims 1 to 12, wherein obtaining the imaging data (1501; 2109) comprises tiling the imaging data (1501; 2109), and wherein processing the imaging data (1501; 2109) comprises processing the tiled imaging data, wherein the method further comprises obtaining the at least one score (1504A, 1504B; 2113) based on the processed tiled imaging data.

14. The method of any one of claims 1 to 13, wherein the at least one machine learning logic (1502 A, 1502B; 1601; 1701, 1703; 1804; 1902) is configured to obtain the at least one score (1504A, 1504B; 2113) based on the imaging data (1501; 2109) in an end-to-end approach without intermediate images.

15. The method of any one of claims 1 to 14, wherein the at least one machine learning logic (1502A, 1502B; 1601; 1701, 1703; 1804; 1902) comprises a first machine-learning logic (1701) configured to generate a virtually stained image based on the imaging data (1501; 2109), and at least one second machine-learning logic (1703; 1804; 1902) configured to provide the at least one score (1504A, 1504B; 2113).

16. The method of claim 15, wherein the at least one second machine-learning logic (1703; 1804; 1902) is configured to provide the score (1504A, 1504B; 2113) based on the virtually stained image (1702).

17. The method of claim 15, wherein the first machine-learning logic (1701) comprises a plurality of layers (1704_1-1704_7) or blocks, and wherein the at least one second machine-learning logic (1902) is configured to provide the at least one score (1504A, 1504B; 2113) based on a representation of the imaging data (1501; 2109) in one of the plurality of layers (1704_1-1704_7) or blocks.

18. The method of claim 17, wherein the one layer or block of the plurality of layers (1704_1-1704_7) or blocks is a bottleneck (1704_4) where the representation has a maximum spatial contraction among the plurality of layers (1704_1-1704_7) or blocks.

19. The method of any one of claims 15 to 18, wherein the at least one second machine-learning logic (1804) is configured to provide regions of interest based on the virtually stained image, (1702), wherein the method further comprises providing the at least one score based on the regions of interest.

20. The method of claim 19, wherein the machine-learning logic (1804) is further configured to provide an image (1802) with the regions of interest (1803) marked therein based on the virtually stained image (1702).(1504A, 1504B; 2113).

21. A device (2110) including at least one machine-learning logic (1502A, 1502B; 1601; 1701, 1703; 1804; 1902), wherein the device is configured to perform the method of any one of claims 1 to 20.

22. A method of training at least one machine-learning logic (1502A, 1502B; 1601; 1701, 1703; 1804; 1902), the method comprising: providing imaging data (1501; 2109) of tissue samples, providing reference scores (1504A, 1504B; 2113) for the imaging data (1501; 2109), and training the at least one machine-learning logic (1502A, 1502B; 1601; 1701, 1703;

1804; 1902) based on the imaging data (1501; 2109) and the reference scores (1504A, 1504B; 2113).

23. The method of claim 22, wherein the imaging data is imaging data of an unlabeled tissue sample or of a tissue sample provided with a simple stain.

24. The method of claim 23, wherein the reference scores are provided based on reference imaging data corresponding to the imaging data, but for a real or virtually stained tissue sample.

25. The method of claim 22, wherein the imaging data is imaging data of a real or virtually stained tissue sample.

26. The method of any one of claims 22 to 25, further comprising providing reference stained images associated with the imaging data (1501; 2109), wherein the training is further based on the reference stained images.

27. The method of claim 26, further comprising providing marked reference images with an indication of regions of interest associated with the stained images, wherein the training is further based on the indications of the regions of interest.

28. The method of any one of claims 22 to 27, wherein the method is adapted for training the at least one machine-learning logic (1502A, 1502B; 1601; 1701, 1703;

1804; 1902) of the device of claim 21.

29. The device of claim 21 , wherein the machine learning logic of the device is trained with the method of any one of claims 22 to 28.

30. The method of any one of claims 1 to 20, wherein the machine learning logic is trained with the method of any one of claims 22 to 28.

31. A device including at least one machine learning logic (1502A, 1502B; 1601; 1701, 1703; 1804; 1902), wherein the device is configured to perform the method of any one of claims 22 to 30.

32. A computer program, comprising a program code, which, when executed on one or more processors, causes execution of the method of any one of claims 1 to 20 or 22 to 30.

33. A tangible storage medium storing the computer program of claim 32.

34. A data carrier signal carrying the program of claim 32.