WO2021198247A1

WO2021198247A1 - Optimal co-design of hardware and software for virtual staining of unlabeled tissue

Info

Publication number: WO2021198247A1
Application number: PCT/EP2021/058277
Authority: WO
Inventors: Alexander Freytag; Christian KUNGEL
Original assignee: Carl Zeiss Ag
Priority date: 2020-03-30
Filing date: 2021-03-30
Publication date: 2021-10-07

Abstract

It is proposed a method for evaluating image modalities for virtual staining of a tissue sample comprising at least one iteration of obtaining multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of imaging data has been acquired using a different image modality of a group of image modalities, obtaining multiple reference images depicting the one or more tissue samples comprising one or more chemical stains, processing the multiple sets of training imaging data in a machine-learning logic, obtaining, from the machine-learning logic and for each one of the one or more training imaging data, training output images comprising one or more virtual stains corresponding to the one or more chemical stains, performing the training of the machine-learning logic by updating parameter values of the machine-learning logic based on a comparison between such reference images and training output images that are associated with corresponding chemical stains and virtual stains, determining a virtual staining accuracy of the trained machine-learning logic for each one of the one or more virtual stains. The evaluating is based on the one or more virtual staining accuracies. Moreover, it is proposed a message for virtual staining and a device for performing at least one of the methods.

Description

Optimal co-design of hardware and software for virtual staining of unlabeled tissue

TECHNICAL FIELD

The present invention relates to a method for virtually staining a tissue sample and a device for tissue analysis.

BACKGROUND

Histopathology is an important tool in the diagnosis of a disease. In particular, histopathology refers to the optical examination of tissue samples.

Typically, histopathological examination starts with surgery, biopsy, or autopsy for obtaining the tissue to be examined. The tissue may be processed to remove water and to prevent decay. The processed sample may then be embedded in a wax block. From the wax block, thin sections may be cut. Said thin sections may be referred to as tissue samples hereinafter.

The tissue samples may be analysed by a histopathologist in a microscope. Heretofore, the tissue samples may be stained with a chemical stain to facilitate the analysis of the tissue sample. In particular, chemical stains may reveal cellular components, which are very difficult to observe in the unstained tissue sample. Moreover, chemical stains may provide contrast.

The most commonly used stain in histopathology is a combination of haematoxylin and eosin (abbreviated H&E). Haematoxylin is used to stain nuclei blue, while eosin stains cytoplasm and the extracellular connective tissue matrix pink. There are hundreds of various other techniques which have been used to selectively stain cells. Recently, antibodies have been used to stain particular proteins, lipids and carbohydrates. Called immunohistochemistry, this technique has greatly increased the ability to specifically identify categories of cells under a microscope. Staining with an H&E stain may be considered as common gold standard for histopathologic diagnosis.

By colouring tissue samples with chemical stains, otherwise almost transparent and indistinguishable sections of the tissue samples become visible for the human eye. This allows pathologists and researchers to investigate the tissue sample under a microscope or with a digital bright-field equivalent image and assess the tissue morphology (structure) or to look for the presence or prevalence of specific cell types, structures or even microorganisms such as bacteria.

Preferably, several chemical stains are used to fully assess the pathology case. Typically, only one chemical stain can be applied to a tissue sample. Thus, if several chemical stains are required for diagnosis, several tissue samples each corresponding to a different section or slice of the wax block have to be prepared. Moreover, different chemical stains may require different staining protocols. Thus, the known chemical staining techniques are labour- and cost-intensive.

WO 2019/154987 A1 discloses a method providing a virtually stained image looking like a typical image of a tissue sample which has been stained with a conventional chemical stain.

Providing a virtually stained image of a tissue sample requires time for acquiring the required digital imaging data and for the calculation of the output image comprising the virtual stain. Virtual staining of tissue samples requires the provision of digital imaging data relating to the to be virtually stained tissue samples. Digital imaging data of tissue samples may be acquired using different image modalities. For some image modalities, the acquisition takes substantial time. Image modalities may generate a large amount of digital imaging data. Other image modalities may require complex and expensive hardware.

SUMMARY

There may be a need for optimizing hardware and software for virtual staining of tissue samples.

Said need may be addressed with the subject matter of the independent claims. Advantageous embodiments are described in the dependent claims.

Tissue samples may relate to thin sections of the wax block comprising an embedded processed sample as described hereinbefore. However, the term tissue sample may also refer to tissue having been processed differently or not having been processed at all. For example, tissue sample may refer to a part of tissue observed in vivo and/or tissue excised from a human, an animal or a plant, wherein the observed tissue sample has been further processed ex vivo, e.g., prepared using a frozen section method. A tissue sample may be any kind of a biological sample. The term tissue sample may also refer to a cell, which cell can be of procaryotic or eucaryotic origin, a plurality of procaryotic and/or eucaryotic cells such as an array of single cells, a plurality of adjacent cells such as a cell colony or a cell culture, a complex sample such as a biofilm or a microbiome that contains a mixture of different procaryotic and/or eucaryotic cell species and/or an organoid.

It is proposed a method for evaluating image modalities for virtual staining of a tissue sample comprising at least one iteration of obtaining multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of digital imaging data has been acquired using a different image modality of a group of image modalities, obtaining multiple reference images depicting the one or more tissue samples comprising one or more chemical stains, processing the multiple sets of training imaging data in a machine-learning logic, obtaining, from the machine-learning logic and for each one of the one or more training imaging data, training output images comprising one or more virtual stains corresponding to the one or more chemical stains, performing the training of the machine-learning logic by updating parameter values of the machine-learning logic based on a comparison between such reference images and training output images that are associated with corresponding chemical stains and virtual stains. Moreover, the method includes determining virtual staining accuracy of the trained machine-learning logic for each one of the one or more virtual stains. The virtual staining accuracies may be used for evaluating the image modalities for virtual staining. Virtual staining accuracies may be determined using at least one loss function has explained further below.

The term chemical staining may also comprise modifying molecules of any one of the different types of tissue sample mentioned above. The modification may lead to fluorescence under a certain illumination (e.g., an illumination under ultra-violet (UV) light). For example, chemical staining may include modifying genetic material of the tissue sample. Chemically stained tissue samples may comprise transfected cells. Transfection may refer to a process of deliberately introducing naked or purified nucleic acids into eukaryotic cells. It may also refer to other methods and cell types. It may also refer to non-viral DNA transfer in bacteria and non-animal eukaryotic cells, including plant cells.

Modifying genetic material of the tissue sample may make the genetic material observable using a certain image modality. For example, the genetic material may be rendered fluorescent. In some examples, modifying genetic material of the tissue sample may cause the tissue sample to produce molecules being observable using a certain image modality. For example, modifying genetic material of the tissue sample may induce the production of fluorescent proteins by the tissue sample. Determining a virtual staining accuracy of the trained machine-learning logic for each of the one or more virtual stains may allow for optimizing the group of image modalities for obtaining specific virtual stains.

Depending on the image modality, the dimensionality of the digital imaging data of the tissue sample may vary. The digital imaging data may be two-dimensional (2-D), one-dimensional (1- D) or even three-dimensional (3-D). If more than one image modality is used for obtaining digital imaging data, a part of the digital imaging data may be two-dimensional and another of the digital imaging data may be one-dimensional or three-dimensional. For instance, microscopy imaging may provide digital imaging data that includes images having spatial resolution, i.e., including multiple pixels. Scanning through the tissue sample with a confocal microscope may provide digital imaging data comprising three-dimensional voxels. Spectroscopy of the tissue sample may result in digital imaging data providing spectral information of the whole tissue sample without spatial resolution. In another embodiment, spectroscopy of the tissue sample may result in digital imaging data providing spectral information for several positions of the tissue sample which results in imaging data comprising spatial resolution but being sparsely sampled.

In an embodiment, the at least one iteration further comprises, depending on the virtual staining accuracy for each of the one or more virtual stains and adding an image modality of the group of image modalities and initiating a new iteration.

In embodiments, the image modalities for acquiring digital imaging data of tissue samples comprise images of the tissue samples in a specific spectral band. For example, a first image modality may refer to an image of the tissue samples in a spectral band corresponding to our red color in the visible spectrum and a different image modality for acquiring digital imaging data may refer to an image of the tissue sample in a different spectral band, for example a spectral band corresponding to green color in the visible spectrum. A hyperspectral scanner may be used for acquiring images of the tissue samples in one or more spectral bands. However, the spectral bands are not limited to spectral bands in the visible spectrum but may also comprise spectral bands in the ultraviolet, and infrared range. The image modalities for acquiring digital imaging data of tissue samples may also comprise a Raman analysis of the tissue samples. In particular, the imaging modalities may comprise some simulated Raman scattering (SRS) analysis of the tissue samples, coherent anti-stokes Raman scattering (CARS) analysis of the tissue samples, surface enhanced Raman scattering (SERS) analysis of the tissue samples. In further embodiments, the image modalities may also comprise fluorescence lifetime imaging microscopy (FLIM) analysis of the tissue samples. The image modalities may also comprise a phase sensitive analysis of the tissue samples. Yet a further example would be transmitted- light or reflected-light microscopy, e.g., for observing cells. Imaging modalities may, as a general rule, imaging tissue in-vivo or ex-vivo. An endoscope may be used to acquire images in-vivo, e.g., a confocal microscope or using endoscopic optical coherence tomography (e.g., scanned or full-field). A confocal fluorescence scanner could be used. Endoscopic two-photon microscopy would be a further imaging modality. A surgical microscope may be used; the surgical microscope may, itself provide for multiple imaging modalities, e.g., microscopic images or fluorescence images, e.g., in specific spectral bands or combinations of two or more wavelengths, or even hyperspectral images. Further, the image modalities may comprise poorer realization sensitive analysis of the tissue samples. It is also thinkable that the image modalities comprise a DCI technology analysis of the tissue sample. Dynamic cell imaging (DCI) may refer to measuring cell metabolism as phase changes with a phase sensitive full field optical coherence tomography setup. DCI technology analysis may be provided by LLTech Inc. (http://lltech.co/the-biopsy- scanner/our-technology).Thus, several different image modalities may be used for optimizing the virtual staining of tissue samples.

In the embodiment, the method further comprises stopping the iterations depending on a comparison of the virtual staining accuracy of the trained machine-learning logic with the staining accuracy of the trained machine-learning logic of the previous iteration.

For example, image modalities may be continuously added to the group of image modalities until the virtual staining accuracy between two iterations does not improve any more with another image modality. Vice versa, image modalities may be removed from the group of image modalities as long as the virtual staining accuracy does not substantially decrease from one iteration to the next iteration. This approach may reduce the number of image modalities within the group of image modalities and thus, the generation of large amounts of data which provide little relevant additional information. It may also be possible to exchange an image modality from the group of image modalities with another image modality. For example, an image modality of the group of image modalities corresponding to a first spectral band may be exchanged with an image modality corresponding to a second spectral band. The first spectral band and the second spectral band may be separate, partially overlapping or completely overlapping. For example, the first spectral band may be comprised within the second spectral band or the second spectral band may be comprised within the first spectral band. Changing one of the limits of a spectral band may be considered as exchanging a first spectral band with a second spectral band.

Evaluating image modalities may further comprise, for N existing image modalities, training N different machine-learning logics, wherein each machine-learning logic is trained with a different group of image modalities and each group of image modalities comprises all but one of N image modalities. Thereafter, the image modality for which the lowest virtual staining accuracy has been determined may be removed. Afterwards, training and removing may be repeated as long as an acceptable virtual staining accuracy is obtained for at least one machine learning logic.

Another embodiment of the method for evaluating image modalities for virtual staining of a tissue may involve associating a weight to every one of the N image modalities of the group of image modalities. Training the machine-learning logic by adding a regularization penalty term of the associated weights. After training, pick a predetermined number of K<N image modalities to be used for training the machine-learning logic for obtaining the virtual stains.

According to an embodiment, the method may further comprise training an inverse machine learning logic. In particular, an inverse machine-learning logic may receive virtually or chemical stained output images and transmit digital imaging data which leads to said output images. The machine-learning logic may also be called virtual stainer and the inverse machine-learning logic may be called virtual destainer.

In an embodiment, the machine-learning logic and the inverse machine-learning logic are implemented using a single invertible neural network. However, it may also be possible to use an inverse machine-learning logic separately from the machine-learning logic. In some embodiments, it may also be possible to couple the machine-learning logic and the inverse machine-learning logic during training to preserve cycle consistency. One possibility to train the machine-learning logic and the inverse machine-learning logic is the cycle consistency approach as used by CycleGANs.

According to an embodiment determining a virtual staining accuracy comprises processing multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of digital imaging data has been acquired using a different image modality of the group of image modalities in a trained machine-learning logic, thereafter, from the trained machine-learning logic output images, for each one of the one or more training imaging data, at least one training output image may be obtained. The training output image may be processed in the trained inverse machine-learning logic. From the trained inverse machine-learning logic output imaging data relating to the one or more tissue samples may be obtained. By comparing, for each image modality of the group of image modalities, the output imaging data with the training imaging data the virtual staining accuracy may be determined.

In an embodiment, removing an image modality of the group of image modalities comprises removing an image modality, for which image modality the difference between the output imaging data and the training imaging data is below a predetermined threshold.

If the difference between the output imaging data and the training imaging data is below a predetermined threshold, it may be assumed that the respective image modality provides only little additional information for virtually staining of the tissue sample.

In another embodiment, the method comprises selecting one or more virtual stains, obtaining multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of digital imaging data has been acquired using a different group of image modalities. Further, the method comprises processing the training imaging data in a trained machine-learning logic, obtaining, from the trained machine-learning logic, output images relating to the one or more tissue samples and comprising the one or more virtual stains, and determining a virtual staining accuracy for each pair of virtual stain and group of image modalities. Further, the method comprises processing the one or more virtual stains and the one or more groups of image modalities in a hardware optimizer machine-learning logic. From the hardware optimizer machine-learning logic a training virtual staining accuracy is obtained for each pair of virtual stains and groups of image modalities. Training of the hardware optimizer machine-learning logic may be performed by updating parameter values of the hardware optimizer machine-learning logic based on a comparison between virtual staining accuracies and training virtual staining accuracies.

The hardware optimizer learning logic may be based on meta-learning, in particular automated machine-learning, AutoML.

As a general rule, meta-learning can be based on at least one of grid search, random search, Bayesian optimization, gradient-free optimization, gradient-based optimization, higher-order optimization, evolutionary optimization, or combinations thereof. For example, meta-learning may involve using an algorithm such as Spearmint which is based on Gaussian process regression. Embodiments may use an algorithm such as SMAC (Sequential Model-based Algorithm Configuration) which is based on random forest regression. Further embodiments may prescribe using a hyperband algorithm which is based on random sampling. Additional embodiments may prescribe using an algorithm such as BOHB (Bayesian optimization (BO) and Hyperband (HB)). Additional embodiments may also prescribe using an algorithm such as RoBO (Robust Bayesian Optimization (BO)).

It is proposed a method for virtually staining a tissue sample comprising selecting one or more virtual stains, selecting a virtual staining accuracy, requiring digital imaging data relating to the tissue sample using a group of image modalities processing the digital imaging data in a trained machine-learning logic, and obtaining, from the machine-learning logic, an output image depicting the tissue sample comprising the one or more virtual stains. The method may be characterised in that the group of image modalities has been selected using a method according to one of the embodiments described above. Thus, the proposed method may use only those image modalities which provide sufficient benefit for obtaining the virtual stains.

In an embodiment, evaluating image modalities may comprise taking an acquisition time and/or acquisition complexity for acquiring digital imaging data for at least one image modality of the group of image modalities into account. If more image modalities, e.g. two spectral channels, would lead to the same virtual staining accuracy than a lesser number of image modalities,

.e.g., only one integral Raman spectrum recording, the former may nevertheless be preferred because the latter may substantially increase the acquisition time or acquisition complexity, e.g., require a more complex acquisition system. In embodiments, the hard ware machine-learning logic may be trained taking into account the acquisition time and/or acquisition complexity corresponding to the respective image modalities.

In particular, the device for tissue analysis may comprise a processor configured to perform a method as described above. Thus, the device for tissue analysis may use only digital imaging data of tissue samples with required image modalities.

In an embodiment, the device for tissue analysis comprises an image acquisition system configured for acquiring the digital imaging data of a tissue sample using one or more image modalities. In particular, the device for tissue analysis may be adapted to specifically only required image modalities. In other examples, the device for tissue analysis may comprise an image acquisition system allowing for acquiring digital imaging data with a large number of image modalities and be controlled to acquire digital imaging data only for the required image modalities.

Furthermore, it is proposed a computer program comprising instructions, which, when the program is executed by a computer, cause a computer to carry out any methods described herein before.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described with respect to the drawings. In the drawings Fig. 1 shows a workflow for staining a tissue sample;

Fig. 2 illustrates a method for evaluating image modalities for virtual staining; and Fig. 3 illustrates a method for virtually staining a tissue sample.

Fig. 4 is a flowchart of a method according to various examples, the method enabling inference of one or more output images depicting a tissue sample including one or more virtual stains;

Fig. 5 schematically illustrates a tissue sample and multiple sets of imaging data depicting the tissue sample according to various examples;

Fig. 6 schematically illustrates a machine-learning logic according to various examples;

Fig. 7 schematically illustrates an example implementation of the machine-learning logic according to various examples;

Fig. 8 schematically illustrates an example implementation of the machine-learning logic according to various examples;

Fig. 9 schematically illustrates an example implementation of the machine-learning logic according to various examples;

Fig. 10 is a flowchart of a method according to various examples, the method enabling training of a machine-learning logic for virtual staining according to various examples;

Fig. 11 schematically illustrates aspects with respect to the training of the machine-learning logic according to various examples; and

Fig. 12 schematically illustrates a method enabling training of and using a virtual staining logic.

DETAILED DESCRIPTION

FIG. 1 illustrates aspects with respect to a workflow for generating images depicting a tissue sample including a stain, e.g., a chemical stain or a virtual stain. FIG. 1 schematically illustrates an example of a histopathology workflow. As explained above, virtual staining can also be applied in other use cases than histopathology. Then, different workflows for generating images can be applicable. For instance, for fluorescence imaging of cells, tissue samples including cell samples may be otherwise acquired and imaged in a respective microscope. Also, in-vivo imaging using an endoscope would be a possible use case for generating imaging data of tissue samples.

As shown in Fig. 1, for histopathology, tissue 2102 may be obtained from a living creature 2101 by surgery, biopsy or autopsy. After some processing steps to remove water and to prevent decay, said tissue 2102 may be embedded in a wax block 2103. From said block 2103, a plurality of slices 2104 may be obtained for further analysis. One slice of said plurality of slices 2104 may also be called a tissue sample 2005.

As mentioned before, the tissue could also include cell samples or in-vivo inspection using, e.g., a surgical microscope or an endoscope.

Before analysing the tissue sample 2005, a chemical stain may be applied to the tissue sample 2005 to obtain a chemically stained tissue sample 2006. Said chemical stain may be an H&E stain. In some embodiments, the tissue sample 2005 may also be directly analysed. A chemically stained tissue sample 2006 may facilitate the analysis. In particular, chemical stains may reveal cellular components, which are very difficult to observe in the unstained tissue sample 2005. Moreover, chemical stains may provide an increased contrast.

Applying a chemical stain may include a-priori transfecting or direct application of a fluorophore such as 5-ALA.

Traditionally, the tissue sample 2005 or 2006 is analysed by an expert using a bright field microscope 2107.

Meanwhile, it has become more common to use image acquisition systems 2108 configured for acquiring digital imaging data of the tissue sample 2105 or the chemically stained tissue sample 2106 using one or more image modalities. Image modalities may comprise images of the tissue sample in one or more specific spectral bands, in particular, spectral bands in the ultra violet, visible and/or infrared range. Image modalities may also comprise a Raman analysis of the tissue samples, in particular a stimulated Raman scattering (SRS) analysis of the tissue sample, a coherent anti-Stokes Raman scattering, CARS, analysis of the tissue sample, a surface enhanced Raman scattering, SERS, analysis of the tissue sample. Further, the image modalities may comprise a fluorescence analysis of the tissue sample, in particular, fluorescence lifetime imaging microscopy. FLIM, analysis of the tissue sample. The image modality may prescribe a phase sensitive acquisition of the digital imaging data. The image modality may also prescribe a polarization sensitive acquisition of the digital imaging data.

The digital imaging data 2109 may be processed in a device for tissue analysis 2110. The device for tissue analysis 2110 may be a computer. The device for tissue analysis 2110 may comprise memory 2111 for (temporarily) storing the digital imaging data 2109 and a processor 2112 for processing the digital imaging data 2109. The device for tissue analysis 2110 may process the digital imaging data 2109 to provide one or more output pictures 2113 which may be displayed on a display 2114 to be analysed by an examiner. The device for tissue analysis 2110 may comprise different types of trained or untrained machine-learning logic for analysing the tissue sample 2105 or the chemically stained tissue sample 2106. The image acquisition system 2108 may be used for providing training data for said machine-learning logic. The output pictures 2113 may depict the tissue sample 2105 with one or more virtual stains.

Fig. 2 illustrates a method for evaluating image modalities for virtual staining of a tissue sample 2620. An image acquisition system 2610 is provided which is configured for acquiring digital imaging data of tissue samples 2620 using a plurality of image modalities. As an explanatory example, the image acquisition system 2610 may comprise three image sensor 2611, 2612 and 2613. The image sensor 2611 may acquire digital imaging data 2631 of tissue samples 2620 in the infrared spectral range, the image sensor 2612 may acquire digital imaging data 2632 of the tissue samples 2620 in the visible spectral range, and the image sensor 2613 may acquire digital imaging data 2633 of the tissue samples 2620 in the ultra-violet spectral range. The different spectral ranges merely serve as an example of different image modalities. Many further and also substantially different other image modalities may be considered providing digital imaging data relating to the tissue samples 2620.

A tissue analyzer 2640 may receive the digital imaging data 2631, 2632, 2633. The digital imaging data 2631, 2632, 2633 may be considered as multiple sets of training imaging data relating to one or more tissue, wherein each set of the multiple sets of training imaging data having been acquired using different image modalities of a group of image modalities. Further, the tissue analyzer 2640 receives multiple reference images 2650 depicting the one or more tissue samples comprising one or more stains. The multiple sets of training imaging data may be processed in a machine-learning logic 2641 of the tissue analyzer 2640 and from the machine-learning logic 2641 for each one of the one or more training imaging data training output images comprising one or more virtual stains corresponding to the one or more chemical stains may be obtained. Then, training of the machine-learning logic 2641 may be performed by updating parameter values of the machine learning logic 2641 based on a comparison of the reference images 2650 and the training output images. Training a machine-learning logic 2641 may further be performed using a method explained further below. In addition to training the machine-learning logic 2641 to be able to provide virtually stained output images based on digital imaging data of unstained tissue samples, the tissue analyzer 2640 may provide for each virtual stain 2661 a virtual staining accuracy 2662. The pair 2660 of the virtual stain 2661 and its respective virtual staining accuracy 2662 may depend on the image modalities 2621, 2622, 2623. Using a different group of image modalities for acquiring the set of digital imaging data may result in different virtual staining accuracies. Thus, providing feedback loop for the obtained pair(s) 2660 of virtual stain 2661 and corresponding virtual staining accuracy 2662 to the tissue analyzer 2640 may allow the tissue analyzer to determine optimal groups of image modalities for a specific virtual stain if a particular virtual staining accuracy is to be achieved.

Fig. 3 illustrates a method for virtually staining a tissue sample. A system 2770 comprising a trained hardware optimizer machine-learning logic may determine which image modality or modalities to use for acquiring the digital imaging data for virtually staining the tissue sample. In the example of an image acquisition system 2710 offering having a detector 2711 for light in the infrared spectral range, a detector 2712 for light in the visible spectral range and a detector 2713 for light in the ultra-violet spectral range, the system 2770 may determine that it is sufficient to acquire digital imaging data 2780 with the detector 2712 for processing in a trained machine-learning logic 2740 to obtain an output picture 2790 of the tissue sample with a virtual stain 2791. In other embodiments, a specific image acquiring system 2710 which is only configured for providing the digital imaging data with the required image modalities for the specific virtual stain / virtual staining accuracy may be provided. Such an image acquiring system 2710 may be less complex and less error prone, which may improve usability of such an image acquiring system 2710 for high throughput of tissue samples.

Fig. 12 illustrates a further method for virtually staining a tissue sample. Tissue samples may be prepared for training a machine learning logic. For example, tissue samples may be excised from a human, an animal or a plant and placed on a sample holder, e.g. a petri dish (step 21111). The tissue sample may refer to a cell, a plurality of cells, a plurality of adjacent cells and/or an organoid. Afterwards, the tissue sample may be chemically stained (step 21121). For example, a chemical stain may be added to the tissue sample. The chemical stain may be only be observable under a certain illumination. In particular, the chemical stain may be fluorescent under illumination with ultra-violet light.

In the alternative, cells of a tissue sample may be transfected (step 21122), i.e. the tissue sample may be chemically stained by transfection. For example, genetic material of the cells may be modified to cause the cell to produce green fluorescent protein (GFP). The tissue sample chemically stained by transfection may then be placed on a sample holder (step 21121), e.g. a petri dish.

In both cases, imaging data of the chemically stained tissue samples may be obtained (step 21130). For this purpose, a microscope may be used. The microscope may be operated using a transmitted light technique. Different imaging modalities may be used for acquiring the imaging data of the chemically stained tissue samples. The chemical stain may only be observable in one or some of the imaging modalities. For example, alternatingly, imaging data using a fluorescence technique (sub-step 21131) and imaging data using a transmitted (visible) light technique (TL technique) (sub-step 21132) may be obtained. The chemical stain may only be observable using the fluorescence technique.

The imaging data obtained using the imaging modality rendering the chemical stain observable may be used as reference imaging data and the imaging data obtained using the imaging modality not showing the chemical stain may be used as training imaging data. For example, the fluorescence images may be used as reference images and the TL images may be used as training images.

In step 21140, the training imaging data and the reference imaging data may be used to train a virtual staining logic 21190. The trained machine learning logic 21190 may than be used to generate, based on imaging data obtained from an unstained tissue sample 21152, an output image 21151 depicting a tissue sample comprising a virtual stain.

The alternating acquisition of training imaging data and reference imaging data may facilitate training of the virtual staining logic. In particular, registration of the imaging data may be easier as the position of the tissue samples may be approximately the same when changing the imaging modalities.

Various techniques described herein generally relate to machine learning. Machine learning, especially deep learning, provides a data-driven strategy to solve problems. Classic inference techniques are able to extract patterns from data based on hand-designed features, to solve problems; an example technique would be regression. However, such classic inference techniques heavily depend on the accurate choice for the hand-designed features, which choice depends on the designer’s ability. One solution to such a problem is to utilize machine learning to discover not only the mapping from features to output, but also the features themselves. This is as training of a machine-learning logic.

Various techniques described herein generally relate to virtual staining of a tissue sample by utilizing a trained machine-learning logic (MLL). The MLL can be implemented, e.g., by a support vector machine or a deep neural network which includes at least one encoder branch and at least one decoder branch.

More specifically, according to various examples, multiple sets of imaging data can be fused and processed by the MLL. This is referred to as a multi-input scenario.

Alternatively or additionally to such a multi-input scenarios, multiple virtually stained images can be obtained (labeled output images hereinafter), from the trained MLL; the multiple virtually stained images can depict the tissue sample including different virtual stains. This is referred to as a multi-output scenario.

As a general rule, examples as summarized in TAB. 1 below can be implemented.

TAB. 1: Various scenarios for input and output of the MLL

For example, the MLL can generate virtual H&E (Hematoxylin and Eosin) stained images of the tissue sample, and/or virtually stained images of the tissue sample highlighting HER2 (human epidermal growth factor receptor 2) proteins and/or ERBB2 (Erb-B2 Receptor Tyrosine Kinase 2) genes.

Another example would pertain to virtual fluorescence staining. For example, in life-science applications, images of cells - e.g., arranged as living or fixated cells in a multi-well plate or another suitable container - may be acquired using transmitted-light microscopy. Also, a reflected light microscope may be used, e.g., in an endoscope or as a surgical microscope. It is then possible to selectively stain certain cell organelles, e.g., nucleus, ribosomes, the endoplasmic reticulum, the golgi apparatus, chloroplasts, or the mitochondria. A fluorophore (or fluorochrome, similarly to a chromophore) is a fluorescent chemical compound that can re-emit light upon light excitation. Fluorophores can be used to provide a fluorescence chemical stain. By using different fluorophores, different chemical stains can be achieved. For example, a Hoechst stain would be a fluorescent dye that can be used to stain DNA. Other fluorophores include 5-aminolevulinic acid (5-ALA), fluorescein, and Indocyanine green (ICG) that can even be used in-vivo. Fluorescence can be selectively excited by using light in respective wavelengths; the fluorophores then emit light at another wavelength. Respective fluorescence microscopes use respective light sources. It has been observed that illumination using light to excite fluorescence can harm the sample; this is avoided when providing virtual fluorescence staining. Virtual fluorescence staining may lead to fluorescence-like images through virtual staining. The virtual fluorescence staining mimics the fluorescence chemical staining, without exposing the tissue to respective excitation light.

The one or more output images depict the tissue sample including respective virtual stains, i.e. , the output images can have a similar appearance as respective images depicting the tissue sample including a corresponding chemical stain. Thus, the virtual stain can have a correspondence in a chemical stain of a tissue sample stained using a staining laboratory process.

By using a multi-input scenario, an increased accuracy for processing the imaging data in the MLL can be achieved. This is because by using multiple sets of imaging data that have been acquired using multiple imaging modalities, different biomarkers or biological structures can be highlighted in each one of the multiple sets. Imaging data can include 2-D images or 1-D or 3-D data.

By using multi-output scenario, a tailored virtual stain or a tailored set of multiple virtual stains can be provided such that a pathologist is enabled to provide an accurate analysis. For example, multiple output images depicting the tissue samples having multiple virtual stains may be helpful to provide a particular accurate diagnosis, e.g., based on multiple types of structures and multiple biomarkers being highlighted in the multiple output images, or multiple organelles of the cells being highlighted.

As a general rule, multi-input scenarios may or may not be combined with multi-output scenarios; and likewise, multi-output scenarios may or may not be combined with multi-input scenarios.

Fig. 4 is a flowchart of a method 3300 according to various examples. For example, the method 3300 according to Fig. 4 may be executed may be executed by at least one processor upon loading program code from a nonvolatile memory. The method facilitates virtual staining of a tissue sample.

Fig. 4 illustrates aspects with respect to virtual staining of a tissue sample. Fig. 4 illustrates aspects with respect to obtaining multiple sets of imaging data depicting the tissue sample by using multiple imaging modalities. Fig. 4 generally relates to multi-input scenario, as described above. Fig. 4 also illustrates aspects with respect to fusing and processing the multiple sets of imaging data in an MLL, and then obtaining (outputting), from the MLL, at least one output image of which each one depicts the tissue sample including a respective virtual stain.

In detail, at block 3301, multiple sets of imaging data depicting a tissue sample are obtained and the multiple sets of imaging data are acquired using multiple imaging modalities.

This is explained in connection with Fig. 5. Fig. 5 depicts a tissue sample 3400 and four sets of imaging data of the tissue sample 3400 acquired using four imaging modalities, i.e. , imaging data set 3401, 3402, 3403 and 3404, respectively. Alternatively, the imaging data set 3401, 3402, 3403 and 3404 can be respectively acquired using the same imaging modality but with different imaging settings or parameters, such as different (low or high) magnification levels, etc.

Referring again to Fig. 4: Each imaging data set 3401-3404 can include multiple instances of imaging data, e.g., multiple images taken at different positions of the sample and/or at different times.

The tissue sample 3400 can be a cancer tissue sample removed from a patient, a tissue sample of other animals or plants.

The multiple imaging modalities can be selected from the group including: hyperspectral microscopy imaging, fluorescence imaging, auto-fluorescence imaging, lightsheet imaging, digital phase contrast; Raman spectroscopy, etc. Further imaging modalities have been discussed above.

Depending on the particular imaging modality, a spatial dimensionality of the imaging data of each set 3401-3404 may vary, e.g., 1-D or 2-D or even 3-D. For instance, microscopy imaging or fluorescence imaging may provide imaging data that include images having spatial resolution, i.e., including multiple pixels. Lightsheet imaging may provide 3-D voxels. On the other hand, where Raman spectroscopy is used, it would be possible that an integral signal not possessing spatial resolution is obtained as the respective set of imaging data (corresponding to a 1-D data point); however, also scanning Raman spectroscopy is known where some 2-D spatial resolution can be provided. For instance, digital phase contrast can be generated using multiple illumination directions and digital post-processing to combine images associated with each one of the multiple illumination directions. See, e.g., US 20170276 923 A1. As a general rule, different imaging modalities may, in some examples, rely on a similar physical observable, e.g., both may pertain to fluorescence imaging or microscopy, but using different acquisition parameters. Example acquisition parameters could include, e.g., illumination type (e.g., brightfield versus dark field microscopy), magnification level, resolution, refresh rate, etc.

Hyperspectral scans help to acquire the substructure of an individual cell to identify subtle changes (morphological change of membrane, change in size of cell components, ...). Adjacent z-slices of a tissue sample can be captured in hyperspectral scans, e.g., by scanning through the probe with a confocal microscope (e.g., a light-sheet microscope, LSM), focusing the light- sheet in LSMs to slightly different z-levels. It is possible to acquire adjacent cell information like what happens in widefield microscopy (integral acquisition). A further class of imaging modalities includes molecularly sensitive methods like Raman, coherent Raman (SRS, CARS), SERS, Fluorescence imaging, FLIM, IR-lmaging. This helps to acquire chemical / molecular information. Yet another technique is dynamic cell imaging to acquire cell metabolism information. Yet a further imaging modality includes phase or polarization sensitive imaging to acquire structural information through contrast changes.

The method 3300 optionally includes pre-processing after obtaining the multiple sets of imaging data, such as one or a combination of the following processing techniques: noise filtering; registration between imaging data of different sets of imaging data, for example, any pairs of the sets, etc..; resizing of the imaging data, etc.

At block 3302, the multiple sets of imaging data are fused and processed by an MLL. The MLL has been trained using supervised learning, semi-supervised learning, or unsupervised learning. A detailed description of a method of performing training of the MLL will be explained later in connection with Fig. 10 and Fig. 11.

As a general rule, various implementations of the MLL are conceivable. In one example, a deep neural network may be used. For example, a U-net implementation is possible. See Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

More generally, the deep neural network can include multiple hidden layers. The deep neural network can include an input layer and an output layer. The hidden layers are arranged in between the input layer and the output layer. There can be a spatial contraction and a spatial expansion implemented by one or more encoder branches and one or more decoder branches, respectively. I.e., the x-y-resolution of respective representations of the imaging data and the output images may be decreased (increased) from layer to layer along the one or more encoder branches (decoder branches). At the same time, feature channels can increase and decrease along the one or more encoder branches and the one or more decoder branches, respectively. The one or more encoder branches and the one or more decoder branches are connected via a bottleneck. At the output layer or layers, the deep neural network can include decoder heads that include an activation function, e.g., a linear or non-linear activation function.

Thus, the MLL can include at least one encoder branch and at least one decoder branch. The at least one encoder branch provides a spatial contraction of respective representatives of the multiple sets of imaging data, and the at least one decoder branch provides a spatial expansion of the respective representatives of the at least one output image.

As a general rule, the fusing of the multiple sets of imaging data is implemented by concatenation or stacking of the respective representatives of the multiple sets of imaging data at at least one layer of the at least one encoder branch. This may be an input layer (a scenario sometimes referred to as early fusion or input fusion) or a hidden layer (a scenario sometimes referred to as middle fusion or late fusion). For middle fusion, it would even be possible that the fusing is implemented at the bottleneck (sometimes referred to as bottleneck fusion). Where there are multiple encoder branches, the connection joining the multiple encoder branches defines the layer at which the fusing is implemented. As a general rule, it is possible that fusing of different pairs of imaging data is implemented at different positions, e.g., different layers.

Details with respect to an implementation of the MLL are illustrated in Fig. 6.

Fig. 6 schematically illustrates the MLL 3500 implemented as a deep neural network according to various examples. As an exemplary architecture of the MLL, Fig. 6 schematically illustrates an overview of the MLL 3500. The MLL 3500 includes an encoder module 3501 having at least one encoder branch, a decoder module 3503 having at least one decoder branch, and a bottleneck 3502 for coupling the encoder module 3501 and the decoder module 3503. Each encoder branch of the encoder module 3501, each decoder branch of the decoder module 3503, and the bottleneck 3502 may respectively include at least one block having at least one layer selected from the group including: convolutional layers, activation function layers (e.g., ReLU (rectified linear unit), Sigmoid, tanh, Maxout, ELU (Exponential Linear Unit), SeLU (Scalable exponential Linear Unit), Softmax and so on), downsampling layers, upsampling layers, normalization layers (e.g., batch normalization, instance normalization, group normalization, channel normalization, etc.), dropout layers, etc.. Thus, each layer defines a respective mathematical operation.

For example, each encoder or decoder branch can include several blocks, each block usually having one or more layers, which may be named as an encoder block and a decoder block, respectively. Every block can include a single layer.

Encoder branches can be built from encoder blocks followed by downsampler blocks. Downsampler blocks may be implemented by using max-pooling, average-pooling, or strided convolution. Upsampler blocks may be implemented by using transposed-convolution, nearest neighbor interpolation, or bilinear interpolation. We also found it helpful to follow them by convolution with activations.

Decoder branches can be built from upsampler blocks followed by decoder blocks. For upsampler blocks, it is possible to apply transposed-convolution, nearest neighbor interpolation, or bilinear interpolation. Especially for the latter two, it has been found that placing several convolution layers thereafter is highly valuable.

More generally, an example encoder or decoder block includes convolutional layers with activation layers and followed by normalization layers. Alternatively, each encoder or decoder block may include more complex blocks, e.g., inception blocks (see, e.g., Szegedy, Christian, et al. "lnception-v4, inception-resnet and the impact of residual connections on learning." Thirty- first AAAI conference on artificial intelligence. 2017), DenseBlocks, RefineBlocks, or having multiple operations in parallel (e.g., convolution and strided convolution) or having multiple operations after each other (e.g., three convolution with activation and then followed by normalization before going to downsampling), etc..

The encoder module 3501 is fed with the multiple sets of imaging data - e.g., the sets 3401, 3402, 3403 and 3404, cf. Fig. 5 - as input. The decoder module 3503 outputs desired one or more output images 4001 depicting the tissue sample including a respective virtual stain (the output images can be labelled virtually stained images).

Fig. 7, Fig. 8, and Fig. 9 schematically illustrate details of three exemplary architectures of the MLL 3500. Different architectures of the MLL 3500 can be used to implement different strategies for fusing the multiple sets of imaging data in a multi-input scenario. Further, different architectures of the MLL 3500 can be used to implement multi-output scenarios. The architectures illustrated in Fig. 7, Fig. 8, and Fig. 9 can be combined with each other: for example, it would be possible that different strategies for fusing the multiple sets of imaging data as illustrated in Fig. 7, Fig. 8, and Fig. 9 are combined with each other, e.g., combining fusing at the input layer with fusing at a hidden layer or at the bottleneck for different sets of imaging data.

With reference to Fig. 7, in the illustrated example, the MLL 3500 includes a single encoder branch 3601 , and the fusing of the multiple sets of imaging data (e.g., inputl , input2 and input3 may be any three of the four imaging data sets 3401, 3402, 3403 of Fig. 5) is implemented by concatenation at an input layer 3610 of the single encoder branch 3600. The single encoder branch 3600, via the bottleneck 3502, connects to three decoder branches 3701, 3702 and 3703, respectively (albeit it would be possible to have a single decoder branch, for a single output scenario). Fig. 7 is, accordingly, a multi-input multi-output scenario using a single encoder branch.

As shown in Fig. 8 and Fig. 9, the MLL 3500 includes multiple encoder branches 3601 - 3604 and each one of the multiple encoder branches 3601 - 3604 is fed with a respective one of the multiple sets of imaging data 3401 - 3404. Fig. 8 and Fig. 9 thus correspond to a multi-input scenario. Fig. 8 is a single-output scenario, and Fig. 9 is a multi-output scenario including multiple decoder branches 3701-3703.

In the illustrated examples of Fig. 8 and Fig. 9, the fusing of the multiple sets of imaging data is implemented by concatenation at at least one hidden layer of the MLL. For example, the extracted features representing the imaging data set 3403 (input3 fed into encoder branch 3603) is fused with the extracted features representing the imaging data set 3401 (inputl fed into encoder branch 3601) at a hidden layer of block 3 of the encoder branch 3601 and then the combination of such two sets of extracted features is fed into block 4 of the encoder branch 3601 for further processing to extract fused features of the combination of the two sets of features. Similarly, the extracted features representing the imaging data set 3402 (input2 fed into encoder branch 3602) is fused with the fused features of the imaging data sets 3401 and 3403 at a hidden layer of block 4 of the encoder branch 3601 and thereby the fused features of the three imaging data sets 3401, 3402 and 3403 are acquired after further processing implemented by block 4 of the encoder 3601.

The MLL includes a bottleneck 3502 in-between the multiple encoder branches 3601 - 3604 and the at least one decoder branch 3701 - 3703. In some examples, the fusing of the multiple sets of imaging data 3401 - 3404 is at least partially implemented by concatenation at the bottleneck 350. This is illustrated in Fig. 8 and Fig. 9, where the extracted features representing the imaging data set 3404 (input4 fed into encoder branch 3604) is fused with the fused features of the three imaging data sets 3401 , 3402 and 3403 at the bottleneck 3502 to obtain fused features of the four imaging data sets 3401 - 3404; then the fused features of the four imaging data sets 3401 - 3404 are fed into respective encoder branches 3701-3703 to obtain desired virtually stained images 4001-4003.

According to various examples, the technique of “skip connections” disclosed by Ronneberger etc. (Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.) is adopted to the MLL 3500. The skip connections provide respective feature sets of the multiple sets of imaging data to corresponding decoders of the MLL 3500 to facilitate feature decoding and pixel-wise information reconstruction on different scales. The bottleneck 3502 and optionally one or more hidden layers are bypassed.

As shown in Fig. 7, several blocks of the encoder branch 3601 directly couple with corresponding blocks of the decoder branches 3701 - 3703 via skip connections 3620. For example, block 1 of the encoder branch 3601 directly provides its output that represents fused features of the combination of inputl , input2 and input3 to block 1 of the decoder branches 3701 and 3702, respectively; block 2 of the encoder branch 3601 directly provides its output which represents second fused features of the combination of inputl, input2 and input3 to block 2 of the decoder branch 3701 ; and block 3 of the encoder branch 3601 provides its output which represents third fused features of the combination of inputl , input2 and input3 to block 3 of the decoder branches 3701 and 3703, respectively.

Skip connections 3620 can, in particular, be used where there are multiple encoder branches 3601 - 3604: with reference to Fig. 8 and Fig. 9, the MLL 3500 includes multiple encoder branches 3601 - 3604. The MLL 3500 includes skip connections 3620 to feed outputs of hidden layers of at least two of the multiple encoder branches 3601 - 3604 to inputs of corresponding hidden layers of the at least one decoder branch 3701 - 3703. For example, as shown in Fig. 8, the MLL 3500 includes one decoder branch 3701 and each block of the decoder branch 3701 is fed with outputs of corresponding blocks of at least two encoder branches among 3601 - 3604 via skip connections 3620. Similarly, with reference to Fig. 9, the MLL 3500 includes multiple decoder branches 3701 - 3703 and some blocks of the three decoder branches receive outputs of corresponding blocks of at least two encoder branches among 3601 - 3604 via skip connections 3620.

Skip connections can - alternatively or additionally - be used where there are multiple decoder branches. The MLL 3500 can include skip connections 3620 to feed outputs of one or more hidden layers of at least one encoder branch to inputs of corresponding hidden layers of the multiple decoder branches.

Now referring again to Fig. 4, at block 3303, at least one output image is obtained from the MLL 3500 and each one of the at least one output image depicts the tissue sample 3400 including a respective virtual stain.

According to various examples, the MLL 3500 includes multiple decoder branches, such as the decoder branches 3701 - 3703 shown in Fig. 7 and Fig. 9, and each one of the multiple decoder branches 3701 - 3703 outputs a respective one of the at least one output images 4001-4003 depicting the tissue sample including a respective virtual stain, e.g. outputl, output2 and output3. For example, the three outputs can respectively be virtual H&E (Hematoxylin and Eosin) stained images of the tissue sample, virtually stained images of the tissue sample highlighting HER2 proteins, and virtually stained images of the tissue sample highlighting antibodies, such as anti-panCK, anti-CK18, anti-CK7, anti-TTF-1, anti-CK20/anti-CDX2, and anti-PSA/anti-PSMA, or other biomarkers. Alternatively, the three outputs can be other virtually stained images depicting the tissue sample including different types of virtual stains. The three different outputs are converted from extracted features of the multiple sets of imaging data 3401 - 3404 by the three decoder branches 3701 - 3703, respectively.

By using multiple sets of imaging data acquired using multiple imaging modalities as input of the MLL 3500, more relevant information of the tissue sample can be extracted by the at least one encoder branch and correlations across the multiple sets of imaging data are taken into account. Thereby; the at least one decoder branch can generate more precise and robust virtually stained images of the tissue sample. In particular, the MLL 3500 with multiple encoder branches can extract, via a specific encoder branch, specific information of the tissue sample from each individual set of imaging data acquired using a specific imaging modality, and thereby provide imaging modality-dependent information to the at least one decoder branch. Thus, the MLL 3500 can output reliable and accurate virtually stained images. Additionally, the MLL 3500 with multiple decoder branches can output relevant virtually stained images of the tissue sample at once. Further, the MLL 3500 with multiple decoder branches can facilitate reduced computational resources: As intermediate computation results are intrinsically shared within the MLL 3500, the number of logic operations are reduced, e.g., if compared to a scenario in which multiple MLLs are used to obtain output images depicting the tissue sample at different virtual stains.

Above, various scenarios for inference using the MLL 3500 have been described. Prior to enabling inference using the MLL 3500, the MLL 3500 is trained. As a general rule, various options are available for training the MLL 3500. For instance, supervised or semi-supervised learning or even unsupervised learning would be possible. Details with respect to the training are explained in connection with Fig. 10. These techniques can be used for training the MLL 3500 described above.

Fig. 10 is a flowchart of a method 3900 according to various examples. Fig. 10 illustrates aspects with respect to training the MLL 3500. For example, the method 3900 according to Fig. 10 may be executed by at least one processor upon loading program code from a nonvolatile memory. The method facilitates virtual staining of a tissue sample.

The MLL 3500 can be trained using supervised learning, such as the method shown in Fig. 10. The method 3900 is for performing a training of the MLL 3500 for virtual staining. In particular, the method 3900 is used to train the MLL 3500 including at least one encoder branch and multiple decoder branches, e.g., the MLL 3500 of Fig. 7 and Fig. 9. While the scenario Fig. 10 illustrates an architecture of the MLL 3500 including multiple decoder branches, similar techniques may be readily applied to a scenario in which the architecture of the MLL 3500 includes multiple encoder branches.

At block 3901, one or more training images depicting one or more tissue samples are obtained. The training images may be part of one or more sets of training imaging data. For example, as shown in Fig. 11, training images 3911, 3921 are obtained for each of two tissue samples 3910 and 3920, respectively.

The tissue samples 3910 or 3920 can be cancer or cancer-free tissue samples removed from a patient, or tissue samples of other animals or plants. The tissue samples could be in-vivo inspected tissue, e.g., using an endoscope. The tissue samples could be ex-vivo inspected cell cultures. As illustrated in Fig. 10, it would be possible that multiple instances of the training images 3911 and 3921 are obtained. For example, different tissue samples (not illustrated in Fig. 11) may be associated with the multiple instances. Thereby, a larger training database is possible.

While in the scenario Fig. 11 only a single set of training images 3911, 3921 is provided, respectively, as a general rule, it would be possible that multiple sets of training images are obtained, the multiple sets being acquired with multiple imaging modalities. For sake of simplicity, hereinafter, the training process will be described in connection with a single imaging modality providing the training images 3911, 3921, respectively, but the examples can be extended to multiple imaging modalities, e.g., by fusing as described above.

The method 3900 of Fig. 10 optionally includes - after obtaining the training images 3911 , 3921 - pre-processing the training images, e.g., using one or a combination of the following image processing techniques: artifacts reduction, such as stitching artifacts, acquisition artifacts, probe contamination, e.g., as air bubbles, dust, etc., registration artifacts, out-of-focus artifacts, etc.; noise filtering; performing a registration 701 between different training images 3911, 3921 (horizontal dotted arrow in Fig. 10); resizing; etc.

At block 3902, multiple reference images 3912-3913, 3922 depicting the one or more tissue samples 3910, 3920 including multiple chemical stains are obtained. The reference images 3912-3913, 3922 serve as a ground truth for training the MLL 3500.

For example, as shown in Fig. 11, for the tissue sample 3910, the reference images 3912-3913 are obtained; and for the tissue sample 3920, the reference image 3922 is obtained. Each reference image 3912-3913, 3922 corresponds to a type of chemical stain, such as H&E, acridine yellow, Congo red and so on. The chemical stains are labeled A, B, and C in Fig. 11. For example, each reference image 3912-3913, 3922 can be obtained by capturing an image of the respective tissue sample 3910, 3920 having been chemically stained using a laboratory staining process that may include, e.g., the tissue specimen being formalin-fixed and paraffin- embedded (FFPE), sectioned into thin slices (typically around 2-10 pm), labelled and stained, mounted on a glass slide and microscopically imaged using, for example, a bright-field microscope.

The reference images 3912-3913, 3922 could be obtained using a fluorescence microscope and appropriate fluorophore. In particular, it is possible to switch on/switch off the respective chemical stain associated with the fluorophore by wavelength-selective excitation. Different fluorophores are excited using different wavelengths and, accordingly, it is possible to selectively excite a given fluorophore. Thereby, it is possible to generate the reference images 3912-3913, 3922 so that they selectively exhibit a certain chemical stain, even if they have been dyed with multiple fluorophores.

Moreover, it is possible to have training images 3911, 3921 which shows a similar structure as the reference images 3912-3913, 3922 by not exciting any fluorophores used to stain the respective tissue sample.

As a general rule, it is not always possible for practical reasons to apply multiple chemical stains to a single tissue sample. For instance, a first reference image may highlight cell cores, another reference image may highlight mitochondrions, due to use of different fluorophores, the different reference images may be acquired from respective columns of a multi-well plate. Thus, multiple reference images may be obtained that depict different tissue samples having different chemical stains. This is why, e.g., the chemical stains of the tissue sample depicted by the reference images 3912-3913, 3922 differ from each other.

At block 3903, the one or more training images 3911 , 3921 are processed in the MLL 3500. As the procedure of processing training images is the same as that of processing images of the multiple sets of imaging data, the block 3903 is similar to block 3302 of method 3300.

Generally, as shown in Fig. 7 and Fig. 9, the at least one encoder of the MLL 3500 extracts relevant features of the one or more training images and transmits the relevant features to the multiple decoder branches 3701 - 3703. Each of the multiple decoder branches 3701 - 3703 converts the relevant features of the one or more training images 3911 , 3921 into corresponding training output images 3981-3983, 3991-3993 depicting the respective tissue sample 3910,

3920 including a respective virtual stain, here virtual stains A*, B*, and C* which correspond to chemical stains A, B, C, respectively.

At block 3904, multiple training output images are obtained from the MLL 3500, for each one of the training images. This corresponds to the multi-output scenario. Each one of the multiple training output images is associated with a respective decoder branch and depicts the respective tissue sample including a respective virtual stain. For example, as illustrated in Fig.

11 , the training output images 3981-3983 are obtained for the training image 3911 ; and the training output images 3991-3993 are obtained for the training image 3921. Generally, as shown in Fig. 7 and Fig. 9, the MLL 3500 includes three decoder branches 3701 - 3703 of which each decoder branch outputs training output images depicting a tissue sample including a respective virtual stain, here virtual stains A*, B*, and C* which correspond to chemical stains A, B, and C.

After obtaining the one or more training output images 3981-3983, 3991-3993 and the multiple reference images 3912-3913, 3922, the method 3900 optionally includes performing a registration 702-703 between the one or more training output images 3981-3983, 3991-3993 and the multiple reference images 3912-3913, 3922.

As illustrated in Fig. 11, there can be intra-sample registrations 702 and inter-sample registrations 703. The intra-sample registrations 702 between the training output images 3981- 3983 and the reference images 3912-3930 that all depict the tissue sample 3910, as well as between the training output images 3991-3993 and the reference image 3922 that all depict the tissue sample 3920. The inter-sample registration 703 can be based on the registration 701 between the training images 3911, 3921.

Such an approach of performing an inter-sample registration 703 between training output images and reference images depicting different tissue samples can, in particular, be helpful where the different tissue samples pertain to adjacent slices of a common tissue probe or pertain to different cell samples of a multi-well plate. Here, it has been observed that the general tissue structure and feature structures are comparable such that the inter-sample registration 703 between such corresponding tissue samples can yield meaningful results. However, other scenarios are conceivable in which a registration between training images and reference images depicting different tissue samples does not yield meaningful results. I.e., inter-sample registration 703 is not always required.

It is not always required to perform all the inter-sample registrations between the training output images 3981-3982 and the reference images 3912-3913, 3922. The method 3900 may optionally be limited pairwise intra-sample registration 702 between each reference image 3912-3913, 3922 and the multiple training output images 3981-3983, 3991-3993 depicting the same tissue sample 3910, 3920 (vertical dashed arrows).

As mentioned above, there are even scenarios conceivable where the training images and the reference images depict the same structures. This can be the case where the chemical stain is generated from fluorescence that can be selectively activated by using respective excitation light of a certain wavelength. Further, by non-wavelength-selective microscopy, a fluorescence contrast can be suppressed. In such a case, an inter-sample or intra-sample registration is not required, because the same structures are inherently imaged.

At block 3905, the training of the MLL is performed by updating parameter values of the MLL based on a comparison between the reference images 3912-3913 and training output images 3981-3983, 3991-3993 that are associated with corresponding chemical stains and virtual stains. The comparison is based on the registrations 702, 703. For instance, a difference in contrast in corresponding spatial regions of the reference images 3912, 3913 and the training output images 3981-3983, 3991-3993 depicting the tissue samples 3910 at corresponding chemical and virtual stains can be determined.

For example, in the scenario Fig. 11 , for the tissue sample 3910, the reference image 3912 is compared with the training output image 3981, because the virtual stain A* corresponds to the chemical stain A, e.g., H&E, etc.. Likewise, the reference image 3913 is compared with the training output image 3982. For the tissue sample 3920, the reference output image 3922 is compared with the training output image 3993. For each comparison, a respective loss can be calculated. The loss can quantify the difference in contrast between corresponding regions of the training output images and the reference images, respectively.

There is no reference image available for the tissue sample 3910 depicting the tissue sample 3910 including the chemical stain C. Thus, a comparison of the training output image 3983 would only be possible with the reference image 3922 which, however, depicts the tissue sample 3920. Thus, this comparison is only possible if the inter-sample registration 703 between the training output image 3983 and the reference image 3922 is available.

In further detail: With reference to Fig. 7 and Fig. 9, for example, the decoder branches 3701 - 3703 output training output images 3981-3983, 3991-3993 depicting the tissue samples 3910, 3920 including virtual stains A*, B*, and C*, respectively. The losses of decoder branches 3701 - 3703 are L1, L2, and L3, respectively, in which L1 is based on a comparison between the training output images 3981, 3991 depicting the tissue samples 3910, 3920 including virtual stain A* and reference images 3912 including chemical stain A, L2 is based on a comparison between the training output images 3982, 3992 depicting the tissue samples 3910, 3920 including the virtual stain B* and reference images 3913 depicting the tissue sample including the chemical stain B, and L3 is based on a comparison between the training output images 3983, 3993 depicting the tissue samples 3910, 3920 including virtual stain C* and reference images 3922 including chemical stain D. The total loss L of the MLL 3500 is a function of L1 , L2, and L3, such as L = a1*L1 + a2*L2 + a3*L3, wherein a1, a2, and a3 are, e.g., manually selected non-negative numbers.

There are various implementations of a loss function possible. For instance, the training of the MLL 3500 may be performed by using other loss functions, e.g., pixel-wise difference (absolute or squared difference) between the reference images 3912-3913 and training output images 3981-3983, 3991-3993 that are associated with corresponding chemical stains and virtual stains; an adversarial loss (i.e. , using a generative adversarial network), or smoothness terms (e.g., total variation). In some implementations a structured similarity index (https://www.ncbi.nlm.nih.giv/pubmed/28924574) may be used as a loss function. Generally, these loss functions can be combined - e.g., in a relatively weighted combination - to obtain a single, final loss function.

In such a scenario, because for each tissue sample 3910 all training output images 3981-3983, 3991-3993 are registered to at least one corresponding reference image 391-3913, 3922, the training of the MLL 3500 can jointly update parameter values of at least one encoder branch 3601 - 3604 and the multiple decoder branches 3701-3703 based on a joint comparison of the multiple reference images and the multiple training output images, such as the loss function L.

As another example, where there is no inter-sample registration 703 available: In such a scenario, the training of the MLL 3500 can include multiple iterations of the method 3900, wherein, for each one of the multiple iterations, the training updates the parameter values of the at least one encoder branch and further selectively updates the parameter values of a respective one of the multiple decoder branches based on a selective comparison of a respective reference image and a respective training output image depicting the tissue same sample including associated chemical and virtual stains. For example, with reference to Fig. 11, during a certain iteration of the training, the first iteration could train the decoder branch 3701 providing the training output images depicting the tissue sample including the virtual stain A*. This could be based on a comparison of the training output image 3981 and the reference image 3912 of the tissue sample 3910. Then, the second iteration could train the decoder branch 3702 providing the training output images depicting the tissue sample including the virtual stain B*. This could be based on a comparison of the training output image 3982 and the reference image 3913 of the tissue sample 3910. Next, the third iteration could train the decoder branch 3703 providing the training output images depicting the tissue sample including the virtual stain C*. This could be based on a comparison of the training output image 3993 and the reference image 3922. Then, the fourth, fifth, and sixth iteration can proceed with further instances of the respective images 3981, 3912; 3982, 3913; and 3993, 3922. Thus, the training updates the parameter values of the at least one encoder branch, and further selectively updates the parameter values of the decoder branches, such as the decoder branches 3701- 3703, based on the total loss L = a1*L1 + a2*L2 + a3*L3 with {a1; a2; a3} being selected from {1 ;0;0} or {0;1 ;0} or {0;0;1}, depending on the currently trained decoder branch 3701-3703.

In some examples, a combination of joint updating of parameter values for multiple decoder branches would be possible, e.g., within each one of the tissue sample 3910 and 3920. In other words, it would be possible to jointly update the parameter values for the decoder branches 3701 and 3702 in a single iteration, because the reference images 3912-3913 depicting the tissue sample 3910 having the virtual stains A* and B* are available with inter-sample registration 702.

According to various examples, the multiple iterations are according to a sequence which alternatingly selects reference images and respective training output images depicting the tissue sample including different associated chemical and virtual stains. I.e., the iterations shuffle between different chemical and virtual stains such that different decoder branches 3701- 3703 are alternatingly trained. An example implementation would be (A-A*, B-B*, C-C*, B-B*, C- C*, A-A*, C-C*, B-B*, ...). A fixed order of stains is not required. For example, this would be different to an approach according to which, firstly, in consecutive iterations all instances of the training output images 3981 are compared with all instances of the reference images 3912, before proceeding to comparing all instances of the training output images 3982 and the reference images 3913. The rationale behind such shuffling through different chemical and virtual stains is to avoid domain-biased training for the at least one encoder branch. For example, where the at least one encoder branch has parameter values that are set based on the comparison is associated with chemical and virtual stain A and A*, only, this can result in parameter values of the encoder branch that are not suited for a domain corresponding to chemical and virtual stain B, B* or C, C*.

Alternatively, according to various examples, the training of the machine-learning logic 3500 includes multiple iterations, wherein, for at least some of the multiple iterations, the training freezes the parameter values of the encoder branches and updates the parameter values of one or more of the multiple decoder branches. Such a scenario may be helpful, e.g., where a pre-trained MLL is extended to include a further decoder branch. Then, it may be helpful to avoid changing of the parameter values of the at least one encoder branch; but rather enforce a fixed setting for the parameter values of the encoder branches, so as to not negatively affect the performance of the pre-trained MLL for the existing one or more decoder branches.

The techniques for training the machine-learning logic 3500 have been explained in connection with a scenario in which the machine-learning logic 3500 includes multiple decoder branches. Similar techniques may be applied to scenarios in which the machine-learning logic 3500 only includes a single decoder branch. Then, it is typically not required to have different samples that illustrate different chemical/virtual stains.

Further, techniques have been described which facilitate training the machine-learning logic 3500 including multiple decoder branches. Similar techniques may be applied to training the machine-learning logic 3500 including multiple encoder branches. Here, as a general observation, typically, it may be possible to obtain reference images as ground truth that depict one and the same tissue sample and that have been acquired using multiple imaging modalities (this is because it is generally possible to measure a tissue sample including a given chemical stain using multiple imaging techniques). However, if this is not the case, then separate encoder branches can be trained separately, as illustrated above in connection with the multiple decoder branches, in particular, by using multiple iterations and defining respective selective loss functions.

Above, some techniques of supervised or semi-supervised learning have been described in which registrations 701-703 between the various images are required. Unsupervised learning would be possible in scenarios in which the chemical stain can be selectively activated using wavelength selective fluorescence. In these cases, registration may be omitted. Further, alternatively or additionally to supervised learning, the MLL 3500 may be trained using a cyclic generative adversarial network (e.g., Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IEEE international conference on computer vision. 2017.) architecture including a forward cycle and a backward cycle, each of the forward cycle and the backward cycle including a generator MLL and a discriminator MLL. Both the generator MLLs of the forward cycle and the backward cycle are respectively implemented using the MLL 3500.

Alternatively, the MLL 3500 may be trained using a generative adversarial network (e.g., Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017; or Kim, Taeksoo, et al. "Learning to discover cross-domain relations with generative adversarial networks." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.) architecture including a generator MLL and a discriminator MLL. The generator MLL is implemented using the MLL 3500.

Summarizing, above, techniques have been described that facilitate implementation of multi input and/or multi-output scenarios for virtual staining. In particular, scenarios have been described in which a single machine-learning logic can be used. Thereby, a flexibility in the processing of input imaging data is provided, thereby facilitating accurate determining of one or more output images depicting a tissue sample including a virtual stain. Further, by using a single machine-learning logic, it is possible to lower memory consumption. Only a single model needs to be stored after training. The dataset size can be reduced for training. Only a single dataset is required for training, because there is only a single model. Although the single dataset needs to be larger than a dataset for a single output image, it is usually smaller than the combination of datasets of all stains (cf. Fig. 11). Further, computational times can be reduced, and accuracy is improved since correlation across stains are taken into account.

Although the invention has been shown and described with respect to certain preferred embodiments, equivalents and modifications will occur to others skilled in the art upon the reading and understanding of the specification. The present invention includes all such equivalents and modifications and is limited only by the scope of the appended claims.

For illustration, above, various scenarios have been described in which the machine-learning logic configured to output multiple output images depicting the tissue sample including multiple virtual stains is implemented using multiple decoder branches. For instance, a conditional neural network may be used. See, e.g., Eslami, Mohammad, et al. "Image-to-lmages Translation for Multi-Task Organ Segmentation and Bone Suppression in Chest X-Ray Radiography." IEEE Transactions on Medical Imaging (2020). Another example implementation relies on a StarGAN, see, e.g., Choi, Yunjey, et al. "Stargan: Unified generative adversarial networks for multi-domain image-to-image translation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

For still further illustration, various examples have been described for a use case pertaining to histopathology. Similar techniques may be used for other types of tissue samples, e.g., cell microscopy ex-vivo or in-vivo imaging, e.g., for micro-chirurgic interventions. Such techniques may be helpful where, e.g., different columns or rows of a multi-well plate include ex-vivo tissue samples of cell cultures that are stained using different fluorophores and thus exhibit different chemical stains. Sometimes, it would be desirable to image a cell culture of a given well of the multi-well plate with multiple stains. Here, stains that are not inherently available chemically, i.e., because the tissue sample in that well has not been stained with the respective fluorophores, can be artificially created as virtual stains using the techniques described herein. This can be based on prior knowledge regarding which chemical stain is available in which well of the multi well plate. Thus, an image of a tissue sample being stained with one or more fluorophores and thus exhibiting one or more chemical stains can be augmented with one or more virtual stains associated with one or more further fluorophores.

Above, MIMO and SIMO scenarios have been described. Other scenarios are possible, e.g., MISO or single-input single-output SISO scenarios. For instance, for the MISO scenario, similar techniques as described for the MIMO scenario are applicable for the encoder part of the MLL.

Some embodiments may be defined by the following examples:

Example 1. A method for evaluating image modalities for virtual staining of a tissue sample comprising

- at least one iteration of

-- obtaining multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of training imaging data has been acquired using a different image modality of a group of image modalities,

-- obtaining multiple reference images depicting the one or more tissue samples comprising one or more chemical stains,

-- processing the multiple sets of training imaging data in a machine-learning logic,

— obtaining, from the machine-learning logic and for each one of the one or more training imaging data, training output images comprising one or more virtual stains corresponding to the one or more chemical stains,

-- performing the training of the machine-learning logic by updating parameter values of the machine-learning logic based on a comparison between such reference images and training output images that are associated with corresponding chemical stains and virtual stains,

-- determining a virtual staining accuracy of the trained machine-learning logic for each one of the one or more virtual stains, wherein the evaluating is based on the one or more virtual staining accuracies. Example 2. The method according to example 1 , wherein the at least one iteration further comprises,

-- depending on the virtual staining accuracy for each one of the one or more virtual stains,

- changing the group of image modalities by one of

— removing an image modality of the group of image modalities,

— replacing an image modality of the group of image modalities, or

— adding an image modality of the group of image modalities and initiating a new iteration.

Example 3. The method according to example 1 or 2, wherein acquiring imaging data of tissue samples using an image modality comprises at least one of

- acquiring images of the tissue samples in a specific spectral band, in particular, spectral bands in the ultra violet, visible and/or infrared range,

- a light microscopic analysis of the tissue samples,

- a structured illumination microscopy, SIM, analysis of the tissue samples,

- a Raman analysis of the tissue samples,

- a confocal Raman analysis of the tissue samples,

- a stimulated Raman scattering, SRS, analysis of the tissue samples

- a coherent anti-Stokes Raman scattering, CARS, analysis of the tissue samples

- a surface enhanced Raman scattering, SERS, analysis of the tissue samples

- a fluorescence analysis of the tissue samples,

- a fluorescence lifetime imaging microscopy, FLIM, analysis of the tissue samples,

- an auto-fluorescence analysis of the tissue samples,

- a phase sensitive analysis of the tissue samples,

- a phase contrast analysis of the tissue samples,

- a polarization sensitive analysis of the tissue samples,

- a digital contrast imaging analysis of the tissue samples,

- an optical coherence imaging, OCI, analysis of the tissue samples,

- an optical coherence tomography, OCT, analysis of the tissue samples,

- a digital holographic imaging, DHM, analysis of the tissue samples,

- a computer tomographic, CT, analysis of the tissue samples,

- a magnetic resonance imaging, MRI, analysis of the tissue samples,

- a multi-photon analysis of the tissue samples, - two-photon excitation fluorescence, TPEF, analysis of the tissue samples,

- second harmonic generation, SHG, analysis of the tissue samples,

- a light-sheet analysis of the tissue samples,

- a microscopy with UV surface excitation, MUSE, analysis of the tissue samples,

- Fourier ptychography analysis of the tissue samples.

Example 4. The method according to any one of examples 1 to 3, further comprising stopping the iterations depending on a comparison of the virtual staining accuracy of the trained machine-learning logic with the virtual staining accuracy of the trained machine-learning logic of a previous iteration.

Example 5. The method according to any one of examples 1 to 4, wherein the method further includes training an inverse machine-learning logic.

Example 6. The method according to example 5, wherein the machine-learning logic and the inverse machine-learning logic have been coupled during training or have been trained separately.

Example 7. The method according to example 5 or 6, wherein the machine-learning logic and the inverse machine-learning logic a implemented using an invertible neural network.

Example 8. The method according to any one of examples 5 to 7, wherein determining a virtual staining accuracy comprises processing multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of imaging data has been acquired using a different image modality of the group of image modalities, in a trained machine-learning logic, obtaining, from the trained machine-learning logic, for each one of the one or more training imaging data, at least one training output image, processing the training output image in a trained inverse machine-learning logic, obtaining, from the trained inverse machine-learning logic, output imaging data relating to the one or more tissue samples, and comparing, for each image modality of the group of image modalities, the output imaging data with the training imaging data. Example 9. The method according to example 8, wherein removing an image modality of the group of image modalities comprises removing an image modality, for which image modality the difference between the output imaging data and the training imaging data is below a predetermined threshold.

Example 10. The method according to any one of examples 1 to 9, comprising selecting one or more virtual stains, obtaining multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of imaging data has been acquired using a different group of image modalities, processing the training imaging data in a trained machine-learning logic, obtaining, from the trained machine-learning logic, output images relating to the one or more tissue samples and comprising the one or more virtual stains, determining a virtual staining accuracy for each pair of virtual stain and group of image modalities, processing the one or more virtual stains and the one or more group of image modalities in a hardware optimizer machine-learning logic, obtaining from the hardware optimizer machine-learning logic for each pair of virtual stain and group of image modalities a training virtual staining accuracy, performing the training of the hardware optimizer machine-learning logic by updating parameter values of the hardware optimizer machine-learning logic based on a comparison between virtual staining accuracies and training virtual staining accuracies.

Example 11. The method according to example 10, wherein the hardware optimizer machine-learning logic is based on meta-learning, in particular automated machine-learning, AutoML.

Example 12. The method according to example 10, wherein meta-learning comprises using at least one of grid search, random search,

Bayesian optimization, gradient-free optimization, gradient-based optimization, higher-order optimization, evolutionary optimization, or combinations thereof.

Example 13. The method according to any one of examples 1 to 12, wherein the method further comprises taking an acquisition time and/or acquisition complexity for acquiring digital imaging data for at least one image modality of the group of image modalities into account.

Example 14. A method for virtually staining a tissue sample comprising selecting one or more virtual stains, selecting a virtual staining accuracy, acquiring imaging data relating to the tissue sample using a group of image modalities, processing the imaging data in a trained machine-learning logic, obtaining, from the machine-learning logic, an output image depicting the tissue sample comprising the one or more virtual stains, wherein the group of image modalities has been selected using a method according to one of the examples 1 to 13.

Example 15. A device for tissue analysis comprising a processor configured to perform a method according to one of the examples 1 to 14.

Example 16. A device for tissue analysis comprising a processor, wherein the processor is configured for evaluating image modalities for virtual staining of a tissue sample, wherein the evaluating comprises - at least one iteration of

-- determining a virtual staining accuracy of the trained machine-learning logic for each one of the one or more virtual stains, wherein the evaluating is based on the one or more virtual staining accuracies.

Example 17. The device according to example 16, wherein the at least one iteration further comprises,

- changing the group of image modalities by one of

— removing an image modality of the group of image modalities,

— replacing an image modality of the group of image modalities, or

Example 18. The device according to example 16 or 17, wherein acquiring imaging data of tissue samples using an image modality comprises at least one of

- a light microscopic analysis of the tissue samples,

- a structured illumination microscopy, SIM, analysis of the tissue samples,

- a Raman analysis of the tissue samples,

- a confocal Raman analysis of the tissue samples,

- a stimulated Raman scattering, SRS, analysis of the tissue samples

- a coherent anti-Stokes Raman scattering, CARS, analysis of the tissue samples

- a surface enhanced Raman scattering, SERS, analysis of the tissue samples

- a fluorescence analysis of the tissue samples,

- an auto-fluorescence analysis of the tissue samples,

- a phase sensitive analysis of the tissue samples,

- a phase contrast analysis of the tissue samples,

- a polarization sensitive analysis of the tissue samples,

- a digital contrast imaging analysis of the tissue samples, - an optical coherence imaging, OCI, analysis of the tissue samples,

- an optical coherence tomography, OCT, analysis of the tissue samples,

- a digital holographic imaging, DHM, analysis of the tissue samples,

- a computer tomographic, CT, analysis of the tissue samples,

- a magnetic resonance imaging, MRI, analysis of the tissue samples,

- a multi-photon analysis of the tissue samples,

- two-photon excitation fluorescence, TPEF, analysis of the tissue samples,

- second harmonic generation, SHG, analysis of the tissue samples,

- a light-sheet analysis of the tissue samples,

- Fourier ptychography analysis of the tissue samples.

Example 19. The device according to any one of examples 16 to 18, wherein the processor is further configured for stopping the iterations depending on a comparison of the virtual staining accuracy of the trained machine-learning logic with the virtual staining accuracy of the trained machine-learning logic of a previous iteration.

Example 20. The device according to any one of examples 16 to 18, wherein the device comprises an inverse machine-learning logic, and wherein the processor is further configured for training the inverse machine-learning logic.

Example 21. The device according to example 20, wherein the processor is configured for coupling the machine-learning logic and the inverse machine-learning logic during training, or wherein the processor is configured for training the machine-learning logic and the inverse machine-learning logic separately.

Example 22. The device according to any one of examples 20 or 21 , wherein the machine-learning logic and the inverse machine-learning logic are implemented using an invertible neural network.

Example 23. The device according to any one of examples 20 to 22, wherein determining a virtual staining accuracy comprises processing multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of imaging data has been acquired using a different image modality of the group of image modalities, in a trained machine-learning logic, obtaining, from the trained machine-learning logic, for each one of the one or more training imaging data, at least one training output image, processing the training output image in a trained inverse machine-learning logic, obtaining, from the trained inverse machine-learning logic, output imaging data relating to the one or more tissue samples, and comparing, for each image modality of the group of image modalities, the output imaging data with the training imaging data.

Example 24. The device according to example 23, wherein removing an image modality of the group of image modalities comprises removing an image modality, for which image modality the difference between the output imaging data and the training imaging data is below a predetermined threshold.

Example 25. The device according to any one of examples 16 to 24, wherein the processor is further configured for selecting one or more virtual stains, obtaining multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of imaging data has been acquired using a different group of image modalities, processing the training imaging data in a trained machine-learning logic, obtaining, from the trained machine-learning logic, output images relating to the one or more tissue samples and comprising the one or more virtual stains, determining a virtual staining accuracy for each pair of virtual stain and group of image modalities, processing the one or more virtual stains and the one or more group of image modalities in a hardware optimizer machine-learning logic, obtaining from the hardware optimizer machine-learning logic for each pair of virtual stain and group of image modalities a training virtual staining accuracy, performing the training of the hardware optimizer machine-learning logic by updating parameter values of the hardware optimizer machine-learning logic based on a comparison between virtual staining accuracies and training virtual staining accuracies.

Example 26. The device according to example 25, wherein the hardware optimizer machine-learning logic is based on meta-learning, in particular automated machine-learning, AutoML.

Example 27. The device according to example 26, wherein meta-learning comprises using at least one of grid search, random search,

Example 28. The device according to any one of examples 16 to 27, wherein the method further comprises taking an acquisition time and/or acquisition complexity for acquiring digital imaging data for at least one image modality of the group of image modalities into account.

Example 29. The device according to any one of examples 16 to 28 selecting one or more virtual stains, selecting a virtual staining accuracy, acquiring imaging data relating to the tissue sample using the evaluated group of image modalities, processing the imaging data in a trained machine-learning logic, obtaining, from the machine-learning logic, an output image depicting the tissue sample comprising the one or more virtual stains.

Example 30. The device for tissue analysis according to any one of examples 16 to 29, further comprising an image acquisition system configured for acquiring digital imaging data of a tissue sample using one or more image modalities.

Example 31. Computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any one of examples 1 to 14. Example 32. A computer program, comprising a program code, which, when executed on one or more processors, causes execution of the method of any one of examples 1 to 15. Example 33. A tangible storage medium storing the computer program of Example 32.

Example 34. A data carrier signal carrying the program of Example 32.

Claims

1. A method for evaluating image modalities for virtual staining of a tissue sample comprising

- at least one iteration of

2. The method according to claim 1, wherein the at least one iteration further comprises,

- changing the group of image modalities by one of

— removing an image modality of the group of image modalities,

— replacing an image modality of the group of image modalities, or

3. The method according to claim 1 or 2, wherein acquiring imaging data of tissue samples using an image modality comprises at least one of

- a light microscopic analysis of the tissue samples,

- a structured illumination microscopy, SIM, analysis of the tissue samples,

- a Raman analysis of the tissue samples,

- a confocal Raman analysis of the tissue samples,

- a stimulated Raman scattering, SRS, analysis of the tissue samples

- a coherent anti-Stokes Raman scattering, CARS, analysis of the tissue samples

- a surface enhanced Raman scattering, SERS, analysis of the tissue samples

- a fluorescence analysis of the tissue samples,

- an auto-fluorescence analysis of the tissue samples,

- a phase sensitive analysis of the tissue samples,

- a phase contrast analysis of the tissue samples,

- a polarization sensitive analysis of the tissue samples,

- a digital contrast imaging analysis of the tissue samples,

- an optical coherence imaging, OCI, analysis of the tissue samples,

- an optical coherence tomography, OCT, analysis of the tissue samples,

- a digital holographic imaging, DHM, analysis of the tissue samples,

- a computer tomographic, CT, analysis of the tissue samples,

- a magnetic resonance imaging, MRI, analysis of the tissue samples,

- a multi-photon analysis of the tissue samples,

- two-photon excitation fluorescence, TPEF, analysis of the tissue samples,

- second harmonic generation, SHG, analysis of the tissue samples,

- a light-sheet analysis of the tissue samples,

- Fourier ptychography analysis of the tissue samples.

4. The method according to any one of claims 1 to 3, further comprising stopping the iterations depending on a comparison of the virtual staining accuracy of the trained machine-learning logic with the virtual staining accuracy of the trained machine-learning logic of a previous iteration.

5. The method according to any one of claims 1 to 4, wherein the method further includes training an inverse machine-learning logic.

6. The method according to claim 5, wherein the machine-learning logic and the inverse machine-learning logic have been coupled during training or have been trained separately.

7. The method according to claim 5 or 6, wherein the machine-learning logic and the inverse machine-learning logic a implemented using an invertible neural network.

8. The method according to any one of claims 5 to 7, wherein determining a virtual staining accuracy comprises processing multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of imaging data has been acquired using a different image modality of the group of image modalities, in a trained machine-learning logic, obtaining, from the trained machine-learning logic, for each one of the one or more training imaging data, at least one training output image, processing the training output image in a trained inverse machine-learning logic, obtaining, from the trained inverse machine-learning logic, output imaging data relating to the one or more tissue samples, and comparing, for each image modality of the group of image modalities, the output imaging data with the training imaging data.

9. The method according to claim 8, wherein removing an image modality of the group of image modalities comprises removing an image modality, for which image modality the difference between the output imaging data and the training imaging data is below a predetermined threshold.

10. The method according to any one of claims 1 to 9, comprising selecting one or more virtual stains, obtaining multiple sets of training imaging data relating to one or more tissue samples, wherein each set of the multiple sets of imaging data has been acquired using a different group of image modalities, processing the training imaging data in a trained machine-learning logic, obtaining, from the trained machine-learning logic, output images relating to the one or more tissue samples and comprising the one or more virtual stains, determining a virtual staining accuracy for each pair of virtual stain and group of image modalities, processing the one or more virtual stains and the one or more group of image modalities in a hardware optimizer machine-learning logic, obtaining from the hardware optimizer machine-learning logic for each pair of virtual stain and group of image modalities a training virtual staining accuracy, performing the training of the hardware optimizer machine-learning logic by updating parameter values of the hardware optimizer machine-learning logic based on a comparison between virtual staining accuracies and training virtual staining accuracies.

11. The method according to claim 10, wherein the hardware optimizer machine-learning logic is based on meta-learning, in particular automated machine-learning, AutoML.

12. The method according to claim 10, wherein meta-learning comprises using at least one of grid search, random search,

13. The method according to any one of claims 1 to 12, wherein the method further comprises taking an acquisition time and/or acquisition complexity for acquiring digital imaging data for at least one image modality of the group of image modalities into account.

14. A method for virtually staining a tissue sample comprising selecting one or more virtual stains, selecting a virtual staining accuracy, acquiring imaging data relating to the tissue sample using a group of image modalities, processing the imaging data in a trained machine-learning logic, obtaining, from the machine-learning logic, an output image depicting the tissue sample comprising the one or more virtual stains, wherein the group of image modalities has been selected using a method according to one of the claims 1 to 13.

15. A device for tissue analysis comprising a processor configured to perform a method according to one of the claims 1 to 14.

16. The device for tissue analysis according to claim 15, further comprising an image acquisition system configured for acquiring digital imaging data of a tissue sample using one or more image modalities.

17. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 14.