DE102021122720A1 - SYNTHETIC FLUORESCENCE IMAGES WITH SAMPLE-SAVING CAPTURE OF REFERENCE IMAGES - Google Patents

Abstract

Techniques are disclosed for training (6001) a machine-learned prediction algorithm (205) that determines synthetic fluorescence images (202) based on measurement images (201) acquired with a microscope.

Description

TECHNICAL AREA

Various examples of the disclosure relate generally to techniques for digitally post-processing measurement images acquired with a microscopic imaging modality. In particular, synthetic fluorescence images can be determined.

BACKGROUND

Fluorescence imaging is used in various areas of "green science" (life sciences). Using fluorescence imaging, it is possible to identify areas and structures in biological samples with great sensitivity and specificity. The specificity is achieved by markers that are designed to bind specifically to certain molecules or cell structures. Specificity can also be achieved by markers that are only activated in specific chemical environments. Specificity can also be achieved by markers produced within the cell itself. This can concern, for example, autofluorescence or markers after a suitable genetic manipulation step. The sensitivity is achieved through the fluorescence of the markers at specific wavelengths. This allows the signal to be filtered in the spectral range.

Typically, a label includes fluorophores or consists of a fluorophore. A fluorophore is a molecule whose electron shell can be excited by light at a specific wavelength and then emits light at a longer wavelength. The use of fluorophores makes it possible to separate excitation and light emission in the spectrum. This allows a signal to be measured that contains only a small proportion of background light. This allows a high signal-to-noise ratio (SNR) to be achieved.

These two advantages of fluorescence imaging - i.e. specificity and sensitivity - make it possible to identify cell structures of a certain type in a corresponding ensemble, for example the cell nucleus or organelle. Another application concerns, for example, the detection of a tumor.

Disadvantages can arise when using microscopic fluorescence imaging. For example, photobleaching occurs due to exposure to light to excite fluorescence. A fluorophore loses the ability to fluoresce. The use of light to excite fluorescence can also cause phototoxicity. This means that the cell structures of a corresponding sample can be damaged. This can falsify the experiment.

In addition, it is known that chemical and/or physiological effects occur as a result of the use of a marker without exposure having to take place. The sample can therefore degrade due to the presence of the markers.

The following publications are known: EP3553165A1 ; Wagner, Nils, et al. "Deep learning-enhanced light-field imaging with continuous validation." Nature Methods 18.5 (2021): 557-563.

BRIEF SUMMARY OF THE INVENTION

Therefore, there is a need for improved techniques to obtain fluorescence images. In particular, there is a need for techniques that overcome at least some of the disadvantages noted above.

This object is solved by the features of the independent patent claims. The features of the dependent claims define embodiments.

Some examples described herein are generally based on the realization that classical hardware-based fluorescence imaging, by using light to excite fluorescence, can have a major impact on the behavior of the samples being examined, in particular on the behavior of cells. On the other hand, the methods described herein are based on the recognition that other imaging modalities are less invasive because less light exposure of the sample is required to acquire corresponding image data. The examples described herein are further based on the knowledge that even with such less invasive imaging modalities there is a correlation between the contrast of pleasing measurement images and a contrast that would be expected for corresponding fluorescence images. It is therefore possible to acquire measurement images by means of a less invasive imaging modality - for example conventional brightfield transmitted light microscopy - without compromising the integrity of the sample. In particular, a large number of measurement images, for example with a high time resolution, can also be recorded.

Various examples described herein rely on the use of a machine-learned (ML) prediction algorithm to determine synthetic fluorescence images. Various examples of the invention relate in particular to the training of parameters of the ML prediction algorithm mus. Various examples are based on the knowledge that pairs of training measurement images and training reference images can be used to train the ML prediction algorithm, with the training reference images being acquired by means of microscopic fluorescence imaging. As a result, the training reference images can be used as prior knowledge or ground truth during training, and the training measurement images can be used as training input. It can then be checked to what extent there is a deviation between the prediction of the ML prediction algorithm and the basic truth, and based on this the parameters of the ML prediction algorithm can be adjusted.

Various examples are also based on the further finding that, in particular, the acquisition of the training reference images can mean that a sample is subjected to stress. This can have several implications. First, it may often be desirable to acquire training reference images over an extended observation period, covering, for example, the length of a cell cycle of a variety of cells in the sample. This is to ensure that the ML prediction algorithm provides precise predictions for different cell stages. On the other hand, if the sample is damaged by the microscopic fluorescence imaging that is used to capture the training reference images, it can happen that the sample itself is changed by capturing the training reference images. Then the ML prediction algorithm can no longer provide an accurate prediction. Such effects also occur in particular when the training measurement images and the training reference images are recorded in a reference experiment. Second: Sometimes it can also happen that the training reference images are to be acquired for a sample that is to be measured in an experiment at the same time (sample-specific training). This means that in addition to the training measurement images and the training reference images, measurement images are also acquired on the sample and the ML prediction algorithm - after performing the training - is then applied to the measurement images in order to predict such synthetic fluorescence images for the sample. These synthetic fluorescence images then have a synthetic fluorescence contrast that corresponds to the fluorescence contrast with which the training reference images with which the ML prediction algorithm was trained were acquired. In such sample-specific training, high light exposure for capturing the training reference images can cause photo-toxicity and/or photo-bleaching, thus falsifying the experiment. The sample can be damaged.

In simple reference scenarios, the light exposure per training reference image could be reduced. Such a reduction in light exposure per training reference image results in a reduced signal-to-noise ratio for the training reference images. It was observed that such a reduced signal-to-noise ratio can reduce the quality of the training of the parameters of the ML prediction algorithm. Such a reduced signal-to-noise ratio is typically observed because the photon noise, dark noise and readout noise do not decrease as much - with reduced light intensity - as the actual signal.

For all of the above reasons, it may be possible to reduce photo-toxicity and/or photo-bleaching when acquiring training reference images using fluorescence microscopic imaging using the techniques described herein.

A computer-implemented method includes driving at least one imaging device to acquire one or more training measurement images in a sample using microscopic imaging. The sample depicts one or more cell structures, i.e. e.g. a large number of cells. The method also includes driving at least one imaging device to capture one or more training reference images of a sample using microscopic fluorescence imaging. The method further includes applying a denoising algorithm to the one or more training reference images. In this way, one or more denoised training reference images are obtained. The computer-implemented method also includes performing a training of parameters of an ML prediction algorithm. The training is performed based on the one or more training measurement images as training input and based on the denoised one or more training reference images as ground truth.

Cell structures can contain e.g. entire cells or cell components, e.g. cell nucleus, nucleoli, etc.

Through such techniques, it may be possible to obtain high-quality denoised training reference images, which allow accurate training of the parameters of the ML prediction algorithm. On the other hand, it may be possible to reduce the light exposure of the sample associated with detecting the one or more training reference images is associated to reduce. For example, a duration of exposure to excitation of fluorescence can be reduced. Alternatively or additionally, the luminous flux of the excitation light can also be reduced. This means that the light exposure per training reference image can be reduced. This may make it possible to reduce the overall accumulated light exposure. Alternatively or additionally, a larger number of training reference images of the sample could be recorded per "light budget" - in comparison to reference scenarios in which no denoising takes place. This in turn allows the training data set to be expanded, allowing for more accurate training.

The training measurement images and the training reference images can be registered with each other or can be registered with each other.

Pairs of training measurement images and training reference images can be captured at a short time interval from one another. In particular, the short time interval can be so short that no significant changes are to be expected at the cellular level of the sample. In this way, registration can be performed easily.

Pairs of training measurement images and training reference images that are used together for performing the training of parameters of the ML prediction algorithm can be registered with each other. This can be possible because, for example, the same optics are used for capturing the training measurement images and for capturing the training reference images. It would also be possible to carry out a digital registration of both images on one another.

The techniques described herein can be used for sample-specific training of the ML prediction algorithm - or for sample-unspecific training of the ML prediction algorithm, where reference training images are acquired on a separate reference sample (i.e. separate from the actual experiment).

For example, it would be conceivable that the computer-implemented method further comprises: after performing the training: applying the ML prediction algorithm to further measurement images of the sample or a further sample imaging one or more further cell structures to obtain synthetic fluorescence images.

These synthetic fluorescence images can have a fluorescence contrast that corresponds to the fluorescence contrast of the one or more training reference images.

In principle, it would be conceivable for at least one imaging device to be controlled in order to capture a number of training measurement images and a number of training reference images distributed over an observation period.

The observation period can, for example, have a length of at least minutes or hours up to days.

The observation period can have a length which is of the same order of magnitude as the length of a cell cycle.

The cell cycle does not have a precisely defined length. Rather, the duration of the cell cycle for an ensemble of cells shows a spread. In general terms, the observation period can cover the length of the cell cycle, ie it can be long enough for at least significant parts of the cell cycle to be imaged.

The observation period can cover, for example, the length of a cell cycle of the cells in the sample.

In a further example, the observation period can also include the duration of an experiment with groups of cells, for example during the growth of spheroids or organoids, for example from primary cells or stem cells.

As already described above, the reduced light exposure per training reference image allows for a longer observation period or a larger number of training reference images per time unit. It would thus be conceivable for at least one imaging device to be controlled in order to acquire a number of training measurement images and a number of training reference images distributed over an observation period whose length covers the length of a cell cycle of the multiplicity of cells.

As a general rule, there are different techniques to implement the denoising algorithm. In one example, it would be conceivable that the denoising algorithm is machine-learned, which means that parameters of the denoising algorithm can be determined in further training.

There are different techniques to train a machine-learned denoising algorithm. For example, in one variant it would be conceivable for the noise reduction algorithm to be machine-learned in a self-supervised training session. Such a self-monitored one Training typically only requires noisy images; however, a basic truth in the form of noise-free images or images with a high signal-to-noise ratio is not required. For example, such training can be designed such that, for such denoising training reference images, a central pixel or region is removed or masked and the denoising algorithm is trained such that the contrast or pixel value of that removed or masked pixel is in the output is restored. Since the noise is typically statistically distributed, noise reduction can be learned in this way. Such techniques are described, for example, in Krull, Alexander, Tim-Oliver Buchholz, and Florian Jug. "Noise2void-learning denoising from single noisy images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. Another technique is described in Broaddus, Coleman, et al. "Removing structured noise with self-supervised blind-spot networks." 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020.

The above examples of denoising algorithms trained using self-supervised training are just a few examples. There are also other examples, for example probabilistic denoising, as described in Krull, Alexander, et al. "Probabilistic noise2void: Unsupervised content-aware denoising." Frontiers in Computer Science 2 (2020): 5.

Another example is denoising with diversification as described in Prakash, Mangal, Alexander Krull, and Florian Jug. "Fully Unsupervised Diversity Denoising with Convolutional Variational Autoencoders." arXiv preprint arXiv:2006.06072 (2020). See also: Prakash, Mangal, et al. "Removing Pixel Noises and Spatial Artifacts with Generative Diversity De-noising Methods." arXiv preprint arXiv:2104.01374 (2021). For example, such techniques employing diversification use variational autoencoder networks to implement the denoising algorithm.

As a general rule, it would be possible for the denoising algorithm to already be pre-trained. This means that the denoising algorithm can be trained based on a training input acquired for a reference sample, ie for a different sample than the sample for which the training reference images are acquired and denoised by the denoising algorithm. Such pre-training is easily possible because the transferability of corresponding parameters of the denoising algorithm from sample to sample can be particularly good, because the noise typically has no or no significant dependency on the sample. So-called transfer learning can be used. However, it would also be conceivable for the self-monitored training to be carried out in a sample-specific manner, ie based on noise reduction training reference images that are recorded for the sample itself.

The denoising algorithm can include a probabilistic model that determines probability distributions for the presence of specific features; these probability distributions can then be sampled. An example would be variational autoencoder networks. By multiple sampling of a latent feature representation (engl. "feature embedding") of the probabilistic model, several noise-free training reference images can be obtained per training reference image. This corresponds to an augmentation of the training data set. Such an augmentation of the training data set allows the ML prediction algorithm to be trained more comprehensively. This allows more robust and accurate predictions to be generated for the synthetic fluorescence images.

It is not necessary in all examples to carry out self-monitored training for the denoising algorithm. Supervised training could also be performed. For this purpose, the denoising algorithm can be trained based on pairs of first and second denoising training reference images. The first denoising training reference images are acquired using microscopic fluorescence imaging and a low signal-to-noise ratio. The second denoising training reference images are recorded using microscopic fluorescence imaging and a high signal-to-noise ratio. The first noise reduction training reference images can be captured in particular with a lower signal-to-noise ratio compared to the second noise reduction training reference images. The first and second denoising training reference images can be registered with each other. This achieves that denoising is achieved by means of an image-to-image transformation when the second denoising training reference images are used as ground truth in training and the first denoising training reference images are used as training input (sometimes referred to as source) . A wide variety of neural networks that are set up for such an image-to-image transformation can be used, for example convolutional networks.

Such supervised training need not be sample-specific. This means that the denoising training reference pic which can be detected by a reference sample. The sample to be measured, for which synthetic fluorescence images are determined, is then not exposed to any additional exposure to light. In any case, it can be desirable to limit the number of second denoising training reference images in order to limit light exposure of the sample (for sample-specific training) or the reference sample. For example, the number of denoising training reference images used for training per sample or per reference sample could be no greater than 50% or no greater than 10% of the number of training reference images.

In order to achieve improved denoising for ML denoising algorithms, it may be possible in the various examples that the denoising algorithm - in addition to the input of the training reference image - receives another input, for example the associated training measurement image and/or context information for the measurement (e.g. the lens used, the light intensity used for excitation, etc.).

Such techniques are based on the knowledge that the training measurement image captured by means of microscopic imaging is additionally available in the applications described herein. Information about the sample is also encoded in its contrast, so that more precise denoising can be generated because the denoising algorithm also has access to this information. When training the denoising algorithm, corresponding denoising training measurement images can also be acquired and passed to the denoising algorithm as an input channel.

In general terms, it is therefore possible for a first input channel of the input of the noise reduction algorithm to include one or more training reference images and a second input channel of the input of the noise reduction algorithm to include one or more training measurement images.

Techniques in which the denoising algorithm is machine-learned have been described above. However, examples would also be conceivable where the noise reduction algorithm is not machine-learned, i.e. is classically parameterized based on heuristics. Examples include homomorphic filtering, Gaussian filtering, non-local means, Poisson fitting, etc.

As a general rule, different implementations for the microscopic imaging to acquire the training measurement images are conceivable. For example, it would be possible that the training measurement images are not captured using fluorescence imaging. The training measurement images can be acquired with transmitted light imaging, for example.

In other examples, the training measurement images can also be acquired, at least in part, with an additional fluorescence imaging that provides a different fluorescence contrast than the fluorescence imaging with which the training reference images are acquired. For example, less sensitive stains such as GFP could be used to predict a more sensitive stain such as tdTomato. Another example concerns the prediction of F-actin based on vinculin.

In such a scenario (training measurement images with fluorescence contrast), it would be possible for the noise reduction algorithm—or also another noise reduction algorithm—to be applied to the training measurement images in order to obtain one or more noise-free training measurement images in this way. The ML prediction algorithm can be trained based on the one or more denoised training measurement images as training input.

If a further denoising algorithm is used specifically for denoising the training measurement images, the aspects described above in connection with the denoising algorithm for the training reference images can also be implemented for the further denoising algorithm. This concerns, for example, the implementation as a machine-learned algorithm, sample-specific or sample-unspecific training, self-supervised or supervised training, etc.

In particular, it would be possible for the training measurement images to be augmented as well. For this purpose, the noise reduction algorithm or the further noise reduction algorithm can be implemented using a probabilistic model (e.g. variation autoencoder network). It is then possible to scan a corresponding latent feature representation of the probabilistic model multiple times per training measurement image in order to obtain a plurality of noise-free training measurement images per training measurement image.

A computer program or a computer program product or a computer-readable storage medium includes program code. The program code can be loaded and executed by a processor. This causes the processor to execute a method. The method includes controlling at least one imaging device in order to use microscopic imaging to generate one or more training measurement images to capture a sample. The sample images a multitude of cells. The method also includes driving at least one imaging device to capture one or more training reference images of a sample using microscopic fluorescence imaging. The method further includes applying a denoising algorithm to the one or more training reference images. In this way, one or more denoised training reference images are obtained. The computer-implemented method also includes performing a training of parameters of an ML prediction algorithm. The training is carried out based on the one or more training measurement images as training input and based on the noise-free and/or, if present, one or more training reference images recorded with little noise as basic truth.

A device includes a processor. The processor is configured to: drive at least one imaging device to acquire one or more training measurement images in a sample using microscopic imaging. The sample images a multitude of cells. The processor is also configured to perform: driving at least one imaging device to acquire one or more training reference images of a sample using fluorescence microscopic imaging. In addition, the processor is configured to perform: applying a denoising algorithm to the one or more training reference images. In this way, one or more denoised training reference images are obtained. In addition, the processor is configured to perform: performing training of parameters of an ML prediction algorithm. The training is performed based on the one or more training measurement images as training input and based on the denoised one or more training reference images as ground truth.

The features presented above and features that are described below can be used not only in the corresponding combinations explicitly presented, but also in further combinations or in isolation, without departing from the scope of protection of the present invention.

character list

• 1 12 illustrates a system having at least one imaging device and a device for digital post-processing of images captured by the at least one imaging device.
• 2 schematically illustrates a digital contrast post-processing of measurement images that are acquired with at least one imaging device.
• 3 Figure 12 is a flow diagram of an example method.

DETAILED DESCRIPTION OF EMBODIMENTS

The properties, features and advantages of this invention described above, and the manner in which they are achieved, will become clearer and more clearly understood in connection with the following description of the exemplary embodiments, which are explained in more detail in connection with the drawings.

The present invention is explained in more detail below on the basis of preferred embodiments with reference to the drawings. In the figures, the same reference symbols designate the same or similar elements. The figures are schematic representations of various embodiments of the invention. Elements depicted in the figures are not necessarily drawn to scale. Rather, the various elements shown in the figures are presented in such a way that their function and general purpose can be understood by those skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as an indirect connection or coupling. A connection or coupling can be implemented wired or wireless. Functional units can be implemented in hardware, software, or a combination of hardware and software.

Techniques are described below for digitally post-processing the contrast of measurement images of biological samples. In particular, it may be possible using the techniques described herein to generate cComputer-implemented synthetic images that emulate a contrast of a particular imaging modality or contrast without actually having to apply the corresponding imaging modality or actually having to detect the particular contrast.

According to various examples described herein, it is possible to generate synthetic images that mimic imaging of stained samples (“virtual staining”). In the case of such imaging of stained samples, the biological sample is conventionally stained with an appropriate dye. This is done in a dyeing process. Here, a dye is in contact with brought to the sample. The dye includes one or more markers. Such markers create a specific contrast in the imaging. For example, the markers could specifically bind to certain parts of the cell, such as the cell nucleus or mitochondria. The markers could be used to visualize tumor tissue. For example, markers that are activated by illumination with a specific wavelength and then emit light at a different wavelength, i.e. fluorescent markers, would be conceivable.

As a further general rule, a wide variety of sample types can be analyzed in the examples described herein. Biological samples can be examined. For example, it would be possible to examine living or dead cells, e.g. in a tissue sample. For example, cell cultures could be examined. The biological sample can thus denote a biological model system, e.g. cells or spheroids (i.e. clusters of general cells, e.g. of tumors or nerve cells) or organoids (i.e. clusters of organ-specific cells). The biological sample can therefore include cells or tissue in particular. A cell culture can be examined. For example, a primary cell culture could be examined, i.e. cells obtained from an organism and maintained alive. Dead cells could also be examined. This sample thus comprises, for example, an ensemble of cell structures.

In the following, techniques in connection with fluorescence imaging are described in particular. These techniques allow the determination of synthetic fluorescence images. Fluorescence images exhibit a contrast characteristic of certain cell structures from a variety of cell structures with high specificity and sensitivity. In this way it is possible to distinguish cell structures of a certain type from other cell structures of a different type.

While various techniques are described below, particularly in the context of staining samples for fluorescence imaging, in other examples it would also be possible to use the staining for other applications, such as histopathology.

Using the techniques described herein, a machine-learned prediction algorithm can be trained to determine synthetic fluorescence images based on measurement images acquired using microscopic imaging. In particular, techniques are described to pre-process training data sets for the machine-learned prediction algorithm before the training is performed. The training data sets can be processed, i.e. their quality can be increased.

In the techniques described herein, it is possible to acquire training reference images of the training data set using fluorescence microscopic imaging for a sample. The training reference images can form the basis for the ground truth of the training.

These training reference images can be acquired in such a way that they have a comparatively low signal-to-noise ratio. A denoising algorithm can then be applied to the one or more training reference images to obtain one or more denoised training reference images. The noise-free training reference images can have a reduced noise component compared to the originally acquired training reference images, ie the noise-free training reference images can have an increased signal-to-noise ratio. Based on these denoised training reference images, the machine-learned prediction algorithm can then be trained to then use it to predict synthetic fluorescence images.

The prediction algorithm can use a correlation between an image contrast of the measurement images and an image contrast of fluorescence images. Such a technique can also be referred to as “virtual staining” because the contrast of the measurement images is changed or enhanced by the digital post-processing using the prediction algorithm, so that the synthetic fluorescence image is obtained. Various advantages can be achieved by such virtual staining, for example reduced exposure time, reduced sample scanning time, increased SNR, reduced photo-toxicity, reduced experimental effort, etc.

The sample can contain a large number of cells. It is therefore a biological sample.

The biological sample can thus denote a biological model system, e.g. cells or spheroids (i.e. clusters of general cells, e.g. of tumors or nerve cells) or organoids (i.e. clusters of organ-specific cells). The biological sample can therefore include cells or tissue in particular. A cell culture can be examined. For example, a primary cell culture could be examined, i.e. cells obtained from an organism and maintained alive. Dead cells could also be examined.

The ML prediction algorithm can be implemented by an artificial neural network (ANN).

An ANN includes a variety of layers that perform various operations. An example of an ANN is a convolutional neural network, which uses convolutional layers that perform convolution of input values with a kernel. The various layers can be connected to one another using suitable weights. Nonlinear activations are conceivable. Pooling operations can be performed: information is discarded there. An example is max-pooling, where only the strongest values of a range (e.g. from 2x2 neighborhood ranges) are kept. ANNs can have a feedforward architecture. For example, the result of one shift can only be passed on to a subsequent shift. If so-called hop connections are present, the output of one layer can be passed on to several subsequent layers. In principle, different types of ANNs can be used, e.g. in particular also generative adversarial networks (GANs) or autoencoder networks, e.g. B. Variation autoencoder networks.

The variation autoencoder network uses an encoder decoder architecture. In one or more coding branches, features in the local space are contracted towards a bottleneck. In one or more decoding branches, there is an expansion of features in the spatial space, away from the bottleneck. The output of the decode branch can reconstruct the input to the encode branch. The bottleneck denotes the latent feature representation. In the variational autoencoder network, the latent feature representation is implemented by probability distributions of the corresponding features. Each probability distribution can be described by a mean and a standard deviation. Accordingly, a two-dimensional vector is obtained. In a sampling operation, these distributions are sampled (this applies in general to probabilistic models) and the sampling result serves as input to the one or more decoding branches.

1 12 schematically illustrates a system 100 according to various examples. The system 100 includes a device 101. The device 101 could be, for example, a computer or a server. The device 101 includes a processor 102 and a memory 103. The device 101 also includes an interface 104. Via the interface 104, the device 101 can receive image data, for example measurement images and/or training reference images and/or training measurement images, from one or multiple imaging devices 111, 112 received. A sample 199 can be imaged by means of the imaging devices 111 , 112 . The training measurement images and training reference images can be used to train an ML prediction algorithm that generates synthetic fluorescence images. Denoising training reference images can also be obtained via the interface 104, with which an ML denoising algorithm can be trained. The processor 102 could also send control data over the interface 104 to the one or more imaging devices 111, 112 to control them to acquire image data. Using the control data, the processor 102 could also adjust the values of one or more imaging parameters, for example lighting parameters.

The processor 102 can be embodied as a central processing unit (CPU) and/or have specialized processing units, for example at least one graphics processing unit (GPU) and/or at least one tensor processing unit (TPU).

In general terms, the processor 102 can be set up to load and execute control instructions from the memory 103 . When the processor 102 loads and executes the control instructions, it causes the processor 102 to perform techniques as described herein. Such techniques include driving imaging device 111 and optionally imaging device 112 to acquire image data. For example, the processor 102 could be set up to control the imaging device 111 in order to acquire one or more training measurement images of a sample by means of microscopic imaging during an observation period. The processor 102 may also be configured to drive the imaging device 111 to acquire multiple training reference images of the sample during the same observation period. In principle, the training reference images can be acquired with a different imaging modality than the measurement images. Depending on the capability of the imaging device 111 regarding the imaging modality used to acquire the training reference images, the processor 102 may also drive the imaging device 112 to acquire the training reference images.

In addition, the processor 102 can be set up based on the control instructions from the memory 103 to determine synthetic fluorescence images based on the measurement images and based on an ML prediction algorithm. This means that processor 102 can perform virtual coloring.

In addition, the processor 102 can be configured to train the ML prediction algorithm. To this end, one or more training measurement images can be used as training input and one or more noise-free training reference images can be used as basic truth.

The processor 102 can also be set up to apply a denoising algorithm to one or more training reference images in order to obtain the one or more denoised training reference images in this way.

The processor 102 can also be configured to train the denoising algorithm.

2 FIG. 12 schematically illustrates aspects related to the virtual coloring of measurement images 201 depicting a sample, the sample comprising an ensemble of cell structures. A measurement image 201 serves as input to the ML prediction algorithm 205, which provides a synthetic fluorescence image 202 as an output.

For example, the prediction algorithm could be implemented by an ANN. The architecture of the ANN could e.g. according to Christiansen, Eric M., et al. "In silico labeling: predicting fluorescent labels in unlabeled images." Cell 173.3 (2018): 792-803; or Ounkomol, Chawin, et al. "Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy." Nature methods 15.11 (2018): 917-920.

Another architecture could be implemented according to a U-network having convolutional layers and hopping connections that override certain layers. sh e.g. Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham; but using grayscale regression as output instead of a classification layer (to get frame-to-frame prediction). The underground network could also be trained with additional adversarial loss contribution. This is also known as the conditional generative adversarial net.

Another example would be an autoencoder network, e.g. Variations autoencoder networks.

Training measurement images 251 and training reference images 252 are used for training 6001 of the prediction algorithm 205.

The training measurement images 251 can be captured using the same imaging modality and the same contrast as the measurement images 201.

The training reference images 252 can be acquired with fluorescence imaging and have a fluorescence contrast that corresponds to the synthetic fluorescence contrast of synthetic fluorescence images 202 .

The training reference images 252 can be denoised by means of a denoising algorithm 209 before they are fed to the training 6001 .

Optionally, the training measurement images 251 can also be denoised by means of the denoising algorithm 209 (or another denoising algorithm).

The denoising algorithm 209 can be trained 6002 by means of pairs of first denoising training reference images 281 and second denoising training reference images 282. These pairs are registered with one another and have different signal-to-noise ratios. However, self-monitored training would also be possible.

3 Figure 12 is a flow diagram of an example method. For example, the method could be performed by a device having a processor and memory when the processor loads and executes control instructions from memory. For example, the method according to 3 from device 101 1 be executed, especially by the processor 102 when this program code loads from the memory 103. Optional blocks are in 3 shown with dashed lines.

In box 3005, one or more training measurement images are obtained (eg, a time series). The one or more training measurement images are recorded in a corresponding measurement during an observation period (cf. 2 , training measurement images 251). For this purpose, an imaging device can be controlled in a suitable manner.

The training measurement images depict a biological sample that includes, for example, an ensemble of cell structures. The cell structures can be exposed to a biological change process. This biological change process can occur distributed over the ensemble and the observation period. The aim of the measurement can therefore be this biological change process to map or measure. In more general terms, the aim of the measurement can be to characterize the biological sample.

Since the biological change process occurs distributed over time and space, that is to say can be observed at different locations at different times, it can be helpful to record the training measurement images with a high repetition rate. This means that many training measurement images are recorded per unit of time. Alternatively or additionally, the training measurement images can also be recorded with a high spatial density, i.e. a Z stack of training measurement images can be recorded (repeatedly), for example. In this way it can be achieved that the biological change process can be mapped. This can be particularly helpful when the duration of each individual occurrence of the biological change process is much smaller than the time spread of the occurrence of the biological change processes in the ensemble.

In order to acquire the training measurement images 3005, the device can suitably control a corresponding imaging device. For example, the processor could send control commands to the imaging device. The control commands can be indicative of points in time for triggering the exposure and/or values of one or more lighting parameters.

In box 3010, a denoising algorithm (cf. 2 , denoising algorithm 209) that is used for denoising the training measurement images.

The denoising algorithm can be trained sample-specifically, e.g. based on further training measurement images. Self-supervised training is possible.

However, a pre-trained denoising algorithm could also be used. This could be trained in a separate experiment using data recorded for a reference sample, again self-monitored. A ground truth could also be used for training, namely pairs of denoising training reference images (cf. 2 : denoising training reference images 281, 282).

Optionally, the trained denoising algorithm can then be applied to the training measurement images in box 3015 in order to obtain such denoised training measurement images.

The denoising of training measurement images can be particularly helpful when the training measurement images themselves are acquired using fluorescence imaging.

Optionally, it would be possible in box 3020 to augment the training measurement images. To do this, a variational autoencoder network can be used for the denoising algorithm in box 3015. By multiple sampling of a latent feature representation of the variation autoencoder network per training measurement image, multiple noise-free training measurement images can be obtained per training measurement image.

In box 3025, one or more training reference images are obtained. For this purpose, an imaging device can be controlled in a suitable manner. For each training measurement image, an associated training reference image can be obtained from box 3005, which images the same sample area of the sample. If the training measurement images and the training reference images are not already inherently registered by using the same optical path for imaging, a digital registration can optionally take place so that the same sample areas are imaged at the same image location.

Fluorescence imaging is used to acquire the training reference images. Exposure parameters, for example light sensitivities of the exciting light or exposure duration, can be set in such a way that the training reference images have a comparatively low signal-to-noise ratio. This means that the light exposure of the sample per training reference image can be comparatively small. The training reference images therefore have a comparatively high level of noise.

Optionally, in box 3030, sample-specific self-supervised training can be performed for the sample and for the denoising algorithm. For box 3030, the aspects discussed above in connection with box 3010 apply.

However, such sample-specific training is not required in all variants. For example, the denoising algorithm applied to the training reference images could be the same as the denoising algorithm applied to the training measurement images. In such a case, the denoising algorithm could already have been trained in box 3010. Alternatively or additionally, it would be conceivable that the denoising algorithm is not trained in a sample-specific manner. This means that the denoising algorithm can be trained on a reference sample. In such a case, self-monitored training would also be conceivable. However, supervised training would also be conceivable. For example would be it is possible for the reference sample to acquire first and second denoising training reference images that form pairs (cf. 2 : denoising training reference images 281, 282). The first noise reduction training reference images have a low signal-to-noise ratio, ie they are recorded with imaging parameters that result in low light exposure per noise reduction training reference image. The second denoising training reference images have a high signal-to-noise ratio, ie are acquired with imaging parameters that cause high light exposure. Then an image-to-image transformation for the denoising algorithm can be trained based on the first denoising training reference images as training input and the second denoising training reference images as ground truth. Optionally, the noise reduction training reference images of the pairs can also be registered with one another. The pairs' denoising training reference images may be oriented such that like pixel coordinates in the pairs correspond to the same physical part of the sample. It is also conceivable to use further pairs of noise reduction training reference images, both of which have a low signal-to-noise ratio, in addition to the pairs of first and second noise reduction training reference images. A combination of self-monitored learning and supervised learning is also possible. It would also be conceivable that the denoising algorithm is not machine-learned, but instead the parameter values of the denoising algorithm are set manually based on heuristics or expert knowledge. Then no separate training is required.

In box 3035, the denoising algorithm can then be applied to the training reference images to obtain denoised training reference images.

As already explained above, the same or a different denoising algorithm can be applied to the training reference images and the training measurement images.

In box 3040, it would be possible for the training reference images to be augmented. It would be conceivable that the noise reduction algorithm is a variation autoencoder network. The latent feature representation of the variation autoencoder network can then be sampled multiple times per training reference image in order to obtain a plurality of noise-free training reference images per training reference image.

When removing noise from the training reference images, the training measurement images that are recorded in box 3005 can also be transferred as a further input channel to the noise removal algorithm. In this way, more precise denoising can be made possible.

Then, in box 3045, the prediction algorithm or, more precisely, its parameters can be trained (cf. 2 , 6001).

Machine learning techniques can be used to train the prediction algorithm. For example, an ANN could be trained. This means that, for example, a gradient descent method can be used, based on a value of a loss function based on a comparison (e.g. a pixel comparison) between the respective noise-free training reference image and the associated possibly noise-free training measurement image Adjusts parameter values of prediction algorithm. Different loss functions can be used, e.g. a sum over the squared deviations between pixel values. A so-called backpropagation can be used to adapt the parameter values.

The denoised training reference images serve as ground truth for the training measurement images. In order to determine the pairs between training reference images and training measurement images that are used to determine the value of the loss function based on a corresponding comparison, the distance between the training reference images and training measurement images in time and/or space can be taken into account. For example, a training reference image could serve as the ground truth for a training measurement image if the training reference image and the training measurement image depict the same area of the sample and were acquired closely in time. A short time interval can denote such a time interval that is so short that the biological change process cannot occur or can only occur with a sufficiently low probability. For example, it may be desirable that a time interval between capturing a training measurement image and capturing a reference image is no greater than 1 second, optionally no greater than 500 ms, further optionally no greater than 10 ms. This is based on the knowledge that such time scales are short enough so that there is no significant change in the sample between the measurement image and the associated reference image.

Sometimes it may be possible that the training reference image and the training measurement image inherently image substantially the same area of the sample with the same pose. This can be the case if the same imaging optics are used - that would be the example of the 1 This is the case when a specific lens of the imaging device 111 is used both for capturing the training reference images and for capturing the training measurement images. If different imaging optics are used, it may be necessary to register the pairs of training reference images and training measurement images with one another, ie to match corresponding sample locations in the images.

However, it is not necessary in all examples that the same area of the sample is inherently imaged or that the training measurement images and the training reference images are registered in order to identify corresponding sample locations. In some examples, it would be possible that, through the appropriate choice of a corresponding loss function, which is taken into account during training, basic morphological properties of the contrast in the synthetic fluorescence images reconstructed based on the training measurement images and the training reference images are compared with one another, independently of the concrete microscopic structures that are imaged. In particular, this can be the case for a loss function that uses a loss contribution with no pixel-based comparison between the synthetic fluorescence images and the training reference images.

Unregistered images can be compared using cyclic generative adverserial networks, for example. There, a non-pixel-related adverserial loss contribution or cycle loss is taken into account in the loss function during training.

As a further example of a non-pixel-based loss contribution, e.g. a semantic information content of the synthetic fluorescence images could be compared with the reference images. The semantic information content can be determined by evaluating the respective images. The semantic information content can, for example, describe a biological property of the sample. For example, the semantic information content could be selected from the following group: number or areal density of certain biological structures; Average size or size distribution of specific biological structures; Average distance between certain biological structures. The biological structures could be e.g. cell components, e.g. cell nuclei or mitochondria. For example, such biological properties could be determined using an object recognition algorithm. This could, for example, in turn be implemented by an ANN. The training in box 3045 can therefore be used to learn correlations between the training measurement images and the training reference images.

Then 3050 measurement images are obtained in box (cf. 2 , measurement images 201). For example, an imaging device can be controlled for this purpose in order to capture the measurement images. In principle, the measurement images can be acquired for the same sample for which the training reference images in box 3025 and the training measurement images in box 3005 were also acquired. This would correspond to sample-specific training of the prediction algorithm in box 3045. However, it would also be conceivable for the measurement images to be recorded for a further sample.

In box 3055, synthetic fluorescence images are then determined based on the measurement images from box 3050 and based on the prediction algorithm, which is previously trained in box 3045. The prediction algorithm thus receives one of the measurement images from block 3015 as input and determines an associated fluorescence image.

Of course, the features of the embodiments and aspects of the invention described above can be combined with one another. In particular, the features can be used not only in the combinations described, but also in other combinations or taken on their own, without departing from the field of the invention.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• EP 3553165 A1 [0007]

Claims (14)

Computer-implemented method that includes: - Controlling (3005) at least one imaging device (111, 112) in order to capture one or more training measurement images (251) of a sample (199) by means of microscopic imaging, which images one or more cell structures, - Controlling (3025) the at least one imaging device (111, 112) in order to capture one or more training reference images (252) of the sample (199) by means of microscopic fluorescence imaging, - applying (3035) a denoising algorithm (209) to the one or more training reference images (252) to obtain one or more denoised training reference images (252), and - Performing (3045) a training (6001) of a machine-learned prediction algorithm (205) based on the one or more training measurement images (251) as training input and based on the denoised one or more training reference images (252) as ground truth. Computer-implemented method claim 1 , wherein the method further comprises: - after performing the training: applying (3055) the machine-learned prediction algorithm (205) to further measurement images (201) of the sample or a further sample that maps one or more further cell structures, to obtain synthetic fluorescence images (202). Computer-implemented method claim 1 or 2 , wherein the at least one imaging device (111, 112) is controlled in order to acquire a plurality of training measurement images (251) and a plurality of training reference images (252) distributed over an observation period, a length of the observation period being a length of a cell cycle of the plurality of cells covers. A computer-implemented method as claimed in any preceding claim, wherein the denoising algorithm (209) is machine learned. Computer-implemented method claim 4 , wherein the denoising algorithm (209) is machine-learned in a self-supervised training (6002, 3010). Computer-implemented method claim 5 , wherein the self-supervised training (6002, 3010) is sample-specific for the sample (199). Computer-implemented method according to one of Claims 4 until 6 , wherein the denoising algorithm (209) comprises a probabilistic model, a plurality of denoised training reference images (252) being obtained by multiple sampling (3040) of a latent feature representation of the probabilistic model per training reference image (252). Computer-implemented method claim 4 , wherein the denoising algorithm (209) is trained based on pairs of first denoising training reference images (281) and second denoising training reference images (282), the first denoising training reference images (281) being generated by means of the microscopic fluorescence imaging and low Signal-to-noise ratio are detected, wherein the second denoising training reference images (292) are captured using the microscopic fluorescence imaging and high signal-to-noise ratio. Computer-implemented method according to one of Claims 4 until 8th , wherein a first input channel of the input of the denoising algorithm (209) comprises the one or more training reference images (252), wherein a second input channel of the input of the denoising algorithm (209) comprises the one or more training measurement images (251). Computer-implemented method according to one of Claims 1 until 3 , where the denoising algorithm (209) is parameterized based on heuristics. Computer-implemented method according to one of the preceding claims, wherein the microscopic imaging with which the one or more training measurement images (251) are captured is a further microscopic fluorescence imaging that is different from the microscopic fluorescence imaging with which the one or more training reference images (252) are captured, the method further comprising: - Applying (3015) the denoising algorithm (209) or a further denoising algorithm (209) to the one or more training measurement images (251) in order to obtain one or more denoised training measurement images (251). Computer-implemented method claim 11 , wherein the denoising algorithm (209) or the further denoising algorithm (209) a proba includes a bilistic model, multiple sampling (3020) of a latent feature representation of the probabilistic model per training measurement image (251) resulting in a plurality of noise-free training measurement images (251) being obtained. Device with a processor configured to perform the following steps: - Controlling at least one imaging device (111, 112) in order to use microscopic imaging to capture one or more training measurement images (251) of a sample that depicts one or more cell structures, - Controlling the at least one imaging device (111, 112) in order to capture one or more training reference images (252) of a sample by means of microscopic fluorescence imaging, - applying a denoising algorithm (209) to the one or more training reference images (252) to obtain one or more denoised training reference images (252), and - performing a training of a machine-learned prediction algorithm (205) based on the one or more training measurement images (251) as training input and based on the denoised one or more training reference images (252) as ground truth. device after Claim 13 , wherein the processor is set up to carry out the method according to one of Claims 1 until 12 to execute.

Images