EP4670011A1

EP4670011A1 - ELECTRONIC DEVICE, METHOD AND COMPUTER PROGRAM

Info

Publication number: EP4670011A1
Application number: EP24705188.1A
Authority: EP
Inventors: Paul Springer; Thimo Emmerich
Original assignee: Sony Europe BV; Sony Group Corp
Current assignee: Sony Europe BV; Sony Group Corp
Priority date: 2023-02-21
Filing date: 2024-02-19
Publication date: 2025-12-31
Also published as: WO2024175539A1

Abstract

An electronic device comprising circuitry configured to determine a refocused object-detector distance based on N holograms captured at N different preset object-detector distances by choosing an optimal joint refocusing result based on N reconstructed object-plane wave functions corresponding to the N holograms.

Description

ELECTRONIC DEVICE, METHOD AND COMPUTER PROGRAM

TECHNICAL FIELD

The present disclosure generally pertains to the field of holographic microscopy, in particular to devices, methods and a computer program for determining a refocused object-detector distance based on a plurality of holograms.

TECHNICAL BACKGROUND

Digital in-line holographic microscopy (DIHM) is digital holography applied with regards to microscopy. DIHM scanners distinguish themselves from other microscopy methods by not directly recording the projected image of an object but instead by recording a hologram. The hologram (also called hologram intensity plane) is recorded by a digital image sensor or by a photodetector. There are several application fields for a DIHM scanner, like a technique known as virtual staining. In virtual staining of an object, a digital representation that is visually equivalent to a chemically stained version of the object is created and thereby histochemical staining can be avoided. Some applications may be improved by applying a multi-height/multi- wavelength DIHM system, which comprises a stack of hologram images consisting of holograms at different object- detector distances and/or different illumination wavelengths.

Based on the recorded hologram the complex field information, that is the corresponding phase and amplitude image at the scanned object, is determined by means of a numerical reconstruction algorithm (e.g., iterative projection). An image forming lens in traditional microscopy is thus replaced by the numerical reconstruction algorithm.

However, it is beneficial if an object-detector distance is known precisely for the numerical reconstruction algorithms to work properly. Therefore, prior to reconstruction, it is necessary to estimate the object- detector distance, a process which is also called refocusing. Improper refocusing on hologram intensity planes may cause outliers leading to complete loss of information of individual hologram intensity plane. Further, refocusing errors may be caused by noisy hologram intensity planes and diverging refocusing results may be obtained in case of multi- wavelength acquisitions.

Therefore, it is desirable to improve the refocusing within a digital in-line holographic microscopy. SUMMARY

According to a first aspect the disclosure provides an electronic device comprising circuitry configured to determine a refocused object-detector distance based on N holograms captured at N different preset object-detector distances by choosing an optimal joint refocusing result based on N reconstructed object-plane wave functions corresponding to the N holograms.

According to a second aspect the disclosure provides a method comprising determining a refocused object-detector distance based on N holograms captured at N different preset object- detector distances by choosing an optimal joint refocusing result based on N reconstructed object-plane wave functions corresponding to the N holograms.

Further aspects are set forth in the dependent claims, the following description and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are explained by way of example with respect to the accompanying drawings, in which:

Fig. 1 schematically shows the basic principle of a digital in-line holographic microscopy (DIHM);

Fig. 2 schematically shows another setup of a multi-height/multi-wavelength digital in-line holographic microscopy (DIHM) scanner in which a phase-shift between interference images is realized by changing the object-sensor distance;

Fig. 3 shows a recorded hologram and a corresponding reconstructed object in the object plane;

Fig. 4 shows a flow chart of an iterative algorithm for reconstruction of phase and amplitude of a single-shot in-line hologram with object-detector-distance;

Fig. 5 is a further flow chart showing the iterative phase and amplitude recovery;

Fig. 6 shows a flow chart of a joint refocusing method for a plurality of holograms;

Fig. 7a shows an object-detector matrix;

Fig. 7b shows a Sobel magnitude metrics matrix;

Fig. 8 shows a flow chart of the process of capturing holograms with different illumination wavelength;

Fig. 9 shows process steps for a virtual staining process operated on an unstained input object;

Fig. 10 shows a flow chart of the calculation of the quantitative dispersion image (QDI); Fig. 11 shows a flow chart of generating a trained classifier for virtual staining;

Fig. 12a shows a flow chart of a process of virtual staining of an object;

Fig. 12b shows a table of QDI pixel values and their corresponding classification;

Fig. 13 shows different virtual staining operations that can be applied to a stained tissue specimen, input to the DIHM scanner;

Fig. 14 shows different virtual staining operations that can be applied to an unstained tissue specimen, input to the DIHM scanner; and

Fig. 15 schematically describes an embodiment of an electronic device which may implement the functionality of the process steps described above.

DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of Fig. 1, some general explanations are made.

The embodiments described below in more detail disclose an electronic device comprising circuitry configured to determine a refocused object-detector distance based on N holograms captured at N different preset object-detector distances by choosing an optimal joint refocusing result based on N reconstructed object-plane wave functions corresponding to the N holograms.

An object-detector distance (also called object-detector distance) may be a distance between an image sensor (also called detector), which records the hologram, and an object of which the hologram is taken. The object may be placed in an object-plane and the hologram may be captured in an image plane. The N different preset object-detector distances may be determined by measuring the distance between the image plane and the object plane or by using a shiftable image sensor and presetting it to N different preset distances from the object plane or the like.

N may be a number of holograms that is captured. N may be a natural number that is 1 or larger than 1, for example 2, or 5 or 10 or more than 10.

Based on a captured hologram, complex field information, that may be a corresponding phase and amplitude image at the scanned object, may be determined by means of a numerical reconstruction algorithm (e.g., iterative projection). The complex field information may be described by the reconstrued object plane wave function (also called transmission function). Therefore, the recorded hologram may be propagated from the detector plane (i.e.,. the image sensor) to the object plane to obtain the reconstrued object plane wave function.

According to some embodiments, an assumed object-detector distance is determined. The assumed object-detector distance may be set as equal to the preset object-detector distance. However, that preset object-detector distance distance may not be very accurately known and therefore the reconstructed object-plane wave function may not be very accurate. Therefore, in the embodiments, the assumed object-detector distance is estimated more precisely and accurately, which is called refocusing and the improved assumed object-detector distance is called refocused object-detector distance.

A joint refocusing result (also called joint refocusing metric) may be based on N reconstructed object-plane wave functions based on N assumed object-detector distances corresponding to the N holograms.

The joint refocusing result may be a joint Sobel magnitude metric.

A refocused object-detector distance may be determined for each of the N holograms.

An optimal joint refocusing result may be a joint refocusing result that is chosen among a plurality of joint refocusing results with regards to a specific optimality criterion. For example, the optimality criterion may be to choose the optimal joint refocusing result as the greatest joint refocusing result among the plurality of joint refocusing results.

Disadvantages of common refocusing algorithms which may perform individual refocusing on holograms (also called hologram intensity planes) may for example be that outliers may lead to complete loss of information of individual holograms, that refocusing errors may be caused by noisy holograms and that diverging refocusing results may occur between multi-wavelength acquisitions.

The electronic device may yield an improved accuracy and robustness of a refocusing result over a complete acquisition stack of the plurality of holograms.

Circuitry may include a processor, a memory (RAM, ROM or the like), a DNN unit, a storage, input means (mouse, keyboard, camera, etc.), output means (display (e.g. liquid crystal, (organic) light emitting diode, etc.), loudspeakers, etc., a (wireless) interface, etc., as it is generally known for electronic devices (computers, smartphones, etc.).

In another embodiment the circuitry may be further configured to reconstruct the N object-plane wave functions corresponding to the N holograms based on N assumed object-detector distances. The assumed object-detector distance may be the distance between the image sensor and the object plane which is assumed for the reconstruction process (for example an iterative angular spectrum method) of the object-plane wave function.

The N assumed object-detector distances may be equal to the N preset object-detector distances. Further, each of the N assumed object-detector distances may be a small value above or a small value below the corresponding preset object-detector distance. A small value may be 1% or 10% or 50% or 100% or the like of the preset object-detector distance.

Each of the N object-plane wave functions corresponding to the N holograms may be reconstructed a plurality of times based on a plurality of different assumed object-detector distances. The plurality of assumed object-detector distances may be partly above and partly below the corresponding preset object-detector distance. For example, the 50% of the plurality of assumed object-detector distances may be above and the 50% may be below the corresponding preset object-detector distance. The plurality of assumed object-detector distances may be distributed equidistantly in an interval centered around the corresponding preset object-detector distance. The interval may be 1% or 10% or 50% or the like of the preset object-detector distance.

In another embodiment the N different preset object-detector distances may be N preset distances between an object in an object plane and a shiftable image sensor configured to capture the N holograms of the object.

The N different preset object-detector distances may be obtained by measuring or setting a first preset object-detector distance and then adding repeatedly an equal (or unequal) step size for each of the following to the object-detector distance according to the formula: z_j = (j — 1) • ^zstep, where z₁ is the first object-detector distance and j is the counter counting from 1, ... , N and z_step is the step size. In this case the step size z_step may be adjustable very precisely so that the first preset object-detector distance z₁ is not known very precisely and by determining a refocused object-detector distance for the first preset object-detector distance z₁, a refocused object-detector distance for the other preset object-detector distance is determined as well.

The refocused object-detector distance may be determined in an iterative process, where the refocused object-detector distance is more precisely estimated with every iteration. The iteration may start with the assumed object-detector distance being the preset object-detector distance for each of the N holograms. For the second and all following iterations, the assumed object-detector distance may be refocused object-detector distance determined in the previous iteration. Then, the process explained above of choosing a plurality of assumed object-detector distances may be repeated with the assumed object-detector distance being the refocused object-detector distance determined in the previous iteration.

In a further embodiment the circuitry may be further configured to determine M joint refocusing results for M respective assumed object-stack distances, and to choose the optimal joint refocusing result from the M joint refocusing results.

An assumed object-stack distance may be an assumed distance between the object and the detector for the first hologram (that is for j = 1) for a current stage k = 1, ... , M. The complete stack of N holograms may be shifted to another assumed object-detector distance for each of the M joint refocusing results, that is shifted to another assumed object-stack distance.

A refocusing result (also called refocusing metric) may be the value that is obtained by applying a refocusing metric to an object plane wave function.

A refocusing metric may be a mathematical function that obtains as input a reconstructed object- plane wave function and provides as output a positive number that may be indicative of how good or bad the reconstructed object-plane wave function is focused, meaning the refocusing metric may determine how well the assumed object-detector distance that was used to reconstruct the reconstructed object-plane wave function was estimated/ assumed with regards to a real (but not yet known distance).

The joint refocusing result may be a value that is obtained by combining a plurality of refocusing results. The joint refocusing result may be indicative of how good or bad a plurality of reconstructed object-plane wave functions are focused, meaning the joint refocusing result may determine how well a plurality of assumed object-detector distances that was used to reconstruct the plurality of reconstructed object-plane wave functions was estimated/ assumed.

M may be a natural number that is 1 or larger than 1, for example 2, or 5 or 10 or more than 10.

Further, the M joint refocusing results, which are used to determine a refocused object-detector distance, may be calculated repeatedly (iteratively) in order to ever determine a refocused object- detector distance more precisely.

According to another embodiment wherein the M assumed object-stack distances may be located at an equidistant spacing within a search range. According to another embodiment the selection of an optimal joint refocusing result may be performed in an iterative way, and wherein, at each level of iteration, the equidistant spacing is refined.

The M assumed object-stack distances may be located within an interval (also called search range), including the preset object-detector distances for each of the N different object-detector distances.

The search range may be chosen based on a coarse-to-fine principle, covering in a first round a large distance around the assumed object-detector distance.

In another embodiment the search range may have a size which is a predetermined percentage of any of the N different preset object-detector distances, for example 1% or 10% or 50% or 100% or the like of any of the N different preset object-detector distance.

Further, the M joint refocusing results, which are used to determine a refocused object-detector distance, may be calculated repeatedly (in an iterative way) in order to ever determine a refocused object-detector distance more precisely. A following iteration may start for each of N holograms with the refocused object-detector distance determined in the previous iteration (or with the preset object detector distances in the first iteration). The M distances within the search range may than be determined centered around each of the N different refocused object-detector distance determined in the previous iteration.

According to another embodiment the selection of an optimal joint refocusing result is performed in an iterative way, and wherein, at each level of iteration, the equidistant spacing is refined

The search range/ interval including the preset object-detector distances (for the first iteration) or the refocused object-detector distance determined in the previous iteration (for all iterations starting with the second iteration) may be refined for each iteration. With ever increasing iteration counter, also called level of iteration, the interval (search range) will become smaller with every iteration, that is becoming refined. For example, the interval will become smaller by factor 2 or factor 5 or factor 10 or the like with every iteration.

A (current) search range may be determined as and may refined (becoming smaller) with every iteration (that is current level of iteration).

According to another embodiment the circuitry may be further configured to determine N assumed object-detector distances based on an assumed object-stack distance and based on the preset object-detector distances. According to another embodiment the circuitry may be further configured to determine a joint refocusing result by combining N different refocusing results each refocusing result being related to a respective one of N assumed object-detector distances.

According to another embodiment the circuitry may be further configured to determine N • M refocusing results based on N • M assumed object-detector distances.

N • M may be a natural number that is 1 or larger than 1, for example 2, or 5 or 10 or more than 10.

According to another embodiment the N • M assumed object-detector distances may be obtained by determining M distances in proximity around each of the N different preset object-detector distances.

The M distances in proximity around each of the N different object-detector distances and their corresponding N holograms may be the same for each of the N different object-detector distances or they may be different for each of the N different object-detector distances and their corresponding N holograms.

The M distances in proximity may mean that the M distances lie within an interval including the preset object-detector distances for each of the N different object-detector distances. The interval may have a size which is a predetermined percentage of any of the N different preset object- detector distances, for example 1% or 10% or 50% or 100% or the like of any of the N different preset object-detector distance.

Further, the M joint refocusing results, which are used to determine a refocused object-detector distance, may be calculated repeatedly (iteratively) in order to ever determine a refocused object- detector distance more precisely. A following iteration may start for each of N holograms with the refocused object-detector distance determined in the previous iteration (or with the preset object detector distances in the first iteration). The M distances may than be determined in proximity around each of the N different refocused object-detector distance determined in the previous iteration. The interval including the preset object-detector distances (for the first iteration) or the refocused object-detector distance determined in the previous iteration (for all iterations starting with the second iteration) may be the same for all iterations or may be different for each iteration. For example, the interval will become smaller with every iteration. For example, the interval will become smaller by factor 2 or factor 5 or factor 10 or the like with every iteration. The number of iterations may be 1 or more than 1 for example 3 or 5 or 10 or the like.

According to another embodiment the N • M assumed object-detector distances are obtained by determining M equidistant distances centered around each of the N different preset object- detector distances.

The M assumed object-detector distances centered around each of the N different preset object- detector distances may be an even or uneven number. If it is an uneven number, (M — 1) /2 assumed object-detector distances may be above the corresponding preset object-detector, and (M — 1)/2 may be below.

If it is an even number M/2, or (M — 1)/2 assumed object-detector distances may be above the corresponding preset object-detector, and M/2, or (M — 1)/2 may be below. The M distances lie within an interval centered around the corresponding preset object-detector distance for each of the N different assumed object-detector distances. The interval may have a size which is a predetermined percentage of any of the N different preset object-detector distances, for example 1% or 10% or 50% or 100% or the like of any of the N different preset object-detector distance. The M distances in proximity may mean a certain percentage of any of the N different preset object-detector distances, for example 1% or 10% or 50% or 100% or the like of any of the N different preset object-detector distance.

Equidistant may mean that the distance between each of two neighboring of the M assumed object-detector distances centered around a respective preset object-detector distance may be the same.

Further, the M joint refocusing results, which are used to determine a refocused object-detector distance, may be calculated repeatedly (iteratively) in order to ever determine a refocused object- detector distance more precisely. A following iteration may start for each of N holograms with the refocused object-detector distance determined in the previous iteration (or with the preset object detector distances in the first iteration). The M distances may than be determined centered around each of the N different refocused object-detector distance determined in the previous iteration. The interval including the preset object-detector distances (for the first iteration) or the refocused object-detector distance determined in the previous iteration (for all iterations starting with the second iteration) may be the same for all iterations for each of the N holograms or may be different for each iteration. For example, the interval will become smaller with every iteration. For example, the interval will become smaller by factor 2 or factor 5 or factor 10 or the like with every iteration. The number of iterations may be 1 or more than 1 , for example 3 or 5 or 10 or the like.

In another embodiment the circuitry may further be configured to choose the optimal joint refocusing result from the M joint refocusing results by comparing the M joint refocusing results to each other.

The comparing may be performed by ordering the M joint refocusing results with regards to the value of the M joint refocusing results.

In another embodiment the circuitry may further be configured to determine the refocused object-detector distance as the assumed object-detector distance corresponding to the greatest joint refocusing results.

In an embodiment, each of the joint refocusing results corresponds to N different assumed object-detector distances which were the basis on which the corresponding object-plane wave functions were determined.

The corresponding N different assumed object-detector distances z_j which correspond to the greatest joint refocusing result are determined as the refocused object-detector distances. If the assumed object-detector distances are determined according to the formula: z_j = (j — 1) • z_step, with a very precisely adjustable step size z_step, the first preset object-detector distance z₁ may be defined as the refocused object-detector distance, from which the other (A — 1) refocused object-detector distance may be derivable.

In another embodiment the joint refocusing result is determined by combining N refocusing results of the N reconstructed object-plane wave functions corresponding to the N holograms.

In another embodiment the joint refocusing result is a weighted sum of the N refocusing results.

A weighted sum may be a sum where each of the summands is multiplied with a different weight before being added together.

According to another embodiment the joint refocusing result is a joint Sobel metric and the N refocusing results are accumulated by means of a weighted sum, in order to obtain the joint refocusing result as where sobel_joint(k) is the joint Sobel metric and w_j,k are the weights, j = 1, ... N is an index for the N holograms and k = 1, ... , M is an index for the number of joint refocusing results. According to another embodiment the weights of the weighted sum are equal. They may for example be equal to a positive number. In another embodiment the sum of all weights may be equal to one.

According to another embodiment the weights of the weighted sum are not equal.

According to another embodiment the non-equal weights are derived from an absolute peak height, a peak significance, an average image or an intensity.

According to another embodiment the refocusing result for the object-plane wave functions is a Sobel magnitude metric.

According to another embodiment the reconstructing of an object-plane wave functions may be performed iteratively.

According to another embodiment the reconstructing of an object-plane wave functions may be based on an angular spectrum method.

According to another embodiment the N holograms may be captured by a digital in-line holography microscopy scanner.

A digital in-line holography microscopy (DIHM) hologram scanner may illuminate an object with illumination light and captures an interference pattern of the illumination light wave that was scattered by the object with the illumination light wave that was non-scattered. An illumination light for a hologram scanner can use the illumination light of a laser.

According to another embodiment the N holograms may be captured with a first illumination light wavelength, a second N holograms are captured with a second illumination light wavelength and a third N holograms are captured with a third illumination light wavelength.

The embodiments further disclose a method comprising determining a refocused object-detector distance based on N holograms captured at N different preset object-detector distances by determining a joint refocusing result based on N reconstructed object-plane wave functions corresponding to the N holograms.

The embodiments described below in more detail discloses a digital in-line holography microscope scanner that may comprise, an image sensor configured to acquire, for each illumination light wavelength of a set of different illumination light wavelengths, respective two or more phase-shifted holograms of an object, wherein the distance between the object and the image sensor is fixed. Embodiments are now described by reference to the drawings.

Realization of a phase shift in a digital in-line holographic microscopy (DIHM) scanner

Fig. 1 schematically shows the basic principle of a digital in-line holographic microscopy (DIHM). An in-line holography laser 100 sends out partially coherent time sequential illumination light beams 101 at different predefined illumination light wavelength λ. An object- plane 103 is located at a distance z₀ from the laser 100. To the object plane 103 it is referred with the coordinates (x, y). Into the object plane 103 an object 102, for example a tissue specimen, is placed. For each of the different predefined illumination light wavelengths, a monochrome image sensor 104, for example a CMOS or a CCD sensor, captures a respective interference pattern (also called hologram or hologram intensity plane) created by superposition of a scattered wave 105 - scattered by the object 102 - and a non-scattered wave front 106 (also called reference wave). The image sensor plane 104, also called detector plane or hologram plane is referred to with the coordinates (X, Y). The non-scattered wave 106 stems from the partially coherent illumination source 10. The monochrome image sensor 104 is placed at a preset distance z₁ from the object-plane 103 (also called object-sensor-distance) in which the object 102 is located.

Fig. 2 schematically shows another setup of a multi-height/multi-wavelength digital in-line holographic microscopy (DIHM) scanner in which a phase-shift between interference images is realized by changing the object-sensor distance is possible. As described in Fig. 1, the in-line holography laser 100 sends out partially coherent time sequential illumination light beams 101 at different a predefined illumination light wavelength λ. An object-plane 103 is located at a distance z₀ from the laser 100. Into this object-plane 103 an object 102, for example a tissue specimen, is placed. For each of the different predefined illumination light wavelengths, a monochrome image sensor 104, for example a CMOS or a CCD sensor, captures a respective interference pattern (also called hologram) created by superposition of a scattered wave 105 - scattered by the object 102 - and a non-scattered wave front 106 (also called reference wave). The image sensor plane 104, also called detector plane or hologram plane is referred to with the coordinates (X, Y). The non-scattered wave 106 stems from the partially coherent illumination source 100. The monochrome image sensor 104 is placed at a preset distance z₁ from the object- plane 103 (also called object-sensor-distance) in which the object 102 is located. In order to capture different interference patterns at the detector plane 104 with different phase shifts, the monochrome image sensor 104 is shiftable in the z-direction by an step size amount z_step by an actuator such as a servomotor (not shown in Fig. 1) and is thus placed at specified positions 104A and 104B. Accordingly, the preset distance z₁ between the image sensor 104 and the object-plane 103 is altered and different interference patterns (holograms) can be recorded at the monochrome image sensor 104 with respective phase shifts. Further, the DIHM scanner may be configured to illuminate the object plane with partially coherent time-sequential illumination with a plurality of different wavelengths, and capture holograms at different object-detector distances z₁ and with different wavelengths X, therefore it is referred to as multi-height/multi- wavelength digital in-line holographic microscopy.

For example, the DIHM scanner of Fig. 1 may be configured to establish a distance z₀ = 5cm between the laser and the object plane and to perform a partially coherent time-sequential illumination with three different wavelengths λ_red = 640nm, λ_green = 530nm and λ_blue = 450nm and with two different phase shifts per wavelength. For example, at wavelength λ_red = 640nm, the scanner captures a first interference image with a distance z₁ = 1mm, and a second interference image at a distance z₁ = 1,015mm with z-shift of z_step = 15μm compared to the first interference image. Further, at wavelength λ_green = 530nm, the scanner captures a third interference image with a distance z₁ = 1mm, and a fourth interference image at a distance z₁ = 1,015mm with a z-shift of 15 μm compared to the third image. Still further, at wavelength λ_blue = 450nm, the scanner captures a fifth interference image with a distance z₁ = 1mm, and a sixth interference image at a distance z₁ = 1,015mm with a z-shift of z_step = 15μm compared to the fifth image. For each of the wavelengths λ_red, λ_green, λ_blue the exemplifying z-shift of ^zstep = 15μm of the above example results in a respective wavelength-dependent phase shift, namely Frac(15μm/640nm) x 360° = 157.5° for λ_red= 640nm, Frac(15μm/530nm) x 360° = 108.7° for λ_green = 530nm, and Frac(15μm/450nm) x 360° = 120.0° for λ_blue = 450nm, where Frac(x) returns the fraction to the right of the decimal point of real number x.

In another embodiment the z shift z_step is realized by other means than actuator such as a servomotor. For example, the z shift z_step is realized by a tuneable phase-shifting liquid crystal filter which is place between the detector plane and the object plane, wherein the by adjusting a control voltage U_c applied to the phase-shifting liquid crystal filter, a phase shift that is exerted on light passing the tuneable phase shifter can be adjusted. In yet another embodiment the z shift ^zstep is realized by inserting optical elements into the light path between the image sensor 104 and the object-plane 103 different refractive indices, and/or elements with different thickness. In yet another embodiment the z shift z_step is realized with a switchable polarizer and a birefringent optical element which is placed between the image sensor 104 and the object-plane 103. A birefringent material is an anisotropic material which has the property that the refractive index depends on the polarization and propagation direction of light. That means, the incoming light beam 11 onto the birefringent optical element is split into two orthogonally polarized components, namely the ordinary component and the extraordinary component, wherein each component is refracted by the birefringent optical element with a different refractive index. Therefore, the light that leaves the birefringent optical element has a phase shift between the two orthogonally polarized components. The birefringent material can be a crystal, for example barium borate, calcite, quartz, ruby, etc. In order to realize a phase shift, the polarizer, which is part of the birefringent optical element, is switched between being permeable for either the ordinary component or the extraordinary component of the incoming light. In this setup a phase shift between at least two acquisitions of an interference patter of an object is obtained by changing the polarization direction of the light (for example from p to s direction or from s to p) that is allowed to pass the polarization filter, that means switching the polarizer.

In the embodiment above a monochrome image sensor is used to generate the inference images. This allows for a high image resolution. However, according to other embodiments, the image sensor may also be a non-monochrome sensor with a colour filter array for arranging RGB colour filters on a square grid of photosensors, e.g., a Bayer filter. A Bayer filter has a mosaic structure, where colour filters, for example red, green and blue, are arranged above a square grid of pixels of an image sensor. For example, the filter pattern is 50% green, 25% red and 25% blue. The laser emits illumination light comprising different wavelengths (for example red, green blue). The sensor with the colour filter is thus able to concurrently capture interference images (holograms) for each of the three different illumination light wavelengths, red, green and blue.

This setup, where the laser 100 sends its beams 101 directly through to object 102 onto the monochrome image sensor 104 where the interference pattern is captured, is called digital in-line holography. In this setup, no lens (lens-free) is used. In this case a pinhole may be added on the illumination side. The interference pattern is also called “hologram” (or “intensity image plan”), wherein each hologram encodes amplitude and phase information of the object 102. In order to realize a proper illumination an aperture may be added to the configuration.

Fig. 3 shows a recorded hologram and a corresponding reconstructed object in object plane. 301 is a recorded hologram of an object in detector plane. 302 shows a corresponding reconstructed object in the object plane. The reconstructed object 302 is described by a phase (x,y) and amplitude t₀(x,y) of an object plane wave function t(x,y) = t₀(x,y) • exp (i (x,y)) (also called transmission function) of the object (for example tissue specimen) in the object plane. It is therefore the goal to reconstruct the complex field information, which is described by the complex valued object plane wave function (also called transmission function) t(x,y) from the recorded hologram H₀(X, Y) in the detector plane. Therefore, the wavefront is propagated from the detector plane to the object plane by an (iterative) numerical algorithm for reconstruction of the phase (x,y) and the amplitude t₀(x,y) is applied (see Figs. 4 and 5).

Reconstruction of the object in the object plane

The goal is to obtain the phase Φ(x,y) and amplitude t₀(x, y) of the object plane wave function t(x,y) ⁼ t₀(x,y) · exp (iΦ(x,y)) of the sample in the object plane. Therefore, an iterative algorithm for reconstruction of phase and amplitude of the object from a single-shot in-line hologram may be applied. The algorithm with regards to Fig. 5 is based on the scientific paper “Reconstruction of purely absorbing, absorbing and phase-shifting, and strong phase-shifting objects from their single-shot in-line holograms.”, by Latychevskaia, Tatiana, and Hans-Werner Fink, published on Applied Optics 54.13 (2015): 3925-3932, and also the papers cited therein.

Fig. 4 shows a flow chart of an iterative algorithm for reconstruction of phase and amplitude of a single-shot in-line hologram with object-detector-distance z₁. In step 401, the iterative procedure is initialized by forming the initial formation of the detector plane wave function U (X, Y) in the detector domain (X, Y) as U X, Y) = A₀(X, Y)exp (iΩ(X, Y)), with initial amplitude A₀(X, Y) being set as and the initial phase distribution Ω(X, Y) being set as Ω(X, Y) = 0. In another embodiment the initial phase distribution Ω(X, Y) may be set to a randomized value. In step 402, the iteration counter i is set to i = 0. In step 403, the detector plane wave function U (X, Y) is propagated from the detector plane (X, Y) to the object plane (x, y) by applying an angular spectrum method transformation ASM to the wavefront distribution U (X, Y) which yields object plane wave function of the sample in the object plane ASM(U (X, Y), z₁) = t(x, y, z₁). The spectrum method transformation ASM is dependent from the object-detector distance z₁ and on the illumination light wavelength λ (see Fig. 1) and may be stated as with ƒ_x, ƒ_y being the spatial frequencies. The object plane wave function t(x, y, z₁) = t₀(x, y, Z₁) exp (i (x, y, Z₁)) of the sample in the object plane may be decomposed as where abso (x, y, z₁) is the absorption property of the object. In step 404, the object plane wave function t(x, y, z₁ is adapted to constraints, which yields an updated object plane wave function t'(x,y) = exp(— abso'(x,y,z₁)) • exp (i '(x,y,z₁)) of the sample in the object plane. The constraints may be to set all negative values in the absorption distribution to zero, to set the values of the object plane wave function outside a predefined support to zero etc. In step 405, the wavefront of the updated object plane wave function t'(x,y,z₁) of the sample in the object plane is propagated from the to the object plane (x, y) to the detector plane X, Y) by applying an inverse angular spectrum method transformation ASM^-1 to the wavefront t'(x,y,z₁), which yields an updated wavefront distribution U’(X, Y in the detector domain (X, Y) as

The inverse spectrum method transformation ASM^-1 may be stated as

In step 406, the updated wavefront distribution U'(X, Y, z₁) in the detector domain (X, Y) is adapted by replacing the amplitude distribution A'(X, Y) again with the initial amplitude

In step 407 the iteration counter is increased by 1, i = i + 1 and the wavefront distribution U (X, Y) is set as the wavefront distribution U(X, Y, z₁) = U" (X, Y, z₁). In step 408 it is asked if the maximum iteration count N_max has not yet been reached, i < N_max? If the answer in step 407 is yes, then it is proceeded with step 403. If the answer in step 408 is no, then it is proceeded with step 409. In step 409, the procedure ends by outputting the updated object plane wave function of the sample in the object plane.

The maximum iteration count N_max may be for example 300 or more or less.

In another embodiment a joint refocusing based on a Sobel magnitude instead of the iterative reconstruction scheme above, a measured hologram with zero-phase assumption is used to determine the reconstruction.

In another embodiment an iterative algorithm for reconstruction of phase and amplitude of the object is based on multiple-shot in-line holograms, as for example described in scientific paper “Iterative phase retrieval for digital holography: tutorial.”, by Latychevskaia, Tatiana published in JOSA A 36.12 (2019): D31-D40. Further, the (inverse) spectrum method transformation may be described in more detail or in an alternative way in the scientific paper “Reconstruction algorithms applied to in-line Gabor digital holographic microscopy.”, by Molony, Karen M., et al., published in Optics Communications 283.6 (2010): 903-909.

Fig. 5 is a further flow chart showing the iterative phase and amplitude recovery. In step 501 the detector plane wave function U (X, Y) in the detector domain (X, Y) as U (X, Y) = A₀(X, Y)exp (iΩ (X, Y)) is obtained with the amplitude A₀(X, Y) being set as A₀(X, Y) = (see step 401 in Fig, 4). In step 502, for the starting of the iteration an initial phase distribution Ω(X, Y) is being set as Ω(X, Y) = 0. In step 503, detector plane wave function U (X, Y) is propagated from the detector plane (X, Y) to the object plane (x, y) by applying an angular spectrum method transformation ASM to the wavefront distribution U (X, Y) (definition of the angular spectrum method transformation ASM see step 403 in Fig. 4). In step 504, the object plane wave function of the sample in the object plane ASM(U (X, Y), z₁ = t(x, y, z₁) is obtained with the object plane wave function t(x, y, z₁) = t₀(x, y, z₁) exp (iΦ(x, y, z₁ ) of the sample in the object plane may be decomposed as t(x, y) = t₀(x, y, z₁) • exp(iΦ (x, y, z₁)) = exp(— abso(x, y, z₁)) • exp(iΦ(x, y, z₁)), where abso(x, y, z₁) is the absorption property of the object. In step 505, the object plane wave function t(x, y, z₁) is adapted to constraints, which yields an updated object plane wave function t'(x,y) = exp(— abso'(x,y,z₁)) • exp (iΦ'(x, y, z₁)) of the sample in the object plane. The constraints may be to set all negative values in the absorption distribution to zero, to set the values of the object plane wave function outside a predefined support to zero etc (see step 404 in Fig. 4). In step 505, the wavefront of the updated object plane wave function t'(x, y, z₁ of the sample in the object plane is propagated from the to the object plane (x, y) to the detector plane (X, Y) by applying an inverse angular spectrum method transformation ASM^-1 to the wavefront t'(x, y, z₁) (definition of the inverse angular spectrum method transformation ASM^-1 see step 405 in Fig. 4). In step 508, an updated wavefront distribution U'(X, Y) in the detector domain (X, Y) is obtained as ASM(U (X, Y), z₁) = U'(X, Y, z₁) = A'(X, Y, z₁ )exp (iΩ'(X, Y, z₁ )). In step 509, the updated wavefront distribution U'(X, Y) in the detector domain (X, Y) is adapted by replacing the amplitude distribution A'(X, Y) again with the initial amplitude which yields U"(X, Y, z₁) = A₀ (X, Y)exp (iΩ' (X, Y, z₁ ), which is defined as U (X, Y) = U"(X, Y) and the iteration continues with step 501. The iteration may for example last for 300 rounds or more or less. When the iteration has finished the reconducted object plane wave function of the sample in the object plane is obtained.

Joint refocusing based on multiple holograms

Prior to reconstruction, it is necessary to estimate the object-detector distance for each of the multi-height acquisitions in order to accurately reconstruct the phase (x, y) and the amplitude t₀(x, y) the z- value z₁ (the spectrum method transformation ASM is dependent on the object- detector distance z₁), wherein the process of determining the object-detector distance is named “refocusing”. Especially the distance z₁ may not be determined very precisely, whereas the distance z_step may be determined and set relatively precisely with different control methods. It is therefore desirable to improve the estimation of the object-detector distance. This is called refocusing.

Below a joint refocusing approach is described, that is estimating the object-detector distance for a plurality of hologram intensity images (also referred to as holograms) (e.g., multi-height and/or multi- wavelength acquisitions). A joint refocusing metric combines the individual refocusing results for each hologram intensity plane over a plurality of captured holograms. The combination of individual multi-height refocusing results is realized by joining the results of corresponding object-detector distances. This correspondence is given by a known or estimated step width in z-direction, z_step.

Fig. 6 shows a flow chart of a joint refocusing method for a plurality of holograms. In step 601, variables for the algorithm are initialized to predetermined values: a refocused object-detector- distance max_distance = z₁, a number of outer iterations (also called number of levels of iterations) layers, a search range around the object-detector-distance search_range, a step size for the joint refocusing step_size, a step size between the plurality of holograms z_step (see Fig. 1), a number determining the plurality of holograms (with different region of interest, ROI) which are refocused jointly N, and a number determining the plurality of different (at different distances z₁) reconstructed object plane wave functions per hologram In step 602, N holograms H_k(X, Y) are captured (see Figs. 1 and 2) with different respective assumed object-detector-distance z_1,k = z₁ + (k — 1) * z_step, for k = 1, ... , N, i.e., each of the plurality of holograms covers a different ROI of the object 102 (see Figs. 1 and 2). In step 603, an iteration counter for the current level of iteration is initialized with current_layer = 1. In step 604, a for-loop is entered (steps 604 - 610) by determining an MxN object-detector distance matrix Z₁ (see Fig. 7a) comprising and with and with

The distance current-distance (j) is also called assumed object-stack distance for each (row) j, which may be seen as an assumed distance between the first hologram (that is for k = 1) at the current stage j. That means the current search range is determined as and is refined with every layer of iteration (that is with every increasing iteration counter current_layer).

In step 605, a M x N matrix T of object plane wave functions T(j, k) = (t(x, y, z_1,j,k is determined, with j = 1, ... , M and k = 1, ... , N, based on z_1,j,k, the N holograms H_k(X, Y) and an initial phase guess Ω_k(X, Y) = 0 (as described with regards to Figs. 4 and 5). In step 606, a

M x M Sobel magnitude metrics matrix SMM of Sobel magnitude metrics is determined (see Fig. 7B below) by applying the Sobel magnitude metric S (definition given below) to the object plane wave functions t(x, y, z_1,j,k) with j = 1, ... , M and k = 1, ... , N, as SMM(J, k) = S(t(x, y, z_1,j,k). In step 607, a joint Sobel magnitude metric sobel_joint(j) (that is a joint refocusing result) is determined over each row of the matrix SMM, as where W_j,k are weights of a weighting matrix W. That means M joint metrics are determined by respectively combining the N metrics SMM (j, k) corresponding to one of the M different assumed object-detector distances (z_1,j,k) (that is making a weighted summation over the entries of each of the M rows of the matrix SMM).

In step 608, the refocused object-detector-distance max _distance is determined as max _distance = z_1,1,J with J = argmax_j=1,...M (sobel_joint(j)), wherein argmax_j=1,...M is the argument of maxima function over all M elements of sobel_joint(j) (sobel_joint(J) is also called the optimal joint refocusing result). In step 609, the for-loop iteration counter for the level of iteration is increased current_layer = current_layer + 1. In step 610 it is asked if the for- loop is finished by asking if current_layer ≤ layers? If the answer in step 610 is yes, then it is proceeded with step 604 and the next round of the for-loop is started. If the answer in step 610 is no, then it is proceeded with step 611. In step 611, the final refocused object-detector-distance max _distance is output. The value max _distance is the refocused object-detector-distance for the hologram (Nr. 1, i.e., k=l) with the estimated object-detector distance z₁. In order to obtain the refocused object-detector-distance for the holograms 2, ... , N the additional positional offset (j — 1) • z_step is added.

After the refocusing process, that is after properly estimating the object-detector-distance the reconstructing of the corresponding object wave functions can be carried out, or the reconstructed object plane wave functions t(x, y, max _distance) corresponding to the refocused object-detector-distance z₁ = max _distance may be utilized for further processing (see Fig. 9).

The Sobel magnitude metric S of an object plane wave function t(x, y, z₁) = t₀ (x, y, z₁) exp (i (x,y,z₁)) is defined as: with being defined as first derivatives in vertical or horizontal direction. The complex valued continuous object plane wave function t(x, y, z₁) is evaluated at the discrete grid points (m, n) for m = 1, ... , A and n = 1, ... , B wherein this grid may be defined by the pixels of the detector plane and A and B are the number of pixels in x and y direction and (m, n) are the center points of the pixels. The Sobel metric is a row-based aggregation. The weights W_j,k may be chosen all equally as W_j,k ≡ c, with c being a positive number. Further, the weights may be non-equal and may be derived from the absolute peak height, peak significance, average image or intensity. Further, the weights may be equal within the rows W(j, : ) of the weighting matrix W.

Fig. 7a shows an object-detector matrix Z₁. The object-detector matrix Z₁ is determined as described above in step 604 (see Fig. 6). The object detector matrix is an M x N matrix (M rows, N columns), where M = 18 is the number of refocusing points per layer defined by M = and N = 10 is the number of captured holograms, with a fixed step size z_step=

0.01.

Fig. 7b shows a Sobel magnitude metrics matrix SMM. The Sobel magnitude metrics matrix SMMis determined as described above in step 606 (see Fig. 6), that is the based on the object- detector matrix Zj shown in Fig. 7a, matrix T comprising the object plane wave functions t(x,y,z_1,j,k) is determined, and based on that the Sobel magnitude metrics matrix SMM is obtained by applying the Sobel magnitude metric to each entry of the matrix T. Therefore, the Sobel magnitude metrics matrix SMM also comprises, as the object-detector matrix Z₁ in Fig. 7a M = 18 rows and N = 10 columns. The sum over each the 18 rows of the Sobel magnitude metrics matrix SMM yields the joint Sobel magnitude metric sobel_joint(f), as exemplarily indicated for the joint Sobel magnitude metric in Fig. 7b.

The procedure of Fig. 6 may be performed with holograms captured at a plurality of different illumination light wavelengths. In one embodiment for each plurality of L different illumination light wavelengths N holograms are obtained wherein for each of the L different illumination light wavelengths the N holograms have the same object-detector-distance z₁ and with the same step size z_step in between them, that is the steps 601 - 607 of Fig. 6 are performed for each of the L different illumination light wavelengths. This yields L different M x N matrices and L • M joint Sobel magnitude metrics sobel_joint(j) . The argmax function is then applied to all of the L • M joint Sobel magnitude metrics sobel_joint(j) . This yields a refocused object-detector- distance max _distance corresponding to one of the L different illumination light wavelengths. The procedure is then started again in step 604 for each of the L different illumination light wavelengths for all with the same determined refocused object-detector-distance max _distance. In one embodiment L = 3, with λ_red, λ_green andλ_blue. That is, multi- wavelength acquisitions of hologram intensity images are considered to be captured at the same fixed object-detector distance.

Fig. 8 shows a flow chart of the process of capturing holograms with different illumination wavelength. In step 801, a hologram is capture with object-detector-distance z₁ at the image sensor 14 for a first wavelength λ_red. In step 802, a hologram is capture object-detector-distance z₁ at the image sensor 14 for a second wavelength λ_green. In step 803, a hologram is capture object-detector-distance z₁ at the image sensor 14 for a third wavelength λ_blue. In step 804, the DIHM scanner setup is changed to a new object-detector-distance z₁ + z_step. This can be achieved as described in embodiment of Fig. 2. In step 805, a hologram is captured object- detector-distance z₁ + z_step at the image sensor 14 for a first wavelength λ_red. In step 806, a hologram is capture object-detector-distance z₁ + z_step at the image sensor 14 for a second wavelength λ_green. In step 807 a hologram is capture object-detector-distance z₁ + z_step at the image sensor 14 for a third wavelength λ_blue. The steps 804 - 806 may be repeated several times.

Embodiment of digital processing of captured holograms

After the refocusing process, that is after properly estimating the object-detector-distance the reconstructing of the corresponding object wave functions can be carried out, or the reconstructed object plane wave functions t(x, y, max _distance) corresponding to the estimated object-detector-distance z₁ = max _distance may be utilized for further processing. There are several applications for DIHM scanners and the reconstructed object plane wave functions can be used, for example, for a virtual staining process operated on an unstained input object (tissue specimen).

Each object 102 (for example a specimen) introduces a delay in the light path when light propagates through it. This amount of delay is also called “phase shift” or “Optical Path Difference”. The DIHM scanners and their operation described above allow to control/introduce a phase shift when capturing interference images by applying different object-detector-distance by applying one or more z-steps z_step. In particular, for each of one or more illumination wavelengths, multiple holograms are captured at with different object-detector-distance z₁ which yields different phases.

Fig. 9 shows process steps for a virtual staining process operated on an unstained input object (tissue specimen). In process step 901, an unstained specimen is input into a DIHM scanner. In process step 902, a number of N phase-shifted holograms (here for example N = 3) - which may each be z_step apart - of the specimen are taken per L different illumination wavelengths (here for example L = 3), wherein the L different illumination wavelengths are applied time-sequentially (see Fig. 8). This results in a total number of N x L holograms. In process step 903, the object plane wave functions are reconstructed (see Figs. 4 and 5) with a precisely estimated correct the object-detector-distance z₁ = max _distance (see Fig. 6). The object plane wave functions t x,y, max _distance ) = t₀(x,y, max _distance )exp (i (x,y, max _distance )) for each of the L = 3 different wavelengths comprise an amplitude t₀(x, y, max _distance ) and a phase (x,y, max _distance ). The phase function (x,y, max _distance ) evaluated at the pixel coordinates of the detector yields a phase value for each pixel which is also called quantitative phase image (QPI). In step 904, the quantitative phase images (QPI) for each of the different illumination wavelengths are collected in a QPI stack, wherein each QPI contains the absolute phase shift, i.e., the optical path delay (PD) through the object per pixel and wavelengths. Each quantitative phase image (QPI) contains the absolute phase shift, i.e., the optical path delay (PD) through the object per pixel and wavelengths. This optical phase shift, or phase delay (PD), which is given for example in radian, can easily be transferred into the optical path difference (OPD) that is caused by each pixel for each wavelength by:

OPD_i[nm] = PD /(2π * λ_i [nm]) wherein the index i is element of the set {red, green, blue}.

Further, the intensity images for the different illumination wavelength are collected in a colour image, for example based on the wavelengths λ_red, λ_green, λ_blue a RGB colour image of the object 102 can be reconstructed. This can for example be done by defining the intensity of each pixel of the colour image by the combined intensity of each component, i.e., per pixel colour image intensity = (1/3 intensity red, 1/3 intensity green, 1/3 intensity blue). Further, a so called quantitative dispersion image (QDI) is generated (see Fig. 10 below for details). In process step 905, a classifier is operated on the QDI. This classifier allows for a “virtual staining” (see Figs. 11-14) of the specimen.

In case that the object is a tissue specimen, the colour image as well as the different QPIs can then be used for virtual staining and classification of each pixel of the tissue specimen, which is described in more detail below.

Construction of Quantitative Dispersion Image (QDI)

In the following an exemplifying process of determining a quantitative dispersion image QDI from a set of quantitative phase images QPI is described. Dispersion in optics describes the phenomenon that the refractive index of a material differs for different wavelength. Constructing a quantitative dispersion image of an object allows to quantize the dispersion of each image pixel.

Therefore, first a refractive index for each pixel and each wavelength of the scanned object 12 is calculated.

Each pixel of a quantitative phase image QPI_i as obtained by the process described in Fig. 7 above describes a phase delay value PD_i, which is typically given in the quantity radian (“rad”) which is a standard unit for measuring angles. The index i represents the illumination wavelength (e.g. i ∈ {red, green, blue}) at which quantitative phase image QPI_i was captured.

The optical path difference OPD_i is directly related to the phase delay value PD; as follows, and may for example be expressed in nanometer, nm:

The optical path difference OPD_i [μm] is the difference between the optical path length

OPL_i,object through the object (tissue specimen) and the optical path length OPL_i,reference through the reference medium (air):

The optical path length OPL_i,object through the object (the tissue specimen, 12 in Figs. 1 to 4) with thickness d and refractive index n_i,object at wavelength i is defined as:

Accordingly, the optical path length OPL_i,reference through a reference medium with thickness d corresponding to the thickness d of the object (tissue specimen) scanned by the DHIM scanner, and with refractive index n_{i reference} at wavelength i is defined as:

Therefore, the refractive index n_i,object of the object at wavelength i can be obtained as:

The optical path difference OPD_i can be normalized regarding the thickness d of the object (tissue specimen) scanned by the DHIM scanner (expressed e.g. in micrometer, μm), which yields the normalized optical path difference NOPD;:

NOPD_i[mm/μm] = OPD_i [mm]/ d[μm]

Using this normalized optical path difference NOPD_i, the refractive index n_i,object of the object at wavelength i can be expressed as: After acquisition of the refractive index n_i,object per illumination wavelength i (here for example i E {red, green, blue}) a quantitative dispersion value QDV_object of the object at a respective pixel of the image can be calculated as:

By calculating this quantitative dispersion value QDV_object for each pixel, the quantitative dispersion image QDI is obtained.

The above described determination of the QDI can be done with any wavelengths that the DIHM scanner operates with. The determination may for example be done for three illumination wavelengths. If, for example, three different wavelengths are applied, these three wavelengths may be ordered as: λ_long > λ_middle > λ_short, and with the corresponding refractive indices n_short, n_middle, n_long , the quantitative dispersion value QDV_object for each pixel can be calculated by:

The same principles apply if less than three or more than three different wavelengths are applied.

Fig. 10 shows a flow chart of the above process of calculating a quantitative dispersion image QDI from a set of quantitative phase images QPI_i. In step 1001, for each wavelength i and for each pixel of the quantitative phase image QPI_i, an optical path difference OPD, is calculated based on a respective phase delay value PD_i of the quantitative phase image QPIi . In step 1002, for each pixel and for each of the wavelengths i, a normalized optical path difference NOPD_i is calculated based on the respective optical path difference PD_i and the thickness d of the object (tissue specimen) scanned by the DHIM scanner. In step 1003, for each pixel and for each of the wavelengths i, a refractive index n_i,object is calculated based on a predetermined refractive index ⁿi,reference of a reference medium and on the respective normalized optical path difference NOPD_i. In step 1004, a quantitative dispersion image QDI is determined by calculating, for each pixel, a quantitative dispersion value QDV_object based on the respective refractive indicesn_i,object of the wavelengths i.

Classifier and Virtual Staining

Virtual staining of an object, for example a tissue specimen, means virtually simulating how an object would look like if it was stained with a respective histochemical dye. In virtual staining of a tissue specimen, a digital representation that is equivalent to a chemically stained (labeled) version of the tissue specimen is created. A classification algorithm (classifier) is trained, which yields a trained classifier (see Fig 11 for more details). The trained classifier receives as an input for each pixel, either the QDI value and/or the QPI value (for each colour) and/or the colour image value, or all or a combination of these three, and outputs for each pixel a probability value that the pixel is stained with a certain staining chemical and thereby a virtually stained colour image is received (see Fig. 12a for more details). For example, if the trained classifier was trained with the histochemical dye Hematoxylin and Eosin, the trained classifier is able to output a probability for each pixel that it is stained by Hematoxylin, a probability for each pixel that it is stained by Eosin and a probability for each pixel that it is stained by neither of the two, wherein this classification is based on (that is the input for the classifier) the QDI value and/or on the QPI value (for each colour) and/or on the colour image value.

As stated above, the quantitative dispersion image QDI is a fast and compact way to condense a “phase characteristic” of the scanned object (102 in Figs. 1 and 2) into one image. This is especially useful if a classification and visualisation is done via a classification learning algorithm, as described below.

A classification and learning algorithm can operate with the quantitative dispersion image QDI as such, or on a combination of the quantitative dispersion image QDI with QPI data and colour image data.

Classification algorithms are known to the skilled person. For example there exist classification algorithms that are based on supervised learning to obtain a trained classifier. For example, linear regression, linear classifiers (Naive Bayes, perceptron, logistic regression), quadratic classifiers, support vector machines (SVM), kernel density estimators, k-nearest neighbor, artificial neural networks (ANN) or more.

Fig. 11 shows a flow chart of generating a trained classifier for virtual staining. In step 1101 the features (or feature vector) for the plurality of different objects are obtained, which are for example the QDIs of the objects and/or the colour images of the objects and/or the QPIs of objects. In step 1102, the objects are stained with a chemical dye (for example with Hematoxylin and Eosin) and (colour) images of the of the stained objects are taken. In step 1103, each pixel of the images of stained objects are labelled based on the chemical reaction with the chemical dye. For example, the pixels that reacted with Hematoxylin are labeled with “H”, the pixels that reacted with Eosin are labeled with “E” and the pixels that did not react with any chemical are labeled with “None”. In step 1104, a data set is generated based on features of objects and the obtained labels. For example, the data set may comprise a pair for each object, comprising the features and the label, is generated. In step 1105, a classifier is trained, via supervised learning, based on generated training data set.

For example if an ANN is used as classifier, the training step 1105 can be done by the skilled person by using any known techniques, such as the original training mechanism for the Long- Short-Term-Memory (LSTM) as described by Gers et al., 2002 in "Learning Precise Timing with LSTM Recurrent Networks" in the Journal of Machine Learning Research 3 (2002), 115-143, which is based on a gradient method to adjust the weights of the LSTM units. During this training process the neural network is preferably presented with a large number of examples (training data). The output layer may be realized as a Softmax layer that assigns normalized decimal probabilities to each class in a multi-class problem.

Fig. 12a shows a flow chart of a process of virtual staining of an object. In step 1201, classification of a stained or unstained object based on the trained classifier and the colour image of the stained or unstained object and/or the QDI of the stained or unstained object and/or the QPIs of the stained or unstained object is performed. In step 1202, virtual staining of the stained or unstained object based on the classification of stained or unstained object and a loop-up-table is performed.

When the trained classifier receives as an input the QDI, the QPIs or the colour image (or all or a combination thereof) of the object it outputs a probability distribution over the different base classes of the histochemical dyes it was trained with, that is a probabilistic value (between [0, ... , 1 ]) of the different base classes for all histochemical dyes that it was trained for. For example, in case the classifier was trained with Hematoxylin and Eosin the base classes are: probability_H= probability that pixel should be virtually stained with Hematoxylin, probability E = probability that pixel should be virtually stained with Eosin and probability _N = probability that none of the two above. Then a lookup table (LUT) can be applied to realize the virtual staining for example for Hematoxylin and Eosin (HE): Virtual_stained_HE_image(R,G,B) = LUT(probability_H, probability E, probability N). For example, the lookup table LUT can determine for a specific pixel that, if the probability H is the highest value among all base classes, the pixel is coloured blue, if the probability E is the highest value among all base classes, the pixel is coloured red, and if the probability N is the highest value among all base classes, the pixel is coloured white. Thereby, a virtually stained colour image of the object, for example, a tissue specimen, is obtained.

The trained classifier can receive a labelled (that is chemically stained) or unlabeled (that is chemically unstained) image of an object as input, wherein if the object (for example a tissue specimen) is labelled the colour image reconstruction provides additional contrast information to the classification. Depending on this, there are different virtual staining operations that are available, for example a virtual staining can be added, or a chemical staining can be virtually removed, or a virtual staining can be transferred, or a chemical staining ca be virtually improved (see Fig. 13 and Fig. 14 for more details). The term virtual staining of an object also implies the above and below mentioned virtual staining operations of the object.

Fig. 12b shows an exemplifying mapping of QDI pixel values to their corresponding classification. Here, a classifier that was trained with QDIs of different objects that were stained Hematoxylin and Eosin and therefore, a direct classification between a pixel value of a QDI and a class, that is Hematoxylin, Eosin or none is possible. In table 1203, for pixel values of the QDI between 0.1 and 0.2 the pixel is classified as being stained with Hematoxylin. In table 1203, for pixel values of the QDI between 0.5 and 0.8 the pixel is classified as being stained with Eosin. In table 1203, for pixel values of the QDI other than the above the pixel is classified as not being stained with either Hematoxylin or Eosin, i.e. none.

Exemplifying virtual staining operations

Fig. 13 shows different virtual staining operations that can be applied to a stained tissue specimen, input to the DIHM scanner. First a digital colour reconstruction of the scanned chemically stained (i.e. labeled) tissue specimen 1101 is reconstructed. In an ADD operation 1302, a digital representation of the scanned chemically stained tissue specimen is created, which is equivalent to a brightfield representation of any additional chemical staining applied to the same scanned stained tissue specimen. In a REMOVE operation 1303, a digital representation of the scanned chemically stained tissue specimen is created, which is equivalent to a brightfield representation of a chemically non-stained tissue specimen with increased imaging contrast. In a TRANSFER operation 1304, a digital representation of the scanned chemically stained tissue specimen is created, which is equivalent to a brightfield representation of any additional chemical staining applied to the same scanned chemically non-stained tissue specimen (that is TRANSFER = REMOVE & ADD). In an IMPROVE operation 1305, a digital representation of the scanned chemically stained tissue specimen is created, which is a highly accurate reproduction of the original scanned chemically stained tissue specimen, which is improved in terms of colour range and resolution.

Fig. 14 shows different virtual staining operations that can be applied to an unstained tissue specimen, input to the DIHM scanner. First a digital colour reconstruction of the scanned chemically non-stained (i.e. unlabeled) tissue specimen 1401 is reconstructed. In a first ADD operation 1402, a digital representation of the scanned chemically non-stained tissue specimen is created, which is equivalent to a brightfield representation of any chemical staining applied to the scanned chemically non-stained tissue specimen. In a second ADD operation 1403, a digital representation of the scanned chemically non-stained tissue specimen is created, which is equivalent to a brightfield representation of any additional chemical staining, other than the first ADD operation, applied to the scanned chemically non-stained tissue specimen. In an IMPROVE operation 1404, a digital representation of the scanned chemically non-stained tissue specimen is created, which is a highly accurate reproduction of the original scanned chemically non-stained tissue specimen, which is improved in terms of colour range and resolution.

To overcome possible resentments and accelerate market introduction of the proposed system one potential embodiment of our proposed system comprises a compatibility mode (IMPROVE), outputting the color image of a specimen as a highly accurate reproduction of the original specimen e.g. in terms of color range and resolution

Computer-implementation of the processes

Fig. 15 schematically describes an embodiment of an electronic device which may implement the functionality of the process steps described above. The electronic device 1500 comprises a CPU 1501 as processor. The electronic device 1500 further comprises a GPU 1506 that is connected to the processor 1501. The electronic system 1500 further comprises an Ethernet interface 1504 which acts as interface for data communication with external devices, as for example a DIHM scanner. The DHIM scanner can also be connected to the electronic device with other standard connection buses, like USB. The electronic device 1500 further comprises a data storage 1502 and a data memory 1503 (here a RAM). The data memory 1503 is arranged to temporarily store or cache data or computer instructions for processing by the processor 1501. The data storage 1502 is arranged as a long-term storage, e.g., for recording a scanned hologram or the labelled (in the sense of labelled for supervised learning) data which is necessary the supervised learning of the classification algorithm. The data storage 1502 and the data memory 1503 may comprise computing instructions that implement the processes described above, e.g. a process of recording and storing a scanned hologram of an object. The computing instructions may further implement a process of obtaining a refocused object-detector-distance and of reconstructing the object plane waves based on the hologram of the object to obtain phase and amplitude information of the object. The computing instructions may further implement the functionality of calculating a colour image, QPIs and a QDI of the object. The computing instructions may further implement the process of training a classification algorithm and the performing virtual staining operations as described above. ***

It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is, however, given for illustrative purposes only and should not be construed as binding. For example, steps 801, 802, 583, or steps 804, 806, 807 in Fig. 8, or the other figures could be exchanged.

In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

Note that the present technology can also be configured as described below:

(1) An electronic device comprising circuitry configured to determine a refocused object- detector distance (max -distance) based on N holograms captured at N different preset object- detector distances (z₁ ... , z₁ + (N—1)z_step) by choosing an optimal joint refocusing result (sobel_joint(J) based on N reconstructed object-plane wave functions (t(x, y, z_1,j,k)) corresponding to the N holograms.

(2) The electronic device of (1), wherein the circuitry is further configured to reconstruct the N object-plane wave functions t(x, y, z_1,j,k) corresponding to the holograms based on N assumed object-detector distances ( z_1,j,k).

(3) The electronic device of (1) or (2), wherein the N preset different object-detector distances (z₁; ... , z₁ + ( — 1)z_step) are N preset distances (z₁; ...,z₁ + (N—1)z_step) between an object (102) in an object plane and an shiftable image sensor (104) configured to capture the N holograms of the object (102).

(4) The electronic device of anyone of (1) to (3), wherein the circuitry is further configured to determine, for each object-plane wave function (t(x, y, z_1,j,k)) a refocusing result

(S(t(x, y, z_1,j,k)) based on an assumed object-detector distance (z_{1,j, k}).

(5) The electronic device of anyone of (1) to (4), wherein the circuitry is further configured to determine M joint refocusing results (sobel_joint(j)) for M respective assumed object-stack distances (current -distance (j)), and to choose the optimal joint refocusing result

(sobel_joint(J)) from the M joint refocusing results (sobel_joint(j)). (6) The electronic device of (5), wherein the M assumed object-stack distances

(current_distance (k)) are located at an equidistant spacing (step_size_layer) within a search range.

(7) The electronic device of (6), wherein the choosing of an optimal joint refocusing result (sobel_joint(J)) is performed in an iterative way, and wherein, at each level of iteration (current Jayer), the equidistant spacing (step_size_layer) is refined.

(8) The electronic device anyone of (4) to (7), wherein the circuitry is further configured to determine a joint refocusing result (sobel_joint(j)) by combining N different refocusing results (S(t(x, y, z_1,j,k)), each refocusing result (S(t(x, y, z_1,j,k)), being related to a respective one of N assumed object-detector distances (z_1,j,k).

(8) The electronic device of anyone of (6) to (8), wherein the circuitry is further configured to determine N assumed object-detector distances based on an assumed object-stack distance current -distance (j)) and based on the preset object-detector distances (z₁; ... , z₁ + (N-1)z_step).

(10) The electronic device of anyone of (4) to (9), wherein the circuitry is further configured to determine N • M refocusing results (S(t(x, y, z_1,j,k)) based on N • M assumed object-detector distances (z_1,j,k).

(11) The electronic device of (10), wherein the N • M assumed object-detector distances (z_1,j,k) are obtained by determining M distances in proximity around each of the N different preset object-detector distances (z₁; ..., z₁ + (N—1)z_step).

(12) The electronic device of (11), wherein the N • M assumed object-detector distances (z_1,j,k) are obtained by determining M equidistant distances centered around each of the N different preset object-detector distances (z₁; ... ,z₁ + (N—1)z_step).

(13) The electronic device of anyone of (5) to (12), wherein the circuitry is further configured to choose the optimal joint refocusing result (sobel_joint(J)) from the M joint refocusing results (sobel_joint(j)).by comparing the M joint refocusing results (sobel_joint(j)) to each other.

(14) The electronic device of (13), wherein the circuitry is further configured to determine the refocused object-detector distance (max -distance) as the assumed object-detector distance corresponding to the greatest joint refocusing result (sobel_joint(J)). (15) The electronic device of anyone of (4) to (14), wherein the joint refocusing result (sobel_joint(j)) is determined by combining the N refocusing results of the N reconstructed object-plane wave functions (t(x,y,z_1,j,k)) corresponding to the N holograms.

(16) The electronic device of (15), where the joint refocusing result (sobel_joint(j)) is a weighted sum (w_j,k) of the N refocusing results.

(17) The electronic device of (15) or (16), where the joint refocusing result (sobel_joint (k)) is a joint Sobel metric (sobel_joint(j)) and the N refocusing results (S(t(x,y,z_1,j,k)) are added together by weights (w_j,k) to obtained to the joint refocusing result (sobel_joint (J )) as where sobel_joint (J is the joint Sobel metric and W_j,k are the weights, k = 1, ... N is an index for the N holograms and j = 1, ... , M is an index for the number of joint refocusing results.

(18) The electronic device of (16) or (17), where the weights (w_j,k) of the weighted sum (WjJ) are equal.

(19) The electronic device of anyone of (16) to (18), where the weights (w_j,k) of the weighted sum (w_j,k) are not equal.

(20) The electronic device of (19), where the non-equal weights (w_j,k) are derived from an absolute peak height, a peak significance, an average image or an intensity.

21. The electronic device of claim 4, wherein the refocusing result (S(t(x,y,z_1,j,k)) for the object-plane wave functions (t(x,y,z_1,j,k) is a Sobel magnitude metric (S(t(x,y,z_1,j,k)).

(22) The electronic device of anyone of (4) to (21), wherein the reconstructing of an object- plane wave functions (t(x,y,z_1,j,k)) is performed iteratively.

(23) The electronic device of anyone of (1) to (22), wherein the reconstructing of an object- plane wave functions (t(x,y,z_1,j,k)) is based on an angular spectrum method.

(24) The electronic device of anyone of (1) to (23), wherein the N holograms are captured by a digital in-line holography microscopy scanner (100).

(25) The electronic device of anyone of (1) to (24), wherein the N holograms are captured with a first illumination light wavelength (λ_blue), a second N holograms are captured with a second illumination light wavelength (λ_green) and a third N holograms are captured with a third illumination light wavelength (λ_red). (26) A method comprising determining a refocused object-detector distance (max _distance) based on N holograms captured at N different preset object-detector distances (z₁; ... ,z₁ + (N—1)z_step) by determining a joint refocusing result (sobel_joint(k)) based on N reconstructed object-plane wave functions t(x,y,z_1,j,k) corresponding to the N holograms. (27) A computer program comprising instructions which are configured to, when executed on a processor, perform the method of claim 26.

Claims

1. An electronic device comprising circuitry configured to determine a refocused object- detector distance based on N holograms captured at N different preset object-detector distances by choosing an optimal joint refocusing result based on N reconstructed object-plane wave functions corresponding to the N holograms.

2. The electronic device of claim 1, wherein the circuitry is further configured to reconstruct the N object-plane wave functions corresponding to the N holograms based on N assumed object-detector distances.

3. The electronic device of claim 1, wherein the N preset different object-detector distances are N preset distances between an object in an object plane and a shiftable image sensor configured to capture the N holograms of the object.

4. The electronic device of claim 1, wherein the circuitry is further configured to determine, for each object-plane wave function a refocusing result based on an assumed object-detector distance.

5. The electronic device of claim 1, wherein the circuitry is further configured to determine M joint refocusing results for M respective assumed object-stack distances, and to choose the optimal joint refocusing result from the M joint refocusing results.

6. The electronic device of claim 5, wherein the M assumed object-stack distances are located at an equidistant spacing within a search range.

7. The electronic device of claim 6, wherein the choosing of an optimal joint refocusing result is performed in an iterative way, and wherein, at each level of iteration, the equidistant spacing is refined.

8. The electronic device of claim 4, wherein the circuitry is further configured to determine a joint refocusing result by combining N different refocusing results each refocusing result, being related to a respective one of N assumed object-detector distances.

9. The electronic device of claim 6, wherein the circuitry is further configured to determine N assumed object-detector distances based on an assumed object-stack distance and based on the preset object-detector distances.

10. The electronic device of claim 4, wherein the circuitry is further configured to determine N • M refocusing results based on N • M assumed object-detector distances.

11. The electronic device of claim 10, wherein the N • M assumed object-detector distances are obtained by determining M distances in proximity around each of the N different preset object-detector distances.

12. The electronic device of claim 11, wherein the N • M assumed object-detector distances are obtained by determining M equidistant distances centered around each of the N different preset object-detector distances.

13. The electronic device of claim 5, wherein the circuitry is further configured to choose the optimal joint refocusing result from the M joint refocusing results by comparing the M joint refocusing results to each other.

14. The electronic device of claim 13, wherein the circuitry is further configured to determine the refocused object-detector distance as the assumed object-detector distance corresponding to the greatest joint refocusing result.

15. The electronic device of claim 4, wherein the joint refocusing result is determined by combining the N refocusing results of the N reconstructed object-plane wave functions corresponding to the N holograms.

16. The electronic device of claim 15, where the joint refocusing result is a weighted sum of the N refocusing results.

17. The electronic device of claim 15, where the joint refocusing result is a joint Sobel metric and the N refocusing results are added together by weights to obtain the joint refocusing result as where sobel_joint(j) is the joint Sobel metric and w_j,k are the weights, k = 1, ... N is an index for the N holograms and j = 1, ... , M is an index for the number of joint refocusing results.

18. The electronic device of claim 16, where the weights of the weighted sum are equal.

19. The electronic device of claim 16, where the weights of the weighted sum are not equal.

20. The electronic device of claim 19, where the non-equal weights are derived from an absolute peak height, a peak significance, an average image or an intensity.

21. The electronic device of claim 4, wherein the refocusing result for the object-plane wave functions is a Sobel magnitude metric.

22. The electronic device of claim 1, wherein the reconstructing of an object-plane wave functions is performed iteratively.

23. The electronic device of claim 1, wherein the reconstructing of an object-plane wave functions is based on an angular spectrum method.

24. The electronic device of claim 1, wherein the N holograms are captured by a digital in- line holography microscopy scanner.

25. The electronic device of claim 1, wherein the N holograms are captured with a first illumination light wavelength, a second N holograms are captured with a second illumination light wavelength and a third N holograms are captured with a third illumination light wavelength.

26. A method comprising determining a refocused object-detector distance based on N holograms captured at N different preset object-detector distances by determining a joint refocusing result based on N reconstructed object-plane wave functions corresponding to the N holograms.

27. A computer program comprising instructions which are configured to, when executed on a processor, perform the method of claim 26.