WO2021198243A1 - Procédé de coloration virtuelle d'un échantillon de tissu et dispositif d'analyse de tissu - Google Patents

Procédé de coloration virtuelle d'un échantillon de tissu et dispositif d'analyse de tissu Download PDF

Info

Publication number
WO2021198243A1
WO2021198243A1 PCT/EP2021/058272 EP2021058272W WO2021198243A1 WO 2021198243 A1 WO2021198243 A1 WO 2021198243A1 EP 2021058272 W EP2021058272 W EP 2021058272W WO 2021198243 A1 WO2021198243 A1 WO 2021198243A1
Authority
WO
WIPO (PCT)
Prior art keywords
tissue sample
tissue
analysis
imaging data
tissue samples
Prior art date
Application number
PCT/EP2021/058272
Other languages
English (en)
Inventor
Alexander Freytag
Christian KUNGEL
Matthias EIBL
Johannes Kindt
Original Assignee
Carl Zeiss Ag
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Carl Zeiss Ag filed Critical Carl Zeiss Ag
Publication of WO2021198243A1 publication Critical patent/WO2021198243A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30024—Cell structures in vitro; Tissue sections in vitro

Definitions

  • the present invention relates to a method for virtually staining a tissue sample and a device for tissue analysis.
  • Histopathology is an important tool in the diagnosis of a disease.
  • histopathology refers to the optical examination of tissue samples.
  • histopathological examination starts with surgery, biopsy, or autopsy for obtaining the tissue to be examined.
  • the tissue may be processed to remove water and to prevent decay.
  • the processed sample may then be embedded in a wax block. From the wax block, thin sections may be cut. Said thin sections may be referred to as tissue samples hereinafter.
  • the tissue samples may be analysed by a histopathologist in a microscope.
  • the tissue samples may be stained with a chemical stain to facilitate the analysis of the tissue sample.
  • chemical stains may reveal cellular components, which are very difficult to observe in the unstained tissue sample.
  • chemical stains may provide contrast.
  • H&E haematoxylin and eosin
  • tissue samples By colouring tissue samples with chemical stains, otherwise almost transparent and indistinguishable sections of the tissue samples become visible for the human eye. This allows pathologists and researchers to investigate the tissue sample under a microscope or with a digital bright-field equivalent image and assess the tissue morphology (structure) or to look for the presence or prevalence of specific cell types, structures or even microorganisms such as bacteria.
  • the known chemical staining techniques are labour- and cost-intensive.
  • WO 2019 / 154987 A1 discloses a method providing a virtually stained image looking like a typical image of a tissue sample which has been stained with a conventional chemical stain.
  • Providing a virtually stained image of a tissue sample requires time for acquiring the required digital imaging data and for the calculation of the output image comprising the virtual stain.
  • tissue samples may relate to thin sections of the wax block comprising an embedded processed sample as described hereinbefore.
  • tissue sample may also refer to tissue having been processed differently or not having been processed at all.
  • tissue sample may refer to a part of tissue observed in vivo and/or tissue excised from a human, an animal or a plant, wherein the observed tissue sample has been further processed ex vivo, e.g., prepared using a frozen section method.
  • a tissue sample may be any kind of a biological sample.
  • tissue sample may also refer to a cell, which cell can be of procaryotic or eucaryotic origin, a plurality of procaryotic and/or eucaryotic cells such as an array of single cells, a plurality of adjacent cells such as a cell colony or a cell culture, a complex sample such as a biofilm or a microbiome that contains a mixture of different procaryotic and/or eucaryotic cell species and/or an organoid.
  • a cell which cell can be of procaryotic or eucaryotic origin, a plurality of procaryotic and/or eucaryotic cells such as an array of single cells, a plurality of adjacent cells such as a cell colony or a cell culture, a complex sample such as a biofilm or a microbiome that contains a mixture of different procaryotic and/or eucaryotic cell species and/or an organoid.
  • It is proposed a method for virtually staining a tissue sample comprising selecting a virtual stain, obtaining digital imaging data of the tissue sample, wherein a digital imaging data of the tissue sample has been acquired using one or more image modalities, determining a region of interest (ROI) of the tissue sample, and providing an output image depicting the tissue sample comprising the virtual stain only in the ROI.
  • ROI region of interest
  • chemical staining may also comprise modifying molecules of any one of the different types of tissue sample mentioned above.
  • the modification may lead to fluorescence under a certain illumination (e.g., an illumination under ultra-violet (UV) light).
  • chemical staining may include modifying genetic material of the tissue sample.
  • Chemically stained tissue samples may comprise transfected cells. Transfection may refer to a process of deliberately introducing naked or purified nucleic acids into eukaryotic cells. It may also refer to other methods and cell types. It may also refer to non-viral DNA transfer in bacteria and non-animal eukaryotic cells, including plant cells.
  • Modifying genetic material of the tissue sample may make the genetic material observable using a certain image modality.
  • the genetic material may be rendered fluorescent.
  • modifying genetic material of the tissue sample may cause the tissue sample to produce molecules being observable using a certain image modality.
  • modifying genetic material of the tissue sample may induce the production of fluorescent proteins by the tissue sample.
  • the dimensionality of the digital imaging data of the tissue sample may vary.
  • the digital imaging data may be two-dimensional (2-D), one-dimensional (1- D) or even three-dimensional (3-D). If more than one image modality is used for obtaining digital imaging data, a part of the digital imaging data may be two-dimensional and another of the digital imaging data may be one-dimensional or three-dimensional.
  • microscopy imaging may provide digital imaging data that includes images having spatial resolution, i.e., including multiple pixels. Scanning through the tissue sample with a confocal microscope may provide digital imaging data comprising three-dimensional voxels.
  • Spectroscopy of the tissue sample may result in digital imaging data providing spectral information of the whole tissue sample without spatial resolution.
  • spectroscopy of the tissue sample may result in digital imaging data providing spectral information for several positions of the tissue sample which results in imaging data comprising spatial resolution but being sparsely sampled.
  • the region of interest may correspond to a contiguous area of the tissue sample or separate areas of the tissue sample.
  • the region of interest may have a well-defined form, e.g., a rectangle. In other embodiments, the region of interest may have an arbitrarily shaped form.
  • Providing an output image depicting the tissue sample comprising the virtual stain only in the region of interest may significantly reduce the time required for providing the output image.
  • a completely virtually stained tissue sample is not required for establishing a diagnosis.
  • the proposed method may allow for providing different virtual stains in different regions of interest of the tissue sample at once. Showing different virtual stains in different regions of interest of the tissue sample at once may allow for a better and more reliable diagnosis. Alternatively or in addition, several output images may be provided depicting the very same tissue sample but with different virtual stains in the region of interest. Providing different chemical stains in the same region of interest of the same tissue sample may not be possible because the tissue sample typically cannot be chemically destained after the first chemical stain has been applied to the tissue sample.
  • providing the output image comprises obtaining further digital imaging data of a tissue sample, wherein the further digital imaging data corresponds to the region of interest or a second region of interest, processing the further digital data in a or the machine-learning logic, and obtaining, from the machine-learning logic, the output image.
  • the region of interest in which the virtual stain is to be applied may differ from the second region of interest in which further digital data is to be obtained.
  • the region of interest in which the virtual stain is to be applied may be larger than the second region of interest for which further digital data is acquired. Similar methods may be applied for determining the region of interest and the second region of interest.
  • the digital imaging data of the tissue sample may be sufficient to determine a region of interest.
  • the further digital imaging data of the tissue sample may improve the virtual staining accuracy in the region of interest. Obtaining the further digital imaging data of the tissue sample only after determining the region of interest may help to reduce the amount of data transmissions for providing the output image.
  • the further digital imaging data may have been acquired using an image modality different from the one or more image modalities used for acquiring the digital imaging data.
  • the digital imaging data may have been acquired using an image modality allowing for a fast acquisition.
  • the digital imaging data may have been acquired using wide-field microscopy.
  • the image modality used for acquiring the digital imaging data may allow for identifying several unknown regions of interest of the tissue sample.
  • the digital imaging data may be sufficient for identifying the region of interest, but not sufficient to provide a virtual stain in the region of interest with the intended virtually staining accuracy.
  • the image modality for the further digital imaging data may refer to a higher resolution of the digital imaging data in the region of interest.
  • the image modality may refer to a different acquisition technique.
  • the image modality for obtaining further digital imaging data may refer to a Raman spectroscopy.
  • the different image modality may refer to an acquisition technique using a different spectral band.
  • the region of interest may refer to a region of the tissue sample, in which region a similar response from the further imaging modality is expected.
  • the imaging modality may provide further digital imaging data being one-dimensional. For example, a molecular composition within a cell nucleus may be assumed to be homogenously distributed and the same spectral response may be expected when spatially probing the cell nucleus using an image modality which is spectral sensitive.
  • the image modalities for acquiring digital imaging data of tissue samples comprise images of the tissue samples in a specific spectral band.
  • a first image modality may refer to an image of the tissue samples in a spectral band corresponding to our red color in the visible spectrum
  • a different image modality for acquiring digital imaging data may refer to an image of the tissue sample in a different spectral band, for example a spectral band corresponding to green color in the visible spectrum.
  • a hyperspectral scanner may be used for acquiring images of the tissue samples in one or more spectral bands.
  • the spectral bands are not limited to spectral bands in the visible spectrum but may also comprise spectral bands in the ultraviolet, and infrared range.
  • the image modalities for acquiring digital imaging data of tissue samples may also comprise a Raman analysis of the tissue samples.
  • the imaging modalities may comprise some simulated Raman scattering (SRS) analysis of the tissue samples, coherent anti-stokes Raman scattering (CARS) analysis of the tissue samples, surface enhanced Raman scattering (SERS) analysis of the tissue samples.
  • the image modalities may also comprise fluorescence lifetime imaging microscopy (FLIM) analysis of the tissue samples.
  • FLIM fluorescence lifetime imaging microscopy
  • the image modalities may also comprise a phase sensitive analysis of the tissue samples. Yet a further example would be transmitted- light or reflected-light microscopy, e.g., for observing cells.
  • Imaging modalities may, as a general rule, imaging tissue in-vivo or ex-vivo.
  • An endoscope may be used to acquire images in-vivo, e.g., a confocal microscope or using endoscopic optical coherence tomography (e.g., scanned or full-field).
  • a confocal fluorescence scanner could be used.
  • Endoscopic two-photon microscopy would be a further imaging modality.
  • a surgical microscope may be used; the surgical microscope may, itself provide for multiple imaging modalities, e.g., microscopic images or fluorescence images, e.g., in specific spectral bands or combinations of two or more wavelengths, or even hyperspectral images.
  • the image modalities may comprise poorer realization sensitive analysis of the tissue samples.
  • the image modalities comprise a DCI technology analysis of the tissue sample.
  • Dynamic cell imaging may refer to measuring cell metabolism as phase changes with a phase sensitive full field optical coherence tomography setup.
  • DCI technology analysis may be provided by LLTech Inc. (http://lltech.co/the-biopsy- scanner/our-technology).Thus, several different image modalities may be used for optimizing the virtual staining of tissue samples.
  • providing digital imaging data of the tissue sample comprises providing digital imaging data of the tissue sample after chemical staining of the tissue sample.
  • determining regions of interest may be facilitated if the tissue sample has been stained with a chemical stain.
  • the chemical stain for staining the tissue sample may be a stain which is regularly applied to tissue samples.
  • the chemical stain may be an H&E stain.
  • the process for applying an H&E stain may be well-established, fast and cheap.
  • it may be easier to perform virtual staining based on a chemically stained tissue sample.
  • chemically staining may allow for acquiring digital imaging data which cannot be acquired from an unstained tissue sample.
  • chemically staining may render spatial differences in the chemical composition of the tissue sample observable. Using already chemically stained samples for determining regions of interest may avoid the need to prepare unstained tissue samples specially for the proposed method. Thus, laboratories preparing the tissue samples may not have to deviate from established workflows.
  • obtaining an output image depicting the tissue sample comprises selecting a subset of the digital imaging data of the tissue sample, wherein the subset corresponds to the region of interest, processing the subset in a machine-learning logic, and obtaining, from the machine-learning logic, the output image.
  • Using only a subset of the available digital imaging data may significantly reduce the time for processing the digital imaging data in the machine-learning logic. Moreover, it may reduce the amount of memory the machine-learning logic requires for performing the processing.
  • determining a region of interest comprises processing the digital imaging data in a or the machine-learning logic, obtaining, from the machine-learning logic, a preliminary output image comprising a preliminary virtual stain of the tissue sample, processing the preliminary output image in an ROI machine-learning logic, and obtaining, from the ROI machine-learning logic, an ROI.
  • determining an ROI comprises processing the digital imaging data in an ROI machine-learning logic, and obtaining, from the ROI machine-learning logic, the ROI.
  • the ROI may be determined based on the original digital imaging data or based on a preliminary output image comprising a preliminary virtual stain of the tissues.
  • determining the ROI is performed by an expert in histopathology.
  • the determining of the ROI is performed by in an automated manner.
  • the determining of the Rl may be performed by an ROI machine-learning logic.
  • the ROI machine-learning logic may comprise a Single Shot MultiBox Detector (SSD) or similar object detection model.
  • SSD Single Shot MultiBox Detector
  • an object detection model may be used with is disclosed in Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016).
  • Annotations for training data may be obtained by asking medical experts to mark regions relevant for diagnosis within the digital imaging data of the tissue samples. Said regions may be virtually stained. In another embodiment, all regions that have been stained with a chemical stain or a preliminary virtual stain may be determined to correspond to regions of interest for training the ROI machine-learning logic. In a weakly supervised setting, a flag or scalar specifying that this image part contains some regions where the virtual stain is sensitive to, may be used.
  • a segmentation model may be used for implementing the ROI machine-learning logic.
  • Usable segmentation models are, for example, known from Ronneberger, O.; P. Fischer & Brox, T., “U-Net: Convolutional Networks for Biomedical Image Segmentation”, Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015, Jegou, S.; Drozdzal, M.; Vazquez, D.; Romero, A. & Bengio, Y., “The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation”, Lin, G.; Milan, A.; Shen, C. & Reid, I.
  • U-Net like models comprising an encoder and a decoder may be used.
  • U-Net like models with or without skip connections may be used.
  • refined nets and/or dense nets may be used.
  • a technique proposed by Jegou et al. "The One Hundread Layers Tiramisu: Fully Convolutional DenseNets for Segmantic Segmentation", arXiv:1611:09326v3 may applied.
  • the models may be applied to sub-crops of the digital imaging data which are later stitched together or blended.
  • the ROI machine-learning logic may implement tiled interference.
  • a certain region of the digital imaging data contains enough pixels which are predicted as stain-sensitive for the virtual stain then this region may be selected as region of interest for virtually staining.
  • Annotations for training data may be obtained similarly to the methods described above. For semantic segmentation, it may not be required to use geometric primitives for annotating training data. In embodiments, it may be sufficient to have weakly annotated training data in place for semi-supervised training.
  • the ROI machine-learning logic may also be possible to train the ROI machine-learning logic to detect regions of interest by using tissue samples that do not contain a region of interest. With respect to cancer diagnosis, this may involve training the ROI machine-learning logic with tissue samples not containing a cancerous region.
  • the ROI machine-learning logic may be trained based on regions for which no annotation of ROIs has been provided. This may be performed using unsupervised techniques like autoencoders.
  • auto encoders variational auto encoders with a clear bottleneck may be used. Said techniques may result in a compact basis representation for healthy tissue samples.
  • a new tissue sample may be processed by the trained ROI machine-learning logic and a difference in reconstruction may be calculated. All regions of the tissue sample with strong reconstructions errors may be potentially ROIs on which a virtual stain should be applied.
  • the output image comprises the preliminary virtual stain in addition to the virtual stain in the region of interest.
  • both virtual stains may provide the expert with important information for diagnosis.
  • an embodiment of the method may prescribe that determining the ROI comprises identifying regions of the tissue sample with meaningful objects.
  • determining the ROI may comprise identifying regions of the tissue samples, to which the virtual stain is sensitive.
  • the device for tissue analysis comprising a processor configured to perform one of the above-disclosed methods.
  • the device for tissue analysis may include an image acquisition system configured for acquiring digital imaging data of the tissue sample using one or more image modalities.
  • the image acquisition system may be configured for selectively illuminating regions of the tissue sample.
  • the image acquisition system may comprise at least one of a spatial light modulator (SLM) and a digital mirror devise (DMD) for selectively illuminating regions of the tissue sample.
  • SLM spatial light modulator
  • DMD digital mirror devise
  • a computer program comprising instructions which, when the program is executed by a computer, causes the computer to carry out one of the above-described methods.
  • FIG. 1 shows a workflow for staining a tissue sample
  • Fig. 2 shows a simplified flowchart for illustrating a method for virtually staining a tissue sample
  • Fig. 3 shows a method for virtually staining a tissue sample
  • Fig. 4 illustrates a further method for virtually staining a tissue sample
  • Fig. 5 shows a further method for virtual staining of a tissue sample
  • Fig. 6 shows a section of an exemplary tissue sample comprising ROIs
  • Fig. 7 is a flowchart of a method according to various examples, the method enabling inference of one or more output images depicting a tissue sample including one or more virtual stains;
  • Fig. 8 schematically illustrates a tissue sample and multiple sets of imaging data depicting the tissue sample according to various examples
  • Fig. 9 schematically illustrates a machine-learning logic according to various examples
  • Fig. 10 schematically illustrates an example implementation of the machine-learning logic according to various examples
  • Fig. 11 schematically illustrates an example implementation of the machine-learning logic according to various examples
  • Fig. 12 schematically illustrates an example implementation of the machine-learning logic according to various examples
  • Fig. 13 is a flowchart of a method according to various examples, the method enabling training of a machine-learning logic for virtual staining according to various examples;
  • Fig. 14 schematically illustrates aspects with respect to the training of the machine-learning logic according to various examples;
  • Fig. 15 schematically illustrates a method enabling training of and using a virtual staining logic.
  • Fig. 1 illustrates aspects with respect to a workflow for generating images depicting a tissue sample including a stain, e.g., a chemical stain or a virtual stain.
  • Fig. 1 schematically illustrates an example of a histopathology workflow.
  • virtual staining can also be applied in other use cases than histopathology.
  • different workflows for generating images can be applicable. For instance, for fluorescence imaging of cells, tissue samples including cell samples may be otherwise acquired and imaged in a respective microscope. Also, in-vivo imaging using an endoscope would be a possible use case for generating imaging data of tissue samples.
  • tissue 2102 may be obtained from a living creature 2101 by surgery, biopsy or autopsy. After some processing steps to remove water and to prevent decay, said tissue 2102 may be embedded in a wax block 2103. From said block 2103, a plurality of slices 2104 may be obtained for further analysis. One slice of said plurality of slices 2104 may also be called a tissue sample 2005.
  • the tissue could also include cell samples or in-vivo inspection using, e.g., a surgical microscope or an endoscope.
  • a chemical stain may be applied to the tissue sample 2005 to obtain a chemically stained tissue sample 2006.
  • Said chemical stain may be an H&E stain.
  • the tissue sample 2005 may also be directly analysed.
  • a chemically stained tissue sample 2006 may facilitate the analysis.
  • chemical stains may reveal cellular components, which are very difficult to observe in the unstained tissue sample 2005.
  • chemical stains may provide an increased contrast.
  • Applying a chemical stain may include a-priori transfecting or direct application of a fluorophore such as 5-ALA.
  • tissue sample 2005 or 2006 is analysed by an expert using a bright field microscope 2107.
  • Image modalities may comprise images of the tissue sample in one or more specific spectral bands, in particular, spectral bands in the ultra violet, visible and/or infrared range.
  • Image modalities may also comprise a Raman analysis of the tissue samples, in particular a stimulated Raman scattering (SRS) analysis of the tissue sample, a coherent anti-Stokes Raman scattering, CARS, analysis of the tissue sample, a surface enhanced Raman scattering, SERS, analysis of the tissue sample.
  • SRS stimulated Raman scattering
  • the image modalities may comprise a fluorescence analysis of the tissue sample, in particular, fluorescence lifetime imaging microscopy. FLIM, analysis of the tissue sample.
  • the image modality may prescribe a phase sensitive acquisition of the digital imaging data.
  • the image modality may also prescribe a polarization sensitive acquisition of the digital imaging data.
  • the digital imaging data 2109 may be processed in a device for tissue analysis 2110.
  • the device for tissue analysis 2110 may be a computer.
  • the device for tissue analysis 2110 may comprise memory 2111 for (temporarily) storing the digital imaging data 2109 and a processor 2112 for processing the digital imaging data 2109.
  • the device for tissue analysis 2110 may process the digital imaging data 2109 to provide one or more output pictures 2113 which may be displayed on a display 2114 to be analysed by an examiner.
  • the device for tissue analysis 2110 may comprise different types of trained or untrained machine-learning logic for analysing the tissue sample 2105 or the chemically stained tissue sample 2106.
  • the image acquisition system 2108 may be used for providing training data for said machine-learning logic.
  • the output pictures 2113 may depict the tissue sample 2105 with one or more virtual stains.
  • Fig. 2 shows a simplified flowchart for illustrating a method for virtually staining a tissue sample.
  • the tissue sample 2201 is analysed using an image acquisition system 2202.
  • the image acquisition system 2202 may be configured for acquiring digital imaging data 2203 using one or more image modalities.
  • the image acquisition system 2202 may be configured for acquiring a wide-field microscope image of the tissue sample 2201. Said image modality may allow for a particularly fast acquisition of digital imaging data of the whole tissue sample 2201.
  • the digital imaging data may then be processed by the device for tissue analysis 2204.
  • the device for tissue analysis 2204 may comprise an ROI machine-learning logic to determine a region of interest 2205 of the tissue sample 2201. Said region of interest 2205 may be analysed by the image acquisition system 2202 or another image acquisition system (not shown) using an additional image modality to obtain further digital imaging data 2206 in the region of interest 2205. The acquisition of the further digital imaging data 2206 using the additional image modality may take more time than the acquisition of the digital imaging data 2203.
  • the digital imaging data 2203 and the further digital imaging data 2206 may be processed by the device for tissue analysis 2204 to obtain an output image 2207 depicting the tissue sample 2201, wherein the output image 2207 comprises a virtual stain 2208 illustrated by a crosshatch pattern only in the region of interest 2205.
  • the virtual stain 2208 may emphasis structures which may be particularly important for diagnosis purposes. Thus, important information may be obtained without the need to acquire the further digital imaging data 2206 for the whole tissue sample 2201. Hence, the total time required for obtaining an output image 2208 comprising the relevant information for diagnosis purposes may be reduced.
  • Fig. 3 shows a further method for virtually staining a tissue sample.
  • Digital image training data 2303 of a chemically stained tissue sample may be obtained from an acquisition system.
  • the chemical stain may be a cheap and well-established H&E stain.
  • the digital imaging data 2303 may be processed by an ROI machine-learning logic 2304 to determine a region of interest 2304 within the digital imaging data 2303.
  • the region of interest 2304 may be a region, wherein specific cells, cell components, nuclei or cytoplasm 2305 are present. Said elements may be identified using pattern recognition, for example.
  • a machine-learning logic 2306 may be used for processing the digital imaging data 2303.
  • the machine-learning logic 2306 may use the digital imaging data in the region of interest 2304 to obtain an output picture 2307 comprising a virtual stain 2308 only in the region of interest 2304.
  • the digital imaging data 2303 may be an output image of a machine-learning logic 2302 comprising a virtual stain instead of a chemical stain as explained herein before.
  • digital imaging data 2301 of an unstained tissue sample may be provided, which is processed by a machine-learning logic 2302 to obtain an output image of the tissue sample comprising a virtual stain.
  • Fig. 4 illustrates a further method for virtually staining a tissue sample 2400.
  • Digital imaging data 2401 of the tissue sample 2400 may be obtained using a microscope 2410 having a large field of view.
  • regions of interest 2402 and 2403 may be identified in the digital imaging data 2401 of the tissue sample 2400.
  • light from a light source 2411 may be directed to a digital mirror device (DMD) 2440 which directs the light through lenses 2432 and 2431 to the tissue sample 2400.
  • DMD digital mirror device
  • the DMD is one of several spatial light modulators (SLM) which could be used.
  • SLM spatial light modulators
  • the spatial light modulator 2440 may be located in an image plane conjugated to the tissue sample plane or in a Fourier plane.
  • the SLM 2440 may be set in such a way that only the ROIs identified herein before are illuminated on the sample.
  • the mask used for the SLM either corresponds to the identified ROIs itself (if located in a conjugated image plane) or a Fourier-transform of that said ROIs (if located in a Fourier plane).
  • the illumination light interacts with the tissue sample 2400.
  • the backscattered or emitted light for example stokes-shifted photons if the imaging modality was Raman imaging, follows the same optical path as the excitation light.
  • illumination and emission light path may be split up by a beam splitter 2415.
  • the backscattered or emitted light may then be directed onto a detector 2460.
  • the detector In the case of Raman imaging, the detector may be a spectrometer.
  • Raman imaging may provide valuable information on the chemical composition of the tissue sample in the region of interest.
  • Raman imaging may provide information on the chemical composition of a cell within the region of interest.
  • a cell itself may consist of several region or sub-compartments wherein the same chemical composition may be expected.
  • a largely homogeneous spatial distribution of molecules typical for cell nuclei may be expected.
  • Other areas with similar internal chemical composition may be the cell membrane or the cytoplasm.
  • the distribution of the chemical composition may be largely homogeneous, little to no additional information may be provided by probing the tissue sample with a high spatial resolution. Thus, combining several regions of interest that belong together not necessarily implies that information is lost. However, if several regions of interest are probed together in a single measurement, more molecules may be excited which may be beneficial for the measurement itself. A stronger signal may be expected and the time for performing Raman spectroscopy may be reduced and less individual measurements may be required.
  • a method for virtual staining a tissue sample may involve acquisition of a bright-field image with enough contrast and spatial resolution to allow for the identification of the three cell components, select a region of interest with one of the above- mentioned methods, for example semantic segmentation, and make three Raman measurements, one for each cell component, nuclei, membrane, cytoplasm, for each individual state, instead of densely sampling the whole field of view with an image modality corresponding to Raman imaging.
  • Fig. 5 shows a further method for virtual staining of a tissue sample 2500.
  • the tissue sample 2500 may comprise several regions of interest which are not known a priori.
  • digital imaging data of the tissue sample is required using an imaging modality A with corresponding costs a.
  • a decision algorithm 2502 determines whether the acquisition of further imaging modalities is beneficial based on the digital imaging data of the tissue sample 2500 obtained in step 2501.
  • further digital imaging data of the tissue sample may be acquired using an imaging modality B with costs b.
  • the further digital imaging data may be acquired only in regions of interest determined by the decision algorithm 2502.
  • the decision algorithm 2502 may determine whether the acquisition of the further digital imaging data may be beneficial for virtual staining of the tissue sample 2500. Generally, acquiring further digital imaging data with higher costs is only performed if the decision algorithm 2502 determines that it is beneficial for obtaining supplementary information which could be used for virtual staining of a tissue sample in a particular region of interest.
  • Fig. 6 shows a section of an exemplary tissue sample 2800 comprising different regions of interest ROIs 2811, 2812, 2823.
  • One square 2801 may correspond to a spatial resolution of an optical system used for acquiring two-dimensional digital imaging data of the tissue sample 2800 using a first image modality.
  • the first image modality may prescribe using a standard sampling of the tissue sample 2800 using a fixed spatial grid and scanning the tissue sample 2800 point-by-point as indicated with 2820.
  • the standard sampling may be used to determine regions of interest 2811, 2812, 2813.
  • a second image modality may prescribe obtaining further digital imaging data by probing a single point 2821 within the region of interest 2811 , wherein the single point 2821 may lie outside the fixed spatial grid of the first image modality.
  • a second scenario which may be explained using the region of interest 2812, may prescribe that further digital imaging data is obtained only for the region of interest using a second image modality, wherein the second image modality uses point scanning with the same spatial grid than the first image modality as indicated with 2822, but is, for example, sensible to a different spectral band.
  • a mask 2823 may be used to obtain further digital imaging data relating to the whole region of interest 2813 with a single measurement.
  • Fig. 15 illustrates a further method for virtually staining a tissue sample.
  • Tissue samples may be prepared for training a machine learning logic.
  • tissue samples may be excised from a human, an animal or a plant and placed on a sample holder, e.g. a petri dish (step 21111).
  • the tissue sample may refer to a cell, a plurality of cells, a plurality of adjacent cells and/or an organoid.
  • the tissue sample may be chemically stained (step 21121).
  • a chemical stain may be added to the tissue sample.
  • the chemical stain may only be observable under a certain illumination.
  • the chemical stain may be fluorescent under illumination with ultra-violet light.
  • cells of a tissue sample may be transfected (step 21122), i.e. the tissue sample may be chemically stained by transfection.
  • the tissue sample may be chemically stained by transfection.
  • genetic material of the cells may be modified to cause the cell to produce green fluorescent protein (GFP).
  • GFP green fluorescent protein
  • the tissue sample chemically stained by transfection may then be placed on a sample holder (step 21121), e.g. a petri dish.
  • imaging data of the chemically stained tissue samples may be obtained (step 21130).
  • a microscope may be used.
  • the microscope may be operated using a transmitted light technique.
  • Different imaging modalities may be used for acquiring the imaging data of the chemically stained tissue samples.
  • the chemical stain may only be observable in one or some of the imaging modalities. For example, alternatingly, imaging data using a fluorescence technique (sub-step 21131) and imaging data using a transmitted (visible) light technique (TL technique) (sub-step 21132) may be obtained.
  • the chemical stain may only be observable using the fluorescence technique.
  • the imaging data obtained using the imaging modality rendering the chemical stain observable may be used as reference imaging data and the imaging data obtained using the imaging modality not showing the chemical stain may be used as training imaging data.
  • the fluorescence images may be used as reference images and the TL images may be used as training images.
  • the training imaging data and the reference imaging data may be used to train a virtual staining logic 21190.
  • the trained machine learning logic 21190 may than be used to generate, based on imaging data obtained from an unstained tissue sample 21152, an output image 21151 depicting a tissue sample comprising a virtual stain.
  • the alternating acquisition of training imaging data and reference imaging data may facilitate training of the virtual staining logic.
  • registration of the imaging data may be easier as the position of the tissue samples may be approximately the same when changing the imaging modalities.
  • Machine learning especially deep learning, provides a data-driven strategy to solve problems.
  • Classic inference techniques are able to extract patterns from data based on hand-designed features, to solve problems; an example technique would be regression.
  • classic inference techniques heavily depend on the accurate choice for the hand-designed features, which choice depends on the designer’s ability.
  • One solution to such a problem is to utilize machine learning to discover not only the mapping from features to output, but also the features themselves. This is as training of a machine-learning logic.
  • Various techniques described herein generally relate to virtual staining of a tissue sample by utilizing a trained machine-learning logic (MLL).
  • the MLL can be implemented, e.g., by a support vector machine or a deep neural network which includes at least one encoder branch and at least one decoder branch. More specifically, according to various examples, multiple sets of imaging data can be fused and processed by the MLL. This is referred to as a multi-input scenario.
  • multiple virtually stained images can be obtained (labeled output images hereinafter), from the trained MLL; the multiple virtually stained images can depict the tissue sample including different virtual stains. This is referred to as a multi-output scenario.
  • TAB. 1 Various scenarios for input and output of the MLL
  • the MLL can generate virtual H&E (Hematoxylin and Eosin) stained images of the tissue sample, and/or virtually stained images of the tissue sample highlighting HER2 (human epidermal growth factor receptor 2) proteins and/or ERBB2 (Erb-B2 Receptor Tyrosine Kinase 2) genes.
  • H&E Hematoxylin and Eosin
  • images of cells - e.g., arranged as living or fixated cells in a multi-well plate or another suitable container - may be acquired using transmitted-light microscopy.
  • a reflected light microscope may be used, e.g., in an endoscope or as a surgical microscope. It is then possible to selectively stain certain cell organelles, e.g., nucleus, ribosomes, the endoplasmic reticulum, the golgi apparatus, chloroplasts, or the mitochondria.
  • a fluorophore (or fluorochrome, similarly to a chromophore) is a fluorescent chemical compound that can re-emit light upon light excitation.
  • Fluorophores can be used to provide a fluorescence chemical stain. By using different fluorophores, different chemical stains can be achieved. For example, a Hoechst stain would be a fluorescent dye that can be used to stain DNA.
  • Other fluorophores include 5-aminolevulinic acid (5-ALA), fluorescein, and Indocyanine green (ICG) that can even be used in-vivo. Fluorescence can be selectively excited by using light in respective wavelengths; the fluorophores then emit light at another wavelength. Respective fluorescence microscopes use respective light sources.
  • Virtual fluorescence staining may lead to fluorescence-like images through virtual staining.
  • the virtual fluorescence staining mimics the fluorescence chemical staining, without exposing the tissue to respective excitation light.
  • the one or more output images depict the tissue sample including respective virtual stains, i.e. , the output images can have a similar appearance as respective images depicting the tissue sample including a corresponding chemical stain.
  • the virtual stain can have a correspondence in a chemical stain of a tissue sample stained using a staining laboratory process.
  • Imaging data can include 2-D images or 1-D or 3-D data.
  • a tailored virtual stain or a tailored set of multiple virtual stains can be provided such that a pathologist is enabled to provide an accurate analysis.
  • multiple output images depicting the tissue samples having multiple virtual stains may be helpful to provide a particular accurate diagnosis, e.g., based on multiple types of structures and multiple biomarkers being highlighted in the multiple output images, or multiple organelles of the cells being highlighted.
  • multi-input scenarios may or may not be combined with multi-output scenarios; and likewise, multi-output scenarios may or may not be combined with multi-input scenarios.
  • Fig. 7 is a flowchart of a method 3300 according to various examples.
  • the method 3300 according to Fig. 7 may be executed may be executed by at least one processor upon loading program code from a nonvolatile memory.
  • the method facilitates virtual staining of a tissue sample.
  • Fig. 7 illustrates aspects with respect to virtual staining of a tissue sample.
  • Fig. 7 illustrates aspects with respect to obtaining multiple sets of imaging data depicting the tissue sample by using multiple imaging modalities.
  • Fig. 7 generally relates to multi-input scenario, as described above.
  • Fig. 7 also illustrates aspects with respect to fusing and processing the multiple sets of imaging data in an MLL, and then obtaining (outputting), from the MLL, at least one output image of which each one depicts the tissue sample including a respective virtual stain.
  • multiple sets of imaging data depicting a tissue sample are obtained and the multiple sets of imaging data are acquired using multiple imaging modalities.
  • FIG. 8 depicts a tissue sample 3400 and four sets of imaging data of the tissue sample 3400 acquired using four imaging modalities, i.e. , imaging data set 3401, 3402, 3403 and 3404, respectively.
  • imaging data set 3401, 3402, 3403 and 3404 can be respectively acquired using the same imaging modality but with different imaging settings or parameters, such as different (low or high) magnification levels, etc.
  • Each imaging data set 3401-3404 can include multiple instances of imaging data, e.g., multiple images taken at different positions of the sample and/or at different times.
  • the tissue sample 3400 can be a cancer tissue sample removed from a patient, a tissue sample of other animals or plants.
  • the multiple imaging modalities can be selected from the group including: hyperspectral microscopy imaging, fluorescence imaging, auto-fluorescence imaging, lightsheet imaging, digital phase contrast; Raman spectroscopy, etc. Further imaging modalities have been discussed above.
  • a spatial dimensionality of the imaging data of each set 3401-3404 may vary, e.g., 1-D or 2-D or even 3-D.
  • microscopy imaging or fluorescence imaging may provide imaging data that include images having spatial resolution, i.e., including multiple pixels.
  • Lightsheet imaging may provide 3-D voxels.
  • Raman spectroscopy it would be possible that an integral signal not possessing spatial resolution is obtained as the respective set of imaging data (corresponding to a 1-D data point); however, also scanning Raman spectroscopy is known where some 2-D spatial resolution can be provided.
  • digital phase contrast can be generated using multiple illumination directions and digital post-processing to combine images associated with each one of the multiple illumination directions. See, e.g., US 2017 / 0276 923 A1.
  • imaging modalities may, in some examples, rely on a similar physical observable, e.g., both may pertain to fluorescence imaging or microscopy, but using different acquisition parameters.
  • Example acquisition parameters could include, e.g., illumination type (e.g., brightfield versus dark field microscopy), magnification level, resolution, refresh rate, etc.
  • Hyperspectral scans help to acquire the substructure of an individual cell to identify subtle changes (morphological change of membrane, change in size of cell components,).
  • Adjacent z-slices of a tissue sample can be captured in hyperspectral scans, e.g., by scanning through the probe with a confocal microscope (e.g., a light-sheet microscope, LSM), focusing the light- sheet in LSMs to slightly different z-levels. It is possible to acquire adjacent cell information like what happens in widefield microscopy (integral acquisition).
  • a further class of imaging modalities includes molecularly sensitive methods like Raman, coherent Raman (SRS, CARS), SERS, Fluorescence imaging, FLIM, IR-lmaging. This helps to acquire chemical / molecular information.
  • Yet another technique is dynamic cell imaging to acquire cell metabolism information.
  • a further imaging modality includes phase or polarization sensitive imaging to acquire structural information through contrast changes.
  • the method 3300 optionally includes pre-processing after obtaining the multiple sets of imaging data, such as one or a combination of the following processing techniques: noise filtering; registration between imaging data of different sets of imaging data, for example, any pairs of the sets, etc.; resizing of the imaging data, etc.
  • the multiple sets of imaging data are fused and processed by an MLL.
  • the MLL has been trained using supervised learning, semi-supervised learning, or unsupervised learning. A detailed description of a method of performing training of the MLL will be explained later in connection with Fig. 13 and Fig. 14.
  • a deep neural network may be used.
  • a U-net implementation is possible. See Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation.” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.
  • the deep neural network can include multiple hidden layers.
  • the deep neural network can include an input layer and an output layer.
  • the hidden layers are arranged in between the input layer and the output layer.
  • feature channels can increase and decrease along the one or more encoder branches and the one or more decoder branches, respectively.
  • the one or more encoder branches and the one or more decoder branches are connected via a bottleneck.
  • the deep neural network can include decoder heads that include an activation function, e.g., a linear or non-linear activation function.
  • the MLL can include at least one encoder branch and at least one decoder branch.
  • the at least one encoder branch provides a spatial contraction of respective representatives of the multiple sets of imaging data
  • the at least one decoder branch provides a spatial expansion of the respective representatives of the at least one output image.
  • the fusing of the multiple sets of imaging data is implemented by concatenation or stacking of the respective representatives of the multiple sets of imaging data at at least one layer of the at least one encoder branch.
  • This may be an input layer (a scenario sometimes referred to as early fusion or input fusion) or a hidden layer (a scenario sometimes referred to as middle fusion or late fusion).
  • middle fusion it would even be possible that the fusing is implemented at the bottleneck (sometimes referred to as bottleneck fusion).
  • the connection joining the multiple encoder branches defines the layer at which the fusing is implemented.
  • it is possible that fusing of different pairs of imaging data is implemented at different positions, e.g., different layers.
  • Fig. 9 schematically illustrates the MLL 3500 implemented as a deep neural network according to various examples.
  • Fig. 9 schematically illustrates an overview of the MLL 3500.
  • the MLL 3500 includes an encoder module 3501 having at least one encoder branch, a decoder module 3503 having at least one decoder branch, and a bottleneck 3502 for coupling the encoder module 3501 and the decoder module 3503.
  • Each encoder branch of the encoder module 3501 , each decoder branch of the decoder module 3503, and the bottleneck 3502 may respectively include at least one block having at least one layer selected from the group including: convolutional layers, activation function layers (e.g., ReLU (rectified linear unit), Sigmoid, tanh, Maxout, ELU (Exponential Linear Unit), SeLU (Scaled exponential Linear Unit), Softmax and so on), downsampling layers, upsampling layers, normalization layers (e.g., batch normalization, instance normalization, group normalization, channel normalization, etc.), dropout layers, etc..
  • each layer defines a respective mathematical operation.
  • each encoder or decoder branch can include several blocks, each block usually having one or more layers, which may be named as an encoder block and a decoder block, respectively. Every block can include a single layer.
  • Encoder branches can be built from encoder blocks followed by downsampler blocks.
  • Downsampler blocks may be implemented by using max-pooling, average-pooling, or strided convolution.
  • Upsampler blocks may be implemented by using transposed-convolution, nearest neighbor interpolation, or bilinear interpolation. We also found it helpful to follow them by convolution with activations.
  • Decoder branches can be built from upsampler blocks followed by decoder blocks.
  • upsampler blocks it is possible to apply transposed-convolution, nearest neighbor interpolation, or bilinear interpolation. Especially for the latter two, it has been found that placing several convolution layers thereafter is highly valuable.
  • an example encoder or decoder block includes convolutional layers with activation layers and followed by normalization layers.
  • each encoder or decoder block may include more complex blocks, e.g., inception blocks (see, e.g., Szegedy, Christian, et al. "lnception-v4, inception-resnet and the impact of residual connections on learning.” Thirty- first AAAI conference on artificial intelligence.
  • DenseBlocks DenseBlocks, RefineBlocks, or having multiple operations in parallel (e.g., convolution and strided convolution) or having multiple operations after each other (e.g., three convolution with activation and then followed by normalization before going to downsampling), etc.
  • the encoder module 3501 is fed with the multiple sets of imaging data - e.g., the sets 3401, 3402, 3403 and 3404, cf. Fig. 8 - as input.
  • the decoder module 3503 outputs desired one or more output images 4001 depicting the tissue sample including a respective virtual stain (the output images can be labelled virtually stained images).
  • Fig. 10, Fig. 11, and Fig. 12 schematically illustrate details of three exemplary architectures of the MLL 3500.
  • Different architectures of the MLL 3500 can be used to implement different strategies for fusing the multiple sets of imaging data in a multi-input scenario.
  • different architectures of the MLL 3500 can be used to implement multi-output scenarios.
  • the architectures illustrated in Fig. 10, Fig. 11, and Fig. 12 can be combined with each other: for example, it would be possible that different strategies for fusing the multiple sets of imaging data as illustrated in Fig. 10, Fig. 11, and Fig. 12 are combined with each other, e.g., combining fusing at the input layer with fusing at a hidden layer or at the bottleneck for different sets of imaging data.
  • the MLL 3500 includes a single encoder branch 3601 , and the fusing of the multiple sets of imaging data (e.g., inputl , input2 and input3 may be any three of the four imaging data sets 3401, 3402, 3403 of Fig. 8) is implemented by concatenation at an input layer 3610 of the single encoder branch 3600.
  • the single encoder branch 3600 via the bottleneck 3502, connects to three decoder branches 3701, 3702 and 3703, respectively (albeit it would be possible to have a single decoder branch, for a single output scenario).
  • Fig. 10 is, accordingly, a multi-input multi-output scenario using a single encoder branch.
  • the MLL 3500 includes multiple encoder branches 3601 - 3604 and each one of the multiple encoder branches 3601 - 3604 is fed with a respective one of the multiple sets of imaging data 3401 - 3404.
  • Fig. 11 and Fig. 12 thus correspond to a multi-input scenario.
  • Fig. 11 is a single-output scenario
  • Fig. 12 is a multi-output scenario including multiple decoder branches 3701-3703.
  • the fusing of the multiple sets of imaging data is implemented by concatenation at at least one hidden layer of the MLL.
  • the extracted features representing the imaging data set 3403 (input3 fed into encoder branch 3603) is fused with the extracted features representing the imaging data set 3401 (inputl fed into encoder branch 3601) at a hidden layer of block 3 of the encoder branch 3601 and then the combination of such two sets of extracted features is fed into block 4 of the encoder branch 3601 for further processing to extract fused features of the combination of the two sets of features.
  • the extracted features representing the imaging data set 3402 (input2 fed into encoder branch 3602) is fused with the fused features of the imaging data sets 3401 and 3403 at a hidden layer of block 4 of the encoder branch 3601 and thereby the fused features of the three imaging data sets 3401 , 3402 and 3403 are acquired after further processing implemented by block 4 of the encoder 3601.
  • the MLL includes a bottleneck 3502 in-between the multiple encoder branches 3601 - 3604 and the at least one decoder branch 3701 - 3703.
  • the fusing of the multiple sets of imaging data 3401 - 3404 is at least partially implemented by concatenation at the bottleneck 350. This is illustrated in Fig. 11 and Fig.
  • the technique of “skip connections” disclosed by Ronneberger etc. (Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation.” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.) is adopted to the MLL 3500.
  • the skip connections provide respective feature sets of the multiple sets of imaging data to corresponding decoders of the MLL 3500 to facilitate feature decoding and pixel-wise information reconstruction on different scales.
  • the bottleneck 3502 and optionally one or more hidden layers are bypassed.
  • several blocks of the encoder branch 3601 directly couple with corresponding blocks of the decoder branches 3701 - 3703 via skip connections 3620.
  • block 1 of the encoder branch 3601 directly provides its output that represents fused features of the combination of inputl , input2 and input3 to block 1 of the decoder branches 3701 and 3702, respectively;
  • block 2 of the encoder branch 3601 directly provides its output which represents second fused features of the combination of inputl, input2 and input3 to block 2 of the decoder branch 3701 ;
  • block 3 of the encoder branch 3601 provides its output which represents third fused features of the combination of inputl, input2 and input3 to block 3 of the decoder branches 3701 and 3703, respectively.
  • Skip connections 3620 can, in particular, be used where there are multiple encoder branches 3601 - 3604: with reference to Fig. 11 and Fig. 12, the MLL 3500 includes multiple encoder branches 3601 - 3604.
  • the MLL 3500 includes skip connections 3620 to feed outputs of hidden layers of at least two of the multiple encoder branches 3601 - 3604 to inputs of corresponding hidden layers of the at least one decoder branch 3701 - 3703. For example, as shown in Fig.
  • the MLL 3500 includes one decoder branch 3701 and each block of the decoder branch 3701 is fed with outputs of corresponding blocks of at least two encoder branches among 3601 - 3604 via skip connections 3620.
  • the MLL 3500 includes multiple decoder branches 3701 - 3703 and some blocks of the three decoder branches receive outputs of corresponding blocks of at least two encoder branches among 3601 - 3604 via skip connections 3620.
  • the MLL 3500 can include skip connections 3620 to feed outputs of one or more hidden layers of at least one encoder branch to inputs of corresponding hidden layers of the multiple decoder branches.
  • At block 3303 at least one output image is obtained from the MLL 3500 and each one of the at least one output image depicts the tissue sample 3400 including a respective virtual stain.
  • the MLL 3500 includes multiple decoder branches, such as the decoder branches 3701 - 3703 shown in Fig. 10 and Fig. 12, and each one of the multiple decoder branches 3701 - 3703 outputs a respective one of the at least one output images 4001-4003 depicting the tissue sample including a respective virtual stain, e.g. outputl, output2 and output3.
  • each one of the multiple decoder branches 3701 - 3703 outputs a respective one of the at least one output images 4001-4003 depicting the tissue sample including a respective virtual stain, e.g. outputl, output2 and output3.
  • the three outputs can respectively be virtual H&E (Hematoxylin and Eosin) stained images of the tissue sample, virtually stained images of the tissue sample highlighting HER2 proteins, and virtually stained images of the tissue sample highlighting antibodies, such as anti-panCK, anti-CK18, anti-CK7, anti-TTF-1, anti-CK20/anti-CDX2, and anti-PSA/anti-PSMA, or other biomarkers.
  • the three outputs can be other virtually stained images depicting the tissue sample including different types of virtual stains.
  • the three different outputs are converted from extracted features of the multiple sets of imaging data 3401 - 3404 by the three decoder branches 3701 - 3703, respectively.
  • the MLL 3500 By using multiple sets of imaging data acquired using multiple imaging modalities as input of the MLL 3500, more relevant information of the tissue sample can be extracted by the at least one encoder branch and correlations across the multiple sets of imaging data are taken into account. Thereby; the at least one decoder branch can generate more precise and robust virtually stained images of the tissue sample.
  • the MLL 3500 with multiple encoder branches can extract, via a specific encoder branch, specific information of the tissue sample from each individual set of imaging data acquired using a specific imaging modality, and thereby provide imaging modality-dependent information to the at least one decoder branch.
  • the MLL 3500 can output reliable and accurate virtually stained images.
  • the MLL 3500 with multiple decoder branches can output relevant virtually stained images of the tissue sample at once.
  • the MLL 3500 with multiple decoder branches can facilitate reduced computational resources: As intermediate computation results are intrinsically shared within the MLL 3500, the number of logic operations are reduced, e.g., if compared to a scenario in which multiple MLLs are used to obtain output images depicting the tissue sample at different virtual stains.
  • the MLL 3500 Prior to enabling inference using the MLL 3500, the MLL 3500 is trained. As a general rule, various options are available for training the MLL 3500. For instance, supervised or semi-supervised learning or even unsupervised learning would be possible. Details with respect to the training are explained in connection with Fig. 13. These techniques can be used for training the MLL 3500 described above.
  • Fig. 13 is a flowchart of a method 3900 according to various examples.
  • Fig. 13 illustrates aspects with respect to training the MLL 3500.
  • the method 3900 according to Fig. 13 may be executed by at least one processor upon loading program code from a nonvolatile memory.
  • the method facilitates virtual staining of a tissue sample.
  • the MLL 3500 can be trained using supervised learning, such as the method shown in Fig. 13.
  • the method 3900 is for performing a training of the MLL 3500 for virtual staining.
  • the method 3900 is used to train the MLL 3500 including at least one encoder branch and multiple decoder branches, e.g., the MLL 3500 of Fig. 10 and Fig. 12. While the scenario Fig. 13 illustrates an architecture of the MLL 3500 including multiple decoder branches, similar techniques may be readily applied to a scenario in which the architecture of the MLL 3500 includes multiple encoder branches.
  • one or more training images depicting one or more tissue samples are obtained.
  • the training images may be part of one or more sets of training imaging data. For example, as shown in Fig. 14, training images 3911 , 3921 are obtained for each of two tissue samples 3910 and 3920, respectively.
  • the tissue samples 3910 or 3920 can be cancer or cancer-free tissue samples removed from a patient, or tissue samples of other animals or plants.
  • the tissue samples could be in vivo inspected tissue, e.g., using an endoscope.
  • the tissue samples could be ex-vivo inspected cell cultures.
  • Fig. 11 it would be possible that multiple instances of the training images 3911 and 3921 are obtained. For example, different tissue samples (not illustrated in Fig. 14) may be associated with the multiple instances. Thereby, a larger training database is possible.
  • Fig. 14 While in the scenario Fig. 14 only a single set of training images 3911, 3921 is provided, respectively, as a general rule, it would be possible that multiple sets of training images are obtained, the multiple sets being acquired with multiple imaging modalities.
  • the training process will be described in connection with a single imaging modality providing the training images 3911, 3921, respectively, but the examples can be extended to multiple imaging modalities, e.g., by fusing as described above.
  • the method 3900 of Fig. 13 optionally includes - after obtaining the training images 3911 , 3921 - pre-processing the training images, e.g., using one or a combination of the following image processing techniques: artifacts reduction, such as stitching artifacts, acquisition artifacts, probe contamination, e.g., as air bubbles, dust, etc., registration artifacts, out-of-focus artifacts, etc.; noise filtering; performing a registration 701 between different training images 3911, 3921 (horizontal dotted arrow in Fig. 13); resizing; etc.
  • image processing techniques e.g., using one or a combination of the following image processing techniques: artifacts reduction, such as stitching artifacts, acquisition artifacts, probe contamination, e.g., as air bubbles, dust, etc., registration artifacts, out-of-focus artifacts, etc.; noise filtering; performing a registration 701 between different training images 3911, 39
  • each reference image 3912-3913, 3922 corresponds to a type of chemical stain, such as H&E, acridine yellow, Congo red and so on.
  • the chemical stains are labeled A, B, and C in Fig. 14.
  • each reference image 3912-3913, 3922 can be obtained by capturing an image of the respective tissue sample 3910, 3920 having been chemically stained using a laboratory staining process that may include, e.g., the tissue specimen being formalin-fixed and paraffin- embedded (FFPE), sectioned into thin slices (typically around 2-10 pm), labelled and stained, mounted on a glass slide and microscopically imaged using, for example, a bright-field microscope.
  • FFPE formalin-fixed and paraffin- embedded
  • the reference images 3912-3913, 3922 could be obtained using a fluorescence microscope and appropriate fluorophore.
  • training images 3911, 3921 which shows a similar structure as the reference images 3912-3913, 3922 by not exciting any fluorophores used to stain the respective tissue sample.
  • a first reference image may highlight cell cores
  • another reference image may highlight mitochondrions
  • the different reference images may be acquired from respective columns of a multi-well plate.
  • multiple reference images may be obtained that depict different tissue samples having different chemical stains. This is why, e.g., the chemical stains of the tissue sample depicted by the reference images 3912-3913, 3922 differ from each other.
  • the one or more training images 3911 , 3921 are processed in the MLL 3500. As the procedure of processing training images is the same as that of processing images of the multiple sets of imaging data, the block 3903 is similar to block 3302 of method 3300.
  • the at least one encoder of the MLL 3500 extracts relevant features of the one or more training images and transmits the relevant features to the multiple decoder branches 3701 - 3703.
  • Each of the multiple decoder branches 3701 - 3703 converts the relevant features of the one or more training images 3911 , 3921 into corresponding training output images 3981-3983, 3991-3993 depicting the respective tissue sample 3910,
  • 3920 including a respective virtual stain, here virtual stains A*, B*, and C* which correspond to chemical stains A, B, C, respectively.
  • multiple training output images are obtained from the MLL 3500, for each one of the training images. This corresponds to the multi-output scenario.
  • Each one of the multiple training output images is associated with a respective decoder branch and depicts the respective tissue sample including a respective virtual stain.
  • the training output images 3981-3983 are obtained for the training image 3911 ; and the training output images 3991-3993 are obtained for the training image 3921.
  • the MLL 3500 includes three decoder branches 3701 - 3703 of which each decoder branch outputs training output images depicting a tissue sample including a respective virtual stain, here virtual stains A*, B*, and C* which correspond to chemical stains A, B, and C.
  • the method 3900 optionally includes performing a registration 702-703 between the one or more training output images 3981-3983, 3991-3993 and the multiple reference images 3912-3913, 3922.
  • the inter-sample registration 703 can be based on the registration 701 between the training images 3911, 3921.
  • Such an approach of performing an inter-sample registration 703 between training output images and reference images depicting different tissue samples can, in particular, be helpful where the different tissue samples pertain to adjacent slices of a common tissue probe or pertain to different cell samples of a multi-well plate.
  • inter-sample registration 703 between such corresponding tissue samples can yield meaningful results.
  • other scenarios are conceivable in which a registration between training images and reference images depicting different tissue samples does not yield meaningful results. I.e., inter-sample registration 703 is not always required.
  • the method 3900 may optionally be limited pairwise intra-sample registration 702 between each reference image 3912-3913, 3922 and the multiple training output images 3981-3983, 3991-3993 depicting the same tissue sample 3910, 3920 (vertical dashed arrows).
  • the training images and the reference images depict the same structures.
  • the chemical stain is generated from fluorescence that can be selectively activated by using respective excitation light of a certain wavelength.
  • a fluorescence contrast can be suppressed.
  • an inter-sample or intra-sample registration is not required, because the same structures are inherently imaged.
  • the training of the MLL is performed by updating parameter values of the MLL based on a comparison between the reference images 3912-3913 and training output images 3981-3983, 3991-3993 that are associated with corresponding chemical stains and virtual stains.
  • the comparison is based on the registrations 702, 703. For instance, a difference in contrast in corresponding spatial regions of the reference images 3912, 3913 and the training output images 3981-3983, 3991-3993 depicting the tissue samples 3910 at corresponding chemical and virtual stains can be determined.
  • the reference image 3912 is compared with the training output image 3981 , because the virtual stain A* corresponds to the chemical stain A, e.g., H&E, etc.
  • the reference image 3913 is compared with the training output image 3982.
  • the reference output image 3922 is compared with the training output image 3993.
  • a respective loss can be calculated. The loss can quantify the difference in contrast between corresponding regions of the training output images and the reference images, respectively.
  • L1 - 3703 are L1 , L2, and L3, respectively, in which L1 is based on a comparison between the training output images 3981, 3991 depicting the tissue samples 3910, 3920 including virtual stain A* and reference images 3912 including chemical stain A, L2 is based on a comparison between the training output images 3982, 3992 depicting the tissue samples 3910, 3920 including the virtual stain B* and reference images 3913 depicting the tissue sample including the chemical stain B, and L3 is based on a comparison between the training output images 3983, 3993 depicting the tissue samples 3910, 3920 including virtual stain C* and reference images 3922 including chemical stain D.
  • the training of the MLL 3500 may be performed by using other loss functions, e.g., pixel-wise difference (absolute or squared difference) between the reference images 3912-3913 and training output images 3981-3983, 3991-3993 that are associated with corresponding chemical stains and virtual stains; an adversarial loss (i.e. , using a generative adversarial network), or smoothness terms (e.g., total variation).
  • loss functions can be combined - e.g., in a relatively weighted combination - to obtain a single, final loss function.
  • the training of the MLL 3500 can jointly update parameter values of at least one encoder branch 3601 - 3604 and the multiple decoder branches 3701-3703 based on a joint comparison of the multiple reference images and the multiple training output images, such as the loss function L.
  • the training of the MLL 3500 can include multiple iterations of the method 3900, wherein, for each one of the multiple iterations, the training updates the parameter values of the at least one encoder branch and further selectively updates the parameter values of a respective one of the multiple decoder branches based on a selective comparison of a respective reference image and a respective training output image depicting the tissue same sample including associated chemical and virtual stains.
  • the first iteration could train the decoder branch 3701 providing the training output images depicting the tissue sample including the virtual stain A*.
  • the second iteration could train the decoder branch 3702 providing the training output images depicting the tissue sample including the virtual stain B*. This could be based on a comparison of the training output image 3982 and the reference image 3913 of the tissue sample 3910.
  • the third iteration could train the decoder branch 3703 providing the training output images depicting the tissue sample including the virtual stain C*. This could be based on a comparison of the training output image 3993 and the reference image 3922.
  • the fourth, fifth, and sixth iteration can proceed with further instances of the respective images 3981, 3912; 3982, 3913; and 3993, 3922.
  • a combination of joint updating of parameter values for multiple decoder branches would be possible, e.g., within each one of the tissue sample 3910 and 3920.
  • the multiple iterations are according to a sequence which alternatingly selects reference images and respective training output images depicting the tissue sample including different associated chemical and virtual stains.
  • the iterations shuffle between different chemical and virtual stains such that different decoder branches 3701- 3703 are alternatingly trained.
  • An example implementation would be (A-A*, B-B*, C-C*, B-B*, C- C*, A-A*, C-C*, B-B*, ... ).
  • a fixed order of stains is not required.
  • the training of the machine-learning logic 3500 includes multiple iterations, wherein, for at least some of the multiple iterations, the training freezes the parameter values of the encoder branches and updates the parameter values of one or more of the multiple decoder branches.
  • the training freezes the parameter values of the encoder branches and updates the parameter values of one or more of the multiple decoder branches.
  • Such a scenario may be helpful, e.g., where a pre-trained MLL is extended to include a further decoder branch. Then, it may be helpful to avoid changing of the parameter values of the at least one encoder branch; but rather enforce a fixed setting for the parameter values of the encoder branches, so as to not negatively affect the performance of the pre-trained MLL for the existing one or more decoder branches.
  • the techniques for training the machine-learning logic 3500 have been explained in connection with a scenario in which the machine-learning logic 3500 includes multiple decoder branches. Similar techniques may be applied to scenarios in which the machine-learning logic 3500 only includes a single decoder branch. Then, it is typically not required to have different samples that illustrate different chemical/virtual stains.
  • the MLL 3500 may be trained using a cyclic generative adversarial network (e.g., Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks.” Proceedings of the IEEE international conference on computer vision. 2017.) architecture including a forward cycle and a backward cycle, each of the forward cycle and the backward cycle including a generator MLL and a discriminator MLL. Both the generator MLLs of the forward cycle and the backward cycle are respectively implemented using the MLL 3500.
  • a cyclic generative adversarial network e.g., Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks.” Proceedings of the IEEE international conference on computer vision. 2017.
  • architecture including a forward cycle and a backward cycle, each of the forward cycle and the backward cycle including a generator MLL and a discriminator MLL. Both the generator MLLs of the forward cycle and the backward cycle are respectively
  • the MLL 3500 may be trained using a generative adversarial network (e.g., Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017; or Kim, Taeksoo, et al. "Learning to discover cross-domain relations with generative adversarial networks.” Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.) architecture including a generator MLL and a discriminator MLL. The generator MLL is implemented using the MLL 3500.
  • a generative adversarial network e.g., Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017; or Kim, Taeksoo, et al. "Learning to discover cross-domain relations with generative adversarial networks.” Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
  • a conditional neural network may be used. See, e.g., Eslami, Mohammad, et al. "Image-to-lmages Translation for Multi-Task Organ Segmentation and Bone Suppression in Chest X-Ray Radiography.” IEEE Transactions on Medical Imaging (2020).
  • Another example implementation relies on a StarGAN, see, e.g., Choi, Yunjey, et al. "Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
  • tissue samples e.g., cell microscopy ex-vivo or in-vivo imaging, e.g., for micro-chirurgic interventions.
  • Such techniques may be helpful where, e.g., different columns or rows of a multi-well plate include ex-vivo tissue samples of cell cultures that are stained using different fluorophores and thus exhibit different chemical stains.
  • stains that are not inherently available chemically i.e., because the tissue sample in that well has not been stained with the respective fluorophores, can be artificially created as virtual stains using the techniques described herein. This can be based on prior knowledge regarding which chemical stain is available in which well of the multi well plate.
  • an image of a tissue sample being stained with one or more fluorophores and thus exhibiting one or more chemical stains can be augmented with one or more virtual stains associated with one or more further fluorophores.
  • MIMO and SIMO scenarios have been described. Other scenarios are possible, e.g., MISO or single-input single-output SISO scenarios.
  • MISO single-input single-output SISO scenarios.
  • similar techniques as described for the MIMO scenario are applicable for the encoder part of the MLL.
  • Example 1 A method for virtually staining a tissue sample comprising selecting a virtual stain, obtaining digital imaging data of the tissue sample, wherein the digital imaging data of the tissue sample has been acquired using one or more image modalities, determining a region of interest, ROI, of the tissue sample, providing an output image depicting the tissue sample comprising the virtual stain only in the ROI.
  • Example 2 The method for virtually staining a tissue sample according to example 1 , wherein providing the output image comprises obtaining further digital imaging data of the tissue sample, wherein the further digital imaging data corresponds to a second region of interest, processing the further digital data in a or the machine-learning logic, and obtaining, from the machine-learning logic, the output image.
  • Example 3 The method for virtually staining a tissue sample according to example 2, wherein the second region of interest is different from the region of interest or the same as the region of interest.
  • Example 4 The method for virtually staining a tissue sample according to any one of examples 1 to 3, wherein the further digital imaging data has been acquired using an image modality different from the one or more image modalities used for acquiring the digital imaging data.
  • Example 5 The method for virtually staining a tissue sample according to any one of examples 1 to 4, wherein providing digital imaging data of the tissue sample comprises providing digital imaging data of the tissue sample after chemical staining of the tissue sample.
  • Example 6 The method for virtually staining a tissue sample according to any one of examples 1 to 5, wherein obtaining an output image depicting the tissue sample comprises selecting a subset of the digital imaging data of the tissue sample, wherein the subset corresponds to the region of interest, processing the subset in a machine-learning logic, and obtaining, from the machine-learning logic, the output image.
  • Example 7 The method for virtually staining a tissue sample according to any one of examples 1 to 6, wherein determining a ROI comprises processing the digital imaging data in a or the machine-learning logic, obtaining, from the machine-learning logic, a preliminary output image comprising a preliminary virtual stain of the tissue sample, processing the preliminary output image in a ROI machine-learning logic, obtaining, from the ROI machine-learning logic, a ROI.
  • Example 8 The method for virtually staining a tissue sample according to example 7, wherein the output image comprises the preliminary virtual stain in addition to the virtual stain in the region of interest.
  • Example 9 The method for virtually staining a tissue sample according to any one of examples 1 to 8, wherein determining the ROI comprises identifying regions of the tissue sample with meaningful objects.
  • Example 10 The method for virtually staining a tissue sample according to any one of examples 1 to 9, wherein determining the ROI comprises identifying regions of the tissue sample, to which the virtual stain is sensitive.
  • Example 11 A device for tissue analysis comprising a processor configured to perform the method according to any one of examples 1 to 10.
  • Example 12 A device for tissue analysis comprising a processor configured for selecting a virtual stain, obtaining digital imaging data of the tissue sample, wherein the digital imaging data of the tissue sample has been acquired using one or more image modalities, determining a region of interest, ROI, of the tissue sample, providing an output image depicting the tissue sample comprising the virtual stain only in the ROI.
  • Example 13 The device for tissue analysis according to example 12 comprising a processor configured for wherein providing the output image comprises obtaining further digital imaging data of the tissue sample, wherein the further digital imaging data corresponds to a second region of interest, processing the further digital data in a or the machine-learning logic, and obtaining, from the machine-learning logic, the output image.
  • Example 14 The device for tissue analysis according to example 13, wherein the second region of interest is different from the region of interest or the same as the region of interest.
  • Example 15 The device for tissue analysis according to any one of examples 12 to 14, wherein the further digital imaging data has been acquired using an image modality different from the one or more image modalities used for acquiring the digital imaging data.
  • Example 16 The device for tissue analysis according to any one of examples 12 to 15, wherein providing digital imaging data of the tissue sample comprises providing digital imaging data of the tissue sample after chemical staining of the tissue sample.
  • Example 17 The device for tissue analysis according to any one of examples 12 to 16, wherein obtaining an output image depicting the tissue sample comprises selecting a subset of the digital imaging data of the tissue sample, wherein the subset corresponds to the region of interest, processing the subset in a machine-learning logic, and obtaining, from the machine-learning logic, the output image.
  • Example 18 The device for tissue analysis according to any one of examples 12 to 17, wherein determining a ROI comprises processing the digital imaging data in a or the machine-learning logic, obtaining, from the machine-learning logic, a preliminary output image comprising a preliminary virtual stain of the tissue sample, processing the preliminary output image in a ROI machine-learning logic, obtaining, from the ROI machine-learning logic, a ROI.
  • Example 19 The device for tissue analysis according to example 18, wherein the output image comprises the preliminary virtual stain in addition to the virtual stain in the region of interest.
  • Example 20 The device for tissue analysis according to any one of examples 12 to 19, wherein determining the ROI comprises identifying regions of the tissue sample with meaningful objects.
  • Example 21 The device for tissue analysis according to any one of examples 12 to 20, wherein determining the ROI comprises identifying regions of the tissue sample, to which the virtual stain is sensitive.
  • Example 22 The device for tissue analysis according to any one of examples 12 to 21 , further comprising an image acquisition system configured for acquiring digital imaging data of the tissue sample using one or more image modalities.
  • Example 23 The device for tissue analysis according to example 22, wherein the image acquisition system is configured for selectively illuminating regions of a tissue sample.
  • Example 24 The device for tissue analysis according to example 23, wherein the image acquisition system comprises at least one of a spatial light modulator, SLM, and a digital mirror device, DMD, for selectively illuminating regions of a tissue sample.
  • the image acquisition system comprises at least one of a spatial light modulator, SLM, and a digital mirror device, DMD, for selectively illuminating regions of a tissue sample.
  • Example 25 The device for tissue analysis according to example 23 or 24, wherein the image acquisition system is configured for selectively illuminating regions of the tissue sample using a point scanning method.
  • Example 26 The device for tissue analysis according to any one of examples 22 to 24, wherein the image acquisition system is configured for acquiring digital imaging data of the tissue sample using a point scanning method.
  • Example 27 The device for tissue analysis according to any one of examples 22 to 26, wherein the image acquisition system is configured for acquiring imaging data of tissue samples using an image modality, wherein the acquiring comprises at least one of
  • spectral bands in the ultra violet, visible and/or infrared range
  • tissue samples - a structured illumination microscopy, SIM, analysis of the tissue samples, - a Raman analysis of the tissue samples,
  • OCI optical coherence imaging
  • OCT optical coherence tomography
  • DHM digital holographic imaging
  • Example 28 Computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any one of examples 1 to 10.
  • Example 29 A tangible storage medium storing the computer program of Example 28.
  • Example 30 A data carrier signal carrying the program of Example 28.

Abstract

L'invention concerne un procédé de coloration virtuelle d'un échantillon de tissu comprenant la sélection d'une tache virtuelle, l'obtention de données d'imagerie numérique de l'échantillon de tissu, les données d'imagerie numérique de l'échantillon de tissu ayant été acquises à l'aide d'une ou de plusieurs modalités d'image, la détermination d'une région d'intérêt (ROI) de l'échantillon de tissu, la fourniture d'une image de sortie représentant l'échantillon de tissu comprenant la tache virtuelle uniquement dans la ROI. En outre, l'invention concerne un dispositif comprenant un processeur configuré pour mettre en œuvre le procédé.
PCT/EP2021/058272 2020-03-30 2021-03-30 Procédé de coloration virtuelle d'un échantillon de tissu et dispositif d'analyse de tissu WO2021198243A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102020108712.8 2020-03-30
DE102020108712 2020-03-30

Publications (1)

Publication Number Publication Date
WO2021198243A1 true WO2021198243A1 (fr) 2021-10-07

Family

ID=75377761

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2021/058272 WO2021198243A1 (fr) 2020-03-30 2021-03-30 Procédé de coloration virtuelle d'un échantillon de tissu et dispositif d'analyse de tissu

Country Status (1)

Country Link
WO (1) WO2021198243A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170276923A1 (en) 2014-08-26 2017-09-28 Carl Zeiss Microscopy Gmbh Phase Contrast Imaging
WO2019154987A1 (fr) 2018-02-12 2019-08-15 F. Hoffmann-La Roche Ag Transformation d'images numériques de pathologie

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170276923A1 (en) 2014-08-26 2017-09-28 Carl Zeiss Microscopy Gmbh Phase Contrast Imaging
WO2019154987A1 (fr) 2018-02-12 2019-08-15 F. Hoffmann-La Roche Ag Transformation d'images numériques de pathologie

Non-Patent Citations (20)

* Cited by examiner, † Cited by third party
Title
CHOI, YUNJEY ET AL.: "Stargan: Unified generative adversarial networks for multi-domain image-to-image translation", PROCEEDINGS OF THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2018
DURALL RPFREUNDT FJKOTHE UKEUPER J.: "German Conference on Pattern Recognition (GCPR", 10 September 2019, article "Object Segmentation Using Pixel-Wise Adversarial Loss", pages: 303 - 316
ESLAMI, MOHAMMAD ET AL.: "Image-to-Images Translation for Multi-Task Organ Segmentation and Bone Suppression in Chest X-Ray Radiography", IEEE TRANSACTIONS ON MEDICAL IMAGING, 2020
HE, K.GKIOXARI, G.DOLLAR, P.GIRSHICK, R.: "Mask r-cnn", PROCEEDINGS OF THE IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (CVPR, 2017
ISOLA, PHILLIP ET AL.: "Image-to-image translation with conditional adversarial networks", PROCEEDINGS OF THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2017
JEGOU ET AL.: "The One Hundread Layers Tiramisu: Fully Convolutional DenseNets for Segmantic Segmentation", ARXIV:1611:09326V3
JEGOU, S.DROZDZAL, M.VAZQUEZ, D.ROMERO, A.BENGIO, Y., THE ONE HUNDRED LAYERS TIRAMISU: FULLY CONVOLUTIONAL DENSENETS FOR SEMANTIC SEGMENTATION
KIM, TAEKSOO ET AL.: "Learning to discover cross-domain relations with generative adversarial networks", PROCEEDINGS OF THE 34TH INTERNATIONAL CONFERENCE ON MACHINE LEARNING-VOLUME 70. JMLR. ORG, vol. 70, 2017
LIN, G.MILAN, A.SHEN, C.REID, I.: "RefineNet: Multi-Path Refinement Networks with Identity Mappings for High-Resolution Semantic Segmentation", CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, ARXIV PREPRINT ARXIV:1611.09326, 2016
LIN, T. Y.GOYAL, P.GIRSHICK, R.HE, K.DOLLAR, P.: "Focal loss for dense object detection", PROCEEDINGS OF THE IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (CVPR, 2017
LISA POLLARO ET AL: "Digital Staining: Microscopy of Live Cells Without Invasive Chemicals", MICROSCOPY TODAY, vol. 23, no. 04, 1 July 2015 (2015-07-01), US, pages 12 - 17, XP055320582, ISSN: 1551-9295, DOI: 10.1017/S1551929515000590 *
LIU WEI ET AL: "SSD: Single Shot MultiBox Detector", 17 September 2016, ICIAP: INTERNATIONAL CONFERENCE ON IMAGE ANALYSIS AND PROCESSING, 17TH INTERNATIONAL CONFERENCE, NAPLES, ITALY, SEPTEMBER 9-13, 2013. PROCEEDINGS; [LECTURE NOTES IN COMPUTER SCIENCE; LECT.NOTES COMPUTER], SPRINGER, BERLIN, HEIDELBERG, PAGE(S) 21 - 37, ISBN: 978-3-642-17318-9, XP047355246 *
LIU, W.ANGUELOV, D.ERHAN, D.SZEGEDY, C.REED, S.FU, C. Y.BERG, A. C.: "SSD: Single shot multibox detector", EUROPEAN CONFERENCE ON COMPUTER VISION, 2016
REDMON, JOSEPHALI FARHADI: "Yolov3: An incremental improvement", ARXIV PREPRINT ARXIV: 1804.02767, 2018
RONNEBERGER, O.P.FISCHERBROX, T.: "U-Net: Convolutional Networks for Biomedical Image Segmentation", MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION (MICCAI, 2015
RONNEBERGER, OLAFPHILIPP FISCHERTHOMAS BROX: "International Conference on Medical image computing and computer-assisted intervention", 2015, INTERNATIONAL CONFERENCE ON MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION, article "U-net: Convolutional networks for biomedical image segmentation"
SZEGEDY, CHRISTIAN ET AL.: "lnception-v4, inception-resnet and the impact of residual connections on learning", THIRTY-FIRST A A AT CONFERENCE ON ARTIFICIAL INTELLIGENCE, 2017
YIJIE ZHANG ET AL: "Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 20 January 2020 (2020-01-20), XP081582487 *
ZHAO, H.SHI, J.QI, X.WANG, X.JIA, J.: "Pyramid scene parsing network", PROCEEDINGS OF THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR, 2017
ZHU, JUN-YAN ET AL.: "Unpaired image-to-image translation using cycle-consistent adversarial networks", PROCEEDINGS OF THE IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION, 2017

Similar Documents

Publication Publication Date Title
de Haan et al. Deep-learning-based image reconstruction and enhancement in optical microscopy
JP7344568B2 (ja) ディープラーニングを使用して無標識蛍光画像をデジタル染色する方法及びシステム
JP7459336B2 (ja) 予測タグ付けを有する画像を生成するための画像処理のためのシステム、デバイス、および方法
US20230030424A1 (en) Method and system for digital staining of microscopy images using deep learning
US11222415B2 (en) Systems and methods for deep learning microscopy
US10824847B2 (en) Generating virtually stained images of unstained samples
WO2020018154A1 (fr) Procédé et système de coloration numérique d'images en phase sans étiquette à l'aide d'un apprentissage profond
CN111699510A (zh) 数字病理学图像的变换
CN109923401B (zh) 高光谱成像系统
Rivenson et al. Deep learning-based virtual histology staining using auto-fluorescence of label-free tissue
US11776124B1 (en) Transforming multispectral images to enhanced resolution images enabled by machine learning
CN113711133A (zh) 基于深度学习的彩色全息显微镜的系统和方法
WO2021198252A1 (fr) Logique de coloration virtuelle
Lim et al. Light sheet fluorescence microscopy (LSFM): past, present and future
WO2021198279A1 (fr) Procédés et dispositifs de notation virtuelle d'échantillons de tissu
WO2021198247A1 (fr) Co-conception optimale de matériel et de logiciel pour coloration virtuelle de tissu non marqué
WO2021198241A1 (fr) Coloration virtuelle à entrées multiples et/ou sorties multiples
WO2021198243A1 (fr) Procédé de coloration virtuelle d'un échantillon de tissu et dispositif d'analyse de tissu
WO2023107844A1 (fr) Coloration immunohistochimique virtuelle sans étiquette de tissu à l'aide d'un apprentissage profond
US20240020955A1 (en) Imaging method and system for generating a digitally stained image, training method for training an artificial intelligence system, and non-transitory storage medium
US20230134734A1 (en) Customizing virtual stain
Mannam Overcoming fundamental limits of three-dimensional in vivo fluorescence imaging using machine learning
CN109754382B (zh) 类h&e影像的合成方法及采用该方法的光学系统
Dai et al. Exceeding the limit for microscopic image translation with a deep learning-based unified framework
Kang Deep Learning Enables Virtual Hematoxylin and Eosin Stained Histological Imaging for 2D and 3D Histopathology

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21716328

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21716328

Country of ref document: EP

Kind code of ref document: A1