WO2025002086A1

WO2025002086A1 - Method for virtual stain and computational pathology based on label-free tissue

Info

Publication number: WO2025002086A1
Application number: PCT/CN2024/101152
Authority: WO
Inventors: Tsz Wai WONG; Zhenghui CHEN; Hei Man WONG; Weixing DAI
Original assignee: Hong Kong University of Science and Technology
Current assignee: Hong Kong University of Science and Technology
Priority date: 2023-06-27
Filing date: 2024-06-25
Publication date: 2025-01-02
Anticipated expiration: 2025-12-27
Also published as: CN120937049A

Abstract

A machine learning (ML) workflow for virtual staining and computational pathology based on a label-free tissue sample is provided to avoid a laborious and time-consuming procedure of pathologist-based interpretation of formalin-fixed and paraffin-embedded histology. A virtual-staining ML model translates a label-free autofluorescence (AF) whole slide image (WSI) of the tissue sample into a virtual hematoxylin and eosin (H&E) -stained WSI, which is further transformed into a WSI virtually stained with a special stain by a stain-transformation ML model. Each of the ML models can be realized as a U-Frame model. A multiple instance learning (MIL) network with attention-based pooling can be applied to the virtual H&E-stained WSI or AF WSI to provide fast and accurate tumor diagnosis. Furthermore, the attention-based MIL network can be configured as a clustering-constrained-attention multiple instance learning (CLAM) model for providing multiclass tumor subtyping, or even multiclass disease typing, in diagnosis of the tissue sample.

Description

Method for Virtual Stain and Computational Pathology Based on Label-Free Tissue

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to, and the benefit of, US Provisional Patent Application No. 63/510, 366 filed on June 27, 2023, the disclosure of which is hereby incorporated by reference in its entirety.

LIST OF ABBREVIATIONS

AF autofluorescence

AI artificial intelligence

CLAM clustering-constrained-attention multiple instance learning

CNN convolutional neural network

DCIS ductal carcinoma in situ

DSMIL dual-stream MIL

EVG Elastin van Gieson

FFPE formalin-fixed and paraffin-embedded

GAN generative adversarial network

H&E hematoxylin and eosin

LUAD lung adenocarcinoma

MIL multiple instance learning

MIL-RNN MIL with RNN

ML machine learning

MT Masson’s trichrome

NA numerical aperture

NSCLC non-small cell lung cancer

PAM photoacoustic microscopy

QPI quantitative phase imaging

RNN recurrent neural network

SRH stimulated Raman histology

UV ultraviolet

WSI whole slide image

TECHNICAL FIELD

The present disclosure generally relates to histological examination of a tissue sample. In particular, the present disclosure relates to virtually staining the tissue sample without a need for physically staining the tissue sample, and relates to performing tumor diagnosis of the tissue sample without a first need for performing a real histochemical staining process of staining the tissue sample and a second need for requiring a pathologist to interpret a histochemically-stained WSI of the tissue sample.

BACKGROUND

The conventional workflow for histopathological examination requires the interpretation from experienced pathologists on histochemically-stained thin slice images, more commonly H&E-stained images. However, the staining process is time-consuming, labor-intensive, and destructive to tissue. In recent years, various virtual staining methods have been developed based on label-free images to substitute the real histochemical staining process, significantly saving the reagents and time costs, and preserving the tissue for further analysis. FFPE histology remains the gold standard in postoperative diagnosis. This procedure requires high-quality thin slice preparation, yet it is also lengthy and laborious, usually taking 3–5 days [1] . For rapid intraoperative assessments as sometimes required in surgeries, the frozen section histology is widely used within 30 minutes. However, the unsatisfactory image quality with freezing artifacts affects the diagnosis [2] . Different advanced slide-free and label-free imaging techniques have been developed to satisfy the demand for rapid histology, including AF microscopy [3] , QPI [4] , PAM [5] , etc.

Various GAN-based models [6] are designed for virtual H&E staining on label-free thin tissue slices and thick unprocessed tissues, including supervised GAN models [3] , [4] , [7] , weakly supervised GAN models [8] and unsupervised GAN models [9] – [11] . Besides the common H&E stain, various special stains are widely used to evaluate a great variety of diseases with additional time and cost. For example, EVG stain can highlight elastic fibers in connective tissue for the diagnosis of vascular diseases, MT stain can visualize the collagen fibers for the diagnosis of non-neoplastic diseases. Supervised GAN-based models are widely practiced in virtual special stains on thin tissue slices. The virtual special stains can be transformed from unlabeled tissue slices [4] , [7] or H&E-stained tissue slices [12] .

Despite the success of different virtual stains, the interpretation of histological images that are mostly H&E-stained images still requires specialized knowledge, which can currently be done by pathologists. With a limited number of trained pathologists worldwide, there is a huge burden on pathologists to handle a large amount of digital WSIs generated every year. Advances in deep-learning-based computational pathology have enabled automatic tumor diagnosis with pathologist-level accuracy and interpretability. The fully supervised model trained with patch-level annotation of WSIs has achieved promising results on different types of tumor datasets, including lymph node metastases from breast cancer [13] , NSCLC subtyping [14] , and brain tumor subtyping [15] . Common supervised models such as VGG16 [16] and inception v3 [17] require millions of patches labeled by experienced pathologists. The labelling task is labor-intensive, costly, and time-consuming. To solve this problem, weakly supervised models using MIL with only slide-level labels are adopted for WSI classification. Deep neural networks combined with different MIL pooling strategies have been applied for tumor diagnosis, including attention-based deep MIL [18] , MIL-RNN [19] , and DSMIL [20] . The probability or attention score of the patches can be visualized as a heatmap to localize tumor tissue regions of the WSIs.

In view of foregoing observations, there is a need in the art for tumor diagnosis of a tissue sample without performing a real histochemical staining process of staining the tissue sample and without requiring a pathologist to interpret a histochemically-stained WSI of the tissue sample.

SUMMARY

A first aspect of the present disclosure is to provide a first method, which is computer-implemented, for virtually staining a tissue sample.

The first method comprises: acquiring an AF WSI of the tissue sample; and using a virtual-staining ML model to generate a virtual H&E-stained WSI from the AF WSI such that the tissue sample is virtually stained with a H&E stain to form the virtual H&E-stained WSI. Advantageously, the virtual H&E-stained WSI is obtained without a need for performing a real histochemical staining process of staining the tissue sample with the H&E stain.

In certain embodiments, the first method further comprises using a stain-transformation ML model to transform the virtual H&E-stained WSI into a virtual specially-stained WSI such that the tissue sample is virtually stained with a special stain other than the H&E stain to form the virtual specially-stained WSI. The virtual specially-stained WSI is advantageously obtained without a need to physically stain the tissue sample with the special stain.

The virtual-staining ML model may be realized as a U-Frame model. The stain-transformation ML model may also be realized as the U-Frame model. The special stain may be a MT stain.

In certain embodiments, the first method further comprises training the virtual-staining ML model before the virtual-staining ML model is used in generating the virtual H&E-stained WSI.

In certain embodiments, the virtual-staining ML model is trained with a first training dataset. The first training dataset is prepared according to whether a thin tissue slice or a thick tissue is used as the tissue sample.

In certain embodiments, the first method further comprises training the stain-transformation ML model before the stain-transformation ML model is used in generating the virtual specially-stained WSI.

In certain embodiments, the stain-transformation ML model is trained with a second training dataset, the second training dataset being prepared according to whether a thin tissue slice or a thick tissue is used as the tissue sample.

A second aspect of the present disclosure is to provide a second method for virtually staining a tissue sample.

The second method comprises: generating, by a fluorescence microscopy system, an AF WSI of the tissue sample; and executing, by one or more computers, a process of virtually staining the tissue sample according to any of the embodiments of the first method.

A third aspect of the present disclosure is to provide a third method, which is computer-implemented, for performing tumor diagnosis of a tissue sample.

The third method comprises: acquiring an AF WSI of the tissue sample; selecting and obtaining a working WSI for tumor diagnosis, wherein the working WSI is selected from the AF WSI and a virtually stained WSI of the tissue sample, and wherein the virtually stained WSI is generated from the AF WSI; cropping the working WSI to retain one or more tissue regions of the working WSI; dividing the one or more tissue regions into a plurality of tissue patches; and using a tumor-diagnosis ML model to process respective tissue patches in the plurality of tissue patches to diagnose any tumor in the working WSI such that tumor diagnosis of the tissue sample is performed without a first need for performing a real histochemical staining process of staining the tissue sample and a second need for requiring a pathologist to interpret a histochemically-stained WSI of the tissue sample.

In certain embodiments, the tumor-diagnosis ML model is realized as a MIL network. The MIL network comprises: a backbone for extracting a plurality of patch-level features of an individual tissue patch in the plurality of tissue patches; and a MIL pooling module implemented with a MIL pooling method for aggregating respective pluralities of patch-level features extracted for the plurality of tissue patches and predicting a score indicating a probability of presence of tumor as observed on the working WSI.

In certain embodiments, attention-based pooling is used as the MIL pooling method such that the MIL network is an attention-based MIL network.

In certain embodiments, the attention-based MIL network is configured as a CLAM model. The CLAM model may be configured to provide multiclass tumor subtyping in tumor diagnosis of the tissue sample. The CLAM model may also be configured to provide multiclass disease typing in diagnosis of the tissue sample.

In certain embodiments, max-pooling or mean-pooling is used as the MIL pooling method.

In certain embodiments, the backbone is realized as a CNN. The backbone may also be realized as a ResNet50 model.

In certain embodiments, the backbone is realized as a vision transformer.

In certain embodiments, the backbone is realized as a diffusion-based model.

In certain embodiments, the tumor-diagnosis ML model is realized as a weakly-supervised ML model.

In certain embodiments, the tumor-diagnosis ML model is realized as a supervised ML model.

In certain embodiments, the third method further comprises after one or more tumors are diagnosed in the working WSI, further using the tumor-diagnosis ML model to perform prognosis prediction of a likelihood of patient outcome. The likelihood of patient outcome may be recurrence likelihood, patient-survival likelihood, and other outcome after a surgical and/or drug treatment.

In certain embodiments, the third method further comprises after one or more tumors are diagnosed in the working WSI, further using the tumor-diagnosis ML model to perform mutation prediction of patient genes.

In certain embodiments, the third method further comprises training the MIL network before the tumor-diagnosis ML model is used in processing the respective tissue patches. In case the tumor-diagnosis ML model is trained with a third training dataset, the third training dataset may be prepared according to whether a thin tissue slice or a thick tissue is used as the tissue sample.

In certain embodiments, the virtually stained WSI is selected to be a virtual H&E-stained WSI of the tissue sample, where the virtual H&E-stained WSI is a WSI resulted from virtually staining the tissue sample with a H&E stain. The obtaining of the working WSI comprises obtaining the virtually stained WSI, wherein the obtaining of the virtually stained WSI comprises using a virtual-staining ML model to generate the virtual H&E-stained WSI from the AF WSI. In certain embodiments, the virtual-staining ML model is realized as a U-Frame model.

In certain embodiments, the virtually stained WSI is selected to be a virtual specially-stained WSI of the tissue sample, where the virtual specially-stained WSI is a WSI resulted from virtually staining the tissue sample with a special stain other than a H&E stain. The obtaining of the working WSI comprises obtaining the virtually stained WSI. The obtaining of the virtually stained WSI comprises: using a virtual-staining ML model to generate a virtual H&E-stained WSI from the AF WSI, wherein the virtual H&E-stained WSI is a WSI resulted from virtually staining the tissue sample with the H&E stain; and using a stain-transformation ML model to transform the virtual H&E-stained WSI into the virtual specially-stained WSI. In certain embodiments, the special stain is a MT stain. In certain embodiments, each of the virtual-staining ML model and stain-transformation ML model is individually realized as a U-Frame model.

A fourth aspect of the present disclosure is to provide a fourth method for performing tumor diagnosis of a tissue sample.

The fourth method comprises: generating, by a fluorescence microscopy system, an AF WSI of the tissue sample; and executing, by one or more computers, a process of performing tumor diagnosis of the tissue sample based on the generated AF WSI according to any of the embodiments of the third method.

Other aspects of the present disclosure are disclosed as illustrated by the embodiments hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a deep-learning-based workflow for virtual staining and computational pathology.

FIG. 2 depicts an exemplary computational pathology workflow using an attention-based MIL model, where in FIG. 2: subplot a depicts an AF image-based tumor diagnosis model; subplot b depicts a H&E-stained image-based tumor diagnosis model; and subplot c depicts the attention heatmap predicted by a well-trained MIL model either on AF image directly or virtual H&E-stained image generated by the U-Frame model.

FIG. 3 depicts histological images of human lung cancer tissue slices, where in FIG. 3: subplot a depicts an AF image of a human lung resection tissue slice; subplot b depicts a ground truth MT-stained image of the same slice; subplot c depicts virtual H&E-stained images generated from the AF image of subplot a by the virtual staining model; subplot d depicts virtual MT-stained images generated from the virtual H&E-stained images of subplot c by the stain transformation model; subplot e depicts the ground truth H&E-stained image of the same slice; and subplot f depicts virtual MT-stained images generated from the ground truth H&E-stained image of subplot e by the stain transformation model.

FIG. 4 depicts histological images of lung adenocarcinoma tissue slices, where in FIG. 4: subplot a depicts an AF image of a human lung resection tissue; subplot b depicts virtual H&E-stained images generated from the AF image of subplot a by the U-Frame model; subplot c depicts the ground truth H&E-stained image of the same slice; subplots d-f depict zoomed-in images of first rectangular regions 410 marked in subplots a-c, respectively; and subplots g-i depict zoomed-in images of second rectangular regions 420 marked in subplots d-f, respectively.

FIG. 5 illustrates tumor region localization and visualization on human lung slices, where in FIG. 5: subplot a illustrates tumor region annotation on a ground truth H&E-stained image by a pathologist; subplot b shows the whole-slide attention heatmap generated from a virtual H&E-stained image by the CLAM model; subplot c shows the whole-slide attention heatmap generated from the ground truth H&E-stained image by the CLAM model; subplots d-f depict zoomed-in images of the first rectangular regions 511 in subplots a-c, respectively; and subplots g-i depict zoomed-in images of the second rectangular regions 512 in subplots a-c, respectively.

FIG. 6 depicts histological images of breast cancer biopsy slices, where in FIG. 6: subplot a depicts an AF image of a human breast biopsy tissue; subplot b depict virtual H&E-stained images generated from the tissue of subplot a by the U-Frame model; subplot c depicts a ground truth H&E-stained image of the same slice; subplots d-f depict zoomed-in images of first rectangular regions 610 in subplots a-c, respectively; and subplots g-i depict zoomed-in images of second rectangular regions 620 in subplots d-f, respectively.

FIG. 7 illustrates tumor region localization and visualization on human breast slices, where in FIG. 7: subplot a depicts tumor region annotation 730 on a ground truth H&E-stained image by a pathologist; subplot b depicts a whole-slide attention heatmap generated from a virtual H&E-stained image by the CLAM model; subplot c depicts a whole-slide attention heatmap generated from ground truth H&E-stained image by CLAM model; subplots d-f depict zoomed-in images of first rectangular regions 711 in subplots a-c, respectively; and subplots g-i depict zoomed-in images of second rectangular regions 712 in subplots a-c, respectively.

FIG. 8 illustrates tumor region localization and visualization on thick human lung samples, where in FIG. 8: subplot a depicts an AF image of thick human lung resection tissue; subplot b depicts a whole-slide attention heatmap generated from a thick AF image by the CLAM model; subplot c depicts reference H&E-stained image of adjacent thin slices with annotation by a pathologist; subplots d-f depict zoomed-in images of first rectangular regions 811 in subplots a-c, respectively; subplots g-i depict zoomed-in images of second rectangular regions 812 in subplots a-c, respectively; subplot j depict AF image of adjacent thin human lung slices; subplot k depicts a whole-slide attention heatmap generated from a thin AF image by the CLAM model; subplot l depicts a ground truth H&E-stained image of thin slices with annotation by a pathologist; subplots m-o depict zoomed-in images of third rectangular regions 813 in subplots j-l, respectively; and subplots p-r depict zoomed-in images of fourth rectangular regions 814 in subplots j-l, respectively.

FIG. 9 provides a comparison of the heatmaps generated by different workflows, where in FIG. 9: subplot a depicts tumor annotation on a ground truth H&E-stained image by a pathologist; subplot b depicts an attention heatmap generated by the AF diagnosis-based workflow; and subplot c depicts an attention heatmap generated by the virtual H&E diagnosis-based workflow.

FIG. 10 depicts a first exemplary workflow in accordance with a first method, which is computer-implemented and disclosed herein, for virtually staining a tissue sample with a special stain to thereby yield a virtual specially-stained WSI of the tissue sample.

FIG. 11 depicts a second exemplary workflow, which incorporates the first exemplary workflow, in accordance with a second method disclosed herein for virtually staining the tissue sample.

FIG. 12 depicts a third exemplary workflow in accordance with a third method, which is computer-implemented and disclosed herein, for performing tumor diagnosis of a tissue sample.

FIG. 13 depicts an exemplary structure of a MIL network used as a tumor-diagnosis ML model for processing a working WSI to perform tumor diagnosis of the tissue sample in the third exemplary workflow.

FIG. 14 depicts a fourth exemplary workflow, which incorporates the third exemplary workflow, in accordance with a fourth method disclosed herein for performing tumor diagnosis of the tissue sample.

Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been depicted to scale.

DETAILED DESCRIPTION

In the present disclosure, automatic tumor diagnosis of a tissue sample without performing a real histochemical staining process of staining the tissue sample and without requiring a pathologist to interpret a histochemically-stained WSI of the tissue sample is developed. In particular, the development of the automatic tumor diagnosis technique is simplified by unifying different virtual stains and computational pathology.

To unify different virtual stains and computational pathology, a novel deep-learning-based method using virtual H&E images of label-free tissue is proposed in the present disclosure. The label-free AF images have negative nuclear contrast under deep-UV excitation, which can be further transformed into virtual H&E-stained images by a virtual staining network. The virtual H&E stain can be further transformed into other special stains using a stain transformation network. The virtual staining and stain transformation networks could be the same model architecture with different staining data for model training. The proposed virtual stain workflow is demonstrated on human lung cancer tissue with virtual H&E stain and MT stain. For the tumor classification and localization on virtual H&E-stained images or potentially AF images, the virtual H&E diagnosis-based workflow and AF diagnosis-based workflow have been developed with the attention-based MIL network, which should be trained on the corresponding large-scale WSIs dataset with the same image type. The versatility of the proposed computational pathology workflow is experimentally demonstrated on thin slices of human lung resection and human breast biopsy tissues.

This general virtual stain and computational pathology method is demonstrated on thin tissue slices using deep learning models, which can be potentially extended for thick unprocessed tissue samples. The virtual H&E-stained images of unprocessed thick tissue using advanced unsupervised virtual staining models [11] can be potentially applied for downstream tasks such as special stains and tumor diagnosis. A similar idea has been demonstrated in the existing work, such as the CNN-based diagnosis on SRH image of unprocessed surgical specimens for intraoperative diagnosis [15] . Moreover, the AF images of unprocessed thick tissue itself can be directly used for tumor diagnosis, which is demonstrated on a small-scale dataset of thick lung cancer samples. Therefore, it is believed that the proposed workflow holds great promise to be generalized for rapid and accurate intraoperative and postoperative pathological examinations.

A. Methods and materials

A.1. General workflow

FIG. 1 depicts a general deep-learning-based workflow 100 for virtual staining and computational pathology on a label-free tissue. For the virtual staining of label-free AF WSIs, we applied a weakly supervised GAN-based model, the U-Frame model [8] , which does not require precise image registration and has shown better performance compared to fully supervised methods. We also adopted the U-Frame model to achieve stain transformation from virtual H&E to other virtual special stains. We applied the attention-based MIL network as a preferred ML model to achieve tumor classification and localization on virtual H&E-stained WSIs or AF WSIs directly. Apart from the attention-based MIL network, which is weakly supervised, other weakly-supervised or even supervised ML models may also be used for tumor classification and localization. Nonetheless, as mentioned above, there are practical advantages of using weakly-supervised ML models over supervised ones.

A.2. Computational pathology

The computational pathology workflow for tumor diagnosis can have two variants including virtual H&E diagnosis-based workflow and AF diagnosis-based workflow (FIG. 2, subplot c) , which can achieve tumor classification and localization comparable to the standard H&E histology interpreted by pathologists (FIG. 2, subplot d) .

Given the large-scale open-source H&E-stained images, we trained the H&E diagnosis model 124 using the open-source H&E-stained images and inference on the virtual H&E-stained images of AF images (FIG. 2, subplots b and c) . This virtual H&E diagnosis-based workflow consists of two deep-learning models for virtual staining and tumor diagnosis. The virtual staining model 115 transforms the AF image 110 into a virtual H&E-stained image 120 equivalent to the FFPE H&E histology, which can be further predicted by a well-trained H&E diagnosis model 124. To evaluate the performance of the diagnosis model 124, the slide-level prediction can be compared with the diagnosis from the pathologists and the heatmap with suspected tumor region generated by the H&E diagnosis model 124 (FIG. 2, subplot c) can be compared with the ground truth H&E-stained images that have been annotated by the pathologists (FIG. 2, subplot d) . To demonstrate the idea of AI diagnosis on label-free tissue slices, we also designed an AF diagnosis-based workflow (FIG. 2, subplot c) , which trained AF diagnosis model 125 using open-source H&E WSIs combined with additional 13 AF WSIs and corresponding slide-level labels (FIG. 2, subplot a) . To demonstrate the possibility of AF diagnosis-based workflow for thick tissue, we also trained two AF diagnosis models 125 purely on a small-scale thick AF WSIs dataset and an adjacent thin AF WSIs dataset, which have comparable performance for tumor localization on attention heatmap (FIG. 8) .

For the tumor diagnosis task, a standard MIL approach can be adopted for tumor versus normal binary classification. The individual WSI can be regarded as the bag and the patches cropped from the WSI are regarded as the instances. Consider that there are M instances in each bag X_n= {x_n, 1, x_n, 2, …, x_n, m} for n=1, …, N and m=1, …, M, the corresponding bag label Y_n is either positive or negative, Y_n∈ {0, 1} , which can be defined by the aggregated unknown instance labels {y_n, 1, y_n, 2, …, y_n, m} , y_n, m∈ {0, 1} . In particular,

For the positive tumor slide, there is at least one tumor patch in that slide. For the negative normal slide, all the patches of this slide are negative. The deep learning models under the MIL framework use pretrained CNN backbone to extract the feature representations of patches. Apart from implementing the backbone with a CNN, other types of ML models, such as a vision transformer and a diffusion-based model, may also be used as the backbone. MIL pooling methods aggregate these patch-level features from the same slide and predict the final score or probability for the whole slide, which is compared with the ground truth slide-level label by a cross-entropy loss function. Therefore, the selection of different MIL pooling methods is of great importance to the model performance. Assume that the deep learning model output the patch-level embedding h_n, m=f (x_n, m) , whereand x_n, m denotes an mth patch of a nth slide, for n=1, …, N and m=1, …, M.

For max-pooling, the slide-level probability p_n is represented by the positive patch with the highest probability among M patches for each WSI. This assumption can be relaxed by using top K patches with the highest probability as positive patches. It follows that p_n is given by

For mean-pooling, the slide-level probability p_n is represented by the mean probability value of all the patches for each WSI, and is given by

For attention-based pooling, the slide-level probability p_n is represented by the weighted average of patch-level embeddings [18] . It follows that

The weights a_n, m can be updated with the neural network during model training, where andare parameters in a fully connected layer. The attention weight reflects the importance of different patches among the slides. In certain embodiments, a_n, m is computed by

In the present disclosure, we apply the CLAM model in the proposed workflow for WSI classification [21] . The tissue regions of WSIs are detected with contours and patched with 512 × 512-pixel size at 20× magnification (FIG. 2, subplots a and b) . These tissue patches are embedded into low-dimensional feature vectors using the ResNet50 model pre-trained on ImageNet. Then, the patch features of the same slide can be aggregated using the gated attention module for the final slide-level prediction. The clustering method with smooth support-vector-machine loss based on the attention score is implemented to enhance the supervision for different classes. Therefore, the combination of the slide-level classification loss and the instance-level clustering loss is minimized during model training, where the attention scores can be learned directly from the data-driven model. The attention heatmap can be used to localize the tumor and normal regions. Compared to the conventional MIL algorithms using max-pooling or mean-pooling or RNN aggregation, the CLAM model is more data-efficient and can be extended for multiclass tumor subtyping or even multiclass disease typing.

A.3. WSI datasets

For the public human breast-cancer lymph node-metastasis dataset, we collected 899 annotated WSIs from CAMELYON16 and CAMELYON17 Grand Challenges [22] . For the public human lung cancer dataset, we collected a total of 1983 lung resection WSIs, including 1059 normal and LUAD WSIs from CPTAC-LUAD projects, 383 normal WSIs from CPTAC-LSCC projects, and 541 LUAD WSIs from TCGA-LUAD projects. These public datasets were used for tumor diagnosis model training.

For the human breast cancer and lung cancer thin slice dataset, we collected the tissue sample from our collaborating hospitals, and acquired the AF images and corresponding H&E-stained images in our laboratory. The human breast cancer tissues were extracted through biopsy and the human lung tissues were resected surgically. These tissues went through the FFPE process. After microtome sectioning, optically thin tissue slices with a thickness of 4 μm were placed on a quartz slide and deparaffined for AF imaging. The label-free AF images were acquired by the AF imaging system. The same slide was stained with H&E and digitalized into WSIs using a whole-slide scanner equipped with a 20×/0.75 NA objective lens (NanoZoomer-SQ, Hamamatsu Photonics K. K) . The AF images and H&E-stained images are used for virtual staining model training. The generated virtual H&E-stained images are used for tumor diagnosis model testing, the detailed information is shown in Table 1.

Table 1. Virtual H&E WSIs dataset.

For the special stain thin slice dataset, we obtained the AF images, H&E-stained images, and MT-stained images of human lung cancer slices. To acquire the MT-stained image of the same slice, the H&E-stained sample is destained by immersion in acid alcohol overnight, followed by a second staining procedure with the standard MT stain. The H&E-stained and MT-stained images are used for stain transformation network training.

For the thick human lung cancer dataset, we collected a total of 33 AF WSIs of thick human lung tissue from 29 patients, including 31 tumor WSIs and 2 normal WSIs. The standard H&E-stained images of the adjacent layer were acquired for reference. For the adjacent thin AF dataset, we collected a total of 43 AF WSIs from 26 patients, including 37 tumor WSIs and 6 normal WSIs.

A.4. Training details

For the virtual H&E staining of AF images and stain transformation from H&E stain to MT stain, we followed the same data preprocessing steps and model implementation in the original U-Frame paper [8] . We used an in-house WSI dataset with AF images, H&E-stained images, and MT-stained images for model training and testing. For tumor classification and localization on H&E-stained images, we only used the default tumor versus normal binary classification mode in the CLAM model for human breast cancer and lung cancer samples. The model was trained purely with the public dataset and tested on our in-house dataset. For tumor diagnosis purely on thick AF WSIs and thin AF WSIs, the normal data was augmented for model training to balance the image class. The models were trained using PyTorch 1.8.1 and Python 3.8.8 on a workstation with Ubuntu 20.04 operating system, Intel Core i9-10980XE CPUs, and Nvidia GeForce RTX 3090 GPUs.

B. Results

B.1. Special stain on virtual H&E of human lung tissue slices

To demonstrate the feasibility of the virtual stain workflow, we validated the virtual H&E and MT stain on the human lung cancer tissue slices with a substantial amount of fibrotic tissue, which the excessive accumulation of collagen fibers can be highlighted in blue by MT staining, using the well-trained virtual staining and stain transformation models.

FIG. 3 shows the label-free grayscale AF image (FIG. 3, subplot a) of a human lung cancer tissue slice with ground truth MT-stained image (FIG. 3, subplot b) and ground truth H&E-stained image (FIG. 3, subplot e) . The virtual H&E-stained image (FIG. 3, subplot c) generated from the AF images (FIG. 3, subplot a) by the virtual staining model has comparable quality as the ground truth H&E-stained image (FIG. 3, subplot e) . The virtual MT-stained images (FIG. 3, subplots d and f) are generated from the virtual H&E (FIG. 3, subplot c) and real H&E (FIG. 3, subplot e) by the same stain transformation model, respectively. These two virtual MT-stained images show similar blue collagen information compared with the ground truth MT-stained image. The results indicate that the virtual special stain workflow can provide important diagnostic features on the label-free images as the conventional special stain method.

B.2. Tumor diagnosis on virtual H&E of human lung resection slices

To illustrate the feasibility of the computational pathology workflow, we first validated the virtual staining and tumor classification on the human lung resection tissue using a well-trained U-Frame model and CLAM model, respectively. FIG. 4 shows the results that a label-free grayscale AF image (FIG. 4, subplot a) of a human lung adenocarcinoma tissue is transformed into a virtual H&E-stained image (FIG. 4, subplot b) , which is comparable to the ground truth H&E-stained image (FIG. 4, subplot c) .

In the zoomed-in first rectangular regions 410 (FIG. 4, subplots d-f) , different features such as large tumor cells and blood can be well identified in the AF image (FIG. 4, subplot d) , corresponding virtual H&E-stained image (FIG. 4, subplot e) and ground truth H&E-stained images (FIG. 4, subplot f) . In the further zoomed-in second rectangular regions 420 (FIG. 4, subplots g-i) , the size and shape of tumor cells are similar among the AF image (FIG. 4, subplot g) , corresponding virtual H&E-stained image (FIG. 4, subplot h) , and the ground truth H&E-stained images (FIG. 4, subplot i) . The virtual staining results show that the U-Frame model has good performance on histological image style transformation for lung cancer tissue with complex cell and tissue structures.

For downstream tumor classification and localization tasks, all 13 samples were correctly classified. Table 2 shows the confusion matrix of the classification results on the testing virtual H&E dataset. FIG. 5 shows the tumor localization results of one lung cancer sample with correct slide-level classification, including the pathologist’s interpretation of ground truth H&E-stained image (FIG. 5, subplot a) , the CLAM model’s prediction on virtual H&E-stained image (FIG. 5, subplot b) , and ground truth H&E-stained images (FIG. 5, subplot c) , respectively.

Table 2. Tumor classification results on human lung resection.

The ground truth tumor region is annotated by a plurality of polygons 530 in FIG. 5, subplot a. For the attention heatmap generated by the CLAM model (FIG. 5, subplots b and c) , most of the darker region with high attention score is highly close to the ground truth annotation, and the lighter region with a low attention score is more likely to be the normal tissue region and background region. In the zoomed-in first rectangular regions 511 (FIG. 5, subplots d-f) and second rectangular regions 512 (FIG. 5, subplots g-i) , the tumor region is localized with high attention in the heatmap generated from virtual H&E-stained image (FIG. 5, subplots e and h) , and ground truth H&E-stained images (FIG. 5, subplots f and i) , which has a high correlation compared to the ground truth annotation (FIG. 5, subplots d and g) , respectively. The promising results indicated that the tumor cell features in the virtual H&E-stained images can be well classified by the CLAM model, and the performance is comparable to the ground truth H&E-stained images.

B.3. Tumor diagnosis on virtual H&E of human breast biopsy slices To demonstrate the versatility of the computational pathology workflow for different tissues treated with different clinical protocols, we validated on the human breast cancer biopsy tissue slices. FIG. 6 shows the histological images of human breast biopsy slices with DCIS. The AF image (FIG. 6, subplot a) is further translated into a virtual H&E-stained image (FIG. 6, subplot b) by the U-Frame model with similar cell distributions, which is equivalent to the ground truth H&E-stained image (FIG. 6, subplot c) .

In the zoomed-in first rectangular regions 610 (FIG. 6, subplot d-f) , the abnormal duct with the neoplastic proliferation of ductal epithelial cells can be observed in the AF image (FIG. 6, subplot d) , corresponding virtual H&E-stained image (FIG. 6, subplot e) and ground truth H&E-stained images (FIG. 6, subplot f) . In the further zoomed-in second rectangular regions 620 (FIG. 6, subplots g-i) , the density of tumor cells and fiber structure are highly correlated among the AF image (FIG. 6, subplot g) , corresponding virtual H&E-stained image (FIG. 6, subplot h) , and ground truth H&E-stained images (FIG. 6, subplot i) . The virtual staining results reveal that the U-Frame model can be adopted for various human tissue samples.

For downstream tumor classification and localization tasks, the CLAM model performed well on virtual H&E-stained images of breast biopsy tissue although the model was trained on the real H&E-stained images of the breast resection tissue. Table 3 shows the confusion matrix of the classification results on the testing virtual H&E WSIs, where 4 samples were correctly classified as tumors while 1 false negative sample was predicted by the diagnosis model.

Table 3. Tumor classification results on a human breast biopsy.

FIG. 7 shows the tumor region interpretation of one breast cancer sample with the correct slide-level prediction by the CLAM model. The attention heatmaps generated from the virtual H&E-stained image (FIG. 7, subplot b) and ground truth H&E-stained images (FIG. 7, subplot c) are almost the same, which can prove the virtual H&E-stained image has the comparable image quality as the standard H&E histology. The attention heatmap (FIG. 7, subplots b and c) can localize most of the DCIS regions accurately compared with the annotation regions in the ground truth H&E-stained image (FIG. 7, subplot a) .

In the zoomed-in first rectangular regions 711 (FIG. 7, subplots d-f) and second rectangular regions 712 (FIG. 7, subplots g-i) , DCIS regions are localized correctly with high attention in the heatmaps generated from virtual H&E-stained image (FIG. 7, subplots e and h) and ground truth H&E-stained images (FIGS. 7, subplots f and i) , which are also marked by pathologists in the ground truth annotation 731 (FIG. 7, subplots d and g) . The results indicated the generalization of the proposed workflow of different human tissue samples.

B.4. Tumor diagnosis on AF images of thick human lung samples

To demonstrate the possibility of the AF diagnosis-based workflow for thick tissue, we validated on thick human lung samples with the reference of adjacent thin human lung slices. FIG. 8 shows the AF images of thick human lung cancer tissue (FIG. 8, subplot a) and adjacent thin slices (FIG. 8, subplot j) , the corresponding attention heatmaps generated by the thick AF diagnosis model and thin AF diagnosis model (FIG. 8, subplots b and k) , and reference H&E-stained images with tumor annotation in polygon 831, 832 (FIG. 8, subplots c and l) , respectively.

Most tumor regions revealed with high attention score in the heatmap of thick AF image (FIG. 8, subplot b) is highly close to the ground truth annotation 831 in the reference H&E (FIG. 8, subplot c) . The heatmap of the adjacent thin AF image (FIG. 8, subplot k) is also well-matched with annotation 832 in the ground truth H&E (FIG. 8, subplot l) . The thick human lung sample and adjacent thin slices share similar diagnostic features in the tumor region (FIG. 8, subplots d and m) and normal region (FIG. 8, subplots g and p) , with the reference to the H&E-stained image (FIG. 8, subplots f and o, FIG. 8, subplots i and r) . The tumor tissue region (FIG. 8, subplot d) and normal tissue region (FIG. 8, subplot g) of the thick AF image (FIG. 8, subplot a) can be well classified by the thick AF diagnosis model with high attention (FIG. 8, subplot e) and low attention (FIG. 8, subplot h) in the heatmap (FIG. 8, subplot b) . The adjacent tumor tissue feature (FIG. 8, subplot m) and normal tissue feature (FIG. 8, subplot p) of the thin AF image (FIG. 8, subplot j) can be distinguished by the thin AF diagnosis model with high attention (FIG. 8, subplot n) and low attention (FIG. 8, subplot q) in the heatmap (FIG. 8, subplot k) . The results indicate that AF diagnosis-based workflow can achieve good performance for thick tissue compared to the AF diagnosis-based workflow for thin tissue slices and pathologists’ interpretation of standard H&E.

C. Discussion

It has been shown that the proposed virtual H&E diagnosis-based workflow enables rapid and interpretable tumor diagnosis on virtual staining of label-free AF images for various thin human tissue slices. The attention heatmap of the tumor tissue sample with correct slide-level prediction already has good performance with the localization of the tumor region comparable to the ground truth annotation by the pathologist (FIGS. 5, subplots a-c, FIG. 7, subplots a-c) . However, some false positive regions with medium attention scores still exist in the predicted heatmap of human lung slices such as the large hemorrhage regions (FIG. 5, subplots a-c) , which is likely caused by the heterogeneity of human lung tissue samples between open-source training data and in-house testing data. Small false negative regions with low attention scores appear in the predicted heatmap of human breast slices (FIG. 7, subplots a-c) , because the poorly differentiated tumor regions in the biopsy slices may not be distinguished by the model. Also, the false negative slide-level prediction still exists in the testing dataset (Table 3) . To improve the generalization and localization accuracy of the H&E diagnosis model, tissue samples with different structures and variations of staining color tone should be considered for H&E training data augmentation.

We have also demonstrated the possibility of AF diagnosis-based workflow using label-free tissue slices directly (FIG. 2, subplot c, and FIG. 9) . Note that the figure is just a simple demonstration of the idea as the AF diagnosis model is trained on limited AF data mixed with large H&E data, the attention heatmap generated by the AF diagnosis model is expected to be not accurate enough. Although we only used the limited AF data for model training, the attention heatmap generated from AF images (FIG. 9, subplot b) can still roughly indicate correct tumor regions compared with the ground truth annotations (FIG. 9, subplot a) and the attention heatmap generated from virtual H&E-stained images (FIG. 9, subplot c) , marked with three blue arrows 920 with different locations. Also, the false positive region in the virtual H&E diagnosis heatmap (FIG. 9, subplot c) is classified correctly as a true negative region in the AF diagnosis heatmap (FIG. 9, subplot b) , compared with ground truth annotations from the pathologist (FIG. 9, subplot a) , which is marked with the brown arrow 910. However, the overall AF diagnosis heatmap (FIG. 9, subplot b) is not as accurate as the virtual H&E diagnosis heatmap (FIG. 9, subplot c) , where the incomplete tumor regions and larger false positive regions appear, which are marked with three green arrows 930. We believe that the performance of the AF diagnosis model can be improved by collecting enough label-free AF WSIs and training the model with label-free image data purely.

Although we validated the computational pathology workflow only on various human thin tissue slices, it is promising that the proposed workflow can be applied to thick and unprocessed tissues. For virtual H&E diagnosis-based workflow, virtual staining for label-free AF images of thick tissue samples is quite challenging because the tissue information between AF images and H&E-stained images cannot be aligned for supervised model training. It is believed that the problem would be solved with the development of unsupervised virtual staining models [11] . With the virtual staining as a bridge, it is easier to interpret the diagnostic features identified by the tumor diagnosis model. For AF diagnosis-based workflow, we demonstrated the usage of slide-free label-free AF images on thick human samples for model training and testing (FIG. 8) , which can further accelerate the process as the virtual staining step is not required. The AF image of adjacent thin slices and corresponding H&E-stained images can be obtained as the diagnostic reference. The pathologists can interpret thick AF images and thin AF images with reference to adjacent H&E-stained images. Given that only slide-label is required for training which can be obtained from the adjacent thin slice reference, the thick AF diagnosis model can be trained like the thin AF diagnosis model. Although the results are encouraging with small-scale training data, there are still many false positive regions in some testing samples. Therefore, a large amount of AF data collection is still needed to improve model performance and generalization.

D. Details of embodiments of the present disclosure

Herein in the specification and appended claims, the terms “virtual staining network” and “virtual-staining ML model” are used interchangeably to mean a ML model for generating a virtual H&E-stained image from an AF image of a biological sample. The virtual H&E-stained image may be a virtual H&E-stained WSI. The AF image may be an AF WSI.

Herein in the specification and appended claims, the terms “stain transformation network” and “stain-transformation ML model” are used interchangeably to mean a ML model for generating a virtual specially-stained image from a H&E-stained image or a virtual H&E-stained image of a biological sample. The (virtual) H&E-stained image may be a (virtual) H&E-stained WSI. The virtual specially-stained image may be a virtual specially-stained WSI.

The terms “thin tissue slice” and “thick tissue” are used for describing a biological tissue. As used herein, “thin tissue slice” means a tissue layer having a thickness of up to 10 micrometers, whereas “thick tissue” means a piece of tissue having a thickness greater than 10 micrometers. In general, a thin tissue slice is on a μm-scale whereas a thick tissue is on a mm-scale or even a larger scale. For instance, a tissue with a thickness of 4μm is deemed a thin tissue slice, while another tissue measuring at least 2mm in thickness is considered to be a thick issue. Most often a thin tissue slice is obtained by sectioning a thick tissue in order that the thin tissue slice is optically thin enough for microscope inspection.

Embodiments of the present disclosure are developed as follows based on the details, examples, applications, etc., of the workflows disclosed above mainly in Sections A-C possibly with generalization and extension.

A first aspect of the present disclosure is to provide a first method, which is computer-implemented, for virtually staining a tissue sample. The tissue sample is label-free.

The first method is illustrated as follows with the aid of FIGS. 10 and 1. FIG. 10 depicts a first exemplary workflow 1000 for virtually staining the tissue sample with the special stain to yield the virtual specially-stained WSI. FIG. 1 depicts a general deep-learning-based workflow for virtual staining and computational pathology.

In certain embodiments that intend to virtually stain the tissue sample with a H&E stain, the workflow 1000 comprises steps 1010 and 1030. In the step 1010, an AF WSI 110 of the tissue sample is acquired. Note that the AF WSI 110 is label-free. A virtual-staining ML model 115 is used in the step 1030 to generate a virtual H&E-stained WSI 120 from the AF WSI 110 such that the tissue sample is virtually stained with the H&E stain to form the virtual H&E-stained WSI 120. Advantageously, the virtual H&E-stained WSI 120 is obtained without a need for performing a real histochemical staining process of staining the tissue sample with the H&E stain.

In certain embodiments that intend to virtually stain the tissue sample with a special stain other than the H&E stain, the workflow 1000 further comprises step 1050. After the virtual H&E-stained WSI 120 is generated in the step 1030, a stain-transformation ML model 122 is used in the step 1050 to transform the virtual H&E-stained WSI 120 into a virtual specially-stained WSI 130 such that the tissue sample is virtually stained with the special stain to form the virtual specially-stained WSI 130. Similarly, it is advantageous that the virtual specially-stained WSI 130 is obtained without a need for physically staining the tissue sample with the special stain.

In certain embodiments, the special stain is a MT stain.

Preferably, the virtual-staining ML model 115 is realized as the U-Frame model. It is also preferable that the stain-transformation ML model 122 is realized as the U-Frame model. Details of the U-Frame model can be found in [8] , the disclosure of which is incorporated herein by reference. Note that the virtual-staining ML model 115 and stain-transformation ML model 122 have the same network structure (as derived from the U-Frame model) , but operating parameters of the two models 115, 122 are different and are determined during model training.

Note that in the steps 1030 and 1050, the virtual-staining ML model 115 and stain-transformation ML model 122 are operated after the two ML models 115, 122 are trained.

In certain embodiments, the virtual-staining ML model 115 used in the step 1030 is a pretrained one. The pretrained virtual-staining ML model is obtained by loading pre-calculated operating parameters into an untrained virtual-staining MT model. This parameter-loading procedure can also be used to obtain a pretrained stain-transformation ML model.

Alternatively, in certain embodiments, the workflow 1000 further comprises one or both of steps 1020 and 1050. The step 1020, which precedes the step 1030 in execution, is used to train the virtual-staining ML model 115. The step 1040 precedes the step 1050 in execution, and is used to train the stain-transformation ML model 122.

As disclosed above, virtual staining accomplished by the workflow 1000 is applicable for both thin tissue slice and thick tissue by using appropriate training datasets in training the two ML models 115, 122. Consider that the virtual-staining ML model 115 is trained with a first training dataset and that the stain-transformation ML model 122 is trained with a second training dataset. Preferably, each of the first and second training datasets is prepared according to whether a thin tissue slice or a thick tissue is used as the tissue sample.

A second aspect of the present disclosure is to extend the workflow 1000 to provide a second method for virtually staining the tissue sample with the special stain other than the H&E stain to thereby yield the virtual specially-stained WSI of the tissue sample.

FIG. 11 depicts a second exemplary workflow 1100, which incorporates the first exemplary workflow 1000, for virtually staining the tissue sample according to the second method. In step 1110 of the second exemplary workflow 1100, a fluorescence microscopy system is used to inspect the tissue sample under illumination of a certain preselected excitation light beam for exciting the tissue sample to produce AF so as to generate the AF WSI 110 of the tissue sample. After the AF WSI 110 is generated in the step 1110, one or more computers are used to execute the first exemplary workflow 1000 realized according to any of the embodiments of the first method.

A third aspect of the present disclosure is to provide a third method, which is computer-implemented, for performing tumor diagnosis of a tissue sample. The tissue sample is label-free.

The third method is illustrated as follows with the aid of FIGS. 12 and 1. FIG. 12 depicts a third exemplary workflow 1200 for performing tumor diagnosis of a tissue sample. Exemplarily, the workflow 1200 comprises steps 1210, 1220, 1230 and 1250.

In the step 1210, an AF WSI 110 of the tissue sample is acquired. Note that the AF WSI 110 is label-free.

In the step 1220, a working WSI of the tissue sample is selected and obtained such that tumor diagnosis 140 is performed based on the working WSI. In particular, the working WSI is selected from a group consisting of the AF WSI 110 and a virtually stained WSI of the tissue sample. The virtually stained WSI is generated from the AF WSI 110. On one hand, performing tumor diagnosis 140 directly on the AF WSI 110 rather than on the virtually stained WSI has an advantage of saving cost and processing time. On the other hand, with virtual staining as a bridge between AF WSI acquisition and tumor diagnosis, it is easier to interpret diagnostic features identified by a tumor diagnosis model. Selecting whether the AF WSI 110 or the virtually stained WSI for tumor diagnosis 140 can be determined by those skilled in the art according to practical situations under consideration.

After the working WSI is identified and obtained in the step 1220, the working WSI is preprocessed in the step 1230 before tumor diagnosis 140 is carried out. The step 1230 includes cropping the working WSI to retain one or more tissue regions of the working WSI. That is, the one or more tissue regions are first located on the working WSI. An existing image-segmentation technique for identifying and bounding tissue regions on a medical image can be used. The step 1230 further includes dividing the one or more tissue regions on the working WSI into a plurality of tissue patches. Most often respective tissue patches in the plurality of tissue patches are of equal size and dimension, e.g., each tissue patch having a size of 512×512 image pixels. As an example for illustration, subplot b of FIG. 2 depicts a set of cropped patches 230 obtained from a tissue region 220 that is identified on a WSI 210.

After the plurality of tissue patches is obtained in the step 1230, a tumor-diagnosis ML model (124 or 125) is used in the step 1250 to process respective tissue patches in the plurality of tissue patches to diagnose any tumor in the working WSI.

In certain embodiments, the tumor-diagnosis ML model is realized as a MIL network. Note that the tumor-diagnosis ML model is the H&E-diagnosis model 124 if the working WSI is the virtual H&E-stained WSI 120; and the tumor-diagnosis ML model is the AF-diagnosis model 125 if the working WSI is the AF WSI 110. As a result, tumor diagnosis 140 of the tissue sample is advantageously performed without a first need for performing a real histochemical staining process of staining the tissue sample and a second need for requiring a pathologist to interpret a histochemically-stained WSI of the tissue sample.

Details of the MIL network are given in Section A. 2. Background references thereof can be found in the published literature, e.g., in [18] - [21] , the disclosures of which are incorporated by reference herein. FIG. 13 depicts, in accordance with the disclosure in Section A. 2, an exemplary MIL network 1300 for realizing the tumor-diagnosis ML model (124 or 125) . The MIL network 1300 comprises a backbone 1310 and a MIL pooling module 1320. The backbone 1310 is used for extracting a plurality of patch-level features of an individual tissue patch in the plurality of tissue patches. The MIL pooling module 1320 is implemented with a MIL pooling method for aggregating respective pluralities of patch-level features extracted for the plurality of tissue patches and predicting a score indicating a probability of presence of tumor as observed on the working WSI. The score may simply be the probability of presence of tumor, or may be a number indicative to a likelihood that any tumor is present in the tissue sample.

As disclosed in Section A. 2, max-pooling, mean-pooling and attention-based pooling can be used as the MIL pooling method. For certain advantages, it is preferable that attention-based pooling is used as the MIL pooling method such that the MIL network 1300 is an attention-based MIL network. Anyway, it is also possible to use max-pooling or mean-pooling as the MIL pooling method.

In certain embodiments, the attention-based MIL network is configured as a CLAM model. Background details of the CLAM model can be found in the published literature, e.g., in [21] , the disclosure of which is incorporated by reference herein. As mentioned in Section A. 2, using the CLAM model instead of using max-pooling, mean-pooling or RNN aggregation offers an advantage of being more data-efficient.

In certain embodiments, the CLAM model used for configuring the attention-based MIL network is configured to provide multiclass tumor subtyping in tumor diagnosis of the tissue sample.

In certain embodiments, the CLAM model is configured to provide multiclass disease typing in diagnosis of the tissue sample. Note that special stains can be used for assisting identification of non-neoplastic diseases, vascular diseases, etc. Performing AI diagnosis with a virtual special stain may lead to an application beyond only tumor subtyping.

Apart from realizing the tumor-diagnosis ML model as the MIL network, in general one may realize the tumor-diagnosis ML model as a weakly-supervised ML model. Alternatively, one may also realize the tumor-diagnosis ML model as a supervised ML model. Note that, as mentioned above, there are practical advantages of using weakly-supervised ML models over supervised ones, such as reducing the costs of preparing training datasets.

In certain embodiments, the tumor-diagnosis ML model (124 or 125) is further used in the step 1250 to, after one or more tumors are diagnosed in the working WSI, perform prognosis prediction of a likelihood of patient outcome, and/or perform mutation prediction of patient genes. In the prognosis prediction, one or more items of information extracted from the working WSI are used to predict the likelihood of patient outcome. The likelihood of patient outcome may be recurrence likelihood, patient-survival likelihood, other outcome after a surgical and/or drug treatment, etc. The one or more items may include tumor grade, tissue component (e.g., stromal content) , mutation information, etc. Mutation prediction results may be used to serve as prescreening to improve cost-efficiency before immunohistochemistry or next-generation sequencing. Examples of prognosis prediction and mutation prediction based on H&E-stained images can be found in [23] and [24] , respectively.

Other implementation details of the third method are elaborated as follows.

In certain embodiments, the backbone 1310 of the MIL network 1300 is realized as a CNN. Furthermore, the backbone 1310 may be realized as a ResNet50 model.

In certain embodiments, the backbone 1310 is realized as a vision transformer.

In certain embodiments, the backbone 1310 is realized as a diffusion-based model. The diffusion-based model, which is used in the field of ML and is also commonly referred to as a diffusion model, is a probabilistic generative model that progressively destructs data by injecting noise, and then learns to reverse this process for sample generation.

Note that in the step 1250, the tumor-diagnosis ML model (124 or 125) is operated to perform tumor diagnosis only after the tumor-diagnosis ML model (124 or 125) is trained.

In certain embodiments, the tumor-diagnosis ML model (124 or 125) used in the step 1250 is a pretrained one. The pretrained tumor-diagnosis ML model is obtained by loading pre-calculated operating parameters into an untrained tumor-diagnosis MT model.

Alternatively, in certain embodiments, the workflow 1200 further comprises step 1240. The step 1240, which precedes the step 1250 in execution, is used to train the tumor-diagnosis ML model (124 or 125) , namely, the MIL network 1300.

Also note that tumor diagnosis accomplished by the workflow 1200 is applicable for both thin tissue slice and thick tissue by using an appropriate training dataset in training the MIL network 1300. Consider that the MIL network 1300 is trained with a third training dataset. Preferably, the third training dataset is prepared according to whether a thin tissue slice or a thick tissue is used as the tissue sample.

The virtually stained WSI may be selected to be a virtual H&E-stained WSI 120 of the tissue sample. Note that the virtual H&E-stained WSI 120 is a WSI resulted from virtually staining the tissue sample with the H&E stain. In the step 1220, the working WSI is selected and obtained. The obtaining of the working WSI may comprise obtaining the virtually stained WSI. The obtaining of the virtually stained WSI may comprise using the virtual-staining ML model 115 as disclosed above to generate the virtual H&E-stained WSI 120 from the AF WSI 110. As mentioned above, the virtual-staining ML model 115 may be realized as the U-Frame model.

Alternatively, the virtually stained WSI may be selected to be a virtual specially-stained WSI 130 of the tissue sample, where the virtual specially-stained WSI 130 is a WSI resulted from virtually staining the tissue sample with a special stain other than the H&E stain. One example of the special strain is a MT stain. In the step 1220, similarly, the obtaining of the working WSI may comprise obtaining the virtually stained WSI. The obtaining of the virtually stained WSI may comprise: (a) using the virtual-staining ML model 115 to generate the virtual H&E-stained WSI 120 from the AF WSI 110; and (b) using a stain-transformation ML model 122 to transform the virtual H&E-stained WSI 120 into the virtual specially-stained WSI 130. As mentioned above, each of the virtual-staining ML model 115 and stain-transformation ML model 122 may be individually realized as the U-Frame model.

A fourth aspect of the present disclosure is to extend the workflow 1200 to provide a fourth method for performing tumor diagnosis of the tissue sample.

FIG. 14 depicts a fourth exemplary workflow 1400, which incorporates the third exemplary workflow 1200, for performing tumor diagnosis of the tissue sample according to the fourth method. In step 1410 of the fourth exemplary workflow 1400, a fluorescence microscopy system is used to inspect the tissue sample under illumination of a certain preselected excitation light beam for exciting the tissue sample to produce AF so as to generate the AF WSI 110 of the tissue sample. After the AF WSI 110 is generated in the step 1410, one or more computers are used to execute the third exemplary workflow 1200 realized according to any of the embodiments of the third method.

Embodiments of the various computer-implemented methods disclosed above are realizable by one or more computers. An individual computer may be a general-purpose computer, a portable computer, a mobile computing device such as a smartphone, a computing server, a cloud server, or any computing device deemed appropriate by those skilled in the art.

The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiment is therefore to be considered in all respects as illustrative and not restrictive. The scope of the invention is indicated by the appended claims rather than by the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

REFERENCES

There follows a list of references that are occasionally cited in the specification. Each of the disclosures of these references is incorporated by reference herein in its entirety.

[1] X. H. Gao et al., “Comparison of Fresh Frozen Tissue With Formalin-Fixed Paraffin-Embedded Tissue for Mutation Analysis Using a Multi-Gene Panel in Patients With Colorectal Cancer, ” Front. Oncol., vol. 10, no. March, pp. 1–8, 2020, doi: 10.3389/fonc. 2020.00310.

[2] Y. Rivenson, K. de Haan, W. D. Wallace, and A. Ozcan, “Emerging Advances to Transform Histopathology Using Virtual Staining, ” BME Front., vol. 2020, pp. 1–11, Aug. 2020, doi: 10.34133/2020/9647163.

[3] Y. Rivenson et al., “Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning, ” Nat. Biomed. Eng., vol. 3, no. 6, pp. 466–477, Jun. 2019, doi: 10.1038/s41551-019-0362-y.

[4] Y. Rivenson, T. Liu, Z. Wei, Y. Zhang, K. de Haan, and A. Ozcan, “PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning, ” Light Sci. Appl., vol. 8, no. 1, p. 23, Dec. 2019, doi: 10.1038/s41377-019-0129-y.

[5] T.T.W. Wong et al., “Fast label-free multilayered histology-like imaging of human breast cancer by photoacoustic microscopy, ” Sci. Adv., vol. 3, no. 5, p. e1602168, May 2017, doi: 10.1126/sciadv. 1602168.

[6] I. J. Goodfellow et al., “Generative Adversarial Nets, ” in Neural Information Processing Systems, 2014, pp. 2672–2680.

[7] Y. Zhang, K. de Haan, Y. Rivenson, J. Li, A. Delis, and A. Ozcan, “Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue, ” Light Sci. Appl., vol. 9, no. 1, p. 78, Dec. 2020, doi: 10.1038/s41377-020-0315-y.

[8] W. Dai, I.H.M. Wong, and T.T.W. Wong, “Exceeding the limit for microscopic image translation with a deep learning-based unified framework, ” PNAS Nexus, 2024, 3, pgae133.

[9] L. Kang, X. Li, Y. Zhang, and T.T.W. Wong, “Deep learning enables ultraviolet photoacoustic microscopy based histological imaging with near real-time virtual staining, ” Photoacoustics, vol. 25, p. 100308, 2022, doi: 10.1016/j. pacs. 2021. 100308.

[10] Y. Zhang et al., “High‐Throughput, Label‐Free and Slide‐Free Histological Imaging by Computational Microscopy and Unsupervised Learning, ” Adv. Sci., vol. 9, no. 2, p. 2102358, Jan. 2022, doi: 10.1002/advs. 202102358.

[11] L. Shi, I. H. M. Wong, C.T.K. Lo, and T.T.W. Wong, “One-side Virtual Histological Staining Model for Complex Human Samples, ” BHI-BSN 2022 -IEEE-EMBS Int. Conf. Biomed. Heal. Informatics IEEE-EMBS Int. Conf. Wearable Implant. Body Sens. Networks, Symp. Proc., 2022, doi: 10.1109/BHI56158.2022.9926959.

[12] K. de Haan et al., “Deep learning-based transformation of H&E stained tissues into special stains, ” Nat. Commun., vol. 12, no. 1, pp. 1–13, 2021, doi: 10.1038/s41467-021-25221-2.

[13] B.E. Bejnordi et al., “Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer, ” JAMA -J. Am. Med. Assoc., vol. 318, no. 22, pp. 2199–2210, 2017, doi: 10.1001/jama. 2017. 14585.

[14] N. Coudray et al., “Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning, ” Nat. Med., vol. 24, no. 10, pp. 1559–1567, Oct. 2018, doi: 10.1038/s41591-018-0177-5.

[15] T. C. Hollon et al., “Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks, ” Nat. Med., vol. 26, no. 1, pp. 52–58, Jan. 2020, doi: 10.1038/s41591-019-0715-9.

[16] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition, ” Int. Conf. Learn. Represent., Sep. 2015, [Online] . Available: http: //arxiv. org/abs/1409.1556

[17] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision, ” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2016-Decem, pp. 2818–2826, 2016, doi: 10.1109/CVPR. 2016.308.

[18] M. Ilse, J.M. Tomczak, and M. Welling, “Attention-based Deep Multiple Instance Learning, ” 2018.

[19] G. Campanella et al., “Clinical-grade computational pathology using weakly supervised deep learning on whole slide images, ” Nat. Med., vol. 25, no. 8, pp. 1301–1309, Aug. 2019, doi: 10.1038/s41591-019-0508-1.

[20] B. Li, Y. Li, and K. W. Eliceiri, “Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning, ” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 14313–14323, 2021, doi: 10.1109/CVPR46437.2021.01409.

[21] M.Y. Lu, D.F.K. Williamson, T.Y. Chen, R.J. Chen, M. Barbieri, and F. Mahmood, “Data-efficient and weakly supervised computational pathology on whole-slide images, ” Nat. Biomed. Eng., vol. 5, no. 6, pp. 555–570, 2021, doi: 10.1038/s41551-020-00682-w.

[22] G. Litjens et al., “1399 H&E-stained sentinel lymph node sections of breast cancer patients: The CAMELYON dataset, ” Gigascience, vol. 7, no. 6, pp. 1–8, 2018, doi: 10.1093/gigascience/giy065.

[23] N. Wahab et al., “AI-enabled routine H&E image based prognostic marker for early-stage luminal breast cancer, ” npj Precision Oncology, volume 7, Article number: 122 (2023) .

[24] M. Chen et al., “Classification and mutation prediction based on histopathology H&E images in liver cancer using deep learning, ” npj Precision Oncology, volume 4, Article number: 14 (2020) .

Claims

A computer-implemented method for virtually staining a tissue sample, the computer-implemented method comprising:

acquiring an autofluorescence (AF) whole slide image (WSI) of the tissue sample; and

using a virtual-staining machine-learning (ML) model to generate a virtual hematoxylin and eosin (H&E) -stained WSI from the AF WSI such that the tissue sample is virtually stained with a H&E stain to form the virtual H&E-stained WSI.
The computer-implemented method of claim 1 further comprising:

using a stain-transformation ML model to transform the virtual H&E-stained WSI into a virtual specially-stained WSI such that the tissue sample is virtually stained with a special stain other than the H&E stain to form the virtual specially-stained WSI.
The computer-implemented method of claim 1, wherein the virtual-staining ML model is realized as a U-Frame model.
The computer-implemented method of claim 2, wherein the stain-transformation ML model is realized as the U-Frame model.
The computer-implemented method of claim 2, wherein the special stain is a Masson’s trichrome (MT) stain.
The computer-implemented method of claim 1 further comprising:

training the virtual-staining ML model before the virtual-staining ML model is used in generating the virtual H&E-stained WSI.
The computer-implemented method of claim 6, wherein the virtual-staining ML model is trained with a first training dataset, the first training dataset being prepared according to whether a thin tissue slice or a thick tissue is used as the tissue sample.
The computer-implemented method of claim 2 further comprising:

training the stain-transformation ML model before the stain-transformation ML model is used in generating the virtual specially-stained WSI.
The computer-implemented method of claim 8, wherein the stain-transformation ML model is trained with a second training dataset, the second training dataset being prepared according to whether a thin tissue slice or a thick tissue is used as the tissue sample.
A method for virtually staining a tissue sample, the method comprising:

generating, by a fluorescence microscopy system, an autofluorescence (AF) WSI of the tissue sample; and

executing, by one or more computers, a process of virtually staining the tissue sample according to the computer-implemented method of any of claims 1-9.
A computer-implemented for performing tumor diagnosis of a tissue sample, the computer-implemented method comprising:

acquiring an autofluorescence (AF) whole slide image (WSI) of the tissue sample;

selecting and obtaining a working WSI for tumor diagnosis, wherein the working WSI is selected from the AF WSI and a virtually stained WSI of the tissue sample, and wherein the virtually stained WSI is generated from the AF WSI;

cropping the working WSI to retain one or more tissue regions of the working WSI;

dividing the one or more tissue regions into a plurality of tissue patches; and

using a tumor-diagnosis ML model to process respective tissue patches in the plurality of tissue patches to diagnose any tumor in the working WSI such that tumor diagnosis of the tissue sample is performed without a first need for performing a real histochemical staining process of staining the tissue sample and a second need for requiring a pathologist to interpret a histochemically-stained WSI of the tissue sample.
The computer-implemented method of claim 11, wherein the tumor-diagnosis ML model is realized as a multiple instance learning (MIL) network, the MIL network comprising:

a backbone for extracting a plurality of patch-level features of an individual tissue patch in the plurality of tissue patches; and

a MIL pooling module implemented with a MIL pooling method for aggregating respective pluralities of patch-level features extracted for the plurality of tissue patches and predicting a score indicating a probability of presence of tumor as observed on the working WSI.
The computer-implemented method of claim 12, wherein attention-based pooling is used as the MIL pooling method such that the MIL network is an attention-based MIL network.
The computer-implemented method of claim 13, wherein the attention-based MIL network is configured as a clustering-constrained-attention multiple instance learning (CLAM) model.
The computer-implemented method of claim 14, wherein the CLAM model is configured to provide multiclass tumor subtyping in tumor diagnosis of the tissue sample.
The computer-implemented method of claim 14, wherein the CLAM model is configured to provide multiclass disease typing in diagnosis of the tissue sample.
The computer-implemented method of claim 12, wherein max-pooling or mean-pooling is used as the MIL pooling method.
The computer-implemented method of claim 12, wherein the backbone is realized as a convolutional neural network (CNN) .
The computer-implemented method of claim 18, wherein the backbone is realized as a ResNet50 model.
The computer-implemented method of claim 12, wherein the backbone is realized as a vision transformer.
The computer-implemented method of claim 12, wherein the backbone is realized as a diffusion-based model.
The computer-implemented method of claim 11, wherein the tumor-diagnosis ML model is realized as a weakly-supervised ML model.
The computer-implemented method of claim 11, wherein the tumor-diagnosis ML model is realized as a supervised ML model.
The computer-implemented method of claim 11 further comprising:

after one or more tumors are diagnosed in the working WSI, further using the tumor-diagnosis ML model to perform prognosis prediction of a likelihood of patient outcome.
The computer-implemented method of claim 11 further comprising:

after one or more tumors are diagnosed in the working WSI, further using the tumor-diagnosis ML model to perform mutation prediction of patient genes.
The computer-implemented method of claim 11 further comprising:

training the tumor-diagnosis ML model before the tumor-diagnosis ML model is used in processing the respective tissue patches.
The computer-implemented method of claim 26, wherein the tumor-diagnosis ML model is trained with a third training dataset, the third training dataset being prepared according to whether a thin tissue slice or a thick tissue is used as the tissue sample.
The computer-implemented method of claim 11, wherein:

the virtually stained WSI is a virtual hematoxylin and eosin (H&E) -stained WSI of the tissue sample, wherein the virtual H&E-stained WSI is a WSI resulted from virtually staining the tissue sample with a H&E stain; and

the obtaining of the working WSI comprises obtaining the virtually stained WSI, wherein the obtaining of the virtually stained WSI comprises using a virtual-staining ML model to generate the virtual H&E-stained WSI from the AF WSI.
The computer-implemented method of claim 28, wherein the virtual-staining ML model is realized as a U-Frame model.
The computer-implemented method of claim 11, wherein:

the virtually stained WSI is a virtual specially-stained WSI of the tissue sample, wherein the virtual specially-stained WSI is a WSI resulted from virtually staining the tissue sample with a special stain other than a hematoxylin and eosin (H&E) stain; and

the obtaining of the working WSI comprises obtaining the virtually stained WSI, wherein the obtaining of the virtually stained WSI comprises:

using a virtual-staining ML model to generate a virtual H&E-stained WSI from the AF WSI, wherein the virtual H&E-stained WSI is a WSI resulted from virtually staining the tissue sample with the H&E stain; and

using a stain-transformation ML model to transform the virtual H&E-stained WSI into the virtual specially-stained WSI.
The computer-implemented method of claim 30, wherein the special stain is a Masson’s trichrome (MT) stain.
The computer-implemented method of claim 30, wherein each of the virtual-staining ML model and stain-transformation ML model is individually realized as a U-Frame model.
A method for performing tumor diagnosis of a tissue sample, the method comprising:

generating, by a fluorescence microscopy system, an autofluorescence (AF) WSI of the tissue sample; and

executing, by one or more computers, a process of performing tumor diagnosis of the tissue sample based on the generated AF WSI according to the computer-implemented method of any of claims 11-32.