EP4728486A1 - Synthesizing immunohistochemistry images and predicting biomarker expression levels using machine learning - Google Patents

Abstract

A method for synthesizing an IHC image based on a histology image of a tissue sample. A computer-implemented system generates a first input specifying at least a portion of the histology image, and processes the first input using a prediction neural network to generate a predicted score indicating an expression level of a molecular marker. The system further processes a second input including data specifying the predicted score using a generative neural network to generate a synthesized IHC image that characterizes a spatial distribution of the molecular marker within the tissue sample. The generative neural network has been trained with a discriminator neural network as components of a conditional generative adversarial network (cGAN) that is conditioned on the predicted score.

Description

SYNTHESIZING IMMUNOHISTOCHEMISTRY IMAGES AND PREDICTING BIOMARKER EXPRESSION LEVELS USING MACHINE LEARNING

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to US Provisional Patent Application No. 63/508,743, filed on June 16. 2023 and European Application No. 23305970.8, filed on June 19, 2023, the disclosure of both of which are hereby incorporated by reference in its entirety.

TECHNICAL FIELD

[0002] This specification generally relates to synthesizing immunohistochemistry images from label -free histology images.

BACKGROUND

[0003] The gold standard for cancer diagnosis is the pathological examination, which often involves staining techniques like hematoxylin-eosin (H&E) and immunohistochemistry (1HC). While H&E staining is commonly used, it can occasionally lead to inaccurate cancer diagnoses when used alone. In contrast, the IHC examination provides additional evidence that supports the diagnostic process. However, access to IHC examination can be challenging, especially in regions with limited resources.

[0004] A machine-learning model is a computational model that learns patterns and relationships in data, and then uses that knowledge to make predictions or generate new data. Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

[0005] A conditional Generative Adversarial Network (cGAN) is a type of machine learning model that combines elements of generative modeling and adversarial training to generate realistic and high-quality samples that are conditioned on specific input conditions. It is an extension of the traditional GAN framework that allows control over the generated output by providing additional information or constraints.

[0006] In a standard GAN, there are two main components: a generator network and a discriminator network. The generator network generates synthetic samples, while the discriminator network distinguishes between real and generated samples. The two networks are trained in a competitive setting, where the generator aims to produce samples that are indistinguishable from real samples, while the discriminator aims to correctly classify the samples.

[0007] In a cGAN, additional input conditions are provided to both the generator and the discriminator. These labels can be auxiliary information, such as class labels, attributes, or other relevant attributes. The generator network takes an input that includes the label information and generates samples conditioned on the provided labels. The discriminator network takes both real samples with their corresponding labels and the generated samples with their generated labels as input. The discriminator is configured to distinguish between real and generated samples and correctly classify the provided labels.

SUMMARY

[0008] This disclosure describes methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for synthesizing an immunohistochemistry (IHC) image and predicting a biomarker expression level based on a histology⁷ image.

[0009] In one aspect, this disclosure provides a first method for synthesizing an IHC image based on a histology⁷ image of a tissue sample. The first method can be implemented by a system including one or more computers. The system generates a first input specifying at least a portion of the histology image, and processes the first input using a prediction neural network to generate a predicted score indicating an expression level of a molecular marker. The prediction neural network has been trained using training data. The system further processes a second input including data specifying the predicted score using a generative neural netw ork to generate a synthesized IHC image that characterizes a spatial distribution of the molecular marker within the tissue sample. The generative neural network has been trained with a discriminator neural network as components of a conditional generative adversarial network (cGAN) that is conditioned on the predicted score. The system outputs one or more of (i) the predicted score or (ii) the synthesized IHC image.

[0010] In some implementations of the first method, the prediction neural network includes (i) a first segmentation network configured to predict a first binary image marking tumor cells in the input image and (ii) a second segmentation network configured to predict a second binary image marking tumor cells expressing the molecular marker in the input image.

[0011] In some implementations of the first method, the molecular marker is Carcinoembryonic Antigen-Related Cell Adhesion Molecule (CEACAM5).

[0012] In some implementations of the first method, the histology' image is a hematoxylin- eosin (H&E) stained histology image.

[0013] In some implementations of the first method, the predicted score is a tumor proportion score that quantifies a percentage of tumor cells that express the molecular marker.

[0014] In some implementations of the first method, the generative model is a diffusion model, and the second input further includes data specifying a noise signal.

[0015] In some implementations of the first method, the prediction neural network includes a U-Net that includes an encoder network, a decoder network, and a bottleneck layer.

[0016] In another aspect, this disclosure provides a second method for training a machinelearning model for synthesizing an IHC image based on a histology⁷ image of a tissue sample. The second method can be implemented by a system including one or more computers. The machine learning model includes (i) a prediction neural network configured to generate a predictive score indicating an expression level of a molecular marker from the histology image, and (ii) a conditional generative adversarial network (cGAN) including a generative neural network and a discriminator neural network. The cGAN is conditioned on the predicted score and configured to generate a synthesized IHC image. The system obtains training data including a set of training examples. Each respective training example includes (i) a respective training input including a histology⁷ image, (ii) a respective IHC image corresponding to the training input, and (iii) a respective annotation label of the predictive score. The system trains the prediction neural network using at least a first portion of the training data, and trains the cGAN using at least (i) a second portion of the training data and (ii) the scores predicted by the prediction neural network.

[0017] In some implementations of the second method, the prediction neural network includes (i) a first segmentation network configured to predict a first binary image marking tumor cells in the input image and (ii) a second segmentation network configured to predict a second binary image marking tumor cells expressing the molecular marker in the input image, and each training example further includes (iv) a respective first segmentation mask image marking tumor cells in the respective histology' image and (v) a respective second segmentation mask image marking tumor cells expressing the molecular marker in the input image in the respective histology image.

[0018] In some implementations of the second method, to obtain the training data, the system receives a histology image of a tissue sample, receives an IHC image of the tissue sample, performs image alignment of the histology image and the IHC image, divides each of the histology^ image and the IHC image into a respective set of image patches, and obtains annotation labels for the predictive score for a selected set of image patches.

[0019] In some implementations of the second method, the system performs an activelearning loop for selecting image patches for annotation. In some implementations, the system can apply collective outliers in the active-learning loop and/or apply domain knowledge in the active-learning loop. The domain knowledge can include one or more of: a biopsy type, an organ location, a cut type, or a cancer cohort.

[0020] In some implementations of the second method, the molecular marker is Carcinoembryonic Antigen-Related Cell Adhesion Molecule (CEACAM5).

[0021] In some implementations of the second method, the histology image is a hematoxylin- eosin (H&E) stained histology' image.

[0022] In some implementations of the second method, the predicted score is a tumor proportion score that quantifies a percentage of tumor cells that express the molecular marker.

[0023] In some implementations of the second method, wherein the generative model is a diffusion model, and the second input further includes data specifying a noise signal. [0024] In some implementations of the second method, wherein the prediction neural network includes a U-Net that includes an encoder network, a decoder network, and a bottleneck layer.

[0025] This disclosure also provides a system including one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the methods described above.

[0026] This disclosure also provides one or more computer storage media storing instructions that when executed by one or more computers, cause the one or more computers to perform the methods described above.

[0027] The subject matter described in this disclosure can be implemented in particular embodiments so as to realize one or more advantages.

[0028] The described techniques provide a machine-learning model for synthesizing an IHC image and predicting a biomarker expression level based on a histology' image.

[0029] The gold standard for cancer diagnosis is the pathological examination, which often involves staining techniques like hematoxylin-eosin (H&E) and immunohistochemistry (IHC). While H&E staining is commonly used, it can occasionally lead to inaccurate cancer diagnoses when used alone. In contrast, the IHC examination provides additional evidence that supports the diagnostic process.

[0030] However, access to the IHC examination can be challenging, especially in regions with limited resources. To address this issue, this specification provides techniques for generating virtual IHC images from H&E-stained images. This approach allows for the extraction of valuable IHC-like information from H&E images, enabling improved accuracy in cancer diagnosis, even in resource-constrained areas. By digitally simulating IHC staining, virtual IHC images can enhance the diagnostic capabilities of pathologists and provide valuable insights to aid in accurate cancer detection, characterization, and treatment evaluation.

[0031] In particular, this specification provides techniques for synthesizing an IHC image and predicting a biomarker expression level of a particular molecular marker CEACAM5. CEACAM5 is a glycoprotein that belongs to the CEA family of cell adhesion molecules. CEACAM5 is primarily expressed in epithelial cells and is found in various tissues, including the colon, gastrointestinal tract, pancreas, lung, and breast. It plays a role in cell adhesion, cell signaling, and immune responses. CEACAM5 is involved in cell-cell interactions and has been implicated in several physiological and pathological processes.

[0032] CEACAM5 is a biomarker that exhibits elevated expression in certain types of cancer. By staining tissue samples with CEACAM5-specific antibodies in IHC, pathologists can detect and quantify the presence of CEACAM5 protein. This aids in cancer diagnosis and provides valuable information about tumor characteristics. For example, CEACAM5 staining in IHC can serve as a diagnostic aid, helping to distinguish between malignant and benign tissues. The differential expression of CEACAM5 in cancerous tissues aids in confirming the presence of cancer and can assist in distinguishing between different subtypes or stages of cancer.

[0033] CEACAM5 expression levels in cancer tissue have been associated with prognosis and disease progression. High expression of CEACAM5 has been linked to more aggressive tumor behavior, increased metastasis, and poorer patient outcomes. Thus, IHC analysis of CEACAM5 can provide insights into the prognosis of cancer patients, guiding treatment decisions and patient management.

[0034] CEACAM5 can also be used as a target for cancer therapy. Monoclonal antibodies specifically designed to bind to CEACAM5 have been developed for immunotherapy approaches, including antibody-drug conjugates (ADCs) and radioimmunotherapy. IHC staining for CEACAM5 helps identify patients who are likely to respond to these targeted therapies, facilitating personalized treatment strategies.

[0035] CEACAM5 staining can be used in combination with other markers in IHC panels to provide a comprehensive characterization of tumor tissue. It can be used alongside other antibodies to assess molecular subtypes, identify cancer-associated proteins, or evaluate tumor heterogeneity.

[0036] By virtually synthesizing the CEACAM5 IHC images from H&E histology images, the techniques described in this specification provide the above benefit of the CEACAM5 biomarker without needing to perform IHC staining which is time-consuming, expensive, and subject to inter-observer variability. [0037] In certain implementations, the described techniques use an active-learning loop to efficiently select image patches for annotation. As a result, the training data can be efficiently generated, requiring fewer annotations and fewer computational resources for generating the training data and performing the training.

[0038] Based on the synthesized IHC image and/or the predicted biomarker expression level, the described system or another system can determine an action (e.g., a future treatment plan) to be taken for the patient, and transmit an output to a device of a clinician to report the treatment efficacy, disease progression, and/or a recommendation for future treatment. Accordingly, near real-time monitoring and treatment planning can be achieved for the management of the disease.

BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG. 1 shows an example environment for generating a synthesized immunohistochemistry' (IHC) image for a tissue sample.

[0040] FIG. 2 is a flow diagram of an example process for generating a synthesized IHC image.

[0041] FIG. 3 is a flow diagram of an example process for training a machine learning model for generating a synthesized IHC image.

[0042] FIG. 4 illustrates a performance comparison between the described technique and a benchmark method for predicting a tumor proportion score that quantifies a percentage of tumor cells that express a molecular marker.

[0043] FIG. 5 is a block diagram of an example computer system.

[0044] Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

[0045] FIG. 1 shows an example environment 100 for an IHC image generation system 120 to generate a synthesized immunohistochemistry (IHC) image 150 for a tissue sample 112 obtained from a subject 110. [0046] The tissue sample 112 can be a biopsy sample obtained from the subject 110. Examples of biopsy samples include core needle biopsies, excisional biopsies, incisional biopsies, punch biopsies, or shave biopsies. The tissue sample 112 can be obtained from various organs or sites of the subject 110 such as the breast, lung, colon, prostate, skin, lymph nodes, liver, or any other organ or tissue suspected of harboring disease. The tissue sample 112 can be prepared for histology imaging and analysis, such as by fixation (e.g., in formalin), embedding (e.g.. in paraffin wax), sectioning (e.g.. using a microtome), and staining (e.g., with hematoxylin and eosin (H&E) stain). The prepared tissue sample can then be imaged using an imaging device, e.g., a whole slide imaging (WSI) scanner to generate the histology image 114.

[0047] The IHC image generation system 120 is configured to process the histology image 114 to generate the synthesized IHC image 150. In this specification, an IHC image refers to an image that specifies the spatial distribution of one or more specific molecular markers (e.g., proteins or antigens) within atissue sample. Conventional IHC images can be generated by staining tissue sections with antibodies that specifically bind to the target proteins or antigens. These antibodies can then be visualized using techniques such as chromogenic detection with a colored enzyme substrate, or fluorescent detection with a fluorescent dye. The resulting IHC images can be used for various purposes, such as disease diagnosis, prognosis, and treatment selection. The synthesized IHC image 150 is a synthesized image that specifies the spatial distribution and localization of the molecular marker synthesized from the histology⁷ image 114 without using the immunohistochemistiy imaging process which is time-consuming, expensive, and subject to inter-observer variability.

[0048] In some particular implementations, the synthesized IHC image 150 is an IHC image based on the CEACAM5 marker. CEACAM5 is a glycoprotein that belongs to the CEA family of cell adhesion molecules. CEACAM5 is primarily expressed in epithelial cells and is found in various tissues, including the colon, gastrointestinal tract, pancreas, lung, and breast. It plays a role in cell adhesion, cell signaling, and immune responses. CEACAM5 is involved in cell-cell interactions and has been implicated in several physiological and pathological processes.

[0049] CEACAM5 is a biomarker that exhibits elevated expression in certain types of cancer. The CEACAM5-based synthesized IHC image 150 can detect and quantify the presence of CEACAM5 protein in the tissue sample. This aids in cancer diagnosis and provides valuable information about tumor characteristics. For example, the CEACAM5- based synthesized IHC image 150 can serve as a diagnostic aid, helping to distinguish between malignant and benign tissues. The differential expression of CEACAM5 in cancerous tissues aids in confirming the presence of cancer and can assist in distinguishing between different subty pes or stages of cancer.

[0050] CEACAM5 expression levels in cancer tissue have been associated with prognosis and disease progression. High expression of CEACAM5 has been linked to more aggressive tumor behavior, increased metastasis, and poorer patient outcomes. Thus, analysis of the CEACAM5-based synthesized IHC image 150 can provide insights into the prognosis of cancer patients, guiding treatment decisions and patient management.

[0051] CEACAM5 can also be used as a target for cancer therapy. Monoclonal antibodies specifically designed to bind to CEACAM5 have been developed for immunotherapy approaches, including antibody-drug conjugates (ADCs) and radioimmunotherapy. Thus, analysis of the CEACAM5-based synthesized IHC image 150 can be used to identify patients who are likely to respond to these targeted therapies, facilitating personalized treatment strategies.

[0052] CEACAM5-based IHC images can be used in combination with other markers in IHC panels to provide a comprehensive characterization of tumor tissue. Thus, the CEACAM5- based synthesized IHC image 150 can be used alongside other antibodies to assess molecular subtypes, identify cancer-associated proteins, or evaluate tumor heterogeneity.

[0053] The IHC image generation system 120 includes a prediction neural network 122 configured to generate a predicted score indicating an expression level of a molecular marker and a generative neural network 124 configured to generate the synthesized IHC image 150. Further details of operations of the IHC image generation system 120 are described with reference to FIG. 2.

[0054] The values of the model parameters of the prediction neural network 122 and the generative neural network 124 are determined by a training system 130. The training system 130 uses a plurality of training examples 140 to train the prediction neural network 122 and the generative neural network 124. In particular, the generative neural network 124 is trained with a discriminator neural network 132 as components of a conditional generative adversarial network (cGAN) 126. Further details of operations of the training system 130 are described with reference to FIG. 3.

[0055] FIG. 2 is a flow diagram illustrating an example process 200 for generating a synthesized IHC image. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, an IHC image generation system 120, e.g., the IHC image generation system 120 of FIG. 1, appropriately programmed in accordance with this disclosure, can perform the process 200.

[0056] At 210, the system generates a first input comprising an input image specifying at least a portion of the histology image. In some cases, the histology image can be a hematoxylin-eosin (H&E) stained histology⁷ image of a tissue sample from a subject. As described with reference to FIG. 1 , the histology image can be a histology image obtained from any appropriate tissue sample, such as a biopsy sample from the subject.

[0057] At 220, the system processes the first input using a prediction neural network to generate a predicted score that indicates an expression level of a molecular marker. In some implementations, the molecular marker can be the CEAC AM5 marker.

[0058] In some implementations, the predicted score characterizes the percentage of tumor cells number that express the biomarker (e.g.. the CEACAM5 membrane staining) above a threshold intensify (e.g., > 2 intensity) in total number (Tu) of viable tumor cells present in the section, i.e., the predicted score is a Biomarker Expression Proportion Score (PS)

PS = C_p/Tu (1).

[0059] The prediction neural network can have any appropriate architecture. In some implementations, the prediction neural network includes (i) a first segmentation network configured to process the histology image to predict a first binary image marking tumor cells in the input image and (ii) a second segmentation network configured to process the histology image to predict a second binary image marking tumor cells expressing the molecular marker in the input image. The first binary' image can be used to compute Tu. and the second binary image can be used to compute C_p. Each of the segmentation networks can include a U-Net that includes an encoder network, a decoder network, and a bottleneck layer. [0060] As will be further described with reference to FIG. 3, the prediction neural network has been trained using training data to predict the predicted score.

[0061] At 230, the system processes a second input comprising data specifying the predicted score using a generative neural network to generate a synthesized IHC image that characterizes a spatial distribution of the molecular marker within the tissue sample.

[0062] The generative neural network can be any appropriate generative model configured to synthesize an image. In some implementations, the generative neural network includes a diffusion model, and the second input further includes data specifying a noise signal. The noise signal can serve as the starting point for the diffusion process, which progressively refines the noise into a structured image. The diffusion model processes the noise signal and iteratively refines the noise signal by applying a series of learned denoising steps. Each step removes some of the noise and adds image structure, gradually transforming the noise signal into a synthesized image. The diffusion model is conditioned on the predicted score. This conditioning mechanism enables the model to incorporate the information from the prediction neural network, ensuring that the generated IHC image is consistent with the predicted expression level of the molecular marker. By leveraging this conditioning, the diffusion model produces synthetic IHC images that are not only visually realistic but also biologically plausible and informative.

[0063] As will be further described with reference to FIG. 3, the generative neural network has been trained with a discriminator neural network as components of a conditional generative adversarial network (cGAN) that is conditioned on the predicted score.

[0064] As described in more details with reference to FIG. 1, the synthesized IHC image can be a synthesized CEACAM5 image. That is, the synthesized IHC image specifies a spatial distribution of the CEACAM5 marker within the tissue sample.

[0065] At 240, the system outputs one or more of (i) the predicted score or (ii) the synthesized IHC image. As described with reference to FIG. 1. the predicted score and/or the synthesized IHC image can be used to aid disease diagnosis (e.g.. distinguishing between malignant and benign tissues), disease characterization (distinguishing between different subtypes or stages of cancer), provide information on disease prognosis (e.g., providing insights into the prognosis of cancer patients and guiding treatment decisions and patient management), identify patients who are likely to respond to particular targeted therapies, and/or be used alongside other antibodies to assess molecular subty pes, identify cancer-associated proteins, or evaluate tumor heterogeneity.

[0066] FIG. 3 is a flow diagram illustrating an example process 300 for training a machine learning model for generating a synthesized IHC image. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system 130, e.g., the training system 130 of FIG. 1, appropriately programmed in accordance with this disclosure, can perform the process 300.

[0067] The machine-learning model for generating the synthesized IHC image includes (i) a prediction neural network configured to generate a predictive score indicating an expression level of a molecular marker from the histology⁷ image, and (ii) a conditional generative adversarial network (cGAN) that includes (a) a generative neural network and (b) a discriminator neural network. The cGAN is conditioned on the predicted score and configured to generate a synthesized IHC image.

[0068] At 310. the system obtains training data including a set of training examples. Each respective training example includes (i) a respective training input comprising a respective histology image, (ii) a respective IHC image corresponding to the respective training input, and (iii) a respective annotation label of the predictive score.

[0069] In some implementations, to obtain the training data, the system obtains a histology image of a tissue sample, obtains an IHC image of the tissue sample, and performs image alignment of the histology⁷ image and the IHC image. In some cases, the image alignment can be performed using image registration techniques. Image registration is the process of transforming different sets of data into one coordinate system. The histology image and the IHC image of the same tissue sample can be misaligned due to various reasons, such as the different staining techniques, tissue deformation during sectioning and mounting, or variations in image acquisition settings. Image registration can help to correct these misalignments.

[0070] Whole slide images (WSIs) present unique challenges due to their large size, varied formats, and multi-channel nature. To address these challenges, the system can read each WSI in smaller, manageable tiles. These tiles can be processed and aligned, and subsequently merged into a single whole-slide image. This approach allows the system to efficiently process large WSIs while minimizing memory usage.

[0071] During image registration, the system can apply both rigid and non-rigid transformations to achieve alignment between the histology and IHC images. The system may use various image registration techniques, such as feature-based registration, intensitybased registration, or a combination of both. Feature-based registration involves identifying and matching distinctive features, such as points, lines, or regions, in the two images. Intensity-based registration, on the other hand, involves directly comparing the pixel intensities of the two images. In some cases, due to variations in tissue sectioning and slide preparation, the registration process prioritizes aligning tissue structures and overall morphology rather than achieving precise pixel-level correspondence.

[0072] In some implementations, each of one or more of the training examples further includes (iv) a respective first segmentation mask image marking tumor cells in the respective histology image and (v) a respective second segmentation mask image marking tumor cells expressing the molecular marker in the histology image in the respective histology image.

[0073] In some cases, the system divides each of the histology image and the IHC image into a respective set of image patches, and obtains annotation labels for the predictive score for a selected set of image patches. In some cases, the system can include patches that partially extend beyond the edges of the WSI, capturing peripheral tissue regions. Additionally, the system can include patches centered on specific coordinates identified by pathologists as areas of interest.

[0074] Any suitable patch sizes, e.g., 256x256 pixels, 512x512 pixels, or 1024x 1024 pixels, can be used. In a particular example, a patch size of 512x512 pixels is used. This size represents a balance between capturing sufficient tissue context and maintaining computational efficiency.

[0075] To ensure a representative dataset, the system can implement a random patch selection process, considering both regional representation and overall balance within the WSI. Furthermore, the system can assess the registration quality at the patch level, discarding any misaligned pairs of histology⁷ and IHC patches to maintain data integrity. As described below, obtaining the training data and training the predicted neural network can be performed in an active-learning loop for selecting image patches for annotation. [0076] At 320, the system trains the prediction neural network using at least a first portion of the training data.

[0077] In some implementations, the prediction neural network includes (i) a first segmentation network configured to predict a first binary image marking tumor cells in the histology image and (ii) a second segmentation network configured to predict a second binary image marking tumor cells expressing the molecular marker in the histology image.

[0078] In some implementations, to address the challenge of annotating a large number of patch dataset, obtaining the training data and training the predicted neural network are performed in an active-learning loop for selecting image patches for annotation. In some cases, the system can apply collective outliers in the active learning loop. In some cases, the system can apply domain knowledge in the active learning loop. The domain knowledge can include, e.g., a biopsy type, an organ location, a cut type, and/or a cancer cohort.

[0079] In particular, the average batch Dice score and domain knowledge information associated with the images can be utilized to identify potential collective outliers, characterized by low model confidence during training. The batch Dice score is a metric that measures the similarity between predicted segmentation masks and the corresponding ground truth masks for a batch of images. The system can assign domain knowledge information to the validation dataset to track Dice scores for different attributes. Active learning can then be conducted iteratively over a series of acquisition iterations (T). In each iteration, an acquisition function guides the selection of a new batch of patches, denoted as Qt, to be annotated. The prediction model is then retrained after each acquisition iteration, incorporating the newly annotated data.

[0080] At 330. the system trains the cGAN using at least (i) a second portion of the training data and (ii) the predictive scores predicted by the prediction neural network. The cGAN includes (a) the generative neural network and (b) the discriminator neural network. The cGAN is conditioned on the predicted score and configured to generate a synthesized IHC image.

[0081] The generative neural network (generator) is configured to generate synthetic IHC images. In some cases, the generative neural network takes as input the predicted scores from the prediction neural network, along with a noise signal, and generates a synthesized image that mimics the spatial distribution of the molecular marker in an actual IHC image. The generator, denoted as G, aims to capture the underlying data distribution of IHC images and produce samples that are indistinguishable from real IHC images.

[0082] The discriminator neural network (discriminator), denoted as D, plays the role of an adversary to the generator. Its task is to estimate the probability that a sample came from the training data rather than being generated by G. For example, the discriminator neural network can take an image as input and output a probability indicating whether it believes the image is real or fake. The discriminator neural network can adopt any appropriate architecture, such as convolutional neural network (CNN) and the multi-scale discriminator that combines multiple discriminators operating at different image scales.

[0083] In some implementations, the cGAN training process follows the two-player minmax game with the objective function V(G, D). D x I PS) represents discriminator outputs for real image samples, given the extra information of proportion score. Similarly, D(G(z I PS)) is the discriminator output for generated image sample by G. This process is shown as mmmax7(£>, G) [log D x I PS)] +

®z~_Pnoise (z) [log (1 S)))] ⁽²⁾

[0084] The goal of the generator is to generate synthetic IHC staining images, taking into account underlying tissue-related biomarker information. Hence, the system provides the biomarker expression proportion score from the prediction model as additional conditioning information to the cGAN. This conditioning guides the generator to produce synthetic IHC images that align with the predicted biomarker expression levels, ensuring the generated images are biologically plausible and relevant.

[0085] The adversarial nature of the cGAN training process drives both the generator and discriminator to improve continuously. Through this iterative competition, the cGAN leams to generate high-quality synthetic IHC images that accurately reflect the spatial distribution of the molecular marker within the tissue sample, conditioned on the information extracted from the histology image by the prediction neural network.

[0086] FIG. 4 illustrates a performance comparison between the described technique and a benchmark method for predicting a tumor proportion score that quantifies a percentage of tumor cells that express a molecular marker. [0087] In particular, WSIs (including H&E-stained tumor and IHC staining images) from 317 patients were collected, where 268 patients' data was used for a 5-fold cross-validation. The rest 49 patients’ data was used as a separate testing dataset. Annotations (tumor areas, CEAC AM5 high areas) for each slide were generated and fine-tuned using the active learning approach.

[0088] The data was partitioned into five folds based on patient IDs and the PS score was predicted using five-fold cross validation. In each fold, C_p and Tu were calculated by two separate U-Net models respectively during the training and validation steps. U-Net models were trained and selected by the best batch dice score. PS scores of the testing dataset in each fold were calculated with the best model. The prediction performance metrics include the AUC (Area Under the Curve) and the AP (Average Precision).

[0089] The performance metrics are compared between the described technique and a benchmarking technique (“MIL”) which is described in Lu, et al., “Data-efficient and weakly supervised computational pathology on whole-slide images,” Nature Biomedical Engineering 5(6), 555-570 (2021). As shown in FIG. 4, the technique described herein performs generally better than the benchmark method.

[0090] FIG. 5 is a block diagram of an example computer system 500 that can be used to perform the operations described above. The system 500 includes a processor 510, a memory 520, a storage device 530, and an input/output device 540. Each of the components 510, 520, 530, and 540 can be interconnected, for example, using a system bus 550. The processor 510 is capable of processing instructions for execution within the system 500. In one implementation, the processor 510 is a single-threaded processor. In another implementation, the processor 510 is a multi -threaded processor. The processor 510 is capable of processing instructions stored in the memory 520 or on the storage device 530.

[0091] The memory 520 stores information within the system 500. In one implementation, the memory 520 is a computer-readable medium. In one implementation, the memory' 520 is a volatile memory’ unit. In another implementation, the memory’ 520 is a non-volatile memory unit.

[0092] The storage device 530 is capable of providing mass storage for the system 500. In one implementation, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (for example, a cloud storage device), or some other large capacity storage device.

[0093] The input/output device 540 provides input/output operations for the system 500. In one implementation, the input/output device 540 can include one or more network interface devices, for example, an Ethernet card, a serial communication device, for example, a RS- 232 port, and/or a wireless interface device, for example, a 502.11 card. In another implementation, the input/output device can include driver devices configured to receive data and send output data to other input/output devices, for example, keyboard, printer and display devices 760. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0094] Although an example processing system has been described in FIG. 5, implementations of the subject matter and the functional operations described in this disclosure can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this disclosure and their structural equivalents, or in combinations of one or more of them.

[0095] This disclosure uses the term "configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this disclosure can be implemented in digital electronic circuitry, in tangibly- embodied computer software or firmware, in computer hardware, including the structures disclosed in this disclosure and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this disclosure can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0096] The term "data processing apparatus’’ refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0097] A computer program, which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g.. one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0098] In this disclosure, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. [0099] Similarly, in this disclosure the term “engine’' is used broadly to refer to a softwarebased system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0100] The processes and logic flows described in this disclosure can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g.. an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0101] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0102] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0103] To provide for interaction with a user, embodiments of the subject matter described in this disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0104] Data processing apparatus for implementing machine-learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine-learning training or production, i.e., inference, workloads.

[0105] Machine learning models can be implemented and deployed using a machine-learning framework, e.g., a PyTorch or a TensorFlow framework.

[0106] Embodiments of the subject matter described in this disclosure can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this disclosure, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0107] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0108] While this disclosure contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this disclosure in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0109] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for synthesizing an immunohistochemistry (IHC) image based on a histology image of a tissue sample, the method comprising: generating a first input comprising an input image specifying at least a portion of the histology image; processing the first input using a prediction neural network to generate a predicted score indicating an expression level of a molecular marker, wherein the prediction neural network has been trained using training data; processing a second input comprising data specifying the predicted score using a generative neural network to generate a synthesized IHC image that characterizes a spatial distribution of the molecular marker within the tissue sample, wherein the generative neural network has been trained with a discriminator neural network as components of a conditional generative adversarial network (cGAN) that is conditioned on the predicted score; and outputting one or more of (i) the predicted score or (ii) the synthesized IHC image.

2. The method of claim 1, wherein the prediction neural network comprises (i) a first segmentation network configured to predict a first binary image marking tumor cells in the input image and (ii) a second segmentation network configured to predict a second binary image marking tumor cells expressing the molecular marker in the input image.

3. The method of claim 1 or claim 2, wherein the molecular marker is Carcinoembryonic Antigen-Related Cell Adhesion Molecule (CEACAM5).

4. The method of any of claims 1-3, wherein the histology image is a hematoxylin-eosin (H&E) stained histology image.

5. The method of any of the preceding claims, wherein the predicted score is a tumor proportion score that quantifies a percentage of tumor cells that express the molecular marker.

6. The method of any of the preceding claims, wherein the generative neural network comprises a diffusion model, and the second input further comprises data specifying a noise signal.

7. The method of any of the preceding claims, wherein the prediction neural network comprises a U-Net that includes an encoder network, a decoder network, and a bottleneck layer.

8. A method for training a machine-learning model for synthesizing an immunohistochemistry (IHC) image based on a histology image of a tissue sample, wherein the machine-learning model comprises (i) a prediction neural network configured to generate a predictive score indicating an expression level of a molecular marker from the histology image, and (ii) a conditional generative adversarial network (cGAN) comprising a generative neural network and a discriminator neural network, wherein the cGAN is conditioned on the predicted score and configured to generate a synthesized IHC image, the method comprising: obtaining training data comprising a set of training examples, each respective training example comprises (i) a respective training input comprising a respective histology image, (ii) a respective IHC image corresponding to the respective training input, and (iii) a respective annotation label of the predictive score; training the prediction neural network using at least a first portion of the training data; and training the cGAN using at least (i) a second portion of the training data and (ii) the predictive scores predicted by the prediction neural network.

9. The method of claim 8, wherein: the prediction neural network comprises (i) a first segmentation network configured to predict a first binary image marking tumor cells in the histology image and (ii) a second segmentation network configured to predict a second binary image marking tumor cells expressing the molecular marker in the histology image; and each of one or more of the training examples further comprises (iv) a respective first segmentation mask image marking tumor cells in the respective histology image and (v) a respective second segmentation mask image marking tumor cells expressing the molecular marker in the histology image in the respective histology image.

10. The method of claim 8 or claim 9, wherein obtaining the training data comprises: receiving a histology image of a tissue sample; receiving an IHC image of the tissue sample; performing image alignment of the histology image and the IHC image; dividing each of the histology image and the IHC image into a respective set of image patches; and obtaining annotation labels for the predictive score for a selected set of image patches.

11. The method of claim 10. wherein obtaining the training data and training the predicted neural network are performed in an active-learning loop for selecting image patches for annotation.

12. The method of claim 11, further comprising: applying collective outliers in the active-learning loop.

13. The method of claim 11 or 12, further comprising: applying domain knowledge in the active-learning loop.

14. The method of claim 13, wherein the domain knowledge comprises one or more of: a biopsy type; an organ location; a cut type; or a cancer cohort.

15. The method of any of claims 8-14, wherein the molecular marker is Carcinoembryonic Antigen-Related Cell Adhesion Molecule (CEACAM5).

16. The method of any of claims 8-15, wherein the histology image is a hematoxylin-eosin (H&E) stained histology image.

17. The method of any of claims 8-16. wherein the predicted score is a tumor proportion score that quantifies a percentage of tumor cells that express the molecular marker.

18. The method of any of claims 8-17. wherein the generative model is a diffusion model, and the second input further comprises data specifying a noise signal.

19. The method of any of claims 8-18, wherein the prediction neural network comprises a U-Net that includes an encoder network, a decoder network, and a bottleneck layer.

20. A computer-implemented method for synthesizing an immunohistochemistry (IHC) image based on a histology image of a tissue sample, the method comprising: generating a first input comprising an input image specifying at least a portion of the histology image; processing the first input using a prediction neural network to generate a predicted score indicating an expression level of a Carcinoembryonic Antigen-Related Cell Adhesion Molecule (CEACAM5) marker, wherein the prediction neural network has been trained using training data; processing a second input comprising data specifying the predicted score using a generative neural network to generate a synthesized IHC image that characterizes a spatial distribution of the CEACAM5 marker within the tissue sample, wherein the generative neural network has been trained with a discriminator neural network as components of a conditional generative adversarial network (cGAN) that is conditioned on the predicted score; and outputting one or more of (i) the predicted score or (ii) the synthesized IHC image.

21. The method of claim 20, wherein the prediction neural network comprises (i) a first segmentation network configured to predict a first binary image marking tumor cells in the input image and (ii) a second segmentation network configured to predict a second binary image marking tumor cells expressing the CEACAM5 marker in the input image.

22. The method of claim 20. wherein the histology image is a hematoxylin-eosin (H&E) stained histology image.

23. A system comprising: one or more computers; and one or more storage devices storing instructions that when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any preceding claim.

24. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-22.