CN117378015A - Predicting actionable mutations from digital pathology images - Google Patents

Abstract

The present disclosure provides a method comprising: a digital pathology image depicting tumor cells sampled from a subject is accessed. A plurality of tiles may be selected from the digital pathology image, wherein each of the tiles depicts a tumor cell. A mutation prediction may be generated for each of the tiles, wherein the mutation prediction represents a prediction of a likelihood that an operable mutation occurs in the tile. Based on the plurality of mutation predictions, a prognosis prediction associated with one or more treatment regimens for the subject may be generated. The prognosis prediction may be based on determining one or more mutated backgrounds of the digital pathology image as unknown drivers or tumor suppressors, oncogene driver mutations, or gene fusions.

Description

Predicting actionable mutations from digital pathology images

Technical Field

The present disclosure relates to cancer diagnosis, treatment, and prognosis.

Background Art

An important step in planning cancer treatment is to identify the treatment options that are most likely to be effective, taking into account both the likelihood of disease progression and patient compliance. Once a malignant tumor has grown beyond a certain size and/or has metastasized, the patient's options may be limited to treatments such as chemotherapy, radiotherapy, targeted therapy, and immunotherapy. Patient compliance with chemotherapy and radiotherapy may be challenging due to severe side effects that make the patient uncomfortable. Patients also face the risk that cancer cells may mutate and become resistant to systemic therapy. Immunotherapy is the newest of these treatment options, which activates the patient's own immune system and represents an important option in managing cancer, although it may only be effective for 20% to 30% of tumors. In general, the following basic principles are applied across different tumor types in order to maximize direct clinical benefits and safeguard future treatment options for patients: As a first step, patients with advanced disease should first be screened for eligibility for targeted therapy. If no mutation target is identified, the patient is offered immunotherapy alone or in combination with traditional chemotherapy. As a last resort, if the combination fails, a combination of chemotherapy can be considered.

Targeted therapies may be preferred, but are only applicable to a small number of tumors due to the rarity of addressable mutations. Based on currently approved therapies, addressable mutations only occur on oncogenes (genes that positively regulate different cell functions). In lung cancer, the most common and actionable mutations are found in the following oncogenes: EGFR, ALK, RET, ROS1, and NTRK. In addition to oncogenes, a second class of genes called tumor suppressors can also cause cancer when mutated. As the name suggests, these genes usually play a role in suppressing cell activity; therefore, mutations in these genes can cause cancer via the loss of inhibition. Ultimately, in clinical practice, the most comprehensive molecular characterization of tumors is performed using next-generation sequencing (NGS) panels. This assay detects mutations only in a subset of genes, with some panels querying as few as 25 genes and others querying up to 500 genes. This represents a small fraction of the 20,000 to 25,000 total genes in the human genome. This may explain why no driver mutations were found in a portion of tumor samples tested via NGS (unknown drivers).

Targeted therapies can include drugs that target the epidermal growth factor receptor (EGFR) as well as gene fusions involving anaplastic lymphoma kinase (ALK), RET, ROS1, and neurotrophic tyrosine receptor kinase (NTRK). For EGFR, although immunohistochemical staining can be used to identify the most common variants (e.g., with coverage up to 97% in patients with EGFR-positive lung adenocarcinoma), molecular testing may be required to identify resistance mutations in patients who have failed EGFR-targeted therapy. Such immunohistochemical stains have not been developed for RET and ROS1, and the performance of immunohistochemical stains for ALK and NTRK can be highly variable and difficult to interpret.

Gene fusions usually require more complex molecular assays, which have a greater coverage of the genome than the more commonly used "hotspot" assays for testing a limited number of loci. In order to target gene fusions, a much wider coverage may be required, resulting in much more expensive tests, which require laboratories to demonstrate much greater technical capabilities. However, some gene fusions (e.g., NTRK fusions) may be extremely rare. Although NTRK fusions have been identified in a variety of tumor types, the frequency of this particular fusion may be less than 1% in the most common cancer indications (such as in lung adenocarcinoma, colorectal cancer, and non-secretory breast cancer). The relative rarity of gene fusions (e.g., ranging from 7% for ALK to less than 0.3% for NTRK in lung adenocarcinoma) constitutes a significant technical and financial obstacle to extensive testing. At present, molecular testing is the only method that can be used to determine whether there is a gene fusion in a patient so far. However, molecular testing is expensive, so patients sometimes avoid arranging molecular testing because patients are likely to not benefit from targeted therapy. Therefore, a considerable proportion of patients may be unlikely to receive the right test to determine whether their tumors carry gene fusions. Therefore, a rapid, robust, and sensitive screening tool is needed to triage tumor samples by the type of test required to determine their preferred and optimal systemic treatment options.

Summary of the invention

Provided herein are systems and methods for using digital pathology technologies to identify actionable mutations, including, for example, but not limited to, oncogenic gene fusions (e.g., ALK, ROS1, RET, and NTRK), wherein the actionable mutations predict mutations for which targeted therapies may be useful and the prognosis of treatment response.

In some cases, the disclosed methods and systems can be applied to detect gene fusion/rearrangement, which is a specific type of rare druggable oncogene mutation event that can be identified across many different cancer types and, if present in tumor tissue samples, can indicate a strong response to certain targeted therapies. Gene fusions include rare druggable mutation events that can occur across many different tumor types and are increasingly being targeted for novel therapies. The identification of gene fusions can be a technically difficult, expensive, and time-consuming process that may ultimately benefit only a few patients carrying such genetic changes; for these reasons, widespread testing may be limited to a few hospitals that have the ability to absorb and provide the technical and financial resources involved in the process. The embodiments disclosed herein can address this gap by creating, training, and using a machine learning model (e.g., a digital pathology screening model) that can predict the presence of oncogene fusions from digital pathology images (such as scanned, stained (e.g., hematoxylin and eosin (H&E) stained) whole slide images (WSIs) depicting cancer tissue/cells (e.g., lung adenocarcinoma)). Furthermore, embodiments disclosed herein may include rapid, inexpensive, and sufficiently accurate screening tools that can be used to guide molecular testing and decision making regarding the use of targeted therapies for individuals in patients, including but not limited to patients with lung adenocarcinoma.

In a specific embodiment, the digital pathology image processing system may access a digital pathology image depicting cancer cells in a specific slice of a biological sample from a subject. The digital pathology image processing system may then identify one or more image patches from the digital pathology image, each image patch depicting one or more clusters of tumor cells (e.g., an area consisting entirely of tumor cells or an area including one or more tumor nests (tumornest) structures surrounded by stroma). In some cases, when the digital pathology image has been divided into a plurality of tiles, the image tile may include a portion of the image tile, a plurality of adjacent tiles, or a combination of one or more adjacent tiles and one or more adjacent portions of the tile. The digital pathology image processing system may generate a mark indicating the possibility (e.g., a binary output or a percentage output) of a cluster of tumor cells depicted by the image tile for each of the plurality of image tiles. In a specific embodiment, the digital pathology image processing system may determine that the digital pathology image includes a depiction of the occurrence of, for example, gene fusion in cancer cells present in the biological sample based on the mark generated for each tile. The digital pathology image processing system may further generate a prognosis prediction for the subject based on the detection of, for example, gene fusion. In some embodiments, a prognostic prediction can include a prediction of the suitability of one or more treatment regimens (eg, chemotherapy or targeted therapy) for a subject.

Methods disclosed herein include: accessing a digital pathology image depicting cancer cells in a specific slice of a biological sample from a subject, and wherein the depicted specific slice is stained with one or more stains; segmenting the digital pathology image into a plurality of image tiles; generating, for each of the plurality of image tiles, a label indicating a likelihood that the image tile depicts a cluster of tumor cells; determining, based on the label generated for each image tile, that the digital pathology image includes a depiction of an occurrence of a gene fusion with respect to the cancer cells; and generating a prognostic prediction for the subject based on the occurrence of the gene fusion with respect to the cancer cells, wherein the prognostic prediction includes a prediction of the suitability of one or more treatment regimens for the subject.

In some embodiments, the method further includes: detecting one or more features from each of the multiple image tiles, wherein the one or more features include one or more of clinical features or histological features, and wherein generating a label for each of the multiple image tiles is based on the one or more features.

In some embodiments, generating a label for each of the plurality of image tiles is based on tumor morphology, wherein the tumor morphology is based on an analysis of one or more of: the presence of signet ring cells, the number of signet ring cells, the presence of hepatoid cells, the number of hepatoid cells, extracellular mucin, or a tumor growth pattern.

In some embodiments, generating labels for each of the multiple image tiles is based on one or more machine learning models, wherein the method further includes: training the one or more machine learning models based on multiple training data, the multiple training data including one or more labeled depictions of clustered tumor cells and one or more labeled depictions of other histological or clinical features.

In some embodiments, the prognostic prediction is generated further based on an analysis of one or more additional digital pathology images, each of the one or more additional digital pathology images depicting an additional specific sample in the biological sample from the subject, and wherein the analyzing includes: determining a likelihood that each of the one or more additional digital pathology images includes a depiction of the occurrence of a gene fusion for a cancer cell; and combining the determinations for each of the one or more additional digital pathology images.

In some embodiments, the method further includes: outputting the prognostic prediction via a graphical user interface, wherein the graphical user interface includes a graphical representation of the digital pathology image, and wherein the graphical representation includes: an indication of the label generated for each of the plurality of image tiles, and a predicted confidence level associated with the corresponding label.

In some embodiments, the method further comprises generating a recommendation associated with use of the one or more treatment regimens.

In some embodiments, specific sections of a biological sample are stained with one or more stains.

In some embodiments, determining that the digital pathology image includes a depiction of the occurrence of a gene fusion with respect to the cancer cell is further based on a weighted combination of the labels generated for each image tile.

In some embodiments, the method further comprises: identifying tumor heterogeneity from the digital pathology image; and measuring the identified tumor heterogeneity, wherein determining that the digital pathology image includes delineation of the occurrence of the gene fusion is further based on the measured tumor heterogeneity.

In some embodiments, identifying tumor heterogeneity comprises classifying mutant tumor cells into phenotypes by identifying morphologically similar cells (e.g., by assessing nuclear heterogeneity). In some embodiments, assessing nuclear heterogeneity comprises quantifying certain features of the cell nucleus to distinguish mutant cells based on nuclear morphological heterogeneity.

In some embodiments, identifying tumor heterogeneity includes: identifying regions of clonal cells by performing a cell-level spatial analysis to assess the spatial distribution. In some embodiments, assessing the spatial distribution includes: measuring spectral distances within a subgraph of a minimum spanning tree of tumor cells, wherein each of the subgraphs represents clustered adjacent cells (e.g., tumor nests), and calculating adjacency spectral distances pairwise across all subgraphs. In some embodiments, each of the subgraphs can be defined by performing outlier detection. In some embodiments, each of the subgraphs can be defined based on segmentation of detected tumor nests.

In some embodiments, identifying tumor heterogeneity comprises: identifying regions of closely adjacent clonal cells by performing a cell-level spatial analysis to assess spatial entropy. In some embodiments, assessing spatial entropy comprises: calculating the frequency with which pairs of cells are identified as morphologically similar for each of a predefined number of distance bins.

Disclosed herein are one or more computer-readable non-transitory storage media embodying software that, when executed, is operable to perform a portion or all of one or more methods disclosed herein.

The system disclosed herein includes: one or more processors; and a non-transitory memory, which contains instructions, which when executed by the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

Methods disclosed herein include: transmitting a request communication from a client computing system to a remote computing system to process a digital pathology image depicting cancer cells in a particular slice of a biological sample from a subject, wherein in response to receiving the request communication from the client computing system, the remote computing system performs operations including: accessing the digital pathology image; segmenting the digital pathology image into a plurality of image tiles; generating a label indicating a likelihood (e.g., a binary output or a percentage output) that the image tile depicts a cluster of tumor cells for each of the plurality of image tiles; determining, based on the label generated for each image tile, that the digital pathology image includes a depiction of an occurrence of a gene fusion with respect to the cancer cells; generating a prognostic prediction for the subject based on the occurrence of the gene fusion with respect to the cancer cells, wherein the prognostic prediction includes a prediction of suitability of one or more treatment regimens for the subject; and providing the prognostic prediction to the client computing system via a response communication; and outputting, by the client computing system, the prognostic prediction in response to receiving the response communication.

The methods disclosed herein include: accessing, by a digital pathology image processing system, a digital pathology image depicting cancer cells in a particular slice of a biological sample from a subject; determining that the digital pathology image includes a depiction of one or more mutations that are mutually exclusive with the occurrence of a gene fusion; determining an absence of the gene fusion with respect to the cancer cell; and generating a prognostic prediction for the subject based on the absence of the gene fusion with respect to the cancer cell, wherein the prognostic prediction includes a prediction of the suitability of one or more treatment regimens for the subject.

BRIEF DESCRIPTION OF THE DRAWINGS

Figures 1A-1D illustrate non-limiting examples of tumor heterogeneity.

2 illustrates a non-limiting example of a network of interactive computer systems that may be used for digital pathology image generation and processing as described herein, according to some embodiments of the present disclosure.

FIG. 3 is a flow chart for a non-limiting example method for predicting gene fusions.

FIG. 4 is a flow chart for a non-limiting example pipeline for detecting gene fusions.

FIG. 5A shows a non-limiting example of a pathology slide image for lung adenocarcinoma.

FIG. 5B shows a non-limiting example of results from quality control.

FIG5C shows a non-limiting example of tumor cluster detection.

FIG. 5D shows a non-limiting example of prediction of gene fusion status.

FIG. 6 shows non-limiting example predictions for ROS1 gene fusion status.

FIG. 7 shows a non-limiting example of a ROC curve for image patch based fusion prediction.

8 is a flow diagram for a non-limiting example method for enabling an end user to request a prognostic prediction.

9 is a flow chart for a non-limiting example method for predicting alternative treatments based on identifying the lack (or absence) of a gene fusion.

FIG. 10A is a schematic diagram of tumor heterogeneity when the mutational context corresponds to only tumor suppressors (such as pathogenic variants in KEAP1, STK11, or TP53); and/or unknown drivers.

FIG. 10B is a schematic diagram of tumor heterogeneity when the mutational context corresponds to oncogenic driver mutations, such as pathogenic variants in EGFR, KRAS, or PIK3CA.

FIG. 10C is a schematic diagram of tumor heterogeneity when the mutational context corresponds to an oncogenic gene fusion such as ALK, RET, ROS1, or NTRK.

FIG. 11 is a flow chart for a non-limiting example method for identifying tumor heterogeneity.

12A-12C depict non-limiting examples of tumor cell annotation, cell nucleus segmentation, and cell nucleus mask generation.

Figures 13A-13E show non-limiting examples of experimental results based on quantifying certain nuclear morphological features to identify gene fusions.

14A-14C illustrate non-limiting examples of digital pathology images showing four different nuclear morphological features corresponding to tumor suppressors and/or unknown drivers.

Figures 15A-15C illustrate non-limiting examples of digital pathology images showing four different nuclear morphological features corresponding to oncogenic driver mutations.

16A-16C illustrate non-limiting examples of digital pathology images showing four different nuclear morphological features corresponding to gene fusions.

FIG. 17A shows a non-limiting example of identification of a subgraph of a minimum spanning tree of tumor cells depicted in a WSI.

FIG. 17B shows a non-limiting example of experimental results for identifying gene fusions based on quantification such as contiguous spectral distances between subgraphs.

18A-18C show non-limiting examples of spatial entropy as measured in three different distributions of tumor cell phenotypes.

FIG. 18D is a schematic diagram of the identification of pairs of tumor cells that co-occur within a specified distance and are classified into a specified phenotype.

Figures 18E to 18F show non-limiting examples of experimental results based on measuring spatial entropy to identify gene fusions, oncogene driver mutations, and tumor suppressors and/or unknown drivers.

FIG. 19 shows an example of a computing system.

DETAILED DESCRIPTION

To identify which patients may benefit from molecular testing for gene fusions, pathologists can view slide images of tumor tissue samples to assess indicators of tumor heterogeneity. Tumor heterogeneity can be observed from the distinct morphological and phenotypic features of different tumor cells, which correspond to genetic mutations that cause cells to become cancerous and grow and spread in the body. Tumor heterogeneity can be manifested as intra-tumor heterogeneity (e.g., within a single tumor nest), or as inter-tumor heterogeneity (e.g., as between nearby tumor nests). In addition to being used as a distinguishing factor between normal and tumor tissue, tumor heterogeneity can also be used as an indicator of disease severity—highly heterogeneous tumors (also known as bizarre tumors) can indicate a poor prognosis due to failure of therapy attributable to resistance acquired through genetic mutations.

Figures 1A to 1D show examples of tumor heterogeneity as visualized in digital pathology images. Figure 1A shows an example of tumor heterogeneity in uterine leiomyosarcoma. The tumor cells look very different from each other. For example, in terms of size, there are some tumor cells that are two, three, or four times the size of other tumor cells. In terms of curvature, some tumor cells have a smooth and rounded appearance, while others are angular (i.e., have at least two re. In other words, they are very heterogeneous. FIG. 1B shows an example of tumor heterogeneity in dermatofibrosarcoma protuberans. In contrast to FIG. 1A , the tumor cells in FIG. 1B are more similar as a population in terms of their shape, easy staining intensity, angles, and size. In sarcomas (such as those depicted in FIG. 1A and FIG. 1B ), high heterogeneity (e.g., as shown in FIG. 1A ) is often associated with aneuploidy (which is an aberration in the number of chromosomes), while monotonicity (e.g., as shown in FIG. 1B ) (aka low tumor heterogeneity) can be associated with gene fusions. FIG. 1C shows an example of high tumor heterogeneity in a lung squamous cell carcinoma sample. FIG. 1D shows an example of high tumor heterogeneity in a lung adenocarcinoma sample.

Patients with gene fusions are usually in the late stage of the disease.Gene fusion/rearrangement is a rare type of oncogene mutation event that can be identified across many different cancer types. However, these mutations are of increasing importance because the presence of certain gene fusions from tumor samples can indicate the possibility of a strong response to certain targeted therapies. In such gene fusions, at the cellular level, cells may appear to have low heterogeneity (i.e., appear to be clones), but at the group level (e.g., colony level), their phenotype (i.e., a set of morphological features, wherein tumor cells sharing a common set of morphological features are considered to belong to a single phenotype) may be aggressive, such as corresponding to the diagnosis at the late stage. There are at least several hypotheses about the correlation of tumor heterogeneity with gene fusion. One hypothesis may be that visual signals indicating gene fusions may be mainly present in tumor nests/cells. Another hypothesis may be that visual signals indicating gene fusions may be strong and spread across all parts of the tumor region. Yet another hypothesis may be that low tumor mutation load may indicate reduced tumor morphological heterogeneity. In any case, the lack of tumor heterogeneity in invasive malignancies may therefore be a sign of gene fusion across tumor types. However, in some cases, the lack of tumor heterogeneity may be difficult to observe with the human eye.

2 illustrates a network 200 of interacting computer systems that may be used for digital pathology image generation and processing as described herein, including prediction of actionable mutations, according to some embodiments of the present disclosure.

The digital pathology image generation system 220 can generate one or more whole slide images (WSI) or other related digital pathology images corresponding to a particular sample. For example, the image generated by the digital pathology image generation system 220 can include a stained section of a biopsy sample. As another example, the image generated by the digital pathology image generation system 220 can include a slide image of a liquid sample (e.g., a blood smear). As another example, the image generated by the digital pathology image generation system 220 can include a fluorescence micrograph, such as a slide image depicting fluorescence in situ hybridization (FISH) after a fluorescent probe has bound to a target DNA or RNA sequence.

Some types of samples (e.g., biopsy tissue, solid sample and/or sample including tissue) can be processed by sample preparation system 221 to fix and/or embed the sample. Sample preparation system 221 can facilitate infiltration of sample with fixative (e.g., liquid fixative, such as formaldehyde solution) and/or embedding material (e.g., histological wax). For example, the sample fixing subsystem can fix the sample by exposing the sample to the fixative for at least a threshold amount of time (e.g., at least 3 hours, at least 6 hours or at least 13 hours). The dehydration subsystem can dehydrate the sample (e.g., by exposing a portion of the fixed sample and/or the fixed sample to one or more ethanol solutions) and may use a clarifying intermediate (e.g., including ethanol and histological wax) to remove the dehydrated sample. The sample embedding subsystem can infiltrate the sample with heated (e.g., so that it is liquid) histological paraffin (e.g., one or more corresponding predefined time periods). Histological wax can include paraffin and possible one or more resins (e.g., styrene or polyethylene). The sample and wax can then be cooled, and the sample infiltrated with wax can then be sealed.

The sample slicer 222 can receive a fixed and embedded sample and can produce a set of slices. The sample slicer 222 can expose the fixed and embedded sample to a cool or cold temperature. The sample slicer 222 can then cut the cooled sample (or its trimmed version) to produce a set of slices. Each slice can have a thickness of (for example) less than 100 μm, less than 50 μm, less than 10 μm or less than 5 μm. Each slice can have a thickness of (for example) greater than 0.1 μm, greater than 1 μm, greater than 2 μm or greater than 4 μm. The cutting of frozen samples can be performed in a warm water bath (for example, at a temperature of at least 30° C., at least 35° C. or at least 40° C.).

Automated staining system 223 can promote the staining of one or more sample sections by exposing each section to one or more staining agents. Each section can be exposed to the staining agent of predefined volume for a predefined time period. In some cases, a single section is exposed to a variety of staining agents simultaneously or sequentially.

Each of one or more stained sections can be presented to an image scanner 224, which can capture the digital image of the section. The image scanner 224 can include a microscope camera. The image scanner 224 can capture digital images at multiple magnification levels (e.g., using a 10x objective lens, a 20x objective lens, a 40x objective lens, etc.). The manipulation of the image can be used to capture the selected portion of the sample within the desired magnification range. The image scanner 224 can further capture the annotations and/or morphological measurements identified by the human operator. In some cases, after capturing one or more images, the section is returned to the automated staining system 223 so that the section can be cleaned, exposed to one or more other stains and imaged again. When using a variety of stains, stains with different color configurations can be selected so that the first region of the first section portion corresponding to the first stain absorbing a large amount of the image can be distinguished from the second region of the second section portion corresponding to the second stain absorbing a large amount of the image (or different images).

It should be understood that in some cases, one or more components of the digital pathology image generation system 220 can be operated in conjunction with a human operator. For example, a human operator can move samples across various subsystems (e.g., the sample preparation system 221 or subsystems of the digital pathology image generation system 220) and/or start or terminate the operation of one or more subsystems, systems, or components of the digital pathology image generation system 220. As another example, a portion or all of one or more components of the digital pathology image generation system (e.g., one or more subsystems of the sample preparation system 221) can be partially or completely replaced with the actions of a human operator.

In addition, it should be understood that while the various described and depicted functions and components of the digital pathology image generation system 220 relate to the processing of solid and/or biopsy samples, other embodiments may relate to liquid samples (e.g., blood samples). For example, the digital pathology image generation system 220 can receive a liquid sample (e.g., blood or urine) slide that includes a base slide, a smeared liquid sample, and a coverslip. The image scanner 224 can then capture an image of the sample slide. Other embodiments of the digital pathology image generation system 220 may involve capturing images of the sample using advanced imaging techniques such as FISH described herein. For example, once a fluorescent probe has been introduced into the sample and bound to a target sequence, an image of the sample can be captured using appropriate imaging for further analysis.

A given sample may be associated with one or more users (e.g., one or more physicians, laboratory technicians, and/or medical providers) during processing and imaging. The associated providers may include, for example, but not limited to, a person who orders a test or biopsy that produces an imaged sample, a person who is entitled to receive the results of a test or biopsy, or a person who performs an analysis of a test or biopsy sample, etc. For example, a user may correspond to a physician, a pathologist, a clinician, or a subject. A user may use one or more user devices 230 to submit one or more requests (e.g., identifying a subject) for processing a sample by a digital pathology image generation system 220 and for processing the resulting image by a digital pathology image processing system 210.

The digital pathology image generation system 220 can transmit the image generated by the image scanner 224 back to the user device 230. The user device 230 then communicates with the digital pathology image processing system 210 to initiate automated processing of the image. In some cases, the digital pathology image generation system 220 provides the image generated by the image scanner 224 directly to the digital pathology image processing system 210, for example, at the instruction of the user of the user device 230. Although not shown, other intermediate devices (e.g., data storage areas connected to the digital pathology image generation system 220 or the server of the digital pathology image processing system 210) can also be used. In addition, for simplicity, only one digital pathology image processing system 210, image generation system 220, and user device 230 are shown in the network 200. The present disclosure contemplates the use of one or more of each type of system and its components without necessarily departing from the teachings of the present disclosure.

The network 200 and associated systems shown in FIG. 2 can be used in various scenarios where the scanning and evaluation of digital pathology images (such as WSI) are essential components of the work. As an example, the network 200 can be associated with a clinical environment, where a user evaluates a sample for possible diagnostic purposes. The user can use a user device 230 to review the image, and then provide the image to the digital pathology image processing system 210. The user can provide additional information to the digital pathology image processing system 210, which can be used to guide or instruct the digital pathology image processing system 210 to analyze the image. For example, the user can provide an expected diagnosis or preliminary assessment of the features within the scan. The user can also provide additional scenarios, such as the type of tissue being examined. As another example, the network 200 can be associated with a laboratory environment where tissue is being examined, for example, to determine the efficacy or potential side effects of a drug. In this scenario, it may be commonplace to submit multiple types of tissues for review to determine the effects of the drug on the whole body. This may bring special challenges to human scan reviewers, who may need to determine various scenarios of the image, which may be highly dependent on the type of tissue being imaged. These scenarios can be optionally provided to the digital pathology image processing system 210.

The digital pathology image processing system 210 can process digital pathology images (including WSI) to classify digital pathology images and generate annotations for digital pathology images and related outputs. As an example, the digital pathology image processing system 210 can process the WSI of a tissue sample or the image tiles of the WSI generated by the digital pathology image processing system 210 to identify the morphological traits that can be observed in clustered tumor cells, and determine the occurrence of gene change events (such as gene fusions based on the identified morphological traits). The digital pathology image processing system 210 can use a sliding window to generate a mask on clustered tumor cells. In addition to its use for identifying clustered tumor cells in WSI, the mask can also be used to measure thickness, determine the length for different endpoints, determine the curvature of the distortion, and measure the volume in the three-dimensional imaging or processing scene. The digital pathology image processing system 210 can then cut the query image into multiple image tiles. The tile generation module 211 can define a set of image tiles for each digital pathology image. To define a set of image tiles, the tile generation module 211 can segment the digital pathology image into a set of image tiles. As embodied herein, the image tiles can be non-overlapping (e.g., each image tile includes pixels of the image that are not included in any other image tile) or overlapping (e.g., each image tile includes some portion of the pixels of the image that are included in at least one other image tile). In addition to the size of each image tile and the stride of the window (e.g., the image distance or pixels between an image tile and a subsequent image tile), features such as whether the image tiles overlap can also increase or decrease the data set for analysis, where more image tiles (e.g., achieved by the use of overlapping or smaller image tiles) increase the potential resolution of the final output and visualization. In some cases, the tile generation module 211 defines a set of image tiles for an image, where each image tile has a predefined size and/or the offset between image tiles is predefined. Continuing with the example of detecting gene fusions or other genetic changes, each pathology slide image can be cropped into image tiles with a width and height of a certain number of pixels. In addition, in some cases, the tile generation module 211 can create multiple groups of image tiles with different sizes, overlaps, step sizes, etc. for each WSI. As an example, in some cases, the width and height (in terms of the number of pixels) of each image tile can be dynamically determined (i.e., not fixed) based on factors such as the assessment task at hand, the query image itself, or any suitable factors. In some embodiments, the digital pathology image itself can contain image tile overlaps that can be generated by imaging techniques. In some cases, average segmentation without image tile overlap can be a preferred solution to balance image tile processing requirements and avoid affecting the embedding generation and weight value generation discussed herein. The image tile size or image tile offset can be determined, for example, by calculating one or more performance metrics (e.g., precision, recall, accuracy, and/or error) for each size/offset, and by selecting an image tile size and/or offset associated with one or more performance metrics above a predetermined threshold and/or associated with one or more performance metrics (e.g., high precision, high recall, high accuracy, and/or low error).

The tile generation module 211 can further define the image tile size according to the type of abnormality being detected. For example, the tile generation module 211 can be configured to understand the type of tissue phenotype or abnormality that the digital pathology image processing system 210 will search for, and can customize the image tile size according to the tissue phenotype or abnormality (and in some cases according to the tissue sample type) to improve detection. For example, the image generation module 211 can determine that when the tissue phenotype or abnormality includes searching for inflammation or necrosis in lung tissue, the image tile size should be reduced to increase the scanning rate, and when the tissue abnormality includes an abnormality of Kupffer cells in liver tissue, the image tile size should be increased to increase the chance of the digital pathology image processing system 210 analyzing Kupffer cells as a whole. In some cases, the tile generation module 211 defines a set of image tiles, wherein the number of image tiles in the group, the size of the image tiles of the group, the resolution of the image tiles for the group, or other relevant attributes are defined for each WSI and maintained as unchanged for each of one or more images.

As embodied herein, the tile generation module 211 can further define a set of image tiles for each digital pathology image along one or more color channels or color combinations. As an example, the digital pathology images received by the digital pathology image processing system 210 can include large-format multi-color channel images having pixel color values (e.g., bit values corresponding to intensity) specified for each pixel of the image in one of the several color channels. Exemplary color specifications or color spaces that can be used include RGB, CMYK, HSL, HSV, or HSB color specifications. A set of image tiles can be defined based on segmenting color channels and/or generating a brightness map or grayscale equivalent for each image tile. For example, for each fragment of the image, the tile generation module 211 can provide a red image tile, a blue image tile, a green image tile, and/or a brightness image tile, or an equivalent of the color specification used. As explained herein, segmenting digital pathology images based on fragments of the image and/or color values of the fragments can improve the accuracy and recognition rate of the model/network used to generate embeddings (e.g., low-dimensional space) for image tiles and digital pathology images and generate digital pathology image classification. In addition, the digital pathology image processing system 210 (e.g., using the tile generation module 211) can convert between color specifications and/or use multiple color specifications to prepare copies of image tiles. Color specification conversion can be selected based on the desired type of image enhancement (e.g., emphasizing or enhancing specific color channels, saturation levels, brightness levels, etc.). Color specification conversion can also be selected to improve the compatibility between the digital pathology image generation system 220 and the digital pathology image processing system 210. For example, a specific image scanning component can provide an output of HSL color specifications, and as described herein, the model used in the digital pathology image processing system 210 can be trained using RGB images. Converting image tiles to compatible color specifications can ensure that image tiles can still be analyzed. Additionally, the digital pathology image processing system may upsample or downsample images provided at a particular color depth (e.g., 8-bit, 1-bit, etc.) for use by the digital pathology image processing system. Additionally, the digital pathology image processing system 210 may cause image tiles to be converted depending on the type of image that was captured (e.g., a fluorescence image may include more detail regarding color intensity or a wider range of colors).

In some cases, the digital pathology image processing system 210 can detect one or more features from each of the multiple image tiles. The one or more features may include, for example, one or more of clinical features or histological features, such as cell type. Therefore, generating a label for each of the multiple image tiles can be based on the one or more features. As an example and not limitation, clinical features may include one or more of the following: patient age at diagnosis, patient sex, patient height, patient weight, patient clinical history, patient sample type, or patient smoking history. As another example and not limitation, histological features may include, for example, growth patterns, such as solid, cribriform, micropapillary, papillary, alveolar, or squamous.

As described herein, the tile embedding module 212 can generate an embedding (e.g., a low-dimensional representation space) in a corresponding feature embedding space for each image tile. The embedding can be represented by the digital pathology image processing system 210 as a feature vector for the image tile. In some cases, the tile embedding module 212 can use a neural network (e.g., a convolutional neural network) to generate a feature vector for each image tile representing the image. In a particular embodiment, the tile embedding neural network can be based on, for example, a ResNet image network trained on a dataset based on natural (e.g., non-medical) images (such as an ImageNet dataset). By using a non-specialized tile embedding network, the tile embedding module 212 can take advantage of known advances in efficiently processing images to generate embeddings. In addition, using a natural image dataset allows the embedding neural network to learn to discern differences between image tile fragments at a holistic level.

In other embodiments, the tile embedding network used by the tile embedding module 212 can be an embedding network customized to handle a large number of image tiles of a large format image (such as a digital pathology WSI). In addition, a custom data set can be used to train the tile embedding network used by the tile embedding module 212. For example, the tile embedding network can be trained using various samples of the WSI, or even trained using samples related to the subject matter (e.g., scans of a specific tissue type) for which the embedding network will generate embeddings. Using a collection of specialized or customized images to train the tile embedding network can allow the tile embedding network to identify finer (e.g., more subtle) differences between image tiles, which can make the distances between image tiles in the feature embedding space more detailed and accurate, but at the potential cost of additional time required to acquire images and/or the computational and economic costs of training multiple tile generation networks for use by the tile embedding module 212. In some cases, the tile embedding module 212 can select from a library of tile embedding networks based on the type of image being processed by the digital pathology image processing system 210.

As described herein, an image tile embedding (e.g., a low-dimensional space) can be generated using a machine learning model (e.g., a deep learning neural network) based on the use of visual features of the image tile. In some cases, the trained machine learning model can therefore act as, for example, an image feature extraction model. The image tile embedding can be further generated based on contextual information associated with the image tile or based on the content displayed in the image tile. For example, the image tile embedding may include one or more features that indicate and/or correspond to the size of the depicted object (e.g., the size of the depicted cell or aberration) and/or the density of the depicted object (e.g., the density of the depicted cell or aberration). The size and density can be measured absolutely (e.g., based on the size represented in pixels or converted from pixels to nanometers), or relative to other image tiles from the same digital pathology image, from a class of digital pathology images (e.g., generated using similar technology or by a single digital pathology image generation system or scanner), or from a related series of digital pathology images. In addition, the image tiles can be classified before the tile embedding module 212 generates an embedding for the image tile, so that the tile embedding module 212 considers the classification when preparing the embedding.

For consistency, in some cases, tile embedding module 212 can generate embeddings of predefined sizes (e.g., a feature vector of 512 elements, a feature vector of 2048 bytes, etc.). In some cases, tile embedding module 212 can generate embeddings of various and arbitrary sizes. Tile embedding module 212 can adjust the size of the embedding based on user instructions, or can select the size based on computational efficiency, accuracy, or other parameters, for example. In particular embodiments, the embedding size can be based on the limitations or specifications of the deep learning neural network that generates the embedding. Larger embedding sizes can be used to increase the amount of information captured in the embedding and improve the quality and accuracy of the results, while smaller embedding sizes can be used to improve computational efficiency.

The digital pathology image processing system 210 can derive different inferences by applying one or more machine learning models to the embedding, that is, inputting the embedding to the machine learning model. As an example, the digital pathology image processing system 210 can identify clustered tumor cells based on a machine learning model trained to identify such structures. In some embodiments, it may not be necessary to crop the image into image tiles, generate embeddings for these image tiles, and then derive inferences based on such embeddings. On the contrary, in some cases, a digital pathology image processing system 210 with sufficient graphics processing unit (GPU) memory can directly apply the machine learning model to the embedding of the WSI to make inferences. In some cases, the output of the machine learning model can be adjusted to the shape of the input image.

The WSI access module 213 can manage requests for access to the WSI from other modules of the digital pathology image processing system 210 and the user device 230. For example, the WSI access module 213 receives a request to identify the WSI based on a specific image tile, an identification for an image tile, or an identification for a WSI. The WSI access module 213 can perform the following tasks: confirm that the WSI is available to the requesting user or module, identify the appropriate database from which to retrieve the requested WSI, and retrieve any additional metadata that the requesting user or module may be interested in. In addition, the WSI access module 213 can effectively handle streaming the appropriate data to the requesting device. As described herein, in some cases, the WSI can be provided to the user device in a partial form based on the possibility that the user will want to see the entire WSI or a portion of the WSI. In some cases, the WSI access module 213 can determine which areas of the WSI are to be provided and how to provide them. In addition, in some cases, the WSI access module 213 can be authorized within the digital pathology image processing system 210 to ensure that no individual component locks or otherwise misuses the database or WSI to the detriment of other components or users.

The tumor heterogeneity assessment module 214 of the digital pathology image processing system 210 can apply one or more techniques to assess the heterogeneity of tumor cells identified in one or more of the WSIs. In some embodiments, assessing tumor heterogeneity includes classifying mutant tumor cells into phenotypes by identifying morphologically similar cells (e.g., by assessing nuclear heterogeneity). In some embodiments, assessing nuclear heterogeneity includes quantifying certain features of the cell nucleus to distinguish mutant cells based on nuclear morphological heterogeneity.

In some embodiments, identifying tumor heterogeneity includes: identifying regions of clonal cells by performing a cell-level spatial analysis to assess spatial distribution. In some embodiments, assessing spatial distribution includes: measuring spectral distances within a subgraph of a minimum spanning tree of tumor cells, wherein each of the subgraphs represents clustered adjacent cells (e.g., tumor nests), and calculating adjacency spectral distances pairwise across all subgraphs. In some embodiments, each of the subgraphs can be defined by performing outlier detection. In some embodiments, each of the subgraphs can be defined based on segmentation of detected tumor nests.

In some embodiments, identifying tumor heterogeneity comprises: identifying regions of closely adjacent clonal cells by performing a cell-level spatial analysis to assess spatial entropy. In some embodiments, assessing spatial entropy comprises: calculating the frequency with which pairs of cells are identified as morphologically similar for each of a predefined number of distance bins.

The gene fusion prediction module 215 of the digital pathology image processing system 210 can apply one or more techniques to predict the likelihood of the presence of a gene fusion (e.g., a binary output or a percentage output). In some embodiments, the gene fusion prediction module 215 can assess and/or aggregate the results of assessing tumor heterogeneity, the results of end-to-end prediction of gene fusions, the results of assessing tumor morphology, and/or the results of other methods to derive a prediction (e.g., a score).

The output generation module 216 of the digital pathology image processing system 210 can generate outputs corresponding to one or more of the image tiles and one or more WSI data sets based on user requests. As described herein, the output can include various visualizations, interactive graphics, and reports based on the type of request and the type of data available. In some embodiments, the output will be provided to the user device 230 for display, but in certain embodiments, the output can be accessed directly from the digital pathology image processing system 210. The output will be based on the existence and access of the appropriate data, so the output generation module will be authorized to access the necessary metadata and anonymous patient information as needed. Like other modules of the digital pathology image processing system 210, the output generation module 214 can be updated and improved in a modular manner so that new output features can be provided to users without a lot of downtime.

The general techniques described herein can be integrated into various tools and use cases. For example, as described, a user (e.g., a pathologist or clinician) can access a user device 230 communicating with a digital pathology image processing system 210 and provide a query image for analysis. The digital pathology image processing system 210 or the connection to the digital pathology image processing system can be provided as an independent software tool or package that searches for corresponding matches, identifies similar features, and generates appropriate outputs for users upon request. As an independent tool or plug-in that can be purchased or licensed on a simplified basis, the tool can be used to enhance the capabilities of a research or clinical laboratory. In addition, the tool can be integrated into the services available to the customers of the digital pathology image generation system. For example, the tool can be provided as a unified workflow, wherein a user receiving an image and/or a previously indexed similar WSI that automatically creates a WSI for a submitted sample is executed or requested to be notable for features previously indexed. Therefore, in addition to improving WSI analysis, these techniques can also be integrated into existing systems to provide additional features that were not previously considered or impossible.

In addition, the digital pathology image processing system 210 can be trained and customized for specific settings. For example, the digital pathology image processing system 210 can be specifically trained to provide insights related to specific types of tissues (e.g., lungs, heart, blood, liver, etc.). As another example, the digital pathology image processing system 210 can be trained to assist in safety assessments, such as determining the level or degree of toxicity associated with a drug or other potential treatment. Once trained for a specific subject or use case, the digital pathology image processing system 210 is not necessarily limited to that use case. Due to the relatively large collection of at least partially labeled or annotated images, training can be performed in specific scenarios, such as toxicity assessments.

The methods and systems disclosed herein can enable a user to easily request a prognosis prediction based on a digital pathology image provided by a user. In some cases, the digital pathology image processing system 210 can transmit a request communication from a client computing system to a remote computing system to process a digital pathology image depicting cancer cells in a specific slice of a biological sample from a subject. In response to receiving a request communication from a client computing system, the remote computing system can perform operations including the following steps. The remote computing system can first access the digital pathology image. The remote computing system can then segment the digital pathology image into a plurality of image tiles, each of which depicts one or more tumor cell clusters. The remote computing system can then generate a mark indicating the possibility of the image tile depicting tumor heterogeneity for each of the plurality of image tiles. The remote computing system can then determine that the digital pathology image includes a depiction of the occurrence of an actionable mutation about a cancer cell based on the mark generated for each image tile. The remote computing system can then generate a prognosis prediction for a subject based on the occurrence of a gene fusion with a response to a cancer cell. In some cases, the prognosis prediction can include a prediction of the suitability of one or more treatment regimens for a subject. The remote computing system may further provide the prognostic prediction to the client computing system via the response communication.In some cases, the client computing system may output the prognostic prediction in response to receiving the response communication.

FIG3 shows an exemplary method 300 for detecting genetic alterations (e.g., gene fusions). The method may include step 310, where the digital pathology image processing system 210 depicted in FIG3 may access a digital pathology image depicting cancer cells in a particular slice of a biological sample from a subject. By way of example and not limitation, the digital pathology image may be a scanned, stained (e.g., hematoxylin and eosin stained) WSI including tumor cells (e.g., lung adenocarcinoma).

At step 320 of FIG. 3 , the digital pathology image processing system 210 may segment the digital pathology image into a plurality of image tiles, each image tile depicting at least one cluster of tumor cells. In a particular embodiment, the tile generation module 211 depicted in FIG. 3 may be used to generate image tiles. The image tiles may be non-overlapping or overlapping. In addition to the size of each image tile and the progressive displacement of the window used to create the image tile, features such as whether the image tiles overlap may also increase or decrease the data set used for analysis, where more image tiles increase the potential resolution of the final output and visualization. In a particular embodiment, each image tile may have a predefined size and/or the offset between image tiles may be predefined. In addition, the tile generation module 211 may create multiple groups of image tiles with different sizes, overlaps, step sizes, etc. for each image. The tile generation module 211 may generate image tiles for each digital pathology image in one or more color channels or for one or more color combinations. Image tiles may be generated based on segmenting color channels and/or generating a brightness map or grayscale equivalent for each image tile. Additionally, the digital pathology image processing system 210 may upsample or downsample images provided at a particular color depth for use by the digital pathology image processing system 210. Additionally, the digital pathology image processing system 210 may cause image tiles to be converted depending on the type of image that has been captured.

At step 330 in FIG. 3 , the digital pathology image processing system 210 may generate a tag indicating the likelihood that an image tile (such as an image tile depicting an actionable mutation) depicts clustered tumor cells (e.g., a tumor region or a tumor nest structure) for each of the multiple image tiles. As an example and not limitation, the digital pathology image processing system 210 may detect one or more features from each of the multiple image tiles. The one or more features may include, for example, one or more of histological features (such as cell types or cell groupings), clinical features, or genomic features. Therefore, generating a tag for each of the multiple image tiles may be based on the one or more features. In a specific embodiment, generating a tag for each of the multiple image tiles may be based on one or more of an image tile-based classification or a multi-instance learning (MIL) classification. Generating a tag for each of the multiple image tiles may be based on the use of one or more trained machine learning models. In certain embodiments, the digital pathology image processing system 210 can train the one or more machine learning models based on a plurality of training data, the plurality of training data comprising: one or more labeled depictions of samples including, for example, tumor regions or tumor nest structures, and one or more labeled depictions of samples not including tumor regions or tumor nest structures. In certain embodiments, generating labels for each of the plurality of image tiles can be based on tissue morphology, for example, tumor morphology. Tumor morphology can be based on, for example, analysis of one or more of the following histological features: presence or number of signet ring cells, presence or number of hepatoid cells, extracellular mucin, or tumor growth pattern.

At step 340 in FIG. 3 , the digital pathology image processing system 210 can determine that the digital pathology image includes a depiction of the occurrence of gene fusions about cancer cells in the image based on the markers generated for each image tile. In a specific embodiment, the digital pathology image processing system 210 can use any of a variety of different methods to effectively determine the gene fusions present. For example, a method may include: combining target gene fusions (e.g., NTRK fusions) with other gene fusions (such as ROS1, ALK, and RET fusions) into a single operational gene fusion cluster. The cluster can be considered as a single category of gene fusions to facilitate detection. In this method, rather than attempting to identify each gene fusion individually, the digital pathology image processing system 210 treats them as a single group, and therefore no longer needs to detect gene fusions that occur individually at a frequency of less than half a percentage. As an example and not limitation, the combined frequency of occurrence of these gene fusions can be about 15%.

Detection of actionable gene fusions can be based on one or more of the following: (i) automated detection of histological features using one or more end-to-end data-driven machine learning models, (ii) identification of mutually exclusive gene mutations (thereby identifying the absence (or absence) of gene fusions), (iii) detection of NTRK gene fusions by grouping NTRK with ALK, ROS1, and RET into a single "actionable gene fusion cluster" and identifying that cluster, (iv) automated detection of histological features associated with ALK, ROS1, and RET (including solid and cribriform growth patterns, extracellular mucin, signet ring cells, goblet cells, and hepatoid cells), (v) identification and elimination of mutational signatures associated with smoking, (vi) identification of low tumor mutational burden, (vii) assessment of tumor heterogeneity, or (viii) identification of pan-tumor or tumor-agnostic actionable gene fusion clusters.

Another approach can include using the molecular landscape and molecular signatures of these tumors. In certain embodiments, the signal for the fusion can be primarily in the tumor nests/cells and be strong and diffuse across the entire tumor region. Thus, in addition to identifying gene fusions directly from the slide, the digital pathology image processing system 210 can also identify gene fusions based on the mutually exclusive distribution of molecular signatures across the tumor.

In certain embodiments, the digital pathology image processing system 210 can indicate the occurrence of a gene fusion to a pathologist, for example, a comparison between a fusion-positive slide image and the same field of view with an overlaid heat map of a gene fusion prediction from that slide. When comparing the two, the pathologist can see how the tumor detection algorithm in some embodiments disclosed herein rejects image tiles that do not contain a tumor. In addition, the confidence metric for the prediction of a gene fusion (as depicted, for example, by the intensity of the heat map) may vary across tumor regions. The confidence metric may be highest in regions with signet ring cells.

At step 350, the digital pathology image processing system 210 can generate a prognosis prediction for the subject based on the occurrence of the gene fusion of the cancer cell detected, wherein the prognosis prediction includes a prediction of the suitability of one or more treatment plans for the subject. The digital pathology image processing system 210 can output the prognosis prediction, for example, via a graphical user interface. As an example and not limitation, the digital pathology image processing system 210 can output a treatment plan assessment. The digital pathology image processing system 210 can generate suggestions associated with the use of the one or more treatment plans. For example, the assessment may be that the patient may have gene fusion. As a subsequent or further step, the digital pathology image processing system 210 can prompt the suggestion of subsequent molecular detection (such as next generation sequencing determination). In some embodiments, one or more steps of the method depicted in Figure 3 can be repeated where appropriate. Although the present disclosure describes and illustrates the specific steps of the method of Figure 3 as occurring in a specific order, the present disclosure contemplates any suitable steps of the method of Figure 3 occurring in any suitable order. In addition, although the present disclosure describes and illustrates an exemplary method for detecting gene fusion (or other genetic alterations) (including specific steps of the method depicted in FIG. 3 ), the present disclosure contemplates any suitable method for detecting gene fusion (including any suitable steps, where appropriate, which may include all steps, some steps, or no steps of the method of FIG. 3 ). In addition, although the present disclosure describes and illustrates specific components, devices, or systems for performing specific steps of the method of FIG. 3 , the present disclosure contemplates any suitable combination of any suitable components, devices, or systems for performing any suitable steps of the method of FIG. 3 .

In some cases, the disclosed methods and systems can be applied to detect gene fusion/rearrangement, which is a specific type of rare druggable oncogene mutation event that can be identified across many different cancer types and, if present in tumor tissue samples, can indicate a strong response to certain targeted therapies. Gene fusions include rare druggable mutation events that can occur across many different tumor types and are increasingly being targeted for novel therapies. The identification of gene fusions can be a technically difficult, expensive, and time-consuming process that may ultimately benefit only a few patients carrying such genetic changes; for these reasons, widespread testing may be limited to a few hospitals that have the ability to absorb and provide the technical and financial resources involved in the process. The embodiments disclosed herein can address this gap by creating, training, and using a machine learning model (e.g., a digital pathology screening model) that can predict the presence of oncogene fusions from digital pathology images (such as scanned, stained (e.g., hematoxylin and eosin-stained) WSIs depicting cancer tissue/cells (e.g., lung adenocarcinoma)). Furthermore, embodiments disclosed herein may include rapid, inexpensive, and sufficiently accurate screening tools that can be used to guide molecular testing and decision making regarding the use of targeted therapies for individual patients, including, but not limited to, patients with lung adenocarcinoma.

In some cases, as noted elsewhere herein, the disclosed methods and systems can be used to identify gene fusions that are increasingly being targeted as novel therapies. Targeted therapies for patients with tumors can include drugs targeting epidermal growth factor receptor (EGFR) and gene fusions involving anaplastic lymphoma kinase (ALK), RET, ROS1, and neurotrophic tyrosine receptor kinase (NTRK). For EGFR, although immunohistochemical staining can be used to identify the most common variants (e.g., its coverage is as high as 97% of EGFR-positive lung adenocarcinoma patients), molecular testing may be required to identify drug-resistant mutations in patients who have failed EGFR targeted therapy. Such immunohistochemical stains for RET and ROS1 have not yet been developed, and immunohistochemical stains for ALK and NTRK may vary greatly and be difficult to interpret. In addition, gene fusions typically require more complex molecular assays, which have a greater coverage of the genome than the more commonly used "hotspot" assays for testing a limited number of loci. In order to target gene fusions, a much wider coverage may be required, resulting in much more expensive testing, which requires laboratories to demonstrate much greater technical capabilities. Therefore, a considerable proportion of patients may be unlikely to receive the correct test to determine the possibility that their tumors carry gene fusions. In addition, some gene fusions (e.g., NTRK fusions) may be extremely rare. Although NTRK fusions have been identified in a variety of tumor types, the frequency of this particular fusion may be less than 1% in the most common cancer indications (such as in lung adenocarcinoma, colorectal cancer, and non-secretory breast cancer). The relative rarity of gene fusions (e.g., ranging from 7% for ALK to less than 0.3% for NTRK in lung adenocarcinoma) constitutes a significant technical and financial obstacle to extensive testing. In fact, studies have shown that the patient groups that benefit most from these drugs are those living near academic institutions, which have the expertise, infrastructure, and budget for complex laboratory tests. At present, molecular testing is the only method that can be used to determine the possibility of gene fusions in patients so far. However, molecular testing is expensive, and patients sometimes avoid arranging molecular testing because of costs and/or unnecessary costs for patients who may not benefit from molecular testing. The current embodiment proposes an improvement to the current system, which is that the current embodiment can be used to identify patients who can benefit from molecular testing. In particular, the digital pathology image processing system described herein can use a digital pathology machine learning model to screen patients who may have gene fusions, and can then provide recommendations for testing those patients using molecular assays. Thus, the disclosed digital pathology image processing system can increase the likelihood of detecting gene fusions in patients, and can reduce the cost of subsequent molecular testing, thereby further benefiting and improving healthcare outcomes for those patients who exhibit gene fusions for whom targeted therapies exist. The digital pathology model can be applicable to any suitable tumor type, although the embodiments disclosed herein contemplate the application of the digital pathology model to lung adenocarcinoma as an exemplary tumor type.

In certain embodiments, the digital pathology image processing system 210 can use different solutions to effectively detect gene fusions. One solution can be to combine target gene fusions (e.g., NTRK fusions) with other gene fusions (such as ROS1, ALK, and RET) into a single operational gene fusion cluster. The cluster can then be considered as a single category of gene fusions. Because the digital pathology image processing system 210 does not attempt to identify each of these gene fusions individually, but rather processes them as a single group, the digital pathology image processing system 210 can no longer process at a frequency of less than half a percentage of each gene fusion. As an example and not limitation, the combined frequency of these gene fusions can be about 15%.

Another approach can include using the molecular landscape and molecular features of these tumors. In certain cases, the signal for fusion can appear primarily in tumor nests/cells and can be strong and spread across tumor regions. Therefore, in addition to identifying fusions directly from the slide, the digital pathology image processing system 210 can also identify gene fusions based on the mutually exclusive distribution of molecular features across tumors. As an example and not limitation, the morphology of lung adenocarcinoma can be mapped onto a molecular landscape, which may include, for example, but not limited to: 17% EGFR-sensitized, 7% ALK, 4% EGFR other, 3% with> 1 mutation, 2% HER2, 2% ROS1, 2% BRAF, 2% RET, 1% NTRK1, 1% PIK3CA, 1% MEK1, 31% unknown oncogene drivers, and 25% KRAS alternation. Among the most common driver mutations in lung adenocarcinoma, only three percent may have more than one mutation, which means that 97% of lung cancer patients carry a single mutation. Therefore, the situation that the driver mutation shows mutual exclusivity is significantly more common, and this feature can be used in a variety of contexts to provide clinical decision-making information in the treatment of cancer patients. In some embodiments, the digital pathology image processing system 210 can access the digital pathology image of the cancer cells in the specific slice of the biological sample from the subject. The digital pathology image processing system 210 can then determine that the digital pathology image includes the depiction of one or more mutations mutually exclusive with the occurrence of gene fusion, and therefore determine that there is no gene fusion about the cancer cell. In some cases, the digital pathology image processing system 210 can further generate a prognosis prediction for the subject based on the absence of gene fusion about the cancer cell. In a specific embodiment, the digital pathology image processing system 210 can further generate a prognosis prediction for the subject based on the absence of gene fusion about cancer. Prognosis prediction can include, for example, a prediction of the suitability of one or more treatment regimens for the subject. Due to this mutual exclusivity, in addition to positively identifying gene fusions, the digital pathology model can also identify more common mutations (such as KRAS and EGFR), and so exclude the presence of gene fusions.

In a particular embodiment, the digital pathology image processing system 210 can detect one or more features from each of the multiple image tiles. The one or more features may include one or more of clinical features or histological features, such as cell types. Therefore, generating a label for each of the multiple image tiles can be based on the one or more features. As an example and not limitation, clinical features may include one or more of a younger age at diagnosis or an estimate of a smoking history. In a particular embodiment, predicting an operable gene fusion can be based on identifying and excluding mutation features associated with smoking. As another example and not limitation, histological features may include growth patterns, such as solid, cribriform, micropapillary, papillary, alveolar, or squamous. In a particular embodiment, predicting or determining the presence of an operable gene fusion can be based on the detection of histological features associated with ALK, ROS1, and RET (including solid and cribriform growth patterns and/or extracellular mucin). As another example and not limitation, predicting or determining an operable gene fusion can be based on the detection of cell types associated with ALK, ROS1, and RET. These cell types may include one or more of signet ring cells, goblet cells, or hepatoid cells. Different features may have different degrees of importance for different tumor types. Automatic detection and quantification of each of these visual features can allow prediction of, for example, ALK, ROS1, RET, and NTRK in, for example, lung adenocarcinoma.

In certain embodiments, another feature of a fusion and tumor that can be used by the digital pathology image processing system 210 (or a digital pathology machine learning model resident therein) to determine the presence of a genetic alteration (e.g., a gene fusion) can be tumor mutation burden (TMB). In some cases, for example, a kinase or oncogene fusion may be associated with a low tumor mutation burden. The primary oncogene driver of a tumor may be a single gene fusion. Therefore, one can expect that morphological signals derived from a single oncogene driver will exist across most tumor cells/regions in a tissue specimen on a slide. End-to-end gene fusion status predictions can also show strong, uniform signals across the entire slide.

In some cases, low tumor mutation load may indicate reduced tumor morphological heterogeneity. Patients may be characterized by having driver mutations, mutations in driver genes, and/or driver fusions (e.g., gene fusions involving driver genes). In some cases, the tumor mutation load in cancer may be driven by driver mutations. In some cases, the tumor mutation load of cancer may also be driven by gene fusions. In some cases, cancers driven by gene fusions may have significantly lower tumor mutation loads. Therefore, low tumor mutation load may be associated with low tumor heterogeneity.

The digital pathology model can be universal across different tumor types. Therefore, the digital pathology image processing system 210 can identify and predict pan-tumor or tumor-agnostic actionable gene fusions based on the digital pathology model. As an example and not limitation, the digital pathology image processing system 210 trains the digital pathology model for ALK fusion and ROS1 fusion, respectively, and the performance is the same. As another example, ALK, ROS1, and RET can be used to rank NTRK signals. For example, even if the digital pathology model has never used NTRK-based training data in training, it can identify NTRK fusions with the same accuracy as it does in the case of ROS1 fusion in the experiments of the embodiments disclosed herein. The universality of the digital pathology model can indicate that the features are consistent across different gene fusions and across different tumor types.

The embodiments disclosed herein may have the technical advantage of using readily available and cheaper materials for analysis compared to corresponding molecular detection. In some embodiments, the slices of biological samples may be stained with one or more stains. As an example and not limitation, the digital pathology image processing system may be used to scan slides stained with hematoxylin and eosin (H&E), for example; the original tissue specimen slides may be easily available for any new or subsequent diagnostic analysis. In contrast, molecular detection may require cutting into tissue blocks to sacrifice some tissues and for sequencing, which may result in the consumption of diagnostic tissue materials. As can be seen, the image data may not be destroyed by using a digital pathology machine learning model to analyze the image data. In some cases, people can use the digital pathology images of the initial diagnostic slides for analysis without the need for additional slides. In some embodiments, a prognosis prediction may be generated based on further analysis of one or more additional digital pathology images. In some cases, each of the one or more additional digital pathology images may depict additional slices of biological samples from a subject. In some embodiments, the analysis may include determining a likelihood that each of the one or more additional digital pathology images includes a depiction of the occurrence of a gene fusion with respect to a cancer cell, and combining the determinations for each of the one or more additional digital pathology images. In some cases, after making a diagnosis with an H&E stained specimen slide, one may need to sacrifice additional unstained specimen slides (e.g., at least 5, 6, 7, 8, 9, 10, or more than 10 unstained specimen slides) for molecular testing.

Embodiments disclosed herein can have another technical advantage of being easy to use. People can scan pathology specimen slides and input the scanned image or the image tile data derived therefrom into the digital pathology machine learning model. The digital pathology machine learning model can then be used to make a prediction of the possibility of the presence of genetic changes (e.g., gene fusion) in biological samples. In some cases, the process may not require any annotations by a pathologist. In some cases, a pathologist may only need to correctly identify the slide as a target tumor type, e.g., lung adenocarcinoma. Embodiments disclosed herein can have another technical advantage of efficiency. As an example and not limitation, the prediction of gene fusion can be completed in about a few minutes, hours, or days (e.g., less than 60 minutes, less than 50 minutes, less than 40 minutes, less than 30 minutes, less than 25 minutes, less than 20 minutes, less than 15 minutes, or less than 10 minutes).

In some cases, predicting or determining the presence of an actionable gene fusion can be based at least in part on the detection of extracellular mucin. It has been reported that excess extracellular mucin indicates a fusion state, and the disclosed methods for predicting gene fusion states can confirm these findings. In some cases, the digital pathology image processing system 210 can predict gene fusion states in detail, identify differences between, for example, resections and biopsies, determine precise segmentation of regions, perform rough detection of image tiles containing extracellular mucin, and transition from tumor region detection to actual gene fusion state prediction. As an example and not limitation, in some cases, transitioning from tumor region detection to actual gene fusion state prediction can include determining the fraction of detected mucin compared to tissue, or determining the fraction of detected mucin compared to tumor.

In some cases, the digital pathology machine learning model may be universally applicable across different tumor types. Therefore, the digital pathology image processing system 210 can be used to identify and predict pan-tumor or tumor-agnostic actionable gene fusions based on the use of the digital pathology machine learning model. For example, the performance of the digital pathology image processing system 210 including a digital pathology machine learning model trained for ALK fusion or for ROS1 fusion is the same. As another example, the signal for NTRK fusion can be sorted with ALK, ROS1, and RET. For example, even if the digital pathology machine learning model is trained without using NTRK-based training data, it can identify NTRK fusion with the same accuracy as it has for the detection of ROS1 fusion in experiments testing the method disclosed herein. The universal applicability of the digital pathology machine learning model can show that the potential image tile features used for prediction are consistent across different gene fusions and across different tumor types.

Fig. 4 shows an exemplary workflow diagram for a process 400 for detecting gene fusions in biological samples (e.g., tissue specimens). The workflow 400 can start with tissue selection 410. In tissue selection 410, the digital pathology image processing system 210 can perform quality control and/or tumor detection. In a specific embodiment, quality control and tumor region detection can include performing supervised classification tasks. For such tasks, image tile level accuracy can be sufficient, and the digital pathology image processing system 210 can use, for example, one binary classifier per task. The results of tissue selection 410 can then be provided as input to an end-to-end classification step 420. In some embodiments, the end-to-end classification 420 can be based on one or more of the classification or multi-instance learning (MIL) classification techniques based on image tiles as described above. As part of the end-to-end classification 420, generating a label for each of the multiple image tiles can be based on one or more machine learning models. In some embodiments, the digital pathology image processing system 210 can train the one or more machine learning models based on multiple training data, which include, for example, one or more labeled depictions of tumor regions or tumor nest structures and one or more labeled depictions of other histological or clinical features.

While performing the end-to-end classification 420, the digital pathology image processing system 210 may perform a tumor morphology analysis 430. In some embodiments, generating a label for each of the multiple image tiles may be based on tumor morphology. The tumor morphology analysis 430 may include an analysis to identify one or more of signet ring cells, hepatoid cells, extracellular mucin, or tumor growth patterns. In some cases, growth pattern analysis may be helpful for gene fusion detection. As an example and not limitation, lung adenocarcinoma may present multiple growth patterns and the proportion of each growth pattern is different. As another example and not limitation, in some cases, solid and cribriform patterns may be associated with gene fusions. In some cases, the digital pathology image processing system 210 may determine the effect of sample collection type (e.g., resection versus biopsy tissue) on growth pattern. Because growth patterns are typically large and homogenous areas, image tile-level classification may be sufficiently accurate. In some embodiments, both signet ring cell detection and hepatoid cell detection may be associated with the presence of gene fusions. In order to detect such cells of interest, the digital pathology image processing system 210 may rely on object detection and positioning. In some cases involving detection of cells of interest, the digital pathology image processing system 210 can determine the relationship between the detected cells and the fusion state, for example, based on the number or type of detected cells. In some cases, the digital pathology image processing system 210 can further perform fine-grained positioning or image tile-level detection of cells.

The digital pathology image processing system 210 may also use other methods 440 for gene fusion detection. By way of example and not limitation, in some cases, the digital pathology image processing system 210 may identify tumor heterogeneity (variability in size, shape, and staining of tumor cells) from a digital pathology image and measure the identified tumor heterogeneity. Accordingly, in some cases, determining that a digital pathology image may include a depiction of the occurrence of a gene fusion may be further based on the measured tumor heterogeneity.

The digital pathology image processing system 210 can then perform an aggregation step 450 on the results from the tumor morphology analysis 430, the end-to-end classification 420, and the other methods 440. Any suitable method can be used to aggregate the results (e.g., using an ensemble classification, or by generating all intermediate results via all subtasks, and then training another classification model that consumes all intermediate results to output a joint prediction). The aggregated results can be used to predict the fusion state 460 of the tissue specimen. In some embodiments, the fusion state prediction can be a weakly supervised classification task (e.g., where a slide-level label can be available). In some cases, the digital pathology image processing system 210 can classify multiple image tiles using a multiple instance learning (MIL) method. In some cases, the digital pathology image processing system 210 can use a simplified strategy that includes assigning slide labels to all image tiles. In a specific embodiment, determining that the digital pathology image includes a depiction of the occurrence of gene fusions about cancer cells can be further based on a weighted combination of labels generated for each image tile. By way of example and not limitation, in some cases, the digital pathology image processing system 210 may use a binary classifier to classify image patches and then determine a slide-level prediction by combining (eg, averaging) all image patch predictions.

In certain embodiments, the digital pathology image processing system 210 can output a prognosis prediction via a graphical user interface. In some cases, the graphical user interface can include a graphical representation of a digital pathology image. In some cases, the graphical representation can include an indication of a mark generated for each of a plurality of image tiles and a predicted confidence level associated with the corresponding mark. In some cases, the output of the digital pathology image processing system 210 can also include other information as follows. As an example and not limitation, the digital pathology image processing system 210 can output a treatment plan assessment. The digital pathology image processing system 210 can generate suggestions associated with the use of one or more treatment plans for a subject or patient derived from a biological sample. For example, an assessment can be that a sample from a given subject or patient may have a gene fusion, so it is recommended to confirm by subsequent molecular assays. As another example and not limitation, the digital pathology image processing system 210 can output a negative result, that is, there is no predicted or detected gene fusion. As another example and not limitation, the digital pathology image processing system 210 can output "not enough for analysis". For example, “insufficient for analysis” may be due to tumor size or slide (e.g., tumor specimen is too small and/or pathology slide quality is affected by the amount of tissue processing artifacts). For example, a microtome blade used to cut tissue sections may create a series of parallel tears across the slide. These types of sampling processing artifacts may prevent digital pathology machine learning models used to analyze pathology slide images from making accurate predictions.

The embodiments disclosed herein can enable users to easily request prognosis predictions based on digital pathology images from a user end. In a specific embodiment, the digital pathology image processing system 210 can transmit a request communication from a client computing system to a remote computing system to process a digital pathology image depicting cancer cells in a specific slice of a biological sample from a subject. In response to receiving a request communication from a client computing system, the remote computing system can perform operations including the following steps. The remote computing system can first access the digital pathology image. The remote computing system can then segment the digital pathology image into a plurality of image tiles, each of which depicts one or more tumor cell clusters. The remote computing system can then generate a mark indicating the possibility of the image tile depicting a tumor area or a tumor nest structure for each of the plurality of image tiles. The remote computing system can then determine that the digital pathology image includes a depiction of the occurrence of gene fusions about cancer cells based on the marks generated for each image tile. The remote computing system can then generate a prognosis prediction for a subject based on the occurrence of gene fusions with a response to cancer cells. In a specific embodiment, the prognosis prediction can include a prediction of the suitability of one or more treatment regimens for a subject. The remote computing system may further provide the prognostic prediction to the client computing system via the response communication.Particular embodiments may further output, by the client computing system, the prognostic prediction in response to receiving the response communication.

Figures 5A to 5D illustrate exemplary actionable fusion predictions in lung adenocarcinoma. Figure 5A illustrates a non-limiting example of a pathology slide image for lung adenocarcinoma. The left image 505 is a slide for a metastatic lung adenocarcinoma with a ROS1 fusion. The right image 510 is a slide for a lung adenocarcinoma with an EGFR mutation. Figure 5B illustrates a non-limiting example of results from quality control. As shown in Figure 5B, the quality control process may identify tissue 515, markers 520, blur 525, and combined image features 550. Figure 5C illustrates a non-limiting example of tumor region detection. As shown in Figure 5C, the darker the region, the more likely it is a tumor region. Figure 5D illustrates a non-limiting example of a prediction of a fusion state. As shown in Figure 5D, the darker the region, the more likely it is to include a gene fusion.

FIG. 6 shows a non-limiting example of a prediction of a ROS1 gene fusion state. The image shown in FIG. 6 is an example of a final output that can be provided to a pathologist. The left image 610 indicates a fusion-positive slide for a metastatic lung adenocarcinoma including a ROS1 fusion. The right image 620 indicates the same field of view with an overlapping heat map of gene fusion predictions from image 610. When comparing the two images, one can see how the tumor detection algorithm rejects image tiles that do not contain a tumor. In addition, the predicted confidence metric (as depicted by the intensity of the heat map) may vary across tumor regions. In an area with signet ring cells, the confidence metric may be the highest. As can be seen, the digital pathology image processing system 210 can provide an output in a format that makes it clear to the pathologist that the digital pathology model is based on interpretable morphological features.

Experiments for actionable fusion prediction in lung adenocarcinoma were conducted to validate the digital pathology model and method disclosed herein. FIG. 7 shows a non-limiting example of a receiver operating characteristic (ROC) curve 710 for image tile-based fusion prediction. The training set for the digital pathology model includes 270 resections. 18.5% of them are fusion-positive, i.e., 50 slides are from 5 patients. Of these fusion-positive slides, 5 slides are ALK fusion-positive and 45 slides are ROS1 fusion-positive. The test set includes 598 resections and biopsies. 11% of them are fusion-positive, i.e., 68 slides. Of these fusion-positive slides, 8 slides are NTRK fusion-positive and 60 slides are ROS1 fusion-positive. For a cutoff value set at 0.5, the performance statistics were as follows: positive predictive value (PPV) was 0.46, and negative predictive value (NPV) was 0.97, with an overall area under the curve (AUC) of 0.89.

In certain embodiments, the extracellular mucin signal in the end-to-end results can be as follows. It is reported that excess extracellular mucin indicates a fusion state, and the end-to-end fusion state prediction can confirm this assumption. Strong signals in the mucin pool can be observable. In certain embodiments, the digital pathology image processing system 210 can additionally predict the fusion state in detail, identify the difference between the resection and the biopsy tissue, determine the precise segmentation of the region, perform a rough detection of the image tiles containing extracellular mucin, and transition from the region to the actual fusion state prediction. As an example and not limitation, the transition from the region to the actual fusion state prediction can include: determining the score of mucin compared to the tissue, or determining the score of mucin compared to the tumor.

In a particular embodiment, the diffuse fusion signal in the end-to-end prediction can be as follows. A kinase or oncogene fusion can be associated with a low tumor mutation load. The primary oncogene driver of a tumor may be a single gene fusion. Therefore, one would expect that the morphological signal derived from a single oncogene driver would be present across most tumor cells/regions on a slide. The end-to-end fusion status prediction can also show a strong uniform signal across the entire slide.

FIG8 illustrates an exemplary method 800 for enabling an end user to utilize a client computing system to request a prognosis prediction based on processing of a digital pathology image from a digital pathology image processing system 210 (by a remote computing system performing steps in method 300). The method may begin at step 810, wherein the digital pathology image processing system 210 depicted in FIG2 may transmit a request communication from a client computing system to a remote computing system to process a digital pathology image depicting cancer cells in a particular slice of a biological sample from a subject, wherein in response to receiving the request communication from the client computing system, the remote computing system performs operations including the following sub-steps. At sub-step 810a, the remote computing system may access the digital pathology image. At sub-step 810b, the remote computing system may segment the digital pathology image into a plurality of image tiles. At sub-step 810c, the remote computing system may generate, for each of the plurality of image tiles, a tag indicating the likelihood that the image tile depicts, for example, a tumor region or a tumor nest structure. At sub-step 810d, the remote computing system may determine that the digital pathology image includes a depiction of the occurrence of gene fusions about cancer cells based on the markers generated for each image tile. At sub-step 810e, the remote computing system may generate a prognosis prediction for the subject based on the occurrence of gene fusions about cancer cells, wherein the prognosis prediction includes a prediction of the suitability of one or more treatment regimens for the subject. At sub-step 810f, the remote computing system may provide the prognosis prediction to the client computing system via a response communication. At step 820, the computing system may output the prognosis prediction in response to receiving the response communication. In some cases, one or more steps of the method depicted in Figure 8 may be repeated where appropriate. Although the present disclosure describes and illustrates specific steps of the method of Figure 8 as occurring in a particular order, the present disclosure contemplates any suitable steps of the method of Figure 8 occurring in any suitable order. Furthermore, although the present disclosure describes and illustrates an exemplary method for enabling an end user to request a prognostic prediction (including specific steps of the method of FIG. 8 ), the present disclosure contemplates any suitable method for enabling an end user to request a prognostic prediction (including any suitable steps, where appropriate, which may include all, some, or no steps of the method of FIG. 8 ). Furthermore, although the present disclosure describes and illustrates specific components, devices, or systems that perform specific steps of the method of FIG. 8 , the present disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of FIG. 8 .

FIG. 9 shows an exemplary method 900 for predicting alternative treatments based on the lack of gene fusions for a group of detected cancer cells. The method may begin at step 910, where the digital pathology image processing system 210 shown in FIG. 2 may access a digital pathology image depicting cancer cells in a specific slice of a biological sample from a subject. At step 920, the digital pathology image processing system 210 may determine that the digital pathology image includes a depiction of one or more mutations that are mutually exclusive with the occurrence of gene fusions. At step 930, the digital pathology image processing system 210 may determine that there is no gene fusion for cancer cells. At step 940, the digital pathology image processing system 210 may generate a prognosis prediction for the subject based on the absence of gene fusions for cancer cells, wherein the prognosis prediction includes a prediction of the suitability of one or more treatment regimens for the subject. In some cases, one or more steps of the method of FIG. 9 may be repeated where appropriate. Although the present disclosure describes and illustrates specific steps of the method of FIG. 9 as occurring in a specific order, the present disclosure contemplates any suitable steps of the method of FIG. 9 occurring in any suitable order. In addition, although the present disclosure describes and illustrates an exemplary method for excluding gene fusions (including specific steps of the method of FIG. 9 ), the present disclosure contemplates any suitable method for identifying the lack (or absence) of gene fusions (including any suitable steps, where appropriate, which may include all, some, or no steps of the method of FIG. 9 ). In addition, although the present disclosure describes and illustrates specific components, devices, or systems for performing specific steps of the method of FIG. 9 , the present disclosure contemplates any suitable combination of any suitable components, devices, or systems for performing any suitable steps of the method of FIG. 9 .

In some embodiments, identifying tumor heterogeneity includes classifying mutant tumor cells into phenotypes based on their morphological features, and assessing the spatial distribution of mutant tumor cells in any of the phenotypes. The mutational context or the way in which tumor cells mutate can vary from a high heterogeneity mutational context (e.g., tumor suppressors and/or unknown drivers, which can be prognostic for response to immunotherapy) to a moderate heterogeneity mutational context (e.g., oncogene mutations, which can be prognostic for response to targeted therapies for oncogene mutations) to a low heterogeneity (aka homogeneous or clonal mutational context (e.g., gene fusions, which can be prognostic for response to a particular type of targeted therapy for a particular gene fusion)).

Figures 10A to 10C are schematic diagrams of exemplary representations of mutational backgrounds, as shown in the upper window, with exemplary representations of corresponding visual features (e.g., morphological features of clustered tumor cells captured in WSI) shown above, as shown in the lower window. As shown, the representation of the mutational background shows how heterogeneity increases or decreases (both overall and within a single phenotype cluster of tumor cells) as tumor cells mutate and multiply. Different tumor cell phenotypes are represented by circles with different fill patterns.

FIG10A is a schematic diagram of an exemplary representation of a mutational context 1010—tumor suppressors and/or unknown drivers—and its corresponding visual features 1020. A high level of heterogeneity, such as between tumor cells, can be observed in morphological features. Exemplary visual features that distinguish the mutational context of tumor suppressors and/or unknown drivers from other mutational contexts include: a high degree of phenotypic variation between tumor cells, and an uneven spatial distribution of tumor cells between different phenotypes. As shown by the cluster 1010a of tumor cells (indicated by dashed lines), the phenotype represented by the black circle can be, for example, but not limited to, a tumor cell with a specific nuclear area range as measured and a certain horizontal angle to the nuclear boundary. Region 1020a of visual feature 1020 depicts an exemplary representation of the corresponding visual features of this exemplary phenotype (i.e., a tumor cell with a specific nuclear area range as measured and a certain horizontal angle to the nuclear boundary).

FIG10B is a schematic diagram of an exemplary representation of a mutational context 1030—oncogenic driver mutations (e.g., EGFR mutations)—and their corresponding visual features 1040. A moderate level of heterogeneity between tumor cells can be observed in morphological features. Exemplary visual features that distinguish a mutational context of oncogenic driver mutations from other mutational contexts include clusters of various phenotypes of tumor cells, and differences in spatial distribution between different clusters of tumor cells, but also tight, equidistant spatial distribution between tumor cells within a cluster. As shown by clusters 1030a, 1030b, and 1030c of tumor cells (indicated by dashed lines), the three phenotypes represented by these clusters can be, for example, but not limited to, three distinct ranges of tumor cells with a ratio of pixels in the nucleus to pixels in a bounding box in a selected region of the field of view. Regions 1040a, 1040b, and 1040c of visual features 1040 (corresponding to clusters 1030a, 1030b, and 1030c, respectively) depict exemplary representations of corresponding visual features of those three exemplary phenotypes.

FIG. 10C is a schematic diagram of an exemplary representation of a mutational context 1050—oncogene fusions (e.g., ALK, NTRK, ROS1, RET)—and their corresponding visual features 1060. A high degree of homogeneity (i.e., low or absent heterogeneity, also known as a clonal appearance) among tumor cells can be observed in morphological features. Exemplary visual features that distinguish a mutational context of oncogene fusions from other mutational contexts include: a low degree (or absence) of phenotypic changes in tumor cells, and a tight, equidistant spatial distribution of tumor cells. As shown by a cluster 1050a of tumor cells (indicated by a dotted line), the phenotype represented by the cluster can be, for example, but not limited to, tumor cells with specific pixel intensity values. Visual features 1060 depict an exemplary representation of the corresponding visual features of the exemplary phenotype (i.e., a homogeneous cluster of cells of a single phenotype).

FIG. 11 is a flow chart for a non-limiting example method 1100 for identifying tumor heterogeneity. At step 1110, a digital pathology image depicting a tumor cell taken from a subject can be accessed by retrieval (e.g., from a database or online repository) or by scanning a WSI using a digital pathology image generation system 220. The digital pathology image may have been previously identified as an image depicting a tumor cell (e.g., by a human pathologist or by a digital pathology image processing system 210). At step 1120, a tile can be selected from a digital pathology image. In these tiles, tumor cells may have been annotated, cell nuclei have been segmented, and cell nucleus masks have been generated. At step 1130, tumor cells can be classified into phenotypic clusters including shared nuclear morphological features based on assessment of nuclear morphological features (e.g., as listed below), each of which corresponds to a different mutation. In some embodiments, other morphological features of tumor cells, their substructures and/or stroma or other nearby cells may also be assessed. At step 1140, tumor cells can be grouped into clusters representing tumor regions or nests by phenotype and location. Grouping tumor cells by phenotype can include assessing tumor cells based on morphological features. Grouping tumor cells by location can include: generating a minimum spanning tree, and splitting the tree into subgraphs by segmenting tumor nests as depicted or using outlier detection by any other suitable technique. At step 1150, the spatial distance between cells of each in the cluster can be quantified. Spatial distances can be quantified by calculating the adjacent spectral distances in pairs for all graphs in the WSI. Spatial distances can also be quantified by measuring spatial entropy to identify closely adjacent clonal cells. At step 1160, the possibility of the presence of actionable mutations can be predicted based on classified tumor cells, identified clusters, or quantified spatial distances. At step 1170, based on tile-level predictions, a prognosis prediction can be generated for a subject, wherein the prognosis prediction includes a prediction of a response to one or more treatment regimens for the subject.

Figures 12A to 12C depict non-limiting examples of tumor cell annotation, nucleus segmentation, and nucleus mask generation. As shown in Figure 12A, the position of each of the tumor cell nuclei can be annotated (e.g., using simple points), and each nucleus is then segmented, as shown in Figure 12B. As shown in the reference numerals in Figure 12B, different nuclei can exhibit different nuclear morphological features. For example, the border of nucleus 1210 is relatively circular, as compared to the oval border of nucleus 1220 or the irregular, partially curved, partially angled borders of nuclei 1230 and 1240. In addition, the relative area of nucleus 1220 appears to be about twice that of nucleus 1210, and about three times the area of nucleus 1230. Next, as shown in Figure 12C, a mask can be generated for each nucleus to define the boundaries within which each nuclear morphological feature is measured.

In some embodiments, tumor heterogeneity is identified by assessing nuclear heterogeneity. In some embodiments, assessing nuclear heterogeneity includes: quantifying certain features of the nucleus to distinguish mutant cells based on nuclear morphology heterogeneity. In some embodiments, the digital pathology image processing system 210 can use any of several different methods to analyze each tumor cell nucleus identified in the WSI. For example, in one method, automatic tumor cell nucleus detection and parameterization can be performed, wherein a trained machine learning model can be used to identify each tumor cell nucleus, measure a set of specified parameters or features for each cell nucleus (as described below), and then compare the population-level distribution of these specified parameters or features. In another example, the method may include performing tumor image segmentation, which may be an assessment based on image tiles. In some cases, the determination of tumor heterogeneity can be performed on the basis of slide prediction (which may include, for example, calculating the percentage of image tiles predicted to be heterogeneous, or the average value of the predicted score for each slide).

In some cases, tumor heterogeneity can be driven by the gene types involved in gene fusion. For tumor suppressor genes, tumor formation can be mediated by loss of function. Such mutations cause cells to escape cell cycle control, which in turn can indirectly promote growth. Over time, this process causes each new generation of daughter cells to accumulate cancer-promoting mutations. On the contrary, for oncogenes, tumor formation can be mediated by gain of function. Overactivation of growth factors can, for example, directly promote growth, resulting in unconstrained growth. It is expected that this process results in an immediate growth advantage that does not require additional mutations. Based on this basic principle, low tumor heterogeneity can be expected in tumors including fusions involving oncogenes such as ALK, ROS1, RET, and NTRK.

One method of assessing tumor heterogeneity may include assessing the morphology of cellular level structures (such as nuclei) in tumor cells. The morphology of the nuclei may be represented by a plurality of image features, which may be organized into categories of image features such as, for example but not limited to, chromatin features, geometric coordinates, basic morphological features, two-dimensional shape features, first-order statistics, "grayscale" (e.g., where "grayscale" refers to the spatial distribution of pixel intensity levels) co-occurrence matrix features, grayscale dependency matrix features, grayscale run matrix features, grayscale size region matrix features, neighboring grayscale hue difference matrix features, high-level nuclear morphology features, and boundary and curvature features.

Each type may include one or more image features. Exemplary image features may include, but are not limited to:

Chromatin features, such as:

○ Heterogeneity of the nucleus (heterogeneity),

○Particle size distribution (cluster),

○The fraction of large particles relative to the total core area (agglomeration),

○Distribution around the nuclear membrane or edge (edge set);

●Geometric coordinates, such as:

○Region coordinates (region_coords),

○x coordinate (x),

○y coordinate (y);

Basic morphological features, such as:

○The area of the cell nucleus (area),

○The convex area of the nucleus (convex_area),

○Eccentricity of the cell nucleus,

○The diameter of the cell nucleus (equivalent_diameter),

○ The ratio of pixels in the nucleus to pixels in the bounding box, either in the total field of view or in a selected region (range) of the field of view,

○ The circumference (perimeter) of the nucleus,

○The ratio of pixels in the nucleus to pixels in the convex hull (solidity),

○ The elongation of the nucleus as measured by the eigenvalues of the inertia tensor (inertia_tensor_eigvals1, inertia_tensor_eigvals2),

○The length of the major axis of the cell nucleus (major_axis_length),

○The length of the minor axis of the cell nucleus (minor_axis_length),

○Hu moment (some specific weighted average of the intensity of image pixels, also known as

"Moments", i.e., translation, rotation, and scale invariant moments) (moments_hu0,

moments_hu1,moments_hu2,moments_hu3,moments_hu4,moments_hu5,moments_hu6),

○Weighted Hu moments (weighted_moments_hu0,

weighted_moments_hu1,weighted_moments_hu2,

weighted_moments_hu3,weighted_moments_hu4,

weighted_moments_hu5,weighted_moments_hu6);

2D shape features, such as:

○2D shape elongation (original_shape2D_Elongation),

○Maximum diameter of two-dimensional shape

(original_shape2D_MaximumDiameter),

○ 2D shape mesh surface (original_shape2D_MeshSurface),

○Ratio of perimeter to surface of two-dimensional shape

(original_shape2D_PerimeterSurfaceRatio),

○ 2D shape pixel surface (original_shape2D_PixelSurface),

○ 2D shape sphericity (original_shape2D_Sphericity),

○ Two-dimensional shape spherical imbalance

(original_shape2D_SphericalDisproportion);

First-order statistics, such as:

○ first order 10th percentile (original_firstorder_10Percentile),

○ The first-order 90th percentile (original_firstorder_90Percentile),

○First-order energy (original_firstorder_Energy),

○ First-order entropy, which specifies the uncertainty or randomness in the image values

(original_firstorder_Entropy),

○ The first-order interquartile range, which measures variability based on quartile splits

(original_firstorder_InterquartileRange),

○ First-order kurtosis, which measures the “peakedness” of the distribution of values

(original_firstorder_Kurtosis),

○First-order maximum (original_firstorder_Maximum),

○First-order mean absolute deviation

(original_firstorder_MeanAbsoluteDeviation),

○First-order mean (original_firstorder_Mean),

○First-order median (original_firstorder_Median),

○First-order minimum (original_firstorder_Minimum),

○First-order range (original_firstorder_Range),

○ The first-order robust mean absolute deviation, which is the average distance of all intensity values from the mean

(original_firstorder_RobustMeanAbsoluteDeviation),

○First-order RMS, which is the average of all squared intensity values

(original_firstorder_RootMeanSquared),

○ First-order skewness, which measures the asymmetry of the distribution of values about the mean

(original_firstorder_Skewness),

○First-order total energy (original_firstorder_TotalEnergy),

○First-order uniformity (original_firstorder_Uniformity),

○First-order variance (original_firstorder_Variance);

Gray-level co-occurrence matrix (GLCM) (a second-order joint probability function describing the image region constrained by the mask) features, such as

○GLCM autocorrelation (original_glcm_Autocorrelation),

○GLCM cluster prominence (original_glcm_ClusterProminence),

○GLCM cluster shadow (original_glcm_ClusterShade),

○GLCM cluster trend (original_glcm_ClusterTendency),

○GLCM contrast (original_glcm_Contrast),

○GLCM correlation (original_glcm_Correlation),

○GLCM difference average (original_glcm_DifferenceAverage),

○GLCM difference entropy (original_glcm_DifferenceEntropy)

○GLCM difference variance (original_glcm_DifferenceVariance),

○GLCM deficit (original_glcm_Id),

○GLCM inverse moment difference (original_glcm_Idm),

○ Normalized GLCM inverse moment difference (original_glcm_Idmn),

○ Normalized GLCM deficit (original_glcm_Idn),

○GLCM correlation information measurement (original_glcm_lmc1,

original_glcm_Imc2),

○GLCM inverse variance (original_glcm_InverseVariance),

○GLCM joint average (original_glcm_JointAverage),

○GLCM joint energy (original_glcm_JointEnergy),

○GLCM joint entropy (original_glcm_JointEntropy),

○GLCM maximum correlation coefficient (original_glcm_MCC),

○GLCM maximum probability (original_glcm_MaximumProbability),

○GLCM sum average (original_glcm_SumAverage),

○GLCM sum entropy (original_glcm_SumEntropy),

○GLCM sum of squares (original_glcm_SumSquares);

Gray-level dependency matrix (quantifies gray-level dependency in an image, where gray-level dependency is defined as the number of connected pixels that depend on a center pixel within a specified distance) features such as:

○GLDM gray level dependency entropy

(original_gldm_DependenceEntropy),

○GLDM relies on heterogeneity

(original_gldm_DependenceNonUniformity),

○Normalized GLDM dependence heterogeneity

(original_gldm_DependenceNonUniformityNormalized),

○GLDM dependency variance (original_gldm_DependenceVariance),

○GLDM grayscale non-uniformity

(original_gldm_GrayLevelNonUniformity),

○GLDM gray level variance (original_gldm_GrayLevelVariance),

○GLDM high gray level emphasis

(original_gldm_HighGrayLevelEmphasis),

○GLDM relies heavily on emphasis

(original_gldm_LargeDependenceEmphasis),

○GLDM relies heavily on high grayscale emphasis

(original_gldm_LargeDependenceHighGrayLevelEmphasis),

○GLDM heavily relies on low grayscale emphasis

(original_gldm_LargeDependenceLowGrayLevelEmphasis),

○GLDM low gray level emphasis

(original_gldm_LowGrayLevelEmphasis),

○GLDM small dependency emphasis

(original_gldm_SmallDependenceEmphasis),

○GLDM small dependency high gray level emphasis

(original_gldm_SmallDependenceHighGrayLevelEmphasis),

○GLDM small dependence low gray level emphasis

(original_gldm_SmallDependenceLowGrayLevelEmphasis);

Gray Level Run Matrix (GLRLM) (quantized gray level run, which is defined as the length of the number of pixels of consecutive pixels with the same gray level value) features such as:

○GLRLM grayscale non-uniformity

(original_glrlm_GrayLevelNonUniformity),

○Normalized GLRLM gray level non-uniformity

(original_glrlm_GrayLevelNonUniformityNormalized),

○ GLRLM gray level variance (original_glrlm_GrayLevelVariance),

○GLRLM high grayscale run-length emphasis

(original_glrlm_HighGrayLevelRunEmphasis),

○GLRLM Long Run Emphasis (LRE)

(original_glrlm_LongRunEmphasis),

○GLRLM long run length high gray level emphasis

(original_glrlm_LongRunHighGrayLevelEmphasis),

○GLRLM long run low gray level emphasis

(original_glrm_LongRunLowGrayLevelEmphasis),

○GLRLM low gray level run-length emphasis

(original_glrlm_LowGrayLevelRunEmphasis),

○GLRM run entropy (original_glrm_RunEntropy),

○GLRM run length non-uniformity),

(original_glrm_RunLengthNonuniformity),

○Normalized GLRLM run-length non-uniformity

(original_glrlm_RunLengthNonUniformityNormalized),

○GLRLM run percentage (original_glrlm_RunPercentage),

○ GLRLM run variance (original_glrlm_RunVariance),

○GLRLM short run emphasis (original_glrlm_ShortRunEmphasis),

○GLRLM short run length high gray level emphasis

(original_glrlm_ShortRunHighGrayLevelEmphasis),

○GLRLM short run low gray level emphasis

(original_glrlm_ShortRunLowGrayLevelEmphasis);

Gray Level Size Zone (GLSZM) (describing the gray level zone in the image area) matrix features, such as:

○GLSZM grayscale non-uniformity

(original_glszm_GrayLevelNonUniformity),

○Normalized GLSZM grayscale non-uniformity

(original_glszm_GrayLevelNonUniformityNormalized),

○GLSZM gray level variance (original_glszm_GrayLevelVariance),

○GLSZM High gray level area emphasis

(original_glszm_HighGrayLevelZoneEmphasis),

○GLSZM emphasizes large areas

(original_glszm_LargeAreaEmphasis),

○GLSZM large area high gray level emphasis

(original_glszm_LargeAreaHighGrayLevelEmphasis),

○GLSZM large area low gray level emphasis

(original_glszm_LargeAreaLowGrayLevelEmphasis),

○GLSZM low gray level area emphasis

(original_glszm_LowGrayLevelZoneEmphasis),

○GLSZM size zone non-uniformity

(original_glszm_SizeZoneNonUniformity),

○Normalized GLSZM size zone non-uniformity

(original_glszm_SizeZoneNonUniformityNormalized),

○GLSZM small area emphasis

(original_glszm_SmallAreaEmphasis),

○GLSZM small area emphasis high gray level emphasis

(original_glszm_SmallAreaHighGrayLevelEmphasis),

○GLSZM small area low gray level emphasis

(original_glszm_SmallAreaLowGrayLevelEmphasis),

○GLSZM zone entropy (original_glszm_ZoneEntropy),

○GLSZM zone percentage (original_glszm_ZonePercentage),

○GLSZM zone variance (original_glszm_ZoneVariance);

Neighborhood Gray Tone Difference Matrix (NGTDM) (describes the difference between a gray value and the average gray value of its neighbors within a certain distance) features, such as:

○NGTDM Busyness (original_ngtdm_Busyness),

○NGTDM roughness (original_ngtdm_Coarseness),

○NGTDM complexity (original_ngtdm_Complexity),

○NGTDM Contrast (original_ngtdm_Contrast),

○NGTDM strength (original_ngtdm_Strength);

●Advanced nuclear morphological features, such as:

○The radius of the elliptical nucleus (ellipse_R_index),

○The major axis of the elliptical nucleus (ellipse_MA_index),

○ The convex perimeter of the nucleus, which measures the curvature of the perimeter

(convexity_perimeter),

○ Nuclear roundness, which measures the circularity (roundness) of the nucleus,

○ The normalized number of connected components remaining when the shape is subtracted from the convex hull

(Ncce_index);

●Boundary (where the boundary feature of the nucleus is the distance distribution from all boundary coordinates to the center of mass of the nucleus) features, such as:

○ Mean (R) ),

○Median (Median (R)),

○Mode (Mode (R)),

○ Maximum value (max_v:maxR(R)),

○ Minimum value (min_v:min(R)),

○ The 25th percentile of the border feature (percentile_25: 25% percentile

(R)),

○ The 75th percentile of the boundary feature (percentile_75: 75% percentile

(R)),

○ Below the 25th percentile for borderline characteristics

(mean_below_percentile_25:mean(R(R<percentile_25))), ○ Above the 75th percentile of the mean boundary feature

(mean_above_percentile_75:mean(R(R>percentile_75))),○The sum of the distances of the boundary features (sum_dist:sum(R)),

○The harmonic mean of the boundary features (harmonic_mean: harmonic

mean(R)),

○3% trimmed mean boundary feature (trimmed_mean_3_percent: 3%

trimmed mean(R)),

○5% trimmed mean boundary feature (trimmed_mean_5_percent: 5%

trimmed mean(R)),

○15% trimmed mean boundary feature (trimmed_mean_15_percent:

15% trimmed mean(R)),

○25% trimmed mean boundary feature (trimmed_mean_25_percent:

25% trimmed mean(R)),

○Standard deviation (std_dev: standard deviation (R) ),

○ Standard deviation by mean (std_dev_by_mean:sR/|<R>|),

○Standard deviation by median (std_dev_by_median:

sR/|median(R)|),

○Standard deviation by mode (std_dev_by_mode:sR/|mode(R)|),

○ Skewness (Skewness(R)),

○ Kurtosis (Kurtosis (R)),

○ The average distance distribution minus the average of the boundary features

(mean_dist_profile_minus_mean:mean(|R-<R>|)),

○ range (range_v:range(X)),

○ Interquartile range (interquartile_range:interquartile range(X)),

○Sum of square distance distribution (sum_dist_profile_square:sum(R2)),

○ Cubic sum distance profile (sum_dist_profile_cube:sum(R3)),

○ Square mean distance distribution (mean_dist_profile_square:mean(R2)),

○ Cubic mean distance profile (mean_dist_profile_cube:mean(R3)),

○ Improved to the fourth power average distance distribution

(mean_dist_profile_raise_to_four:mean(R4)),

○ Improved to the fifth power average distance distribution

(mean_dist_profile_raise_to_five:mean(R5)),

○ The sum of the distance distribution minus the average power of 2

(sum_dist_profile_minus_mean_pow2:sum(|R-<R>|2)),

○ The sum of the distance distribution minus the average power of 3

(sum_dist_profile_minus_mean_pow3:sum(|R-<R>|3)),

○ The average distance distribution minus the average power of 2

(mean_dist_profile_minus_mean_pow2:mean(|R-<R>|2)),

○ The average distance distribution minus the average power of 3

(mean_dist_profile_minus_mean_pow3:mean(|R-<R>|3)),

○ The average distance distribution minus the average power of 4

(mean_dist_profile_minus_mean_pow4:mean(|R-<R>|4)),

○ The average distance distribution minus the average power of 5

(mean_dist_profile_minus_mean_pow5:mean(|R-<R>|5)),

○Number of peaks (number_of_peaks),

○ Gini coefficient (Gini coefficient);

Curvature features, such as:

○ Mean curvature (c_mean:mean(k) ),

○Median curvature (c_median:median(k)),

○ Mode curvature (c_mode: mode(k)),

○ Maximum curvature (c_max_v:max(k)),

○ Minimum curvature (c_min_v:min(k)),

○ 25th percentile curvature (c_percentile_25: 25% percentile (k)),

○ 75th percentile curvature (c_percentile_75: 75% percentile(k)), ○ 25th percentile below the mean curvature

(c_mean_below_percentile_25:mean(k(k<

c_percentile_25))),

○ Above the 75th percentile of mean curvature

(c_mean_above_percentile_75:mean(k(k>

c_percentile_75))),

○Sum distance (c_sum_dist:sum(k)),

○ Harmonic mean (c_harmonic_mean:harmonic mean(k)),

○3% trimmed mean curvature (c_trimmed_mean_3_percent: 3%

trimmed mean(k)),

○3% trimmed mean curvature (c_trimmed_mean_5_percent: 5%

trimmed mean(k)),

○15% trimmed mean curvature (c_trimmed_mean_15_percent:

15% trimmed mean(k)),

○25% trimmed mean curvature (c_trimmed_mean_25_percent:

25% trimmed mean(k)),

○Standard deviation (c_std_dev: standard deviation (k) ),

○ Standard deviation by mean (c_std_dev_by_mean:sk/|<k>|),

○Standard deviation by median (c_std_dev_by_median:

sk/|median(k)|),

○ standard deviation by mode (c_std_dev_by_mode:sk/|mode(k)|), ○ skewness (c_skewness:skewness(k)),

○Kurtosis(c_kurtosis:kurtosis(k)),

○ Mean curvature (c_mean:mean(|k-<k>|)),

○The range of curvature (c_range_v:range(k)),

○ interquartile range (c_interquartile_range:interquartile range(k)), ○ sum of square curvature profile (c_sum_curvature_profile_square:

sum(k2)),

○ Cubic sum curvature profile (c_sum_curvature_profile_cube:

sum(k3)),

○ Square mean curvature profile (c_mean_curvature_profile_square:

mean(k2)),

○Cubic mean curvature profile (c_mean_curvature_profile_cube:

mean(k3)),

○The average curvature distribution increased to the fourth power

(c_mean_curvature_profile_raise_to_four:mean(k4)),

○The average curvature distribution increased to the fifth power

(c_mean_curvature_profile_raise_to_five:mean(k5)),

○ The sum of the curvature distribution minus the average power of 2

(c_sum_curvature_profile_minus_mean_pow2:sum(|k-

<k>|2)),

○ The sum of the curvature distribution minus the average power of 3

(c_sum_curvature_profile_minus_mean_pow3:sum(|k-

<k>|3)),

○ The mean curvature distribution minus the mean power of 2

(c_mean_curvature_profile_minus_mean_pow2:mean(|k-

<k>|2)),

○ The mean curvature distribution minus the mean power of 3

(c_mean_curvature_profile_minus_mean_pow3:mean(|k-

<k>|3)),

○ The mean curvature distribution minus the mean power of 4

(c_mean_curvature_profile_minus_mean_pow4:mean(|k-

<k>|4)),

○ Mean curvature distribution minus the average power of 5

(c_mean_curvature_profile_minus_mean_pow5:mean(|k-

<k>|5)),

○Number of peaks (c_number_of_peaks:number of peaks),

○Gini coefficient (c_gini_coefficient:gini coefficient(k)).

One or more statistical metrics can be used to evaluate image features. One or more feature selection processes can be used to select image features associated with oncogene drivers. Non-limiting example statistical metrics are standard deviation, quadratic entropy averaging the differences between two randomly drawn samples, Kolmogorov-Smirnov based on the distance between the normal distribution of the sample and the empirical distribution function, and the percentage of outliers (e.g., the percentage of values outside the range of two times the standard deviation of the mean). In some embodiments, the selected image feature can have the highest correlation with the oncogene driver among the multiple image features. The oncogene driver can be a fusion, a mutation, or an unknown driver.

In some embodiments, exemplary selected nuclear morphology image features may be included in four exemplary categories. One category of shape-related features may include features targeting the geometry of the cell nucleus. By way of example and not limitation, the captured attributes may include the size and shape of individual cell nuclei and their image moments (weighted averages of image pixel intensities). Exemplary selected features in this category may include:

●The area of the cell nucleus (area),

The ratio of pixels in the nucleus to pixels in the bounding box, either in the total field of view or in a selected region (range) of the field of view,

The ratio of pixels in the nucleus to pixels in the convex hull (solidity),

Hu moment (a certain weighted average of the intensities of the image pixels, also known as "moments", i.e., translation, rotation, and scale invariant moments) (moments_hu0),

●Weighted Hu moments (weighted_moments_hu0, weighted_moments_hu1, weighted_moments_hu2);

● 2D shape features, such as 2D shape perimeter to surface ratio (original_shape2D_PerimeterSurfaceRatio);

●The radius of the elliptical nucleus (ellipse_R_index),

●The major axis of the elliptical nucleus (ellipse_MA_index)

The convexity perimeter of the nucleus, which measures the curvature of the perimeter (convexity_perimeter), and

• The normalized number of connected components remaining when the shape is subtracted from the convex hull (Ncce_index).

The category of intensity distribution related features may include features that capture the statistical properties of the distribution of image intensities (pixel values) in images of individual cell nuclei. Exemplary selected features in this category may include:

● first-order 90th percentile (original_firstorder_90Percentile),

● the first-order minimum (original_firstorder_Minimum), and

• First-order entropy, which specifies the uncertainty or randomness in the image values (original_firstorder_Entropy).

The category of texture-related features may include: features that target the quantification of texture by analyzing the spatial relationship between pixels of a cell nucleus image and their values (in sub-regions). As described below, the gray-level co-occurrence matrix (GLCM) describes the second-order joint probability function of an image region constrained by a mask. The gray-level dependency matrix (GLDM) quantifies the gray-level dependency in an image, where the gray-level dependency is defined as the number of connected pixels that depend on a center pixel within a specified distance. The gray-level run matrix (GLRLM) quantifies the gray-level run, which is defined as the length of the number of pixels of consecutive pixels with the same gray-level value. Exemplary selected features in this category may include:

GLCM deficit (original_glcm_Id),

GLCM contrast (original_glcm_contrast),

GLCM joint entropy (original_glcm_JointEntropy),

●GLCM sum entropy (original_glcm_SumEntropy);

GLDM gray level dependency entropy (original_gldm_DependenceEntropy),

● Normalized GLDM dependency non-uniformity (DNUN), which measures the similarity of dependencies across images and is normalized

(original_DependenceNonUniformityNormalized),

Small Dependency Emphasis (SDE), which measures the distribution of small dependencies

(original_gldm_SmallDependenceEmphasis);

●GLRLM Long Run Emphasis (LRE) (original_glrlm_LongRunEmphasis),

●GLRLM long run length low gray level emphasis

(original_glrm_LongRunLowGrayLevelEmphasis),

●GLRM run length non-uniformity (original_glrm_RunLengthNonuniformity),

GLRM run entropy (original_glrm_RunEntropy);

The category of boundary curvature related features may include features derived by analyzing the boundary curvature of the cell nucleus using different statistical methods. Exemplary selected features in this category may include:

● Mean curvature (c_mean),

●Median curvature (c_median),

●25th percentile curvature (c_percentile_25),

●75th percentile curvature (c_percentile_75),

● Above the 75th percentile of the mean curvature (c_mean_above_percentile_75),

3% trimmed mean curvature (c_trimmed_mean_3_percent),

5% trimmed mean curvature (c_trimmed_mean_5_percent),

15% trimmed mean curvature (c_trimmed_mean_15_percent),

25% trimmed mean curvature (c_trimmed_mean_25_percent),

●Interquartile range curvature (c_interquartile_range), and

●Gini coefficient of curvature (c_gini_coefficient).

Figures 13A to 13E show non-limiting examples of experimental results based on quantifying certain nuclear morphological features to identify gene fusions. As shown in Figures 13A to 13E, comparison of these specific nuclear morphological features can provide statistically significant distinctions between gene fusions, oncogene driver mutations, and tumor suppressors and/or unknown drivers.

FIG14A to FIG14C illustrate non-limiting examples of digital pathology images showing four different nuclear morphological features corresponding to tumor suppressors and/or unknown drivers. FIG14A to FIG14C illustrate exemplary WSIs for which the nuclear morphological features of “area”, “first-order entropy”, and “run non-uniformity” are highly indicative of the degree of tumor heterogeneity that may exist in the mutational background of tumor suppressors and/or unknown drivers, respectively. “Area” may be the surface area of the observed nuclei. “First-order entropy” may be a first-order statistic describing the distribution of pixel intensities (i.e., values) within the considered image region. Thus, entropy specifies the uncertainty/randomness in the observed pixel values. “Run non-uniformity” may be a run length metric that quantifies the grayscale run in an image. A grayscale run is defined as the length of the number of pixels of consecutive pixels with the same grayscale value. In the grayscale run length matrix p(i,j|θ), the (i,j)th element describes the number of times j that grayscale level i appears consecutively in the direction specified by θ. The run non-uniformity measure analyzes the distribution of possible runs with a given run based on the gray-level run matrix.

Figures 15A-15C show non-limiting examples of digital pathology images showing four different nuclear morphology features corresponding to oncogenic driver mutations. Figures 15A-15C show exemplary WSIs for which the nuclear morphology features of "Area", "First Order Entropy" and "Run Heterogeneity" are each highly indicative of the degree of tumor heterogeneity that may exist in the mutational background of oncogenic driver mutations.

Figures 16A-16C show non-limiting examples of digital pathology images showing four different nuclear morphological features corresponding to gene fusions. Figures 16A-16C show exemplary WSIs for which the nuclear morphological features of "Area", "First Order Entropy" and "Run Heterogeneity" are each highly indicative of the degree of tumor heterogeneity that may exist in the mutational context of the gene fusion.

In some embodiments, identifying tumor heterogeneity includes: identifying the region of clonal cells by performing cell-level spatial analysis to assess spatial distribution. In some embodiments, assessing spatial distribution includes: measuring the spectral distance within the subgraph of the minimum spanning tree of tumor cells, wherein each of the subgraphs represents clustered adjacent cells (e.g., tumor nests), and calculating the adjacent spectral distance in pairs across all subgraphs. Figure 17A shows a non-limiting example of the identification of the subgraph of the minimum spanning tree of tumor cells depicted in the WSI. After generating the minimum spanning tree connecting all cells in the field of view or WSI, a subgraph (e.g., corresponding to a tumor nest) can be identified. Each subgraph can be defined by performing outlier detection and/or defined based on the segmentation of the detected tumor nest. Once the adjacent spectral distance across all subgraphs is paired, an assessment can be made about the spatial distribution of tumor cells. More equidistantly distributed tumor cells can correspond to the region of homogeneous tumor cells (i.e., cells with specific gene fusions). Figure 17B shows a non-limiting example of the experimental results of identifying gene fusions based on quantification such as the adjacent spectral distance between subgraphs. As shown in Figure 17B, comparison of neighbor spectral distances across subgraphs can provide statistically significant distinctions between gene fusions, oncogene driver mutations, and tumor suppressors and/or unknown drivers.

In some embodiments, identifying tumor heterogeneity includes: identifying regions of closely adjacent clonal cells by performing a cell-level spatial analysis to assess spatial entropy. Figures 18A to 18C show non-limiting examples of spatial entropy as measured in three different distributions of tumor cell phenotypes. As shown in Figures 18A to 18C, measures of non-spatial entropy (i.e., Shannon entropy) cannot reflect the effect of location on the heterogeneity of the distribution of occurrences, while spatial entropy (i.e., Altieri entropy) accurately reflects the differences in heterogeneity between occurrences as plotted spatially along the x and y axes in Figures 18A to 18C.

In some embodiments, assessing spatial entropy may include specifying a set of distinct distance bins (each bin representing a range of distances between a pair of tumor cells), identifying all pairs of tumor cells belonging to each of the distance bins, specifying each of the distance bins to calculate the frequency at which paired tumor cells are identified as morphologically similar, and then calculating a weighted sum of all bin frequency values. The weight applied to each distance bin may correspond to the number of paired tumor cells in the distance bin. The set of distance bins may be limited to those bins representing a range of distances with a maximum distance below a specified threshold. FIG. 18D is a schematic diagram of the identification of paired tumor cells that coexist within a specified distance and are classified in a specified phenotype.

Figures 18E to 18F show non-limiting examples of experimental results based on measuring spatial entropy to identify gene fusions, oncogene driver mutations, and tumor suppressors and/or unknown drivers. As shown in Figures 18E to 18F, spatial entropy can provide statistically significant distinctions between gene fusions, oncogene driver mutations, and tumor suppressors and/or unknown drivers.

FIG. 19 illustrates an exemplary computer system 1900. In a particular embodiment, one or more computer systems 1900 perform one or more steps of one or more methods described or illustrated herein. In a particular embodiment, one or more computer systems 1900 provide functionality described or illustrated herein. In a particular embodiment, software running on one or more computer systems 1900 performs one or more steps of one or more methods described or illustrated herein, or provides functionality described or illustrated herein. A particular embodiment includes one or more portions of one or more computer systems 1900. Herein, references to computer systems may include computing devices, and vice versa, where appropriate. Additionally, references to computer systems may include one or more computer systems, where appropriate.

The present disclosure contemplates any suitable number of computer systems 1900. The present disclosure contemplates computer systems 1900 in any suitable physical form. By way of example and not limitation, computer system 1900 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more thereof. Where appropriate, computer system 1900 may include one or more computer systems 1900; may be integrated or distributed; may span multiple locations; may span multiple machines; may span multiple data centers; or may reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1900 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not limitation, one or more computer systems 1900 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 1900 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations.

In a particular embodiment, computer system 1900 includes a processor 1902, a memory 1904, a storage device 1906, an input/output (I/O) interface 1908, a communication interface 1910, and a bus 1912. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

In a particular embodiment, the processor 1902 includes hardware for executing instructions, such as those that constitute a computer program. By way of example and not limitation, to execute instructions, the processor 1902 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 1904, or storage 1906; may decode and execute these instructions; and may then write one or more results to an internal register, an internal cache, memory 1904, or storage 1906. In a particular embodiment, the processor 1902 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1902 that includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 1902 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1904 or storage 1906, and the instruction cache may speed up the retrieval of those instructions by the processor 1902. The data in the data cache may be: a copy of the data in the memory 1904 or storage device 1906 for operation by instructions executed at the processor 1902; the result of the previous instruction executed at the processor 1902 for access or writing to the memory 1904 or storage device 1906 by subsequent instructions executed at the processor 1902; or other suitable data. The data cache may speed up the read or write operation of the processor 1902. The TLB may speed up the virtual address translation of the processor 1902. In a particular embodiment, the processor 1902 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1902 including any suitable number of any suitable internal registers. Where appropriate, the processor 1902 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 1902. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

In a particular embodiment, the memory 1904 includes a main memory for storing instructions for the processor 1902 to execute or data for the processor 1902 to operate on. As an example and not limitation, the computer system 1900 may load instructions from the storage device 1906 or another source (such as, for example, another computer system 1900) to the memory 1904. The processor 1902 may then load the instructions from the memory 1904 to an internal register or an internal cache. To execute the instructions, the processor 1902 may retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 1902 may write one or more results (which may be intermediate results or final results) to the internal register or the internal cache. The processor 1902 may then write one or more of those results to the memory 1904. In a particular embodiment, the processor 1902 executes only instructions in one or more internal registers or internal caches or in memory 1904 (rather than storage 1906 or elsewhere), and operates only on data in one or more internal registers or internal caches or in memory 1904 (rather than storage 1906 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 1902 to the memory 1904. Bus 1912 may include one or more memory buses, as described below. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 1902 and the memory 1904 and facilitate access to the memory 1904 requested by the processor 1902. In a particular embodiment, the memory 1904 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port or multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1904 may include one or more memories 1904. Although this disclosure describes and illustrates particular memories, this disclosure contemplates any suitable memories.

In a particular embodiment, the storage device 1906 includes a mass storage device for data or instructions. By way of example and not limitation, the storage device 1906 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more thereof. Where appropriate, the storage device 1906 may include a removable or non-removable (or fixed) medium. Where appropriate, the storage device 1906 may be inside or outside the computer system 1900. In a particular embodiment, the storage device 1906 is a non-volatile solid-state memory. In a particular embodiment, the storage device 1906 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more thereof. The present disclosure contemplates a mass storage device 1906 in any suitable physical form. Where appropriate, storage 1906 may include one or more memory control units that facilitate communications between processor 1902 and storage 1906. Where appropriate, storage 1906 may include one or more storage devices 1906. Although this disclosure describes and illustrates particular storage devices, this disclosure contemplates any suitable storage devices.

In a particular embodiment, the I/O interface 1908 includes hardware, software, or both, which provides one or more interfaces for communicating between the computer system 1900 and one or more I/O devices. Where appropriate, the computer system 1900 may include one or more of these I/O devices. One or more of these I/O devices can enable communication between a person and the computer system 1900. As an example and not limitation, the I/O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a camera, another suitable I/O device, or a combination of two or more thereof. The I/O device may include one or more sensors. The present disclosure contemplates any suitable I/O device and any suitable I/O interface 1908 for them. Where appropriate, the I/O interface 1908 may include one or more device or software drivers so that the processor 1902 can drive one or more of these I/O devices. Where appropriate, the I/O interface 1908 may include one or more I/O interfaces 1908. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.

In a particular embodiment, the communication interface 1910 includes hardware, software, or both that provides one or more interfaces for communication (such as, for example, packet-based communication) between the computer system 1900 and one or more other computer systems 1900 or one or more networks. By way of example and not limitation, the communication interface 1910 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (such as a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 1910 for it. By way of example and not limitation, the computer system 1900 may communicate with one or more parts of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more thereof. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1900 may be connected to a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more thereof. Where appropriate, the computer system 1900 may include any suitable communication interface 1910 for any of these networks. Where appropriate, the communication interface 1910 may include one or more communication interfaces 1910. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure encompasses any suitable communication interface.

In certain embodiments, bus 1912 includes hardware, software, or two mutually coupled components of computer system 1900. By way of example and not limitation, bus 1912 may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a HyperTransport (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or another suitable bus or a combination of two or more thereof. Where appropriate, bus 1912 may include one or more buses 1912. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus.

Herein, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more thereof. Where appropriate, the computer-readable non-transitory storage medium may be a volatile storage medium, a non-volatile storage medium, or a combination of a volatile storage medium and a non-volatile storage medium.

As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Furthermore, as used herein, "and" means both jointly and severally, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

The scope of the present disclosure covers all changes, substitutions, variations, alterations and modifications to the exemplary embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the corresponding embodiments herein as including specific parts, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of any parts, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, in the appended claims, references to devices or systems or parts of devices or systems adapted to, arranged to, capable of, configured to, enabled to, operable to or operated to perform specific functions, cover the device, system, component, whether or not it or the specific function is activated, turned on or unlocked, as long as the device, system or component is so adapted, arranged, capable of, configured to, enabled to, operable or operated. In addition, although the present disclosure describes or illustrates specific embodiments as providing specific advantages, specific embodiments may not provide these advantages, some advantages or all advantages.

Implementation

1. A method comprising:

access to digital pathology images depicting tumor cells sampled from the subject;

selecting a plurality of tiles from the digital pathology image, wherein each of the tiles depicts a tumor cell;

generating a mutation prediction for each of the tiles, wherein the mutation prediction represents a prediction of the likelihood that an actionable mutation will occur in the tile; and

A prognostic prediction associated with one or more treatment regimens for the subject is generated based on the plurality of mutation predictions.

2. The method of claim 1, wherein generating the mutation prediction comprises:

One or more features are detected from each of the plurality of tiles, wherein the one or more features include one or more of clinical features or histological features, and wherein generating a label for each of the plurality of tiles is based on the one or more features.

3. The method of claim 2 or 3, wherein generating the mutation prediction is based on tumor morphology, wherein the tumor morphology is based on analysis of one or more of: the presence of signet ring cells, the number of signet ring cells, the presence of hepatoid cells, the number of hepatoid cells, extracellular mucin, or tumor growth pattern.

4. The method of any one of claims 1 to 3, wherein generating the mutation prediction is based on one or more machine learning models, wherein the method further comprises: training the one or more machine learning models based on a plurality of training data, the plurality of training data comprising one or more labeled depictions of tumor cells and one or more labeled depictions of other histological or clinical features.

5. The method of any one of claims 1 to 4, wherein generating the prognostic prediction is based on generating mutation predictions for tiles from one or more additional digital pathology images, each of the one or more additional digital pathology images depicting an additional specific sample from the biological samples of the subject, and wherein analyzing comprises:

generating a mutation prediction for each of the tiles from the one or more additional digital pathology images; and

A combined prognostic prediction is generated for the subject based on all of the mutation predictions.

6. The method according to any one of claims 1 to 5, further comprising:

The prognostic prediction is outputted via a graphical user interface, wherein the graphical user interface includes a graphical representation of the digital pathology image, and wherein the graphical representation includes an indication of the mutation prediction generated for each of the plurality of tiles and a predicted confidence level associated with the prognostic prediction.

7. The method according to any one of claims 1 to 6, further comprising:

A recommendation associated with use of the one or more treatment regimens is generated.

8. The method according to any one of claims 1 to 7, wherein specific sections of the biological sample are stained with one or more stains.

9. The method of any one of claims 1 to 8, wherein generating the prognostic prediction is further based on a weighted combination of the mutation predictions generated for the tile.

10. The method of any one of claims 1 to 9, wherein generating mutation predictions for tiles depicting tumor cells comprises:

The tumor cells are classified into phenotypes, each of which corresponds to a different mutation class.

11. The method of claim 10, wherein classifying the tumor cells into phenotypes comprises:

identifying nuclear heterogeneity in the image patch; and

The identified nuclear heterogeneity is quantified, wherein generating the mutation prediction is further based on the quantified nuclear heterogeneity.

12. The method of claim 10 or 11, wherein generating the mutation prediction comprises:

Cell-level spatial analysis is performed to assess spatial distribution, wherein the spatial distribution indicates regions of clonal cells and the spatial arrangement of cells within each of the regions of clonal cells.

13. The method of claim 12, wherein assessing the spatial distribution comprises:

measuring spectral distances within subgraphs of minimum spanning trees of the tumor cells, wherein each of the subgraphs represents a tumor nest;

Adjacency spectral distances are calculated pairwise across all of the subgraphs.

The method of claim 13 , wherein each of the subgraphs is defined by performing outlier detection.

15. The method of claim 13 or 14, wherein each of the sub-graphs is defined based on segmentation of detected tumor nests.

16. The method of any one of claims 10 to 15, wherein generating the mutation prediction comprises:

Regions of closely adjacent clonal cells in the tiles were identified by performing cell-level spatial analysis to assess spatial entropy.

17. The method of claim 16, wherein assessing spatial entropy comprises:

specifying a set of distinct distance bins, wherein each of the distance bins corresponds to a range of distances between pairs of tumor cells;

for each of the distance bins, identifying pairs of the tumor cells, wherein a distance between the tumor cells in each of the pairs falls within the distance range corresponding to the distance bin;

For each of the distance bins, the frequency with which pairs of tumor cells were identified as morphologically similar was calculated;

for each possible pair of tumor cells classified into a phenotype, classifying the paired tumor cells into one of a predefined number of bins, each of the bins representing a distance between the spatial locations of each of the cells in the pair;

For each of the bins, the frequency of classification of the pairs of tumor cells in the bin is calculated.

18. The method of claim 17, wherein the weight applied to each distance bin corresponds to the number of pairs of tumor cells in the distance bin.

19. A method according to claim 17 or 18, wherein the set of distance bins is restricted to only those bins representing distance ranges with a maximum distance below a specified threshold.

20. The method of any one of claims 1 to 19, wherein said generating said prognostic prediction comprises:

The mutational background of the digital pathology image is determined to be an unknown driver or tumor suppressor, wherein the heterogeneity level of the clustered tumor cells is high, and wherein the prognostic prediction is associated with a treatment regimen including immunotherapy.

21. The method of any one of claims 1 to 20, wherein said generating said prognostic prediction comprises:

The mutational context of the digital pathology image is determined to be oncogenic driver mutations, wherein the level of heterogeneity of the clustered tumor cells is intermediate, and wherein the prognostic prediction is associated with a treatment regimen that includes a targeted therapy corresponding to the mutation.

22. The method according to any one of claims 1 to 21, wherein:

If an actionable mutation is present in at least one of the tiles, the one or more treatment regimens include a targeted therapy associated with the actionable mutation;

Otherwise the one or more treatment regimens comprise immunotherapy.

23. One or more computer-readable non-transitory storage media embodying software that, when executed, is operable to:

access to digital pathology images depicting tumor cells sampled from the subject;

selecting a plurality of tiles from the digital pathology image, wherein each of the tiles depicts a tumor cell;

generating a mutation prediction for each of the tiles, wherein the mutation prediction represents a prediction of the likelihood that an actionable mutation will occur in the tile; and

A prognostic prediction associated with one or more treatment regimens for the subject is generated based on the plurality of mutation predictions.

24. The computer-readable non-transitory storage medium of claim 23, wherein generating the mutation prediction comprises:

One or more features are detected from each of the plurality of tiles, wherein the one or more features include one or more of clinical features or histological features, and wherein generating a label for each of the plurality of tiles is based on the one or more features.

25. The computer-readable non-transitory storage medium of claim 23 or 24, wherein generating the mutation prediction is based on tumor morphology, wherein the tumor morphology is based on analysis of one or more of: the presence of signet ring cells, the number of signet ring cells, the presence of hepatoid cells, the number of hepatoid cells, extracellular mucin, or tumor growth pattern.

26. A computer-readable, non-transitory storage medium according to any one of claims 23 to 25, wherein generating the mutation prediction is based on one or more machine learning models, wherein the computer-readable, non-transitory storage medium further comprises: training the one or more machine learning models based on a plurality of training data, the plurality of training data comprising one or more labeled depictions of tumor cells and one or more labeled depictions of other histological or clinical features.

27. The computer-readable non-transitory storage medium of any one of claims 23 to 26, wherein generating the prognostic prediction is based on generating mutation predictions for tiles from one or more additional digital pathology images, each of the one or more additional digital pathology images depicting an additional specific sample from the biological samples of the subject, and wherein analyzing comprises:

generating a mutation prediction for each of the tiles from the one or more additional digital pathology images; and

A combined prognostic prediction is generated for the subject based on all of the mutation predictions.

28. The computer-readable non-transitory storage medium according to any one of claims 23 to 27, further comprising:

The prognostic prediction is outputted via a graphical user interface, wherein the graphical user interface includes a graphical representation of the digital pathology image, and wherein the graphical representation includes an indication of the mutation prediction generated for each of the plurality of tiles and a predicted confidence level associated with the prognostic prediction.

29. The computer-readable non-transitory storage medium according to any one of claims 23 to 28, further comprising:

A recommendation associated with use of the one or more treatment regimens is generated.

30. The computer readable non-transitory storage medium of any one of claims 23 to 29, wherein a specific section of the biological sample is stained with one or more stains.

31. The computer-readable non-transitory storage medium of any one of claims 23 to 30, wherein generating the prognostic prediction is further based on a weighted combination of the mutation predictions generated for the tile.

32. The computer-readable non-transitory storage medium of any one of claims 23 to 31, wherein generating a mutation prediction for a tile depicting a tumor cell comprises:

The tumor cells are classified into phenotypes, each of which corresponds to a different mutation class.

33. The computer readable non-transitory storage medium of claim 32, wherein classifying the tumor cells into phenotypes comprises:

identifying nuclear heterogeneity in the image patch; and

The identified nuclear heterogeneity is quantified, wherein generating the mutation prediction is further based on the quantified nuclear heterogeneity.

34. The computer-readable non-transitory storage medium of claim 32 or 33, wherein generating the mutation prediction comprises:

Cell-level spatial analysis is performed to assess spatial distribution, wherein the spatial distribution indicates regions of clonal cells and the spatial arrangement of cells within each of the regions of clonal cells.

35. The computer-readable non-transitory storage medium of claim 34, wherein assessing the spatial distribution comprises:

measuring spectral distances within subgraphs of minimum spanning trees of the tumor cells, wherein each of the subgraphs represents a tumor nest;

Adjacency spectral distances are calculated pairwise across all of the subgraphs.

36. The computer-readable non-transitory storage medium of claim 35, wherein each of the subgraphs is defined by performing outlier detection.

37. The computer-readable non-transitory storage medium of claim 35 or 36, wherein each of the subgraphs is defined based on segmentation of detected tumor nests.

38. The computer-readable non-transitory storage medium of any one of claims 32 to 37, wherein generating the mutation prediction comprises:

Regions of closely adjacent clonal cells in the tiles were identified by performing cell-level spatial analysis to assess spatial entropy.

39. The computer-readable non-transitory storage medium of claim 38, wherein assessing spatial entropy comprises:

specifying a set of distinct distance bins, wherein each of the distance bins corresponds to a range of distances between pairs of tumor cells;

for each of the distance bins, identifying pairs of the tumor cells, wherein a distance between the tumor cells in each of the pairs falls within the distance range corresponding to the distance bin;

For each of the distance bins, the frequency with which pairs of tumor cells were identified as morphologically similar was calculated;

for each possible pair of tumor cells classified into a phenotype, classifying the paired tumor cells into one of a predefined number of bins, each of the bins representing a distance between the spatial locations of each of the cells in the pair;

For each of the bins, the frequency of classification of the pairs of tumor cells in the bin is calculated.

40. The computer-readable non-transitory storage medium of claim 39, wherein a weight applied to each distance bin corresponds to a number of pairs of tumor cells in the distance bin.

41. The computer-readable non-transitory storage medium of claim 39 or 40, wherein the set of distance bins is limited to only those bins representing distance ranges with a maximum distance below a specified threshold.

42. The computer-readable non-transitory storage medium of any one of claims 23 to 41, wherein said generating said prognostic prediction comprises:

The mutational background of the digital pathology image is determined to be an unknown driver or tumor suppressor, wherein the heterogeneity level of the clustered tumor cells is high, and wherein the prognostic prediction is associated with a treatment regimen including immunotherapy.

43. The computer-readable non-transitory storage medium of any one of claims 23 to 42, wherein said generating said prognostic prediction comprises:

The mutational context of the digital pathology image is determined to be oncogenic driver mutations, wherein the level of heterogeneity of the clustered tumor cells is intermediate, and wherein the prognostic prediction is associated with a treatment regimen that includes a targeted therapy corresponding to the mutation.

44. The computer-readable non-transitory storage medium of any one of claims 23 to 43, wherein:

If an actionable mutation is present in at least one of the tiles, the one or more treatment regimens include a targeted therapy associated with the actionable mutation;

Otherwise the one or more treatment regimens comprise immunotherapy.

45. A system comprising: one or more processors; and a non-transitory memory coupled to the processors, the non-transitory memory comprising instructions executable by the processors, the processors when executing the instructions being operable to:

access to digital pathology images depicting tumor cells sampled from the subject;

selecting a plurality of tiles from the digital pathology image, wherein each of the tiles depicts a tumor cell;

generating a mutation prediction for each of the tiles, wherein the mutation prediction represents a prediction of the likelihood that an actionable mutation will occur in the tile; and

A prognostic prediction associated with one or more treatment regimens for the subject is generated based on the plurality of mutation predictions.

46. The system of claim 45, wherein generating the mutation prediction comprises:

One or more features are detected from each of the plurality of tiles, wherein the one or more features include one or more of clinical features or histological features, and wherein generating a label for each of the plurality of tiles is based on the one or more features.

47. The system of claim 45 or 46, wherein generating the mutation prediction is based on tumor morphology, wherein the tumor morphology is based on analysis of one or more of: the presence of signet ring cells, the number of signet ring cells, the presence of hepatoid cells, the number of hepatoid cells, extracellular mucin, or tumor growth pattern.

48. The system of any one of claims 45 to 47, wherein generating the mutation prediction is based on one or more machine learning models, wherein the system further comprises: training the one or more machine learning models based on a plurality of training data, the plurality of training data comprising one or more labeled depictions of tumor cells and one or more labeled depictions of other histological or clinical features.

49. The system of any one of claims 45 to 48, wherein generating the prognostic prediction is based on generating mutation predictions for tiles from one or more additional digital pathology images, each of the one or more additional digital pathology images depicting an additional specific sample from the biological samples of the subject, and wherein analyzing comprises:

generating a mutation prediction for each of the tiles from the one or more additional digital pathology images; and

A combined prognostic prediction is generated for the subject based on all of the mutation predictions.

50. The system of any one of claims 45 to 49, further comprising:

The prognostic prediction is outputted via a graphical user interface, wherein the graphical user interface includes a graphical representation of the digital pathology image, and wherein the graphical representation includes an indication of the mutation prediction generated for each of the plurality of tiles and a predicted confidence level associated with the prognostic prediction.

51. The system of any one of claims 45 to 50, further comprising:

A recommendation associated with use of the one or more treatment regimens is generated.

52. The system of any one of claims 45 to 51, wherein specific sections of the biological sample are stained with one or more stains.

53. The system of any one of claims 45 to 52, wherein generating the prognostic prediction is further based on a weighted combination of the mutation predictions generated for the tile.

54. The system of any one of claims 45 to 53, wherein generating a mutation prediction for a tile depicting a tumor cell comprises:

The tumor cells are classified into phenotypes, each of which corresponds to a different mutation class.

55. The system of claim 54, wherein classifying the tumor cells into phenotypes comprises:

identifying nuclear heterogeneity in the image patch; and

The identified nuclear heterogeneity is quantified, wherein generating the mutation prediction is further based on the quantified nuclear heterogeneity.

56. The system of claim 54 or 55, wherein generating the mutation prediction comprises:

Cell-level spatial analysis is performed to assess spatial distribution, wherein the spatial distribution indicates regions of clonal cells and the spatial arrangement of cells within each of the regions of clonal cells.

57. The system of claim 56, wherein assessing the spatial distribution comprises:

measuring spectral distances within subgraphs of minimum spanning trees of the tumor cells, wherein each of the subgraphs represents a tumor nest;

Adjacency spectral distances are calculated pairwise across all of the subgraphs.

58. The system of claim 57, wherein each of the subgraphs is defined by performing outlier detection.

59. The system of claim 57 or 58, wherein each of the sub-graphs is defined based on segmentation of detected tumor nests.

60. The system of any one of claims 54 to 59, wherein generating the mutation prediction comprises:

Regions of closely adjacent clonal cells in the tiles were identified by performing cell-level spatial analysis to assess spatial entropy.

61. The system of claim 60, wherein assessing spatial entropy comprises:

specifying a set of distinct distance bins, wherein each of the distance bins corresponds to a range of distances between pairs of tumor cells;

for each of the distance bins, identifying pairs of the tumor cells, wherein a distance between the tumor cells in each of the pairs falls within the distance range corresponding to the distance bin;

For each of the distance bins, the frequency with which pairs of tumor cells were identified as morphologically similar was calculated;

for each possible pair of tumor cells classified into a phenotype, classifying the paired tumor cells into one of a predefined number of bins, each of the bins representing a distance between the spatial locations of each of the cells in the pair;

For each of the bins, the frequency of classification of the pairs of tumor cells in the bin is calculated.

62. The system of claim 61, wherein a weight applied to each distance bin corresponds to the number of pairs of tumor cells in the distance bin.

63. A system according to claim 61 or 62, wherein the set of distance bins is limited to only those bins representing distance ranges with a maximum distance below a specified threshold.

64. The system of any one of claims 45 to 63, wherein said generating said prognostic prediction comprises:

The mutational background of the digital pathology image is determined to be an unknown driver or tumor suppressor, wherein the heterogeneity level of the clustered tumor cells is high, and wherein the prognostic prediction is associated with a treatment regimen including immunotherapy.

65. The system of any one of claims 45 to 64, wherein said generating said prognostic prediction comprises:

The mutational context of the digital pathology image is determined to be oncogenic driver mutations, wherein the level of heterogeneity of the clustered tumor cells is intermediate, and wherein the prognostic prediction is associated with a treatment regimen that includes a targeted therapy corresponding to the mutation.

66. A system according to any one of claims 45 to 65, wherein:

If an actionable mutation is present in at least one of the tiles, the one or more treatment regimens include a targeted therapy associated with the actionable mutation;

Otherwise the one or more treatment regimens comprise immunotherapy.

Claims (24)

1. A method comprising: a digital pathology image processing system: access digital pathology images depicting tumor cells sampled from the subject; Selecting a plurality of tiles from the digital pathology image, wherein each of the tiles depicts tumor cells; Generating a mutation prediction for each of the tiles, wherein the mutation prediction represents a prediction of the likelihood of an actionable mutation occurring in the tile; and A prognostic prediction related to one or more treatment regimens for the subject is generated based on the plurality of mutation predictions. 2. The method of claim 1, wherein generating the mutation prediction includes: One or more features are detected from each of the plurality of tiles, wherein the one or more features include one or more of clinical features or histological features, and wherein for the plurality of tiles Each of the blocks generates a mark based on the one or more characteristics. 3. The method of claim 1, wherein generating the mutation prediction is based on tumor morphology, wherein the tumor morphology is based on analysis of one or more of: the presence of signet ring cells, signature The number of ring cells, the presence of hepatoid cells, the number of hepatoid cells, extracellular mucin, or tumor growth pattern. 4. The method of claim 1, wherein generating the mutation prediction is based on one or more machine learning models, wherein the method further comprises: training the one or more machine learning models based on a plurality of training data The plurality of training data includes one or more labeled depictions of tumor cells and one or more labeled depictions of other histological or clinical features. 5. The method of claim 1, wherein generating the prognostic prediction is based on generating mutation predictions for tiles from one or more additional digital pathology images, each of the one or more additional digital pathology images One depicts an additional specific sample from the biological sample from the subject, and wherein the analysis includes: generating mutation predictions for each of the tiles from the one or more additional digital pathology images; and A combined prognostic prediction is generated for the subject based on all of the mutation predictions. 6. The method of claim 1, further comprising: The prognostic prediction is output via a graphical user interface, wherein the graphical user interface includes a graphical representation of the digital pathology image, and wherein the graphical representation includes: An indication of the mutation prediction is generated, and a predicted confidence level associated with the prognostic prediction. 7. The method of claim 1, further comprising: Recommendations associated with use of the one or more treatment regimens are generated. 8. The method of claim 1, wherein specific sections of the biological sample are stained with one or more stains. 9. The method of claim 1, wherein generating the prognostic prediction is further based on a weighted combination of the mutation predictions generated for the tile. 10. The method of claim 1, wherein generating mutation predictions for tiles depicting tumor cells comprises: The tumor cells are classified into phenotypes, each of which corresponds to a different mutation class. 11. The method of claim 10, wherein classifying the tumor cells into phenotypes includes: identifying nuclear heterogeneity in the tile; and The identified nuclear heterogeneity is quantified, wherein generating the mutation prediction is further based on the quantified nuclear heterogeneity. 12. The method of claim 10, wherein generating the mutation prediction comprises: Cell-level spatial analysis was performed to assess the spatial distribution indicative of the area of clonal cells and the spatial arrangement of cells within each of the areas of clonal cells. 13. The method of claim 12, wherein assessing spatial distribution includes: measuring spectral distances within subgraphs of a minimum spanning tree of said tumor cells, wherein each of said subgraphs represents a tumor nest; Adjacency spectral distances are calculated pairwise across all subgraphs. 14. The method of claim 13, wherein each of the subgraphs is defined by performing outlier detection. 15. The method of claim 13, wherein each of the subgraphs is defined based on segmentation of detected tumor nests. 16. The method of claim 10, wherein generating the mutation prediction comprises: Regions of closely adjacent clonal cells in the tile were identified by performing cell-level spatial analysis to assess spatial entropy. 17. The method of claim 16, wherein assessing spatial entropy includes: specifying a set of distinct distance bins, wherein each of said distance bins corresponds to a range of distances between pairs of tumor cells; For each of the distance bins, pairs of the tumor cells are identified, wherein the distance between the tumor cells in each of the pairs falls within the range corresponding to the distance bin. within distance range; For each of the distance bins, calculate the frequency with which pairs of tumor cells are identified as morphologically similar; For each possible pair of tumor cells classified into a phenotype, the pair of tumor cells is classified into one of a predefined number of bins, each of the bins representing a the distance between the spatial locations of each of the cells; For each of the bins, the frequency of classification of the pair of tumor cells in the bin is calculated. 18. The method of claim 17, wherein the weight applied to each distance bin may correspond to the number of paired tumor cells in the distance bin. 19. The method of claim 17, wherein the set of distance bins may be limited to those bins that represent a distance range with a maximum distance below a specified threshold. 20. The method of claim 1, wherein generating the prognostic prediction includes: The mutational context of the digital pathology image is determined to be an unknown driver or tumor suppressor, where the heterogeneity level of clustered tumor cells is high, and where the prognostic prediction is relevant to a treatment regimen including immunotherapy. 21. The method of claim 1, wherein generating the prognostic prediction includes: Determining that the mutational background of the digital pathology image is an oncogene driver mutation, wherein the heterogeneity level of the clustered tumor cells is moderate, and wherein the prognostic prediction is relevant to a treatment regimen including a targeted therapy corresponding to the mutation . 22. The method of claim 1, wherein: If an actionable mutation is present in at least one of the tiles, the one or more treatment options include a targeted therapy associated with the actionable mutation; Otherwise the one or more treatment regimens include immunotherapy. 23. One or more computer-readable non-transitory storage media embodying software that when executed is operable to: access digital pathology images depicting tumor cells sampled from the subject; Selecting a plurality of tiles from the digital pathology image, wherein each of the tiles depicts tumor cells; Generating a mutation prediction for each of the tiles, wherein the mutation prediction represents a prediction of the likelihood of an actionable mutation occurring in the tile; and A prognostic prediction related to one or more treatment regimens for the subject is generated based on the plurality of mutation predictions. 24. A system, comprising: one or more processors; and non-transitory memory coupled to the processors, the non-transitory memory including instructions executable by the processors, the processing The processor when executing the instructions is operable to: access digital pathology images depicting tumor cells sampled from the subject; Selecting a plurality of tiles from the digital pathology image, wherein each of the tiles depicts tumor cells; Generating a mutation prediction for each of the tiles, wherein the mutation prediction represents a prediction of the likelihood of an actionable mutation occurring in the tile; and A prognostic prediction related to one or more treatment regimens for the subject is generated based on the plurality of mutation predictions.