CN114981898A

CN114981898A - Systems and methods for delivering digital biomarkers and genomic packages

Info

Publication number: CN114981898A
Application number: CN202180011451.3A
Authority: CN
Inventors: J·苏; J·洛克; P·舒弗勒; C·卡南; T·富克斯; L·格雷迪
Original assignee: Paige Artificial Intelligence Co
Current assignee: Paige Artificial Intelligence Co
Priority date: 2020-01-28
Filing date: 2021-01-27
Publication date: 2022-08-30
Also published as: JP7228210B2; AU2023254922A1; BR112022014531A2; AU2021213741A1; WO2021154869A1; WO2021154869A8; US11475990B2; JP2023502158A; KR20220111725A; EP4097737A1; US20210233642A1; KR102553095B1; US20230005597A1; JP2023041889A; CA3161263A1

Abstract

Disclosed are methods and systems for receiving one or more digital images associated with a tissue specimen, a related case, a patient, and/or a plurality of clinical information, determining one or more of a prediction, a recommendation, and/or a plurality of data for the one or more digital images using a machine learning system that has been trained using a plurality of training images to predict biomarkers and a plurality of genomic suite elements, and determining whether to record an output and at least one visualization region as part of a history of medical records within a clinical reporting system based on the prediction, the recommendation, and/or the plurality of data.

Description

Systems and methods for delivering digital biomarkers and genomic packages

RELATED APPLICATIONS

This application claims priority from U.S. provisional application No. 62/966, 659, filed on 28/1/2020, the entire disclosure of which is hereby incorporated by reference in its entirety. .

Technical Field

Various embodiments of the present invention generally relate to the development of Artificial Intelligence (AI) techniques to detect biomarkers, genomic features, treatment resistance, and other relevant features necessary for additional testing of pathology specimens (specimens). More specifically, certain embodiments of the present disclosure relate to systems and methods for predicting, identifying, or detecting biomarkers and genomic features of prepared tissue specimens. The present disclosure further provides systems and methods for creating a predictive model of the slide predictive signature from an unseen slide.

Background

A pathologist may require multiple steps, the cost and time of production to receive the results of a biomarker or genome Panel (Genomics Panel). For biomarker results, (a) the pathologist may notice the appropriate or suspicious portion of the patient (b) the lab may receive a request for slide staining; (c) laboratory cutting of the block or finding the appropriate unstained slide; (d) the part is stained. And (e) electronically recording the test into the case and giving it to the pathologist for final review. For genome suites, (a) a request for molecular testing can be given to a pathologist; (b) the pathologist can select the slides from which to sequence; (c) prompting that tissue re-cutting is to be performed; (d) prompt to scrape off the tumor from the previous biopsy incision based on the pathologist's profile; (e) the genome in the scraped tumor tissue can be sequenced; and (f) a gene report can be created. These processes can be expensive and time consuming.

Both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. The background description provided herein is for the purpose of generally presenting the context of the disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art or prior art suggestions by inclusion in this section.

Disclosure of Invention

In accordance with certain aspects of the present disclosure, systems and methods for predicting a biomarker and/or at least one genomic feature in a digital image associated with a tissue specimen are disclosed.

A computer-implemented method for processing an electronic image corresponding to a specimen includes: receiving one or more digital images associated with a tissue specimen, a related case, a patient, and/or a plurality of clinical information; determining one or more of a prediction, a recommendation, and/or a plurality of data for the one or more digital images using a machine learning system that has been trained using a plurality of training images to predict biomarkers and a plurality of genomic suite elements; and determining whether to record an output and at least one visualization area as part of a case history within a clinical reporting system based on the prediction, the recommendation, and/or the plurality of data.

A system for processing an electronic image corresponding to a specimen, comprising: a memory storing instructions; and at least one processor executing the instructions to perform a process comprising receiving one or more digital images associated with a tissue specimen, a related case, a patient, and/or a plurality of clinical information; determining one or more of a prediction, a recommendation, and/or a plurality of data for the one or more digital images using a machine learning system that has been trained using a plurality of training images to predict biomarkers and a plurality of genomic suite elements; and determining whether to record an output and at least one visualization area as part of a case history within a clinical reporting system based on the prediction, the recommendation, and/or the plurality of data.

A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to implement a method for processing an electronic image corresponding to a specimen, the method comprising: the method includes receiving one or more digital images associated with a tissue specimen, a related case, a patient, and/or a plurality of clinical information, determining one or more of a prediction, a recommendation, and/or a plurality of data for the one or more digital images using a machine learning system that has been trained using a plurality of training images to predict biomarkers and a plurality of genomic suite elements, and determining whether to record an output and at least one visualization region as part of a case history within a clinical reporting system based on the prediction, the recommendation, and/or the plurality of data.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments disclosed, as claimed.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and together with the description, serve to explain the principles of the disclosed embodiments:

fig. 1A illustrates an exemplary block diagram of a system and network for detecting biomarkers and/or at least one genomic feature according to an exemplary embodiment of the present disclosure;

fig. 1B illustrates an exemplary block diagram of a biomarker detection platform for predicting biomarker and genomic stack features using machine learning, in accordance with an embodiment of the present disclosure;

fig. 1C illustrates an exemplary block diagram of a slide analysis tool according to an exemplary embodiment of the present disclosure;

fig. 2A is a flow diagram illustrating an example method for detecting biomarkers and/or at least one genomic feature using a machine learning system, according to one or more example embodiments of the present disclosure;

fig. 2B is a flow diagram illustrating an example method for training a machine learning system to detect biomarkers and/or at least one genomic feature in accordance with one or more example embodiments of the present disclosure;

fig. 3 is a flow diagram illustrating an exemplary method for visualizing a positive biomarker lesion in accordance with one or more exemplary embodiments of the present disclosure;

fig. 4 is a flowchart illustrating an exemplary method for visualizing a tumor region to guide a molecular pathologist, according to one or more exemplary embodiments of the present disclosure.

Fig. 5 is a flow diagram illustrating an example method for reporting a predicted development of anti-tumor resistance in accordance with one or more example embodiments of the present disclosure;

fig. 6 depicts exemplary options for a user to view visualizations and/or reports in accordance with one or more exemplary embodiments of the present disclosure;

fig. 7 depicts an exemplary system that can perform the techniques presented herein.

Detailed Description

Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

The systems, devices and methods disclosed herein are described in detail by way of example and with reference to the accompanying drawings. The examples discussed herein are merely examples and are provided to help explain the apparatuses, devices, systems, and methods described herein. Unless specifically designated as mandatory, none of the features or components shown in the drawings or discussed below should be considered mandatory for any particular implementation of any of these devices, systems or methods.

Moreover, for any method described, whether or not the method is described in connection with a flowchart, it should be understood that any explicit or implicit ordering of steps performed in the performance of the method does not imply that the steps must be performed in the order presented, unless otherwise specified or required by the context, but may instead be performed in a different order or in parallel.

As used herein, the term "exemplary" is used in the sense of "exemplary" rather than "ideal," and further, the terms "a" and "an" herein do not denote a limitation of quantity, but rather denote the presence of one or more of the referenced item.

Pathology refers to the study of disease and the cause and effect of the disease. More specifically, pathology refers to performing tests and analysis for diagnosing disease. For example, a tissue sample may be placed on a slide for viewing under a microscope by a pathologist (e.g., a physician who is an expert in analyzing the tissue sample to determine if any abnormalities are present). That is, a pathology specimen may be cut into multiple sections, stained, and prepared into slides for examination and diagnosis by a pathologist. When the diagnostic results on the slide are uncertain, the pathologist can schedule additional levels of cutting, staining or other tests to gather more information from the tissue. The technician(s) may then create a new slide(s) that may contain additional information that the pathologist uses in making a diagnosis. The process of creating additional slides can be time consuming not only because it can involve retrieving a tissue block, cutting it to make a new slide, and then staining the slide, but also because it can be batch processed for multiple orders. This can significantly delay the final diagnosis given by the pathologist. Furthermore, even after a delay, there may still be no guarantee that the new slide(s) will have sufficient information to give a diagnosis.

The pathologist can evaluate cancer and other disease pathology slides separately. The present disclosure presents the use of AI to detect and predict biomarker and genomic set characteristics. In particular, the present disclosure describes various exemplary user interfaces available in a workflow, as well as AI tools that can be integrated into the workflow to accelerate and improve pathologist work.

For example, a computer may be used to analyze images of tissue samples to quickly identify whether additional information is needed about a particular tissue sample, and/or to highlight to a pathologist the areas he or she should more carefully view. Thus, the process of obtaining additional stained slides and testing can be done automatically before being reviewed by a pathologist. This can provide a fully automatic slide preparation line when paired with an automatic slide segmentation and staining machine. This automation has at least the following benefits: (1) minimizing the amount of time wasted by a pathologist in determining that a slide is insufficient to make a diagnosis, (2) minimizing the (average total) time from specimen acquisition to diagnosis by avoiding additional time between scheduling additional tests and generating additional tests, (3) reducing the amount of time per re-cut and the amount of material wasted by allowing re-cuts while a piece of tissue (e.g., a pathological specimen) is in the cutting station, (4) reducing the amount of tissue material wasted/discarded during slide preparation, (5) reducing the cost of slide preparation by partially or fully automating the procedure, (6) allowing automatic custom cutting and staining of slides that will result in a more representative/informative slide from the sample, (7) allowing more slides to be generated per piece of tissue, facilitating a more informed diagnosis by reducing the overhead of requesting additional tests by the pathologist, and/or (8) identify or verify the correct attributes of the digital pathology image (e.g., with respect to specimen type), and/or the like.

The process of using a computer to assist a pathologist is called computational pathology. Computational methods for calculating pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. The AI may include, but is not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. By using computational pathology, life can be saved by helping pathologists improve the accuracy, reliability, efficiency and accessibility of diagnosis. For example, computational pathology can be used to help detect slides suspected of cancer, allowing pathologists to examine and confirm their initial assessment before giving a final diagnosis.

As described above, the computational pathology processes and apparatus of the present disclosure may provide an integrated platform that allows for fully automated processes including data acquisition, processing, and viewing of digital pathology images via a web browser or other user interface, while integrating with a Laboratory Information System (LIS). Further, cloud-based data analysis of patient data may be used to aggregate clinical information. The data may come from hospitals, clinics, field researchers, etc. And may be analyzed by machine learning, computer vision, natural language processing, and/or statistical algorithms to perform real-time monitoring and forecasting of health patterns at multiple geo-specific levels.

The present disclosure relates to systems and methods for quickly and correctly identifying and/or verifying specimen type of a digital pathology image or any information related to a digital pathology image without having to access a LIS or similar information database. One embodiment of the present disclosure can include a system trained to identify various characteristics of digital pathology images based on a dataset of prior digital pathology images. The trained system may provide classification of the specimen shown in the digital pathology image. Classification can be helpful in providing treatment or diagnostic prediction(s) for a patient associated with a specimen.

The systems and methods of the present disclosure can use artificial intelligence to detect scan slides with any characteristics that can be a prerequisite for further testing (e.g., the most highly molecular or aggressive tumor volumes of human epidermal growth factor receptor 2/estrogen receptor/progesterone receptor (HER 2/ER/PR)). This feature detection can be done at the case, part or block level of the specimen. The results may be available via any user interface (e.g., by viewer, report, by Laboratory Information System (LIS), etc.). The systems and methods of the present disclosure may also provide immediate visualization of predicted Immunohistochemical (IHC) results, genomic suites, AI-derived information (e.g., treatment resistance), and the like from one or more digital pathology specimen images acquired from a patient. This can provide turn around time and cost benefits for both the hospital and the patient. In addition to showing the results of a digital IHC or digital genome suite, the present system may further manage the reimbursement elements of the purchase. This may provide additional efficiencies for hospitals and patients.

The present disclosure includes one or more embodiments of a slide analysis tool. The input to the tool may include the digital pathology image and any associated additional input. The output of the tool may include global and/or local information about the specimen. The specimen may comprise a biopsy or surgical resection specimen.

An exemplary global output of the disclosed tool(s) may contain information about the entire image, e.g., specimen type, overall quality of the cut of the specimen, overall quality of the glass pathology slide itself, and/or histomorphological characteristics. Exemplary local outputs may indicate information in particular regions of the image, e.g., particular image regions may be classified as having blurriness or cracks in the slide. The present disclosure includes embodiments for developing and using the disclosed slide analysis tool(s), as described in further detail below.

Fig. 1A illustrates a block diagram of a system and network for determining specimen characteristics or image characteristic information related to digital pathology images using machine learning, according to an exemplary embodiment of the present disclosure.

In particular, fig. 1A illustrates an electronic network 120 that may be connected to a server of a hospital, laboratory, and/or doctor's office, etc. For example, a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and/or a laboratory information system 125, etc., may each be connected to the electronic network 120, such as the internet, through one or more computers, servers, and/or handheld mobile devices. According to an exemplary embodiment of the present disclosure, the electronic network 120 may also be connected to a server system 110, the server system 110 may include a processing device configured to implement the biomarker detection platform 100, the disease detection platform 100 includes a slide analysis tool for determining specimen characteristics or image characteristic information related to digital pathology images, and creating a genome suite using machine learning, according to an exemplary embodiment of the present disclosure.

The physician server 121, the hospital server 122, the clinical trials server 123, the research laboratory server 124, and/or the laboratory information system 125 may create or otherwise obtain images of cytological specimen(s), histopathological specimen(s), slide(s) of cytological specimen(s), digitized images of slide(s) of histopathological specimen(s), or any combination thereof, for one or more patients. The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and/or laboratory information system 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy or cytology information, and the like. The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and/or laboratory information system 125 may transmit digitized slide images and/or patient-specific information to the server system 110 over the electronic network 120. Server system 110 may include one or more storage devices 109 for storing images and data received from at least one of a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and/or a laboratory information system 125. Server system 110 may also include a processing device for processing images and data stored in one or more storage devices 109. The server system 110 may further include one or more machine learning tools or capabilities. For example, according to one embodiment, the processing device may include a machine learning tool for the biomarker detection platform 100. Alternatively or additionally, the present disclosure (or portions of the systems and methods of the present disclosure) may be executed on a local processing device (e.g., a laptop).

The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and/or laboratory information system 125 refer to systems used by pathologists to examine images of slides. In a hospital environment, the organization type information may be stored in the laboratory information system 125. However, correct tissue classification information is not always paired with image content. Furthermore, even if the LIS is used to access specimen types of digital pathology images, the label may be incorrect due to the fact that many components of the LIS may be manually entered, leaving a large margin of error. According to an example embodiment of the present disclosure, the specimen type may be identified without requiring access to the library information system 125, or may be identified to potentially calibrate the library information system 125. For example, a third party may anonymously access image content in the LIS that does not have a corresponding specimen type tag stored therein. Furthermore, access to LIS content may be limited due to its sensitive content.

Fig. 1B illustrates an exemplary block diagram of a biomarker detection platform for predicting biomarker and genomic stack features using machine learning, in accordance with an embodiment of the present disclosure.

In particular, fig. 1B depicts components of the biomarker detection platform 100 according to one embodiment. For example, the biomarker detection platform 100 may include a slide analysis tool 101, a data acquisition tool 102, a slide uptake tool 103, a slide scanner 104, a slide manager 105, a storage 106, and a viewing application tool 108.

As described below, the slide analysis tool 101 refers to a process and system for processing digital images associated with tissue specimens and analyzing slides using machine learning, according to an exemplary embodiment.

Data acquisition tool 102 refers to a process and system for facilitating the transfer of digital pathology images to various tools, modules, components, and devices for classifying and processing digital pathology images, according to an exemplary embodiment.

According to an exemplary embodiment, slide uptake tool 103 refers to a process and system for scanning pathology images and converting them into digital form. The slide may be scanned with the slide scanner 104 and the slide manager 105 may process the image on the slide into a digitized pathology image and store the digitized image in the memory 106.

According to an exemplary embodiment, the viewing application tool 108 refers to a process and system for providing specimen characteristic or image characteristic information related to digital pathology image(s) to a user (e.g., a pathologist). Information may be provided through various output interfaces (e.g., a screen, a monitor, a storage device, and/or a web browser, etc.).

The slide analysis tool 101 and each of its components can transmit and/or receive digitized slide images and/or patient information to the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125 over the electronic network 120. Further, the server system 110 can include one or more storage devices 109 for storing images and data received from at least one of the slide analysis tool 101, the data acquisition tool 102, the slide ingestion tool 103, the slide scanner 104, the slide manager 105, and the viewing application tool 108. Server system 110 may also include a processing device for processing images and data stored in the storage device. The server system 110 may further include one or more machine learning tools or capabilities, for example, due to the processing device. Alternatively or additionally, the present disclosure (or portions of the systems and methods of the present disclosure) may be executed on a local processing device (e.g., a laptop).

Any of the above devices, tools, and modules may be located on a device that may be connected to electronic network 120 through one or more computers, servers, and/or handheld mobile devices, such as the internet or a cloud service provider.

Fig. 1C illustrates an example block diagram of the slide analysis tool 101 according to an example embodiment of the present disclosure. Slide analysis tool 101 may include a training image platform 131 and/or a target image platform 135.

According to one embodiment, the training image platform 131 may create or receive training images for training a machine learning system to efficiently analyze and classify digital pathology images. For example, the training images may be received from any one or any combination of server system 110, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and/or laboratory information system 125. The images used for training may be from real sources (e.g., humans, animals, etc.) or may be from synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains, such as (but not limited to) H & E, hematoxylin alone, IHC, molecular pathology, and the like; and/or (b) a digitized tissue sample from a 3D imaging device, such as a micct.

The training image ingest module 132 may create or receive a data set including one or more training images corresponding to either or both of an image of human tissue and a graphically rendered image. For example, the training images may be received from any one or any combination of server system 110, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and/or laboratory information system 125. The data set may be maintained on a digital storage device. The quality score determiner module 133 can identify Quality Control (QC) problems (e.g., defects) of the training images at a global or local level, which can greatly affect the usability of the digital pathology images. For example, the quality score determiner module may use information about the entire image, such as the specimen type, the overall quality of the specimen cut, the overall quality of the glass pathology slide itself, or histomorphological characteristics, and determine an overall quality score for the image. The treatment identification module 134 can analyze the images of the tissue and determine which digital pathology images have a treatment effect (e.g., post-treatment) and which images do not have a treatment effect (e.g., pre-treatment). It is useful to identify whether a digital pathology image has a therapeutic effect, since previous therapeutic effects in the tissue may affect the morphology of the tissue itself. Most LIS do not specifically retain this property, and therefore it may be desirable to classify specimen types with prior treatment effects.

According to one embodiment, the target image platform 135 may include a target image input module 136, a specimen detection module 137, and an output interface 138. The target image platform 135 may receive the target image and apply a machine learning model to the received target image to determine characteristics of the target specimen. For example, the target image may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and/or the laboratory information system 125. The target image capture module 136 may receive a target image corresponding to a target specimen. The specimen detection module 137 may apply a machine learning model to the target image to determine characteristics of the target specimen. For example, the specimen detection module 137 may detect a specimen type of the target specimen. The specimen detection module 137 may also apply a machine learning model to the target image to determine a quality score for the target image. In addition, the specimen detection module 137 may apply a machine learning model to the target specimen to determine whether the target specimen is pre-treatment or post-treatment.

The output interface 138 may be used to output information about the target image and the target specimen (e.g., to a screen, monitor, storage device, web browser, etc.).

Fig. 2A is a flow diagram illustrating an example method for predicting biomarkers and at least one genomic suite element using a machine learning system in accordance with one or more example embodiments of the present disclosure. For example, the example method 200 (i.e., step 202-212) may be performed by the slide analysis tool 101 automatically or in response to a request from a user.

According to one embodiment, the exemplary method 200 for predicting a biomarker and at least one genomic stack element may include one or more of the following steps. In step 202, the method may include receiving one or more digital images associated with a tissue specimen, a related case, a patient, and/or a plurality of clinical information. The tissue specimen may include a histological specimen, and the patient information may include a specimen type, a case and patient ID, a portion within a case, a rough description, and the like. The plurality of clinical information may include the assigned pathologist, whether the relevant specimen is available for testing, etc. The digital image may be received into a digital storage device (e.g., a hard drive, a network drive, cloud storage, Random Access Memory (RAM), etc.).

In step 204, the method may include determining a prediction and/or visualization of the one or more digital images using a machine learning system that has been trained using a plurality of training images to predict the biomarker and the at least one genomic suite element. The machine learning system may additionally output the recommendation and/or the data to the electronic storage device.

In step 206, the method may include generating a notification to the user indicating that a prediction and/or visualization is available. The notification may include a visual display, a pop-up window, or other suitable alert.

In step 208, the method may include generating options for a user to view the predictions and/or visualizations. As described below, this option may include an exemplary screen display as illustrated in fig. 6.

In step 210, the method may include generating at least one display of at least one recommended treatment path based on the prediction and/or visualization. The at least one recommended treatment path may include a validated treatment path, a new treatment path, a clinical treatment path, etc., or a subsequent step (e.g., a clinical trial, a professional physician visit, etc.) based on the generated prediction. Visualization of digital immunohistochemistry or genomic suite results may be accomplished using a variety of methods, including but not limited to:

a. overlaying at least one region of interest on top of the original image;

b. visualization side by side;

c. a report with a quantization metric; and

d. the digital test runs were summarized using the results.

The visualization of recommendations may include an interactive web interface where users (e.g., pathologists, oncologists, patients, etc.) may learn more about a particular recommendation (e.g., open clinical trials, treatment-specialized hospitals/physicians, etc.) via direct links and sources (e.g., websites, literature, etc.) of the interface. Alternatively, the visualization may include a report where the user may view an aggregated, immutable report, which may include, but is not limited to, the following elements:

a. history of patients

b. Case summary

c. Summary of diagnosis

d. Digital and/or "manual" test results

e. Subsequent steps are suggested for the patient based on the digital test results.

The method may group similar patients (e.g., patients with similar morphological patterns, similar biomarker expression, similar genomic profiles, similar treatment pathways, or other similarities) together as a reference for a given case to support the decision process for a particular case. The visualization of similar patients may or may not be related to the recommended treatment path for the case. A user (e.g., pathologist, oncologist, patient, etc.) may learn more about a particular patient and its results (e.g., from clinical trials, medications, etc.). The results may be visualized through an interactive web interface (e.g., filtering, sharing, saving, etc.) or through reports, as disclosed above.

The results may take the form of a consolidated report that includes the report prediction and related information (e.g., PDF). An exemplary report may contain one or more of the following elements:

a. history of patients

b. Patient summary

c. Case summary

d. The digital test is completed

e. Digital test results

f. Synthetic summary of results and significance of results to patients

g. Visualization of statistics based on similar patient outcomes (e.g., information charts, interactive websites, etc.)

h. Summary of related and/or recent literature

i. Suggested follow-up steps (e.g., clinical trials, drugs, chemotherapy, etc.), and the like.

In step 212, the method may include determining whether to record the output and at least one visualization area as part of a case history within the clinical reporting system based on the prediction and/or visualization. The method may further include integrating the recommendation and visualization into a final diagnostic report of the specimen.

Fig. 2B is a flow diagram illustrating an example method for training a machine learning system to predict biomarkers and at least one genomic suite element in accordance with one or more example embodiments of the present disclosure. For example, the example method 220 (i.e., step 221-235) may be performed by the slide analysis tool 101 automatically or in response to a request from a user.

According to one embodiment, the exemplary method 220 is used to train a machine learning system to predict biomarkers and at least one genome stack element. In step 221, the method may include receiving one or more digital images associated with a tissue specimen, a related case, a patient, and/or a plurality of clinical information. The tissue specimen may include a histological specimen, and the patient information may include a specimen type, a case and patient ID, a portion within the case, a rough description, and the like. The plurality of clinical information may include the assigned pathologist, whether the relevant specimen is available for testing, etc. The digital image may be received into a digital storage device (e.g., a hard drive, a network drive, cloud storage, Random Access Memory (RAM), etc.).

In step 223, the method may include developing a system to store and archive a plurality of processed images associated with a plurality of patient data.

In step 225, the method may include storing the plurality of processed images in a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, RAM, and the like.

In step 227, the method may include generating at least one recommendation for a treatment path based on the plurality of processed images. The treatment pathway may include clinical trials, treatments, and the like. The recommendation may be patient specific and may be based on at least one relevant feature of the plurality of stored images and patient data (e.g., patient diagnosis, medical history, demographic data). The recommendations for treatment paths may include or be based on clinical practice guidelines, which may be customized based on patient demographics, pre-approved stage drugs or therapies, clinical practice, and the like.

In step 229, the method can include generating a prediction of the biomarker and the at least one genomic package element.

In step 231, the method may include generating a list of at least one recommended treatment path based on the prediction. The list of at least one recommended treatment path may include medication treatments, clinical trials, etc., as well as relevant information based on the predicted biomarkers and genomic package elements (e.g., success rate, location of treatment, etc.).

In step 233, the method can include converting the prediction and the at least one recommended treatment path into a form that can be visualized and interpreted by a user (e.g., pathologist, patient, oncologist, etc.). The method may additionally include outputting or displaying the at least one result in various effective formats (e.g., interactive, structured, templated, static, etc.) depending on the user and use case.

In step 235, the method may include outputting the one or more predicted values and the treatment path recommendation to a user interface. Depending on the user and use case, the output or display results may be in various effective formats (e.g., interactive, structured, templated, static, etc.).

Fig. 3 is a flow diagram illustrating an example method for visualizing positive biomarker lesions using and training a machine learning system in accordance with one or more example embodiments of the present disclosure. Visualization of biomarkers (e.g., IHC markers, genome stack) can help a pathologist understand how a computational assay is performed.

Exemplary methods

300 and 320 may be used to visually display detected positive biomarker lesions. The exemplary methods 300 and 320 (i.e., steps 301 and 321) may be performed by the slide analysis tool 101 automatically or in response to a request from a user.

According to one embodiment, an exemplary method 300 for training a machine learning system to visualize a positive biomarker lesion may include one or more of the following steps. In step 301, the method may include receiving one or more digital images associated with a tissue specimen and corresponding information. The one or more digital images may include a histology slide. The corresponding information may include relevant information (e.g., specimen type, available fraction, rough description, etc.), clinical information (e.g., diagnosis, biomarker information, etc.), and patient information (e.g., demographic data, gender, etc.).

In step 303, the method may include developing a system to store and archive a plurality of digital images and corresponding patient data. The corresponding patient data may include images from screening, follow-up, results, and the like.

In step 305, the method may include storing a plurality of digital images and corresponding patient data in a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, RAM, and the like.

In step 307, the method may include generating at least one recommendation for a treatment path based on the at least one relevant feature of the plurality of digital images. The treatment path may include clinical trials, treatments, etc. determined for a patient based on at least one relevant factor (e.g., patient diagnosis, medical history, demographic data, etc.).

In step 309, the method may include predicting at least one biomarker and genomic stack element.

In step 311, the method may include generating a list of at least one recommended treatment pathway based on the predicted biomarkers and the genomic suite elements. The recommended treatment path (e.g., drug, clinical trial, etc.) and any relevant information (e.g., success rate, treatment location, etc.) may be based on predicted biomarkers and genomic package elements.

In step 313, the method may include converting the one or more predicted values or recommendations into a form that may be visualized or interpreted by a user (e.g., pathologist, patient, oncologist, etc.).

In step 321, the method may include receiving one or more digital images associated with a tissue specimen, a plurality of related cases, and patient information from a clinical system. The pathology specimen (e.g., histological specimen), related case and patient information (e.g., specimen type, case and patient ID, part within case, rough description, etc.), and information from the clinical system (e.g., assigned pathologist, specimen available for testing, etc.) into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.).

In step 323, the method may include generating at least one of a prediction, a recommendation, and/or a plurality of data for one or more digital images.

In step 325, the method may include generating a notification indicating that at least one of the prediction, the recommendation, and/or the plurality of data is available. Additionally, visualization for immunohistochemistry or genomic suites may also be available.

In step 327, the method may include providing the user with an option to select a visualization and/or report to view. The user may be a pathologist.

In step 329, the method may include generating a visualization of a recommended treatment path based on at least one of the prediction, the recommendation, and/or the plurality of data. The treatment path (e.g., validated, new, clinical, etc.) or subsequent steps (e.g., clinical trials, professional physician visits, etc.) may be based on the output/generated predictions. Visualization of digital immunohistochemistry or genomic suite results may include one or more of the following:

a. overlaying a positive region of interest (e.g., outline, gradient with color mapped to algorithmic prediction, etc.) on an original image

b. Side-by-side comparison of images with digital IHC or genome suite predictive displays with images without predictive displays

c. A prioritized list of all positive foci (e.g., a slide show of an image crop, an interface that allows a user to jump from one focus to another, etc.) identified as positive areas for a biomarker or mutation of interest

d. All tests are aggregated into one final output (e.g., score, result, recommendation, etc.) or a report listing the final output for each digital test.

In step 331, the method can include recording a visualization as part of a case history within a clinical reporting system.

In step 333, the method may include integrating one or more test results in a final diagnostic report associated with the tissue specimen.

Fig. 4 is a flowchart illustrating an exemplary method for visualizing a tumor region using and training a machine learning system to guide a molecular pathologist, according to one or more exemplary embodiments of the present disclosure. Visualizing the malignant tissue regions on a digitized pathology slide may help a molecular pathologist evaluate the optimal downstream tests. Exemplary embodiments may be used to select the best zone for downstream testing.

Exemplary methods

400 and 420 may be used to visualize a tumor region to guide a molecular pathologist. The exemplary methods 400 and 420 (i.e., steps 401-413 and 421-433) may be performed by the slide analysis tool 101 automatically or in response to a request from a user.

According to one embodiment, an exemplary method 400 for training a machine learning system to visualize a tumor region to guide a molecular pathologist may include one or more of the following steps. In step 401, the method may include receiving one or more digital images associated with a tissue specimen and corresponding information. The one or more digital images may include a histology slide. The corresponding information may include relevant information (e.g., specimen type, available fraction, rough description, etc.), clinical information (e.g., diagnosis, biomarker information, etc.), and patient information (e.g., demographic data, gender, etc.).

In step 403, the method may include developing a system to store and archive a plurality of digital images and corresponding patient data. The corresponding patient data may include images from screening, follow-up, results, and the like.

In step 405, the method may include storing a plurality of digital images and corresponding patient data in a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, RAM, and the like.

In step 407, the method may include generating at least one recommendation for a treatment path based on at least one relevant feature of the plurality of digital images. The treatment pathway may include clinical trials, treatments, etc. for the patient based on at least one relevant factor (e.g., patient diagnosis, medical history, demographic data, etc.).

In step 409, the method may include predicting a tumor region on the plurality of digital images.

In step 411, the method may include generating a list of at least one recommended treatment path based on the predicted tumor region. The recommended treatment path (e.g., medication, clinical trial, etc.) and any relevant information (e.g., success rate, treatment location, etc.) may be based on the predicted tumor region.

In step 413, the method may include converting the one or more predicted values or recommendations into a form that may be visualized or interpreted by a user (e.g., pathologist, patient, oncologist, etc.).

In step 421, the method can include receiving one or more digital images associated with a tissue specimen, a plurality of related cases, and patient information from a clinical system. Pathology specimens (e.g., histological specimens), related case and patient information (e.g., specimen type, case and patient ID, parts within case, rough description, etc.), and information from clinical systems (e.g., assigned pathologist, specimens available for testing, etc.) into digital storage (e.g., hard drive, network drive, cloud storage, RAM, etc.).

In step 423, the method may include generating at least one of a prediction, a recommendation, and/or a plurality of data for the one or more digital images.

In step 425, the method may include generating a notification indicating that at least one of the predictions, recommendations, and/or the plurality of data is available. Additionally, immunohistochemistry or visualization of genomic suites may also be available.

In step 427, the method may include providing the user with an option to select a visualization and/or report to view. The user may be a pathologist.

In step 429, the method may include generating a visualization of the recommended treatment path based on at least one of the prediction, recommendation, and/or plurality of data. The treatment path (e.g., validated, new, clinical, etc.) or subsequent steps (e.g., clinical trials, professional physician visits, etc.) may be based on the output/generated predictions. Visualization of the digital tumor probe results may include one or more of the following:

a. overlay over the positive region of interest on the original image (e.g., contours, gradations with colors mapped to algorithmic predictions, etc.). The overlay can be registered to subsequent images to guide the user to scrape the tumor for sequencing

b. Side-by-side comparison of images with predictive display with images without predictive display

c. A prioritized list of top regions (e.g., tumor with highest mutation load, etc.). The priority list may include reports summarizing all portions analyzed for tumor specific characteristics (e.g., tumor mutational burden) with the predictions.

In step 431, the method may include recording the visualization as part of a case history within a clinical reporting system.

In step 433, the method may include integrating one or more test results in a final diagnostic report associated with the tissue specimen.

Fig. 5 is a flow diagram illustrating an example method for reporting a predicted development of anti-tumor resistance using and training a machine learning system in accordance with one or more example embodiments of the present disclosure. Anti-tumor resistance occurs when cancer cells resist and survive despite anti-cancer therapy. This ability can evolve in cancer during the course of treatment. Therapies that predict that cancer will be the most refractory to acquiring resistance may improve treatment and survival of patients. Certain cancers may develop resistance to multiple drugs during treatment. This information may be provided to identify treatments that are likely to be ineffective. Exemplary embodiments may be used to report the predicted development of anti-tumor resistance.

Exemplary methods

500 and 520 may be used to predict the development of anti-tumor resistance. The exemplary methods 500 and 520 (i.e., steps 501-511 and 521-533) may be performed by the slide analysis tool 101 automatically or in response to a request from a user.

According to one embodiment, an exemplary method 500 for training a machine learning system to visualize a tumor region to guide a molecular pathologist may include one or more of the following steps. In step 501, the method may include receiving one or more digital images associated with a tissue specimen and corresponding information. The one or more digital images may include a histology slide. The corresponding information may include relevant information (e.g., specimen type, available fraction, rough description, etc.), clinical information (e.g., diagnosis, biomarker information, etc.), and patient information (e.g., demographic data, gender, etc.).

In step 503, the method may include developing a system for storing and archiving a plurality of digital images and corresponding patient data. The corresponding patient data may include images from screening, follow-up, results, and the like.

In step 505, the method may include storing a plurality of digital images and corresponding patient data in a digital storage device. The digital storage device may include a hard disk drive, a network drive, cloud storage, RAM, and the like.

In step 507, the method may include predicting current or future resistance to at least one treatment pathway or at least one drug. The prediction may use AI, testing, etc. The AI may infer this information using various inputs including demographic data, digital images of (stained) tissue containing tumors, patient medical history, and the like.

In step 509, the method may include generating a list of at least one treatment predicted to be unlikely to be effective.

In step 511, the method may include generating a list of at least one recommended treatment path based on the predicted tumor region. The recommended treatment path (e.g., medication, clinical trial, etc.) and any relevant information (e.g., success rate, treatment location, etc.) may be based on the predicted tumor region.

In step 511, the method may include converting the one or more predicted values or recommendations into a form that may be visualized or interpreted by a user (e.g., pathologist, patient, oncologist, etc.).

In step 521, the method may include receiving one or more digital images associated with a tissue specimen, a plurality of related cases, and patient information from a clinical system. Pathology specimens (e.g., histological specimens), related case and patient information (e.g., specimen type, case and patient ID, parts within case, rough description, etc.), and information from clinical systems (e.g., assigned pathologist, specimens available for testing, etc.) into digital storage (e.g., hard drive, network drive, cloud storage, RAM, etc.).

In step 523, the method may include generating at least one efficacy prediction and/or a plurality of data for the one or more digital images.

In step 525, the method may include generating a notification indicating that a prediction and visualization of at least one treatment unlikely to be effective is available.

In step 527, the method can include providing the user with an option to select a visualization and/or report to view. The user may be a pathologist.

In step 529, the method may include generating a visualization of at least one treatment that is unlikely to be effective based on the prediction. Visualization of information may be provided via:

a. an interactive web interface, wherein a user (e.g., pathologist, oncologist, patient, etc.) can learn more about at least one particular recommendation (e.g., open clinical trial, treatment-specialized hospital/physician, etc.) via a direct link to the interface and the source (e.g., website, literature, etc.)

b. Reports in which a user may view aggregated, immutable reports, which may include, but are not limited to, the following elements:

i. history of patients

Case summary

Summary of diagnosis

Numerical and/or "manual" test results

v. subsequent steps suggested to the patient based on the digital test results.

In step 531, the method can include recording a visualization as part of a case history within a clinical reporting system.

In step 533, the method may include integrating one or more test results in a final diagnostic report associated with the tissue specimen.

Fig. 6 depicts exemplary options for a user to view visualizations and/or reports according to one or more exemplary embodiments of the present disclosure. In display 60, an example report with a display of slide scoring results is shown. Display 65 shows an exemplary window with an option for a user to order a digital IHC running on a slide.

As shown in fig. 7, device 700 may include a Central Processing Unit (CPU) 720. CPU 720 may be any type of processor device including, for example, any type of special purpose or general purpose microprocessor device. As will be appreciated by those skilled in the relevant art, CPU 720 may also be a single processor in a multi-core/multi-processor system (such a system operating alone), or in a cluster of computing devices operating in a cluster or server farm. CPU 720 may be connected to a data communication infrastructure 710 such as a bus, message queue, network, or multi-core messaging scheme.

Device 700 may also include a main memory 740, such as Random Access Memory (RAM), and may also include a secondary memory 730. The secondary memory 730, such as Read Only Memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may comprise, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, or a flash memory, among others. The removable storage drive in this example reads from and/or writes to a removable storage unit in a well-known manner. Removable storage may include floppy disks, magnetic tapes, optical disks, etc. which are read by and written to by removable storage drives. As will be appreciated by one skilled in the relevant art, such removable storage units typically include a computer usable storage medium having stored therein computer software and/or data.

In alternative implementations, the secondary memory 730 may include similar means for allowing computer programs or other instructions to be loaded into the device 700. Examples of such components may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, and other removable storage units and interfaces that allow software and data to be transferred from the removable storage unit to device 700.

The device 700 may also include a communication interface ("COM") 760. Communications interface 760 allows software and data to be transferred between device 700 and external devices. Communications interface 760 may include a modem, a network interface (such as an ethernet card), a communications port or PCMCIA slot and card, etc. Software and data transferred via communications interface 760 may be in the form of signals which may be electronic, electromagnetic, optical or other signals capable of being received by communications interface 760. These signals may be provided to communications interface 760 via a communications path of device 700, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channels.

The hardware elements, operating systems and programming languages of such equipment are conventional in nature and it is assumed that those skilled in the art are sufficiently familiar with this. Device 700 may also include input and output ports 750 for connecting to input and output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, the various server functions may be implemented in a distributed manner across a plurality of similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming of a computer hardware platform.

Throughout this disclosure, references to components or modules generally refer to items that may be logically combined together to perform a function or a group of related functions. Like reference numerals are generally intended to refer to the same or similar components. The components and modules may be implemented in software, hardware, or a combination of software and hardware.

The tools, modules, and functions described above may be performed by one or more processors. A "storage" type medium may include any or all of a tangible memory, such as a computer or processor, or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., that may provide non-transitory storage for software programming at any time.

The software may communicate over the internet, a cloud service provider, or other telecommunications network. For example, communication may enable loading of software from one computer or processor into another computer or processor. As used herein, unless limited to a non-transitory tangible "storage" medium, terms such as a computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

The foregoing general description is exemplary and explanatory only and is not restrictive of the disclosure. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only.

Claims

1. A computer-implemented method for processing an electronic image corresponding to a specimen, the method comprising:

receiving one or more digital images associated with a tissue specimen, a related case, a patient, and/or a plurality of clinical information;

determining one or more of a prediction, a recommendation, and/or a plurality of data for the one or more digital images using a machine learning system that has been trained using a plurality of training images to predict biomarkers and a plurality of genomic suite elements; and

based on the prediction, the recommendation, and/or the plurality of data, determining whether to record an output and at least one visualization area as part of a case history within a clinical reporting system.

2. The computer-implemented method of claim 1, wherein the method further comprises generating a notification indicating that a prediction or visualization is available for the one or more digital images.

3. The computer-implemented method of claim 1, wherein the method further comprises generating an option for a user to view the prediction or the visualization.

4. The computer-implemented method of claim 1, wherein the method further comprises generating one or more displays of at least one recommended treatment based on the prediction.

5. The computer-implemented method of claim 4, wherein the one or more displays may be generated by one of a plurality of methods: an overlay of at least one region of interest layered on top of the original image, a side-by-side visualization, a report with at least one quantification method, and a summary of at least one digital test run with multiple results.

6. The computer-implemented method of claim 1, wherein visualization of digital immunohistochemistry or genome stack results comprises one of a plurality of methods: an overlay of at least one region of interest layered on top of the original image, a side-by-side visualization, a report with at least one quantitative metric, and a summary of a digital test run with at least one result.

7. The computer-implemented method of claim 1, wherein the recommendation of the at least one treatment pathway is based on a plurality of clinical practice guidelines.

8. The computer-implemented method of claim 1, wherein generating at least one prediction comprises:

receiving one or more digitized images of a pathology specimen, related information, clinical information, and patient information;

developing a system for storing and archiving a plurality of images and a plurality of corresponding patient data;

determining at least one predicted biomarker and at least one predicted genomic set element based on the plurality of images and the plurality of corresponding patient data;

generating a list of recommended treatment paths based on the plurality of predicted biomarkers and the genomic suite element; and

converting the one or more predictors and the at least one treatment path recommendation into a user-readable form.

9. The computer-implemented method of claim 8, further comprising outputting the one or more predicted values and the at least one treatment path recommendation to a user interface.

10. A system for processing an electronic image corresponding to a specimen, the system comprising:

at least one memory storing instructions; and

at least one processor configured to execute the instructions to perform operations comprising:

11. The system of claim 10, wherein the method further comprises generating a notification indicating that a prediction or visualization of the one or more digital images is available.

12. The system of claim 10, wherein the method further comprises generating an option for a user to view the prediction or the visualization.

13. The system of claim 10, wherein the method further comprises generating one or more displays of at least one recommended treatment based on the prediction.

14. The system of claim 10, wherein the one or more displays may be generated by one of a plurality of methods: coverage of at least one region of interest layered on top of the original image, side-by-side visualization, a report with at least one quantification method, and a summary of at least one digital test run with multiple results.

15. The system of claim 10, wherein visualization of digital immunohistochemistry or genomic suite results comprises one of a plurality of methods: an overlay of at least one region of interest layered on top of the original image, a side-by-side visualization, a report with at least one quantitative metric, and a summary of a digital test run with at least one result.

16. The system of claim 10, wherein the recommendation of the at least one treatment path is based on a plurality of clinical practice guidelines.

17. The system of claim 10, wherein generating at least one prediction comprises:

generating a list of recommended treatment paths based on the plurality of predicted biomarkers and the genomic set element; and

18. The system of claim 10, further comprising outputting the one or more predictive values and the at least one therapy pathway recommendation to a user interface.

19. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to implement a method for monitoring the health of a population of people, the method comprising:

20. The method of claim 19, wherein the method further comprises generating a notification indicating that a prediction or visualization of the one or more digital images is available.