BR112022014530B1 - SYSTEMS AND METHODS FOR PROCESSING ELECTRONIC IMAGES FOR COMPUTATIONAL DETECTION METHODS - Google Patents

Abstract

SYSTEMS AND METHODS FOR PROCESSING ELECTRONIC IMAGES FOR COMPUTATIONAL DETECTION METHODS. Disclosed are systems and methods for receiving one or more electronic slide images associated with a tissue sample, wherein the tissue sample is associated with a patient and/or medical case, partitioning a first slide image from one or more electronic blade into a plurality of cutouts, detect a plurality of tissue regions of the first slide image and/or plurality of cutouts to generate a fabric mask, determine whether any of the plurality of cutouts correspond to non-woven fabric, remove any of the plurality of cutouts of cutouts that are determined to be nonwoven, determine a prediction, with the use of a machine learning prediction model, for at least one label for one or more electronic slide images, wherein the machine learning prediction model machine was generated by processing a plurality of training images and outputting the prediction from the trained machine learning prediction model.

Description

RELATED REQUEST(S)

[0001] This application claims priority to Provisional Application No. US 62/966,716 filed on January 28, 2020, the complete disclosure of which is incorporated herein by reference in its entirety.

FIELD OF REVELATION

[0002] Various embodiments of the present disclosure generally relate to creating a prediction model to predict labels for tissue samples prepared through electronic image processing. More specifically, particular embodiments of the present disclosure relate to systems and methods for predicting, identifying, or detecting diagnostic information about prepared tissue samples. The present disclosure further provides systems and methods for creating a prediction model that predicts unseen slide labels.

BACKGROUND

[0003] The performance of machine learning and deep learning models for histopathology may be limited by the volume and quality of the annotated examples used to train these models. Large-scale experiments on supervised image classification problems have shown that model performance continues to improve, reaching an order of 50 million training examples. Manually annotating this volume of data can be prohibitively expensive, both in time and cost, and can be a severe limitation to ensuring that systems function at a clinically relevant level and generalize across institutions.

[0004] The previous general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. The background description provided in this document is intended to generally present the context of the disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted as prior art, or suggestions of prior art, by inclusion in this section.

SUMMARY

[0005] In accordance with certain aspects of the present disclosure, systems and methods are disclosed for developing weakly supervised multi-label and multi-task learning for computational biomarker detection in digital pathology.

[0006] A computer-implemented method for processing an electronic image corresponding to a sample includes: receiving one or more digital images associated with a tissue sample, receiving one or more electronic slide images associated with a tissue sample, wherein the tissue sample is associated with a patient and/or medical case; partitioning a first slide image of one or more electronic slide images into a plurality of slices; detecting a plurality of tissue regions of the first slide image and/or plurality of cutouts to generate a tissue mask; determining whether any of the plurality of cutouts corresponds to nonwoven fabric; removing any of the plurality of cutouts that are determined to be nonwoven; determining a prediction, using a machine learning prediction model, for at least one label for one or more electronic slide images, wherein the machine learning prediction model was generated by processing a plurality of images training; and output the prediction from the trained machine learning prediction model.

[0007] A system for processing an electronic image corresponding to a sample includes memory storage instructions; and at least one processor executing instructions to perform a process including receiving one or more digital images associated with a tissue sample, receiving one or more electronic slide images associated with a tissue sample, wherein the tissue sample is associated with a patient and/or medical case; partitioning a first slide image of one or more electronic slide images into a plurality of slices; detecting a plurality of tissue regions of the first slide image and/or plurality of cutouts to generate a tissue mask; determining whether any of the plurality of cutouts corresponds to nonwoven fabric; removing any of the plurality of cutouts that are determined to be nonwoven; determining a prediction, using a machine learning prediction model, for at least one label for one or more electronic slide images, wherein the machine learning prediction model was generated by processing a plurality of images training; and output the prediction from the trained machine learning prediction model.

[0008] A non-transitory computer-readable medium that stores instructions that, when executed by a processor, cause the processor to perform a method for processing an electronic image corresponding to a sample includes: receiving one or more digital images associated with a sample of tissue, receiving one or more electronic slide images associated with a tissue sample, wherein the tissue sample is associated with a patient and/or medical case; partitioning a first slide image of one or more electronic slide images into a plurality of slices; detecting a plurality of tissue regions of the first slide image and/or plurality of cutouts to generate a tissue mask; determining whether any of the plurality of cutouts corresponds to nonwoven fabric; removing any one of the plurality of cutouts that is determined to be nonwoven; determining a prediction, using a machine learning prediction model, for at least one label for one or more electronic slide images, wherein the machine learning prediction model was generated by processing a plurality of images training; and output the prediction from the trained machine learning prediction model.

[0009] It should be understood that both the previous general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The attached drawings, which are incorporated and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

[0011] Figure 1A illustrates an exemplary block diagram of a system and network for creating a prediction model in accordance with an exemplary embodiment of the present disclosure.

[0012] Figure 1B illustrates an exemplary block diagram of a prediction model platform, in accordance with an exemplary embodiment of the present disclosure.

[0013] Figure 1C illustrates an exemplary block diagram of a slide analysis tool according to an exemplary embodiment of the present disclosure.

[0014] Figure 2A is a flowchart illustrating an exemplary method for using a prediction model created by a trained machine learning system, in accordance with one or more exemplary embodiments of the present disclosure.

[0015] Figure 2B is a flowchart illustrating an exemplary method for training a weakly supervised slice-level learning module in a trained machine learning system, in accordance with one or more exemplary embodiments of the present disclosure.

[0016] Figure 2C is a flowchart illustrating an exemplary method for training a weakly supervised aggregation module in a trained machine learning system, in accordance with one or more exemplary embodiments of the present disclosure.

[0017] Figure 3 is a flowchart illustrating an exemplary method for training and using a machine learning system to simultaneously detect and classify prostate cancer, in accordance with one or more exemplary embodiments of the present disclosure.

[0018] Figure 4 is a flowchart illustrating an exemplary method for training and using a machine learning system for tumor quantification in prostate needle biopsies, in accordance with one or more exemplary embodiments of the present disclosure.

[0019] Figure 5 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict a cancer subtype, in accordance with one or more exemplary embodiments of the present disclosure.

[0020] Figure 6 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict a surgical margin in accordance with one or more exemplary embodiments of the present disclosure

[0021] Figure 7 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict a bladder cancer biomarker, in accordance with one or more exemplary embodiments of the present disclosure.

[0022] Figure 8 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict a pancancer diagnosis, in accordance with one or more exemplary embodiments of the present disclosure.

[0023] Figure 9 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict organ toxicity, in accordance with one or more exemplary embodiments of the present disclosure.

[0024] Figure 10 illustrates an exemplary connected components algorithm, according to one embodiment of the disclosure.

[0025] Figure 11 represents an exemplary system that can perform techniques presented in this document.

DESCRIPTION OF MODALITIES

[0026] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the attached drawings. Whenever possible, the same reference numbers will be used on all drawings to refer to the same or similar parts.

[0027] The systems, devices and methods disclosed in this document are described in detail by means of examples and with reference to the figures. The examples discussed herein are examples only and are provided to assist in explaining the apparatus, devices, systems and methods described herein. None of the features or components shown in the drawings or discussed below should be considered mandatory for any particular implementation of any such device, system or method unless specifically designated as mandatory.

[0028] Furthermore, for any methods described, regardless of whether the method is described in conjunction with a flow diagram, it should be understood that, unless otherwise specified or required by the context, any explicit or implicit ordering of steps performed executing a method does not imply that these steps must be executed in the order presented, but they can be executed in a different order or in parallel.

[0029] As used herein, the term "exemplary" is used in the sense of "example", rather than "ideal". Furthermore, the terms "a" and "an" herein do not denote a quantity limitation, but rather denote the presence of one or more of the referenced items.

[0030] Pathology refers to the study of diseases, as well as the causes and effects of the disease. More specifically, pathology refers to the performance of tests and analyzes that are used to diagnose diseases. For example, tissue samples may be placed on slides to be viewed under a microscope by a pathologist (e.g., a doctor who specializes in analyzing tissue samples to determine whether abnormalities exist). That is, pathology samples can be cut into multiple sections, stained, and prepared as slides for a pathologist to examine and make a diagnosis. When a diagnostic finding on a slide is uncertain, a pathologist may order additional cut levels, stains, or other tests to gather more information from the tissue. The technician (or technicians) may create a new slide (or slides) that may contain additional information for the pathologist to use in diagnosis. This process of creating additional slides can be time-consuming, not only due to the fact that it may involve retrieving the tissue block, cutting it to make a new slide, and staining the slide, but also due to the fact that it may be pooled. for multiple orders. This can significantly delay the pathologist's final diagnosis. Additionally, even after the delay, there may be no guarantee that the new slide (or slides) will have enough information to provide a diagnosis.

[0031] Pathologists can evaluate cancer and other disease pathology slides alone. The present disclosure presents a consolidated workflow to improve the diagnosis of cancer and other diseases. The workflow can integrate, for example, slide evaluation, tasks, image analysis and cancer detection artificial intelligence (AI), annotations, queries and recommendations on a workstation. In particular, the present disclosure describes several exemplary user interfaces available in the workflow, as well as AI tools that can be integrated into the workflow to streamline and improve the work of a pathologist.

[0032] For example, computers can be used to analyze an image of a tissue sample to quickly identify whether additional information may be needed about a specific tissue sample and/or to highlight for a pathologist an area in which he or she should look further. closely. This way, the process of obtaining additional stained slides and exams can be done automatically before being reviewed by a pathologist. When paired with automatic slide segmentation and staining machines, this can provide a fully automated slide preparation pipeline. This automation has at least the benefits of (1) minimizing the amount of time wasted by a pathologist determining that a slide is insufficient to make a diagnosis, (2) minimizing the time (average total) from sample acquisition to diagnosis, avoiding additional time between ordering additional tests and their production, (3) reducing the amount of time per cut and the amount of wasted material, allowing cuts to be performed while tissue blocks (e.g., samples of pathology) are on a cutting table, (4) reduce the amount of wasted/discarded tissue material during slide preparation, (5) reduce the cost of slide preparation by partially or fully automating the procedure, (6) allow the automatic custom cutting and staining of slides that would result in slides that are more representative/informative of the samples, (7) allow for greater volumes of slides to be generated by tissue clipping, contributing to more informed/accurate diagnoses, reducing the overhead of requesting additional testing additional for a pathologist and/or (8) identify or verify the correct properties (e.g., belonging to a sample type) of a digital pathology image, etc.

[0033] The process of using computers to assist pathologists is known as computational pathology. Computing methods used for computational pathology may include, without limitation, statistical analysis, autonomous or machine learning, and AI. AI may include, but is not limited to, deep learning, neural networks, classifications, clustering, and regression algorithms. Through the use of computational pathology, lives can be saved by helping pathologists improve their diagnostic accuracy, reliability, efficiency and accessibility. For example, computational pathology can be used to aid in the detection of suspected cancer slides, thereby allowing pathologists to verify and confirm their initial assessments before providing a final diagnosis.

[0034] As described above, the computational pathology processes and devices of the present disclosure can provide an integrated platform that enables a fully automated process, including data ingestion, processing and viewing of digital pathology images via a web browser or another user interface, while integrating with a laboratory information system (LIS). Additionally, clinical information can be aggregated using cloud-based data analytics of patient data. Data can come from hospitals, clinics, field researchers, etc., and can be analyzed by machine learning, computer vision, natural language processing and/or statistical algorithms to do real-time monitoring and prediction of health patterns in multiple levels of geographic specificities.

[0035] Histopathology refers to the study of a sample that has been placed on a slide. For example, a digital pathology image may be comprised of a digitized image of a microscope slide containing the sample (e.g., a smear). One method a pathologist can use to analyze an image on a slide is to identify nuclei and classify whether a nucleus is normal (e.g., benign) or abnormal (e.g., malignant). To assist pathologists in identifying and classifying nuclei, histological stains can be used to make cells visible. Many dye-based staining systems have been developed, including the periodic acid-Schiff reaction, Masson's trichrome, nissl and methylene blue, and hematoxylin and eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method, with hematoxylin staining the cell nucleus blue, eosin staining the cytoplasm and extracellular matrix pink, and other tissue regions taking on variations of these colors. In many cases, however, H&E-stained histological preparations do not provide sufficient information for a pathologist to visually identify biomarkers that can aid in diagnosis or guide treatment. In this situation, techniques such as immunohistochemistry (IHC), immunofluorescence, in situ hybridization (ISH) or fluorescence in situ hybridization (FISH) can be used. IHC and immunofluorescence involve, for example, the use of antibodies that bind to specific antigens in tissues, allowing visual detection of cells expressing specific proteins of interest, which can reveal biomarkers that are not reliably identifiable to trained pathologists. based on analysis of H&E stained slides. ISH and FISH can be employed to assess gene copy number or the abundance of specific RNA molecules, depending on the type of probes employed (e.g., DNA probes for gene copy number and RNA probes for expression assessment). of RNA). If these methods also do not provide sufficient information to detect some biolabels, tissue genetic testing can be used to confirm whether a biomarker is present (e.g., overexpression of a specific protein or gene product in a tumor, amplification of a particular gene in cancer).

[0036] A digitized image can be prepared to show a stained microscope slide, which can allow a pathologist to manually view the image on a slide and estimate a number of abnormal cells stained in the image. However, this process can be time-consuming and can lead to errors in identifying abnormalities, as some abnormalities are difficult to detect. Computational processes and devices can be used to assist pathologists in detecting abnormalities that would otherwise be difficult to detect. For example, AI can be used to predict biomarkers (such as overexpression of a protein and/or gene product, amplification or mutations of specific genes) from salient regions in digital images of tissues stained using H&E and other data-based methods. dyes. Tissue images can be whole slide images (WSI), images of tissue cores within microarrays, or selected areas of interest within a tissue section. Through the use of staining methods such as H&E, these biomarkers can be difficult for humans to visually detect or quantify without the help of additional testing. Using AI to infer these biomarkers from digital tissue images has the potential to improve patient care, as well as be faster and less expensive.

[0037] The detected biomarkers or imaging alone can be used to recommend specific cancer drugs or drug combination therapies to be used to treat a patient, and AI can identify which drugs or drug combinations are unlikely to be used. successful studies correlating detected biomarkers with a database of treatment options. This can be used to facilitate automatic recommendation of immunotherapy drugs to target a patient's specific cancer. Furthermore, this could be used to enable personalized cancer treatment for specific subsets of patients and/or rarer cancer types.

[0038] As described above, the computational pathology processes and devices of the present disclosure can provide an integrated platform that enables a fully automated process, including data ingestion, processing and viewing of digital pathology images via a web browser or another user interface, while integrating with a laboratory information system (LIS). Additionally, clinical information can be aggregated using cloud-based data analytics of patient data. Data can come from hospitals, clinics, field researchers, etc., and can be analyzed by machine learning, computer vision, natural language processing and/or statistical algorithms to do real-time monitoring and prediction of health patterns in multiple levels of geographic specificities.

[0039] The digital pathology images described above may be stored with tags and/or labels pertaining to the properties of the sample or digital pathology image and such tags/labels may be incomplete. Consequently, the systems and methods disclosed herein provide for at least one label of a collection of digital images.

[0040] The performance of machine learning and deep learning models for histopathology may be limited by the volume and quality of the annotated examples used to train these models. Large-scale experiments on supervised image classification problems have shown that model performance continues to improve, reaching an order of 50 million training examples. Most clinically relevant tasks in pathology involve much more than classification, however. When a pathologist provides a diagnosis, the diagnosis may take the form of a report that contains many heterogeneous interrelated fields and pertains to an entire slide or set of slides. In oncology, these fields may include presence of cancer, cancer grades, tumor quantification, cancer grade group, presence of various features important for cancer staging, etc. In preclinical drug research animal studies, these fields may include the presence of toxicity, the severity of toxicity, and the type of toxicity. Obtaining the annotations necessary to train most supervised deep learning models may involve a pathologist labeling individual pixels, indentations (e.g., one or more relatively small rectangular regions in a slide image), or regions of interest (e.g., polygons ) of the slide image with an appropriate annotation. For each report field, a different set of training annotations can be used. Furthermore, a typical digital pathology slide can contain on the order of 10 gigapixels, or more than 100,000 slices. Manually annotating this volume of data can be prohibitively expensive, both in time and cost, and can be a severe limitation to ensuring that systems function at a clinically relevant level and generalize across institutions. Consequently, there is a desire to generate training data that can be used for histopathology.

[0041] Embodiments of the present disclosure can overcome the above limitations. In particular, the embodiments disclosed herein may use weak supervision, in which a deep learning model may be trained directly from a pathologist's diagnosis, rather than additional labeling of each pixel or crop in a digital image. A machine learning or deep learning model may comprise a machine learning algorithm, in some embodiments. One technique can determine binary cancer detection, however, the techniques discussed in the present paper further disclose, for example, how a deep learning system can be trained in a weakly supervised multi-label, multi-task setting to perform classification, subtyping, inferring various disease attributes simultaneously, and more. This allows systems to be trained directly from diagnostic reports or test results without the need for extensive annotations, reducing the number of training labels required by five orders of magnitude or more.

[0042] The disclosed systems and methods can automatically predict sample or image properties without relying on stored tags or labels. Furthermore, systems and methods are disclosed for quickly and correctly identifying and/or verifying a sample type of a digital pathology image, or any information related to a digital pathology image, without necessarily accessing a database of analogous information or LIS. An embodiment of the present disclosure may include a system trained to identify various properties of a digital pathology image, based on prior digital pathology image datasets. The trained system can provide a classification for a sample shown in a digital pathology image. Classification can help provide prediction (or predictions) of treatment or diagnosis for a patient associated with the sample.

[0043] This disclosure includes one or more embodiments of a slide analysis tool. Input to the tool can include a digital pathology image and any additional relevant inputs. The tool outputs can include global and/or local information about the sample. A specimen may include a biopsy or surgical resection specimen.

[0044] Figure 1A illustrates a block diagram of a system and network for determining sample property or image property information pertaining to the digital pathology image (or images), using machine learning, in accordance with an exemplary embodiment of the present disclosure.

[0045] Specifically, Figure 1A illustrates an electronic network 120 that can be connected to servers in hospitals, laboratories and/or doctors' offices, etc. For example, medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124 and/or laboratory information systems 125, etc., may be connected to an electronic network 120, such as the Internet, via of one or more computers, servers and/or portable mobile devices. According to an exemplary embodiment of the present application, the electronic network 120 may also be connected to server systems 110, which may include processing devices that are configured to implement a disease detection platform 100, which includes a disease analysis tool. blade 101 to determine sample property or image property information pertaining to the digital pathology image (or images) and using machine learning to classify a sample, in accordance with an exemplary embodiment of the present disclosure.

[0046] Medical servers 121, hospital servers 122, clinical trials servers 123, research laboratory servers 124 and/or laboratory information systems 125 may create or obtain images of one or more patient cytology samples, histopathology samples , slide (or slides) of the cytological specimen (or cytological specimens), digitized images of the slide (or slides) of the histopathological specimen (or histopathological specimens), or any combination thereof. Medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124, and/or laboratory information systems 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer, family history, previous biopsy or cytology information, etc. Medical servers 121 , hospital servers 122 , clinical trial servers 123 , research laboratory servers 124 , and/or laboratory information systems 125 may transmit scanned slide images and/or patient-specific information to server systems 110 over the electronic network 120. Server systems 110 may include one or more storage devices 109 for storing images and data received from at least one of medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124, and/or laboratory information systems 125. Server systems 110 may also include processing devices for processing images and data stored on one or more storage devices 109. Server systems 110 may further include one or more machine learning tools or capabilities . For example, the processing devices may include a machine learning tool for a disease detection platform 100, in accordance with one embodiment. Alternatively or additionally, the present disclosure (or portions of the system and methods of the present disclosure) may be performed on a local processing device (e.g., a laptop-type computer).

[0047] Medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124, and/or laboratory information systems 125 refer to systems used by pathologists to review slide images. In hospital settings, tissue type information can be stored in laboratory information systems 125. However, correct tissue classification information is not always paired with image content. Additionally, even if an LIS is used to access the sample type for a digital pathology image, this label may be incorrect due to the fact that many components of an LIS can be entered manually, leaving a large margin for error. In accordance with an exemplary embodiment of the present disclosure, a sample type may be identified without needing to access library information systems 125, or may be identified to possibly correct library information systems 125. For example, a third party may have anonymous access to image content without the corresponding sample type label stored in the LIS. Additionally, access to LIS content may be limited due to its sensitive content.

[0048] Figure 1B illustrates an exemplary block diagram of a disease detection platform 100 for determining sample property or image property information pertaining to the digital pathology image (or images), using machine learning. . For example, the disease detection platform 100 may include a slide analysis tool 101, a data ingestion tool 102, a slide ingestion tool 103, a slide scanner 104, a slide manager 105, a storage 106 and a visualization application tool 108.

[0049] Slide analysis tool 101, as described below, refers to a process and system for processing digital images associated with a tissue sample and using machine learning to analyze a slide, according to an exemplary embodiment.

[0050] Data ingestion tool 102 refers to a process and system for facilitating a transfer of the digital pathology images to the various tools, modules, components and devices that are used to classify and process the digital pathology images, from according to an exemplary embodiment.

[0051] Slide input tool 103 refers to a process and system for scanning pathology images and converting them into a digital form, in accordance with an exemplary embodiment. The slides can be digitized with the slide scanner 104 and the slide manager 105 can process the images on the slides into digitized pathology images and store the digitized images in storage 106.

[0052] Visualization application tool 108 refers to a process and system for providing a user (e.g., a pathologist) with sample ownership or image ownership information pertaining to a digital pathology image (or images), of according to an exemplary embodiment. Information may be provided through various output interfaces (e.g., a screen, a monitor, a storage device, and/or a web browser, etc.).

[0053] The slide analysis tool 101, and each of its components, can transmit and/or receive scanned slide images and/or patient information to server systems 110, medical servers 121, hospital servers 122, clinical trials 123, research laboratory servers 124, and/or laboratory information systems 125 via an electronic network 120. Additionally, server systems 110 may include one or more storage devices 109 for storing images and data received from at least one of slide analysis tool 101, data ingestion tool 102, slide input tool 103, slide scanner 104, slide manager 105, and visualization application tool 108. Server 110 may also include processing devices for processing images and data stored on the storage devices. Server systems 110 may further include one or more machine learning tools or resources, for example due to processing devices. Alternatively or additionally, the present disclosure (or portions of the system and methods of the present disclosure) may be performed on a local processing device (e.g., a laptop-type computer).

[0054] Any of the above devices, tools, and modules may be located on a device that may be connected to an electronic network 120, such as the Internet or a cloud service provider, through one or more computers, servers, and/or or portable mobile devices.

[0055] Figure 1C illustrates an exemplary block diagram of a slide analysis tool 101 in accordance with an exemplary embodiment of the present disclosure. The slide analysis tool 101 may include a training image platform 131 and/or a target image platform 135.

[0056] The training image platform 131, in accordance with one embodiment, may create or receive training images that are used to train a machine learning system to effectively analyze and classify digital pathology images. For example, training images may be received from any one or any combination of server systems 110, medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124, and/or laboratory information systems. 125. Images used for training can come from real sources (e.g. humans, animals, etc.) or they can come from synthetic sources (e.g. graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with a variety of stains, such as (without limitation) H&E, Hemotoxylin alone, IHC, molecular pathology, etc.; and/or (b) tissue samples scanned from a 3D imaging device such as microCT.

[0057] The training image input module 132 may create or receive a data set comprising one or more training images corresponding to one or both images of a human tissue and images that are graphically rendered. For example, training images may be received from any one or any combination of server systems 110, medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124, and/or laboratory information systems. 125. This data set can be maintained on a digital storage device. The quality score determiner module 133 can identify quality control (QC) issues (e.g., imperfections) for the training images at a global or local level that can greatly affect the usability of a digital pathology image. For example, the quality score determination module may use information about an entire image, e.g., the type of sample, the overall quality of the sample section, the overall quality of the glass pathology slide itself, or features of the morphology of the tissue and determine an overall image quality score. The treatment identification module 134 can analyze tissue images and determine which digital pathology images have treatment effects (e.g., post-treatment) and which images do not have treatment effects (e.g., pre-treatment). It is useful to identify whether a digital pathology image has treatment effects due to the fact that prior treatment effects on the tissue can affect the morphology of the tissue itself. Most LIS do not explicitly track this characteristic, and therefore it may be desirable to classify sample types with prior treatment effects.

[0058] According to one embodiment, the target image platform 135 may include a target image input module 136, a sample detection module 137, and an output interface 138. The target image platform 135 may receiving a target image and applying the machine learning model to the received target image to determine a characteristic of a target sample. For example, the target image may be received from any one or any combination of server systems 110, medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124, and/or laboratory information systems. 125. The target image input module 136 may receive a target image corresponding to a target sample. The sample detection module 137 may apply the machine learning model to the target image to determine a characteristic of the target sample. For example, the sample detection module 137 may detect a sample type of the target sample. The sample detection module 137 may also apply the machine learning model to the target image to determine a quality score for the target image. Additionally, the sample detection module 137 may apply the machine learning model to the target sample to determine whether the target sample is pre-treatment or post-treatment.

[0059] Output interface 138 can be used to send information about the target image and target sample (e.g., to a screen, monitor, storage device, web browser, etc.).

[0060] Figure 2A is a flowchart illustrating an exemplary method for using a prediction model created by a trained machine learning system, in accordance with one or more exemplary embodiments of the present disclosure. For example, an exemplary method 200 (steps 202-210) may be performed by slide analysis tool 101 automatically or in response to a user request.

[0061] According to one embodiment, the exemplary method 200 for using a prediction model may include one or more of the following steps. In step 202, the method may include receiving one or more digital images associated with a tissue sample, wherein the one or more digital images comprise a plurality of slide images. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM) or any other suitable storage device.

[0062] In step 204, the method may include partitioning one of the plurality of slide images into a collection of slices for the plurality of slide images.

[0063] In step 206, the method may include detecting a plurality of tissue regions from a background of a plurality of slide images to create a tissue mask and removing at least one snip from the collection of snips that is detected as non-woven. The cutout that is non-woven may include a slide image background. This can be accomplished in a number of ways, including: methods based on thresholds based on color, color intensity, texture features, or the Otsu method, followed by execution of a connected components algorithm; segmentation algorithms such as k-means, graph cuts, mask region convolutional neural network (Mask R-CNN); or any other suitable methods.

[0064] In step 208, the method may include determining a prediction, using a machine learning system, for a label for the plurality of slide images corresponding to a patient or medical case, the machine learning system was generated by processing a plurality of training examples to create a prediction model. Training examples may comprise a set of one or more digital slide images and a plurality of target labels.

[0065] In step 210, the method may include output from the training machine learning system prediction model that predicts at least one label from at least one slide that was not used to train the machine learning system and output the prediction for an electronic storage device.

[0066] Figure 2B is a flowchart illustrating an exemplary method for training a weakly supervised slice-level learning module in a trained machine learning system, in accordance with one or more exemplary embodiments of the present disclosure. The weakly supervised learning module can train a model to make slice-level predictions using blade-level training labels. For example, an exemplary method 220 (steps 222-230) may be performed by slide analysis tool 101 automatically or in response to a user request.

[0067] According to one embodiment, the exemplary method 220 for using a prediction model may include one or more of the following steps. In step 222, the method may include receiving a collection of digital images associated with a training tissue sample into a digital storage device, wherein the collection of digital images comprises a plurality of training slide images. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM) or any other suitable storage device.

[0068] In step 224, the method may include receiving a plurality of synoptic annotations comprising one or more labels for each of the plurality of training slide images. Labels can be binary, multi-level binary, categorical, ordinal, or real.

[0069] In step 226, the method may include partitioning one of the plurality of training slide images into a collection of training slices for the plurality of training slide images.

[0070] In step 228, the method may include detecting at least one background tissue region of the plurality of training slide images to create a training tissue mask and removing at least one training slice from the collection of training slices. which is detected as non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms such as k-means, graph cuts, Mask R- CNN or any other suitable method. general approaches to training a model in a weak supervision environment, but any suitable approach to training the model can be used.

[0071] In step 230, the method may include training a prediction model under weak supervision to infer at least one multi-label clipping level prediction using at least one synoptic label. There can be four general approaches to training a model in a weak supervision environment, but any suitable approach to training the model can be used.

[0072] 1. Multiple Instance Learning (MIL) can be used to train a snip-level prediction model for binary or categorical labels by learning to identify snips that contain a blade target label. This identification can be accomplished by finding salient clippings (e.g., top-score clippings based on annotations or synoptic labels received in each training iteration) and using these clippings to update a classifier using the label (or labels) from synoptic training received associated with each salient cutout. For example, the classifier can be trained to identify cancer based on a collection of overlapping snips. As the salient clips are determined, the synoptic labels can be used to update the clip-level labels. This cut-level label and classifier can determine or provide a label for a group of cuts. MIL can also be used to train a machine learning model to extract diagnostic features for other downstream tasks such as cancer classification, cancer subtyping, biomarker detection, etc.

[0073] 2. Multiple Instance Multiple Label Learning (MIMLL) may be a slice-level prediction model that comprises a generalization of MIL that treats each slice as a set of slices that can be associated with multiple labels, rather than of just a single binary label as in MIL. These slide labels can come from a pathologist's diagnostic report, genetic testing, immunological testing, or other measurements/assays. The MIMLL model can be trained to select slices that correspond to each of the training synoptic labels belonging to the set of one or more slides. The present embodiment may involve MIMLL training a neural network (e.g., a Convolutional Neural Network (CNN), capsule network, etc.) through iteration of the following steps: A. For each label of the labels to be predicted, select the most relevant set of clippings using a scoring function. The scoring function can be formulated to classify multiple pieces simultaneously. For example, with multiple binary labels, a CNN can be run on each slice that attempts to predict each of the multiple binary labels for each slice in a set of slides, and the slices with the outputs closest to 1 for one or more of the labels can be selected. B. Use the selected clippings to update the CNN model weights with respect to the associated label assignments. Each label can have its own output layer in the model.

[0074] Similar to the MIL model, the MIMLL model can also be used to extract diagnostic features for other downstream tasks.

[0075] 3. Self-supervised learning can use a small amount of snip-level training data to create an initial snip-based classifier using supervised learning. This initial classifier can be used to initialize a complete training process by alternating the following: A. Reassign clipping labels in the training set using predictions from the current clipping-based model. B. Update the template for each snip with the latest label assignments.

[0076] 4. Unsupervised clustering can learn to group similar instances without the use of target labels. Slide cutouts can be treated as instances and the number of clusters can be pre-specified or automatically learned by the algorithm. Such clustering algorithms may include, without limitation, the following methods: A. Expectation Maximization (EM) B. Majorization Maximization (MM) C. K-Nearest Neighbor (KNN) D. Hierarchical Clustering E. Agglomerative Clustering

[0077] The resulting model can be used to extract diagnostic features for use by the blade level prediction module.

[0078] Figure 2C is a flowchart illustrating an exemplary method for training a weakly supervised aggregation module in a trained machine learning system, in accordance with one or more exemplary embodiments of the present disclosure. For example, an exemplary method 240 (steps 242-244) may be performed by slide analysis tool 101 automatically or in response to a user request.

[0079] According to one embodiment, the exemplary method 240 for training the weakly supervised aggregation module may include one or more of the following steps. In step 242, the method may include receiving a plurality of predictions or a plurality of vectors of at least one feature from a weakly supervised slice-level learning module for the collection of training slices.

[0080] In step 244, the method may include training a machine learning model to receive, as input, the plurality of predictions or the plurality of vectors of at least one feature of the weakly supervised slice-level learning module for the collection of clippings. This aggregation module can train a multi-task blade-level aggregation model to receive slice-level inputs and produce a final prediction for the input of slices in a system and/or blade images in a system. A general form of the model may be composed of multiple outputs (e.g., multitask learning), and each label may be binary, categorical, ordinal, or real. Crop-level inputs may include image features of any type, including, without limitation: A. Outputs (e.g., feature vectors or embeddings) from the weakly supervised model B. CNN features C. Scale Invariant Feature Transform (SIFT) D. Robust and Accelerated Features (SURF) E. Rotation Invariant Feature Transform (RIFT) F. Oriented FAST and Rotated BRIEF (ORB)

[0081] The multi-task blade-level aggregation model of the aggregation module can take several forms, including, without limitation: A. Fully connected neural network trained with multiple groups of output tasks B. CNN C. Fully convolutional neural networks D . Recurrent neural network (RNN), including closed recurrent units (GRU) and long-short-term memory networks (LSTM) E. Graph neural networks F. Transformer networks G. Random forest, boosted forest, XGBoost, etc.

[0082] Figure 3 is a flowchart illustrating an exemplary method for training and using a machine learning system to simultaneously detect and classify prostate cancer, in accordance with one or more exemplary embodiments of the present disclosure. Cancer grading can measure the differentiation of cancer cells from normal tissue and can be assessed at the local level by inspecting cell morphology as well as slide-level summaries containing the relative amounts of grades. Classification can be performed as part of a pathologist's diagnostic report for common cancers such as prostate, kidney, and breast. Exemplary methods 300 and 320 can be used to train and use a machine learning system to simultaneously detect and classify prostate cancer.

[0083] According to one embodiment, exemplary methods 300 and 320 may include one or more of the following steps. In step 301, the method may include receiving one or more digital images of a stained prostate tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0084] In step 303, the method may include receiving at least one label for one or more digital images, wherein at least one label contains an indication of the presence of cancer and a grade of cancer. The grade of cancer may comprise a primary and a secondary Gleason grade.

[0085] In step 305, the method may include partitioning each one or more digital images into a collection of clippings.

[0086] In step 307, the method may include detecting at least one region of fabric from a background of each one or more digital images to create a fabric mask and removing at least one cutout that is non-woven. Detection of fabric regions and removal of non-woven cutouts can be performed by thresholding methods based on color, color intensity, texture features, Otsu method, etc., followed by execution of a connected components algorithm. Thresholding can provide labels in tissue versus non-woven regions for one or more pixels of each incoming slide image, based on the thresholding method. The connected components algorithm can detect image regions or pixels connected to each other, to detect tissue versus non-woven regions in entire image regions, slide images, or slides. Detection of fabric regions and removal of non-woven cutouts can also be performed by segmentation algorithms such as k-means, graph cuts, Mask R-CNN, etc.

[0087] In step 309, the method may include training a machine learning model to predict whether cancer is present and a grade of cancer for one or more digital images. Training can be accomplished in a variety of ways, including, without limitation: A. Training a CNN to predict primary, secondary, and/or tertiary grades using a MIMLL model as disclosed above, for example, by treating each blade as a set of slices associated with multiple labels, from selecting blades that correspond to synoptic training labels, scoring each slice for its relevance to a label, and updating the CNN model weights against the associated label assignments. The trained CNN can be used to extract embeddings from each slice in a set of slides, to train a multitask aggregator (e.g., the aggregation model disclosed previously) to predict the presence of cancer, cancer Gleason grade group, and/or the primary, secondary and tertiary grade of each cutting or blade. Alternatively, the prediction output from each tile can be used and aggregated with hand-drawn post-processing methods, for example, having each tile vote for each grade and getting the majority of votes. B. Use a MIL model, classify each clipping as cancerous or benign, and transfer the classification labels to the “pure” cases where the primary/secondary/tertiary grades are the same grade. Train a clipping level classifier with the transferred labels using supervised learning. Refine the model using self-supervised learning as revealed in the weakly supervised learning module above. C. Extract features/embeddings from each snip and then use the multi-task aggregator (e.g. the aggregation model revealed above) to predict the presence of cancer, cancer Gleason grade group and/or grade primary, secondary and tertiary. Embeddings can be from a pre-trained CNN, random features, features from an unsupervised clustering model, SIFT, ORB, etc.

[0088] In step 321, the method may include receiving one or more digital images of a stained prostate sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, a cloud storage, a RAM, etc.

[0089] In step 323, the method may include partitioning one or more digital images into a collection of clippings.

[0090] In step 325, the method may include detecting at least one region of fabric from a background of a digital image to create a fabric mask and removing at least one cutout that is non-woven. Detection can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms such as k-means, graph cuts, Mask R-CNN, or any other suitable method.

[0091] In step 327, the method may include applying a trained machine learning model to the collection of clippings to predict the presence of cancer and a grade of cancer. The cancer grade may comprise a Gleason cancer grade group and/or a primary, secondary, and tertiary grade group.

[0092] In step 329, the method may include outputting a prediction, for example, to an electronic storage device.

[0093] Figure 4 is a flowchart illustrating an exemplary method for training and using a machine learning system for tumor quantification in prostate needle biopsies, in accordance with one or more exemplary embodiments of the present disclosure. Tumor quantification for prostate needle biopsies may consist of estimating the total and relative volumes of cancer for each cancer grade (e.g., a Gleason grade). Tumor quantification can play an important role in understanding the composition and severity of prostate cancer, and can be a common element in diagnostic pathology reports. Tumor size quantification can traditionally be performed manually with a physical ruler on a glass slide. Manual quantification in this manner can suffer from inaccuracy and consistency. Exemplary methods 400 and 420 can be used to train and use a machine learning system to quantify a tumor in prostate needle biopsies.

[0094] According to one embodiment, exemplary methods 400 and 420 may include one or more of the following steps. In step 401, the method may include receiving one or more digital images of a stained prostate tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0095] In step 403, the method may include receiving at least one real-value tumor quantification label for each of the one or more digital images, wherein the at least one real-value tumor quantification label contains an indication of a primary degree and a secondary degree. The label may also include a respective volume, a respective length and a respective size of the tumor in one or more digital images.

[0096] In step 405, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0097] In step 407, the method may include detecting at least one region of fabric from a background of each one or more digital images to create a fabric mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0098] In step 409, the method may include training a machine learning model to produce a cancer classification prediction, as described in exemplary method 300. Tumor quantification estimates may be estimated in a number of ways, including, without limitation: A. Count the number of degree indentations and geometrically estimate their volume and ratios in relation to the volume of benign tissue. B. Training a model using a blade-level classification module, for example, as described in exemplary method 300. Such a model may receive, as input, slice-level diagnostic features from a classification prediction model machine learning cancer model (e.g., the model trained in exemplary method 300) and output each tumor quantification metric using a real-valued regression model.

[0099] In step 421, the method may include receiving one or more digital images of a stained prostate sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0100] In step 423, the method may include partitioning one or more digital images into a collection of clippings.

[0101] In step 425, the method may include detecting at least one region of fabric from a background of a digital image to create a fabric mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0102] In step 427, the method may include applying a trained machine learning model to the collection of clippings to calculate a tumor quantification prediction. The forecast can be sent to an electronic storage device. Tumor quantification can be in the form of size metrics or percentages.

[0103] In step 429, the method may include outputting a prediction to an electronic storage device.

[0104] Figure 5 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict a cancer subtype, in accordance with one or more exemplary embodiments of the present disclosure. Many cancers have several subtypes. For example, in breast cancer, it can be determined whether a cancer is invasive or not, whether it is lobular or ductal, and whether several other attributes are present, such as calcifications. This method of predicting a cancer subtype may include prediction of multiple, non-exclusive categories, which may involve the use of multi-label learning.

[0105] According to one embodiment, exemplary methods 500 and 520 may include one or more of the following steps. In step 501, the method may include receiving one or more digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0106] In step 503, the method may include receiving a plurality of labels for one or more digital images, wherein the plurality of labels and/or a biomarker from the tissue sample. In a breast cancer sample, a relevant biomarker may be the presence of calcifications, presence or absence of cancer, ductal carcinoma in situ (DCIS), invasive ductal carcinoma (IDC), inflammatory breast cancer (IBC), Paget's disease breast cancer, angiosarcoma, phyllodes tumor, invasive lobular carcinoma, lobular carcinoma in situ and various forms of atypia. Labels may not necessarily be mutually exclusive and multiple subtypes may be observed simultaneously.

[0107] In step 505, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0108] In step 507, the method may include detecting at least one region of fabric from a background of each one or more digital images to create a fabric mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0109] In step 509, the method may include training a machine learning model to predict a form and/or subtype of cancer for each slice and/or slide. Machine learning model training can be carried out using the MIMLL model revealed above. The trained subtype prediction machine learning model can be refined using a blade-level prediction model (e.g., an aggregation model) as disclosed above. The slide-level prediction model can receive, as input, cut-level subtype predictions from a MIMLL model and output slide-level predictions indicating the presence of each cancer subtype.

[0110] In step 521, the method may include receiving one or more digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0111] In step 523, the method may include detecting at least one fabric region of a background of each one or more digital images to create a fabric mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0112] In step 525, the method may include partitioning one or more digital images into a collection of clippings and discarding any clippings that do not contain tissue.

[0113] In step 527, the method may include computing a cancer subtype prediction from the collection of clippings and sending the prediction to an electronic storage device.

[0114] Figure 6 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict a surgical margin in accordance with one or more exemplary embodiments of the present disclosure. When a tumor is surgically removed from a patient, it may be important to assess whether the tumor has been completely removed by analyzing the margin of tissue surrounding the tumor. The width of this margin and the identification of any cancerous tissue in the margin can play an important role in determining how a patient can be treated. Training a model to predict margin width and composition can take the form of multi-task multi-label learning.

[0115] According to one embodiment, exemplary methods 600 and 620 may include one or more of the following steps. In step 601, the method may include receiving one or more digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0116] In step 603, the method may include receiving a plurality of labels for one or more digital images, wherein the plurality of labels indicates a tumor margin and whether a margin is positive (e.g., tumor cells are found in the margin). , negative (e.g., the margin is completely free of cancer), or close (e.g., not definitely positive or negative).

[0117] In step 605, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0118] In step 607, the method may include detecting at least one region of fabric from a background of each one or more digital images to create a fabric mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0119] In step 609, the method may include training a machine learning model to predict a detection, presence, or grade of cancer, as disclosed above.

[0120] In step 621, the method may include receiving one or more digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0121] In step 623, the method may include detecting at least one region of fabric from a background of each one or more digital images to create a fabric mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0122] In step 625, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0123] In step 627, the method may include computing a surgical margin, tumor margin size, or predicting the tumor composition of the clippings. The method may also include outputting the forecast to an electronic storage device.

[0124] Figure 7 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict a bladder cancer biomarker, in accordance with one or more exemplary embodiments of the present disclosure. Bladder cancer is one of the most common cancers in the world. If bladder cancer is detected, the pathologist can also determine whether muscularis propria is present in any of the slides where bladder cancer is detected. Muscularis propria is a layer of smooth muscle cells that form a significant portion of the bladder wall. Detecting the presence or absence of muscularis propria is an important step in determining whether bladder cancer is invasive or not. The modality performs cancer detection and muscularis propria detection, but can be extended to any number of binary classification tasks.

[0125] According to one embodiment, exemplary methods 700 and 720 may include one or more of the following steps. In step 701, receiving one or digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0126] In step 703, the method may include receiving a plurality of labels for one or more digital images, wherein the plurality of labels indicates the presence or absence of cancer or the presence/absence of muscularis propria.

[0127] In step 705, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0128] In step 707, the method may include detecting at least one tissue region of a background of each one or more digital images to create a cloth mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0129] In step 709, the method may include training a machine learning model, for example, through the use of a weakly supervised learning module (as disclosed above) to train a MIMLL model and aggregating output scores indicating the presence/absence of cancer or the presence/absence of muscularis propria in several sections. Alternatively, an aggregation model can be trained to predict multiple labels from each image, slice, or slide, using embeddings from each slice.

[0130] In step 721, the method may include receiving one or more digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0131] In step 723, the method may include detecting at least one tissue region of a background of each one or more digital images to create a cloth mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0132] In step 725, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0133] In step 727, the method may include computing a muscularis propria prediction or invasive cancer prediction from the collection of clippings. The method may also include outputting the forecast to an electronic storage device.

[0134] Figure 8 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict a pancancer diagnosis, in accordance with one or more exemplary embodiments of the present disclosure. Although machine learning has been successfully used to create good models for predicting cancer in common cancer types, predictions for rare cancers are challenging due to the fact that there may be less training data. Another challenge is predicting where a cancer originated when it is metastatic, and sometimes determination is not possible. Knowing the tissue of origin can help guide cancer treatment. The modality enables pan-cancer prediction and source cancer prediction with the use of a single machine learning model. By training on many tissue types, the method can achieve an understanding of tissue morphology so that it can effectively generalize to rare cancer types where very little data may be available.

[0135] According to one embodiment, exemplary methods 800 and 820 may include one or more of the following steps. In step 801, receiving one or digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0136] In step 803, the method may include receiving a plurality of data denoting a type of tissue shown in each of the digital images received for a patient.

[0137] In step 805, the method may include receiving a set of binary labels for each digital image indicating the presence or absence of cancer.

[0138] In step 807, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0139] In step 809, the method may include detecting at least one tissue region of a background of each one or more digital images to create a cloth mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0140] In step 811, the method may include organizing at least one pan-cancer prediction output for a patient into a binary list. One element of the list may indicate the presence of any cancer and other elements of the list may indicate the presence of each specific type of cancer. For example, a prostate cancer sample may have a positive indicator for general cancer, a positive indicator for prostate cancer, and negative indicators for all other results corresponding to other tissues (e.g. lung, breast, etc. .). A patient for whom all slides are benign may have the label list containing all negative indicators.

[0141] In step 813, the method may include training a machine learning model to predict a binary vector for the patient. The machine learning model may comprise a MIMLL model as described above, wherein a weakly supervised learning module may train a MIMLL model. Additionally, the method may include aggregating pan-cancer prediction outputs from MIMLL across multiple slices, using an aggregation model (as disclosed above). Alternatively, an aggregation model can be trained to predict (multiple) pan-cancer prediction labels using embeddings from each snip.

[0142] In step 821, the method may include receiving one or more digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0143] In step 823, the method may include receiving a plurality of data denoting a type of tissue shown in each of the digital images received for a patient.

[0144] In step 825, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0145] In step 827, the method may include detecting at least one tissue region of a background of each one or more digital images to create a cloth mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0146] In step 829, the method may include computing a pan-cancer prediction using a trained machine learning model. The machine learning model may comprise the trained MIMLL model and/or aggregation model (as disclosed above). Exemplary outputs may include, without limitation: A. Pan-cancer prediction: The cancer presence output (or outputs) may be used to determine the presence of cancer regardless of tissue type, even for tissue types not observed during the training. This can be useful for rare cancers when there may not be enough data available to train a machine learning model. B. Cancer Source Prediction: Cancer subtype output (or outputs) can be used to predict a metastatic cancer source by identifying the largest subtype output. If one of the cancer outputs for a subtype is sufficiently larger than the tissue input type for the system, this may indicate to a pathologist that the output is the parent cancer. For example, if a bladder tissue sample shows cancer by the machine learning model (or models), but the subtype outputs prostate cancer, this may indicate to a pathologist that the cancer found in the bladder may be prostate cancer. metastasized rather than cancer that originated in the bladder.

[0147] In step 831, the method may include saving the prediction to an electronic storage device.

[0148] Figure 9 is a flowchart illustrating an exemplary method for training and using a machine learning system to predict organ toxicity, in accordance with one or more exemplary embodiments of the present disclosure. In preclinical animal studies for drug development, pathologists determine whether any toxicity is present, the form of toxicity, and/or the organs in which the toxicity may be found. The modality allows you to make these predictions automatically. One challenge with preclinical work is that one slide may contain multiple organs to save glass during preparation.

[0149] According to one embodiment, exemplary methods 900 and 920 may include one or more of the following steps. In step 901, receiving one or digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0150] In step 903, the method may include receiving a plurality of binary labels indicating the presence or absence of toxicity and/or a type or severity of toxicity.

[0151] In step 905, the method may include receiving a presence or absence of toxicity to at least one organ and/or its type or severity.

[0152] In step 907, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0153] In step 909, the method may include detecting at least one fabric region of a background of each one or more digital images to create a fabric mask and removing at least one cutout that is non-woven. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0154] In step 911, the method may include organizing at least one toxicity prediction output for a patient into a binary list. One element of the list may indicate the presence or type of any toxicity found on the slide, and other elements of the list may indicate the presence/type of toxicity in each organ.

[0155] In step 913, the method may include training a machine learning model to predict a binary vector for the patient. The machine learning model may comprise a MIMLL model as described above, wherein a weakly supervised learning module may train a MIMLL model. Additionally, the method may include aggregating MIMLL toxicity prediction outputs across multiple slices, using an aggregation model (as disclosed above). Alternatively, an aggregation model can be trained to predict toxicity prediction labels using embeddings from each clipping.

[0156] In step 921, the method may include receiving one or more digital images associated with a tissue sample on a digital storage device. The digital storage device may comprise a hard drive, a network drive, cloud storage, random access memory (RAM), etc.

[0157] In step 923, the method may include partitioning each of the one or more digital images into a collection of clippings.

[0158] In step 925, the method may include detecting at least one tissue region of a background of each one or more digital images to create a cloth mask and removing at least one cutout that is non-woven. Further processing can begin without the non-woven cutouts. This can be achieved in a number of ways, including, without limitation: thresholding methods, based on color, color intensity, texture features, Otsu's method, or any other suitable method, followed by execution of a connected components algorithm; and segmentation algorithms like k-means, graph cuts, Mask R-CNN or any other suitable method.

[0159] In step 927, the method may include computing a toxicity prediction using a trained machine learning model. The machine learning model may comprise the trained MIMLL model and/or aggregation model (as disclosed above). Exemplary outputs may include, but are not limited to: A. Presence of toxicity: A presence of toxicity output may be used to determine the presence and/or severity of toxicity, regardless of tissue type across the entire slide. B. Organ toxicity prediction: An organ toxicity output can be used to determine in which organ toxicity can be found.

[0160] In step 929, the method may include saving the toxicity prediction to an electronic storage device.

[0161] Figure 10 illustrates an exemplary connected components algorithm, according to one embodiment of the disclosure. The connected components algorithm can aggregate features into image regions. For example, thresholding may produce a binary image (e.g. black and white). An algorithm or connected components model can identify multiple regions in the image, for example, 3 regions (green, red, brown) at the pixel level. Each pixel can belong to a slice and a component (green, red, or brown) in the specific deployment using connected components. Aggregation can occur in several ways, including majority voting (e.g., for all clippings in the green component vote, resulting in green having a value of 1) or a learned aggregator (e.g., in which a feature vector can be extracted of each cut and input to a component aggregator module run for each component, so the cuts in the green component would be fed into a component aggregator module that can output a degree number). A CNN can produce a prediction (e.g., a number) for a snip, a feature vector for a snip that describes its visual properties, or both.

[0162] As shown in Figure 11, device 1100 may include a central processing unit (CPU) 1120. CPU 1120 may be any type of processor device including, for example, any type of special purpose device or a device general purpose microprocessor. As will be appreciated by those skilled in the art, the CPU 1120 may also be a single processor in a multicore/multiprocessor system, wherein such a system operates alone or in a cluster of computing devices operating in a server cluster or tower. The CPU 1120 may be connected to a data communications infrastructure 1110, e.g., a bus, message queue, network, or multi-core message passing scheme.

[0163] Device 1100 may also include a main memory 1140, e.g., random access memory (RAM), and may also include a secondary memory 1130. The secondary memory 1130, e.g., a read-only memory (ROM), it could be, for example, a hard disk drive or a removable storage drive. Such a removable storage unit may comprise, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory or the like. The removable storage drive in this example reads from and/or writes to a removable storage drive in a well-known manner. Removable storage may comprise a floppy disk, magnetic tape, optical disk, etc., which is read and written by the removable storage drive. As will be appreciated by those skilled in the art, such removable storage unit generally includes a computer usable storage medium having stored computer software and/or data.

[0164] In alternative implementations, secondary memory 1130 may include similar means for allowing computer programs or other instructions to be loaded into device 1100. Examples of such means may include a program cartridge and a cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket and other removable storage drives and interfaces, which allow software and data to be transferred from a removable storage drive to the device 1100.

[0165] The device 1100 may also include a communications interface (“COM”) 1160. The communications interface 1160 allows software and data to be transferred between the device 1100 and external devices. The communication interface 1160 may include a modem, a network interface (such as an Ethernet card), a communication port, a PCMCIA slot and card, or the like. The software and data transferred through the communication interface 1160 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by the communication interface 1160. These signals may be provided to the communication interface 1160 via a communication path of the device 1100, which may be implemented using, for example, wire or cable, optical fiber, a telephone line, a cellular telephone link, an RF link, or other communication channels.

[0166] The hardware elements, operating systems and programming languages of such equipment are conventional in nature, and those skilled in the art are presumed to be adequately familiar with the same. Device 1100 may also include input and output ports 1150 for connecting with input and output devices such as keyboards, mice, touch screens, monitors, monitors, etc. Of course, the various server functions can be deployed in a distributed manner across multiple platforms to distribute the processing load. Alternatively, servers can be implemented by appropriately programming a computer hardware platform.

[0167] Throughout this disclosure, references to components or modules generally refer to items that can logically be grouped together to perform a function or group of related functions. Similar reference numbers are generally intended to refer to the same or similar components. Components and modules can be implemented in software, hardware, or a combination of software and hardware.

[0168] The tools, modules and functions described above can be executed by one or more processors. "Storage" type media may include any or all of computers' tangible memory, processors, or the like, or associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, that can provide storage at any time for programming of software.

[0169] The software may be communicated over the Internet, a cloud service provider, or other telecommunications networks. For example, communications may allow software to be loaded from one computer or processor to another. As used herein, unless restricted to tangible, non-transitory “storage” media, terms such as computer or machine readable “medium” refer to any medium that participates in providing instructions to a processor for execution.

[0170] The previous general description is only exemplary and explanatory, and not restrictive of the disclosure. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. The specification and examples are intended to be considered as examples only.

Claims (20)

1. Computer-implemented method for processing electronic slide images corresponding to a tissue sample, the method being characterized by the fact that it comprises: receiving one or more electronic slide images associated with a tissue sample, wherein the sample of tissue is associated with a patient and/or medical case; partitioning a first slide image of the one or more electronic slide images into a plurality of slices; determine a prediction, using a machine learning prediction model, for at least one label for the one or more electronic slide images, wherein the machine learning prediction model was generated by processing a plurality of training images by: receiving a plurality of synoptic annotations comprising one or more labels for each of the plurality of training images; partitioning one of the plurality of training images into a plurality of training slices for the plurality of training images; segmenting at least one region of tissue from a background of the one or more electronic slide images to create a training tissue mask; removing at least one of the plurality of cutouts that are detected as nonwovens; and using the machine learning prediction model under weak supervision to infer at least one multi-label clipping level prediction using at least one label from the plurality of synoptic annotations. 2. Computer-implemented method according to claim 1, characterized by the fact that the plurality of cutouts that are determined to be non-woven are further determined to be a background of the fabric sample. 3. The computer-implemented method of claim 1, further comprising: detecting a plurality of tissue regions from the first slide image and/or a plurality of records by segmenting the tissue regions from a background of the one or more electronic slide images to generate a fabric mask. 4. The computer-implemented method of claim 3, further comprising: upon segmenting the tissue regions from the background, generating the tissue mask, wherein the segmentation uses feature-based thresholds of color/intensity and/or texture. 5. The computer-implemented method of claim 1, wherein the plurality of training images comprises a plurality of electronic slide images and a plurality of target labels. 6. The computer-implemented method of claim 1, wherein processing the plurality of training images comprises: receiving a collection of digital images associated with at least one training tissue sample, wherein the collection of digital images comprises a plurality of images of electronic training slides; receiving a plurality of synoptic annotations comprising one or more labels for each of the plurality of electronic training slide images; partitioning one of the plurality of training electronic slide images into a plurality of training slices for the plurality of training electronic slide images; and segmenting at least one region of tissue from a background of the one or more electronic slide images to create a training tissue mask. 7. The computer-implemented method of claim 6, wherein training the machine learning prediction model under weak supervision comprises using at least one multiple instance learning (MIL), multiple instance learning multi-label learning (MIMLL), self-supervised learning and unsupervised clustering. 8. The computer-implemented method of claim 6, wherein processing the plurality of training images to generate the machine learning prediction model further comprises: receiving a plurality of predictions or a plurality of vectors of at least one feature of a weakly supervised slice-level learning module for the plurality of training slices; training a machine learning model to receive, as input, the plurality of predictions or the plurality of vectors of the at least one feature of the weakly supervised slice-level learning module for the plurality of training slices; and providing a plurality of labels for a slide or a patient sample, using the plurality of training slices. 9. The computer-implemented method of claim 8, wherein at least one of the plurality of labels is binary, categorical, ordinal, or real-valued. 10. The computer-implemented method of claim 8, wherein training the machine learning model to receive, as input, the plurality of predictions or the plurality of vectors of the at least one feature of the Weakly supervised slice-level learning for the plurality of training slices comprises a plurality of image features. 11. The computer-implemented method of claim 1, wherein the trained machine learning prediction model predicts at least one label using at least one unseen slide. 12. System for processing images of electronic slides corresponding to a tissue sample, wherein the system is characterized by the fact that it comprises: at least one memory storing instructions; and at least one processor configured to execute instructions for performing operations comprising: receiving one or more electronic slide images associated with the tissue sample, wherein the tissue sample is associated with a patient and/or medical case; determining a prediction, using a machine learning prediction model, for at least one label for the one or more electronic slide images, the machine learning prediction model generated by processing a plurality of slide images training electronics by: receiving a plurality of synoptic annotations comprising one or more labels for each of the plurality of images from electronic training slides; partitioning one of the plurality of training electronic slide images into a plurality of training slices for the plurality of training electronic slide images; segmenting at least one region of tissue from a background of the one or more electronic slide images to create a training tissue mask; removing at least one of the plurality of training cutouts that are detected as nonwovens; and using the machine learning prediction model under weak supervision to infer at least one multi-label clipping level prediction using at least one label from the plurality of synoptic annotations. 13. System according to claim 12, characterized by the fact that the plurality of training cuts that are determined to be non-woven are further determined to be a background of the fabric sample. 14. System according to claim 12, characterized by the fact that detecting, by the trained machine learning system, a plurality of tissue regions of the plurality of cutouts by segmenting the tissue regions from a background of the one or plus images of electronic slides to generate a fabric mask. 15. System, according to claim 14, characterized by the fact that it further comprises: by segmenting the tissue regions from the background, generating the tissue mask, wherein the segmentation uses thresholds based on color features/ intensity and/or texture. 16. The system of claim 12, wherein the plurality of training electronic slide images comprises a plurality of electronic slide images and a plurality of target labels. 17. The system of claim 12, wherein processing the plurality of training electronic slide images comprises: receiving a collection of digital images associated with at least one training tissue sample, wherein the collection of digital images comprises the plurality of images from electronic training slides. 18. System according to claim 12, characterized by the fact that the use of the machine learning prediction model under weak supervision comprises the use of at least one of MIL, MIMLL, self-supervised learning and unsupervised clustering. 19. The system of claim 17, wherein processing the plurality of training electronic slide images to generate the machine learning prediction model further comprises: receiving a plurality of predictions or a plurality of vectors of at least one feature of a weakly supervised slice-level learning module for the plurality of training slices; training a machine learning model to receive, as an input, the plurality of predictions or the plurality of vectors of the at least one feature of the weakly supervised slice-level learning module for the plurality of training slices; and providing a plurality of labels for a slide or a patient sample, using the plurality of training slices. 20. A non-transitory computer-readable medium that stores instructions that, when executed by a processor, cause the processor to execute a method for processing electronic slide images corresponding to a tissue sample, wherein the method comprises: receiving one or more electronic slide images associated with the tissue sample associated with a patient and/or a medical case; partitioning a first slide image of the one or more electronic slide images into a plurality of slices; determining whether any of the plurality of cutouts corresponds to nonwoven fabric; removing any of the plurality of cutouts that are determined to be nonwoven; and determining a prediction, using a machine learning prediction model, for at least one label corresponding to the patient and/or medical case for the one or more electronic slide images, wherein the learning prediction model was generated by processing a plurality of training images by: predicting a plurality of labels for a slide or a patient sample, using the plurality of training slices; segmenting at least one region of tissue from a background of the one or more electronic slide images to create a training tissue mask; removing at least one of the plurality of cutouts that are detected as nonwovens; and using the machine learning prediction model under weak supervision to infer at least one multi-label clipping level prediction using at least one label from a plurality of synoptic annotations.