BR112022013004B1 - COMPUTER-IMPLEMENTED METHODS FOR PROCESSING ELECTRONIC IMAGES USING IN VITRO SAMPLES AND SYSTEM FOR PROCESSING ELECTRONIC IMAGES - Google Patents

Abstract

SYSTEMS AND METHODS FOR ANALYZING ELECTRONIC IMAGES FOR QUALITY CONTROL. The present invention relates to systems and methods that are described to receive a digital image corresponding to a target sample associated with a category of pathology, determine a quality control (QC) machine learning model to predict a quality designation based on one or more artifacts, provide the digital image as an input to the QC machine learning model, receive the quality designation for the digital image as an output of the machine learning model, and issue the quality designation of the digital image. A quality assurance (QA) machine learning model can predict a disease designation based on one or more biomarkers. The digital image can be fed to the QA model which can issue a disease designation. An external designation can be compared with the disease designation and a comparison result can be output.

Description

Related Request(s)

[0001] This application claims priority to U.S. Provisional Application No. 62/957,517 filed on January 6, 2020, the complete description of which is incorporated herein by reference in its entirety.

Field of Invention

[0002] Various embodiments of the present disclosure generally relate to image-based sample quality control (QC) and related image processing methods for quality assurance (QA). More specifically, particular embodiments of the present disclosure relate to systems and methods for processing images to provide QC feedback before the sample is evaluated, and also to provide QA to supplement sample evaluation and diagnosis.

Fundamentals of Invention

[0003] Laboratory quality control and digital pathology quality assurance are crucial to the successful entry, processing, diagnosis and archiving of patient samples. Current methods for quality assurance include (1) second review of cancer cases diagnosed for the first time, (2) periodic reviews of discordant or changed diagnoses by a quality assurance committee, or (3) random review of a subset of cases. These methods are not complete, mainly retrospective and manual.

[0004] Conventional techniques for quality control and assurance can be improved with more systematic quality control and assurance. However, systematic quality assurance is currently impractical and inefficient because it would require duplicated efforts by two pathologists. Such duplicated efforts would be prohibitive in terms of resources and time. As a result, there is a desire for viable systemic quality control and assurance.

[0005] The previous general description and the following detailed description are exemplary and explanatory only and are not restrictive of the description. The description of fundamentals provided in this document is intended to generically present the context of the description. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted as prior art, or suggestions of prior art, by inclusion in this section.

Summary of the Invention

[0006] In accordance with certain aspects of the present disclosure, systems and methods are described for implementing QC and QA measures for digital pathology images.

[0007] A method for issuing quality designations includes receiving a digital image corresponding to a target sample associated with a pathology category, wherein the digital image is an image of human tissue and/or an image algorithmically generated to replicate human tissue, determine a quality control (QC) machine learning model, the QC machine learning model generated by processing a plurality of training images, associated with the pathology category, to predict a quality designation based on a or more artifacts, provide the digital image as an input to the QC machine learning model, receive the quality designation for the digital image as an output from the machine learning model, and submit the image quality designation digital.

[0008] A method for verifying disease designations includes receiving at least one digital image corresponding to a target sample associated with a category of pathology, wherein the digital image is an image of human tissue and/or an image generated algorithmically to replicate tissue human, determine a quality assurance (QA) machine learning model, the QA machine learning model generated by processing a plurality of training images, from the pathology category, to predict a disease designation with based on one or more biomarkers, provide the digital image as input to the QA machine learning model, receive disease designation for the digital image as an output from the QA machine learning model, receive an external designation for the digital image, compare the disease designation with the external designation, and output a comparison result based on the comparison of the disease designation with the external designation.

[0009] A system for verifying disease designations includes memory storage instructions and a processor executing the instructions to perform a process including receiving at least one digital image corresponding to a target sample associated with a pathology category, wherein the digital image is an image of human tissue and/or an image generated algorithmically to replicate human tissue, determine a quality assurance (QA) machine learning model, the QA machine learning model generated by processing a plurality of images of training, from the pathology category, to predict a disease designation based on one or more biomarkers, provide the digital image as an input to the QA machine learning model, receive the disease designation for the digital image as an output from the QA machine learning model, receive an external designation for the digital image, compare the disease designation with the external designation, and output a comparison result based on the comparison of the disease designation with the external designation .

Brief Description of the Drawings

[0010] The attached drawings, which are incorporated and form part of this specification, illustrate several exemplary embodiments and, together with the description, serve to explain the principles of the described embodiments.

[0011] Figure 1A illustrates an exemplary block diagram of a system and network for implementing QA / QC tools with digital images, in accordance with an exemplary embodiment of the present description.

[0012] Figure 1B illustrates an exemplary block diagram of a machine learning model according to an exemplary embodiment of the present description.

[0013] Figure 2A is a flowchart illustrating an exemplary method for using a QC machine learning model, in accordance with an exemplary embodiment of the present description.

[0014] Figure 2B is a flowchart illustrating an exemplary method for using a QA machine learning model, in accordance with an exemplary embodiment of the present description.

[0015] Figure 3 illustrates an exemplary block diagram of a training module according to an exemplary embodiment of the present description.

[0016] Figure 4 illustrates a diagram for implementing QA/QC analysis at different stages of a diagnosis, according to an exemplary embodiment of the present description.

[0017] Figure 5 is a flowchart of an exemplary embodiment of a QC implementation, according to an exemplary embodiment of the present description.

[0018] Figure 6 is a flowchart of an exemplary embodiment of a QA implementation, according to an exemplary embodiment of the present description.

[0019] Figure 7 is a flowchart of an exemplary embodiment of another QA implementation, in accordance with an exemplary embodiment of the present description.

[0020] Figure 8 represents an exemplary system that can perform techniques presented in this document.

Detailed Description of the Invention

[0021] Reference will now be made in detail to the exemplary embodiments of the present description, examples of which are illustrated in the attached drawings. Wherever possible, the same reference numbers will be used in all drawings to refer to the same or similar parts.

[0022] The systems, devices and methods described in this document are described in detail by means of examples and with reference to the figures. The examples discussed here are examples only and are provided to assist in explaining the apparatus, devices, systems and methods described herein. None of the features or components shown in the drawings or discussed below should be considered mandatory for any specific implementation of any such device, system or method unless specifically designated as mandatory.

[0023] Furthermore, for any methods described, regardless of whether the method is described in conjunction with a flowchart, it should be understood that, unless otherwise specified or required by the context, any explicit or implicit ordering of steps performed in the Executing a method does not imply that these steps must be executed in the order presented, but they can be executed in a different order or in parallel.

[0024] As used in this document, the term “exemplary” is used in the sense of “example”, rather than “ideal”. Furthermore, the terms “one” and “one” here do not denote a quantity limitation, but denote the presence of one or more of the aforementioned items.

[0025] Pathology refers to the study of diseases. More specifically, pathology refers to the performance of tests and analyzes that are used to diagnose diseases. For example, tissue samples may be placed on slides to be viewed under a microscope by a pathologist (e.g., a doctor who specializes in analyzing tissue samples to determine whether abnormalities exist). That is, pathology samples can be cut into multiple sections, stained, and prepared as slides for a pathologist to examine and make a diagnosis. When unsure of a diagnostic finding on a slide, the pathologist may order additional cut levels, staining, or other tests to gather more information from the tissue. The technician(s) can then create a new slide(s) that can contain additional information for the pathologist to use in diagnosis. This process of creating additional slides can be time-consuming, not only because it can involve retrieving the block of tissue, cutting it to make a new slide, and then staining the slide, but also because it can be grouped together for multiple requests. This can significantly delay the pathologist's final diagnosis. Additionally, even after the delay, there may be no guarantee that the new slide(s) will have enough information to provide a diagnosis.

[0026] Pathologists can evaluate cancer and other disease pathology slides alone. This description presents a consolidated workflow to improve the diagnosis of cancer and other diseases. The workflow can integrate, for example, slide evaluation, tasks, image analysis and cancer detection artificial intelligence (AI), annotations, queries and recommendations on a workstation. In particular, the present disclosure describes several exemplary AI tools that can be integrated into the workflow to streamline and improve a pathologist's work.

[0027] For example, computers can be used to analyze an image of a tissue sample to quickly identify whether the quality of the tissue sample is sufficient for its intended purpose (i.e., quality control) and can additionally analyze the image of the tissue sample. tissue sample to determine a result to complement the determination made by another entity, such as a pathologist (i.e., quality assurance). Thus, the process of obtaining stained slides and testing can be done automatically before being reviewed by a pathologist. When paired with automatic slide quality review and result determination, this can provide a fully automated slide preparation and evaluation system in parallel with pathologist review. This automation has at least the benefits of (1) minimizing the amount of time spent by a pathologist determining the findings of a slide using slides that do not meet a quality threshold, (2) minimizing the (average total) time since sample acquisition to diagnosis, avoiding additional time performing manual analysis or questionable slides, (3) reduce the amount of repeated tissue evaluations by automatically determining slide quality before it is presented for pathologist review, (4) reduce the cost of repeat biopsies and pathologist review by providing slides that meet a quality threshold, (5) eliminate or mitigate the need for a second or subsequent pathologist diagnostic review, (6) reduce the likelihood of an incorrect diagnosis, (7) increase the probability of a proper diagnosis based on a double confirmation, and/or (8) identify or verify correct properties (e.g., belonging to a sample type) of a digital pathology image.

[0028] The process of using computers to assist pathologists is called computational pathology. Computing methods used for computational pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI can include, but is not limited to, deep learning, neural networks, classifications, clustering, and regression algorithms. By using computational pathology, lives can be saved by helping pathologists improve their diagnostic accuracy, reliability, efficiency, and accessibility. For example, computational pathology can be used to aid in the detection of suspected cancer slides, thereby allowing pathologists to verify and confirm their initial assessments before providing a final diagnosis.

[0029] Histopathology refers to the study of a sample that has been placed on a slide. For example, a digital pathology image may be comprised of a digitized image of a microscope slide containing the sample (e.g., a smear). One method a pathologist can use to analyze an image on a slide is to identify nuclei and classify whether a nucleus is normal (e.g., benign) or abnormal (e.g., malignant). To assist pathologists in identifying and classifying nuclei, histological stains can be used to make cells visible. Many dye-based staining systems have been developed, including periodic acid-Schiff reaction, Masson's trichrome, nissl and methylene blue, and Hemotoxylin and Eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method, with hematoxylin staining the cell nucleus blue, eosin staining the cytoplasm and extracellular matrix pink, and other tissue regions taking on variations of these colors. In many cases, however, H&E-stained histological preparations do not provide sufficient information for a pathologist to visually identify biomarkers that can aid in diagnosis or guide treatment. In this situation, techniques such as immunohistochemistry (IHC), immunofluorescence, in situ hybridization (ISH) or fluorescence in situ hybridization (FISH) can be used. IHC and immunofluorescence involve, for example, the use of antibodies that bind to specific antigens in tissues, allowing visual detection of cells expressing specific proteins of interest, which can reveal biomarkers that are not reliably identifiable to pathologists trained with based on the analysis of slides stained with H&E. ISH and FISH can be employed to assess gene copy number or the abundance of specific RNA molecules, depending on the type of probes employed (e.g., DNA probes for gene copy number and RNA probes for expression assessment). of RNA). If these methods also fail to provide sufficient information to detect some biomarkers, tissue genetic testing can be used to confirm whether a biomarker is present (e.g., overexpression of a specific protein or gene product in a tumor, amplification of a particular gene in cancer).

[0030] A digitized image can be prepared to show a stained microscope slide, which can allow a pathologist to manually view the image on a slide and estimate a number of abnormal stained cells in the image. However, this process can be time-consuming and can lead to errors in identifying abnormalities, as some abnormalities are difficult to detect. Computational processes using machine learning models and devices can be used to assist pathologists in detecting abnormalities that would otherwise be difficult to detect. For example, AI can be used to predict biomarkers (such as overexpression of a protein and/or gene product, amplification or mutations of specific genes) from salient regions within digital images of tissues stained using H&E and other dye-based methods. . Tissue images can be whole slide images (WSI), images of tissue cores within microarrays, or selected areas of interest within a tissue section. Using staining methods like H&E, these biomarkers can be difficult for humans to visually detect or quantify without the help of additional testing. Using AI to infer these biomarkers from digital tissue images has the potential to improve patient care, as well as be faster and less expensive.

[0031] As described above, the computational pathology processes and devices of the present disclosure can provide an integrated platform that allows a fully automated process, including data ingestion, processing and viewing of digital pathology images via a web browser or another user interface, while integrating with a laboratory information system (LIS). Additionally, clinical information can be aggregated using cloud-based data analytics of patient data. Data can come from hospitals, clinics, field researchers, etc., and can be analyzed by machine learning, computer vision, natural language processing, and/or statistical algorithms to do real-time monitoring and prediction of health patterns in multiple levels of geographic specificity.

[0032] Implementations of the subject matter described include systems and methods for systematic QC for pathology sample preparation and QA for diagnostic quality, throughout a histopathology workflow. With an automated and systematic QC and QA mechanism, quality can be guaranteed throughout the entire workflow of each case. Systematic QC and QA have the potential to provide efficiency and improve diagnostic quality.

[0033] In a detection-based workflow and quality system described herein, input to the system may include one or several digitized pathology images and any additional relevant inputs. System outputs may include global and/or local information about the sample, with respect to quality control and assurance.

[0034] According to one implementation, a QC machine learning model can be generated to determine the quality of digitized pathology images. Qualification may include generating a quality designation for each digitized pathology image where the quality designation is either an approved designation or a rejected designation. An approved designation may result from a QC machine learning model output that indicates that no artifacts were found in the digitized pathology image or that no artifacts found exceed an applicable artifact threshold. A rejected assignment may result from QC machine learning model output that indicates that an artifact or an artifact quantity above an applicable artifact threshold was found.

[0035] The QC machine learning model can be trained based on supervised, partially supervised or unsupervised training images from the same pathology category as the respective digitized pathology images input into the QC machine learning model . Pathology categories may include, but are not limited to, histology, cytology, frozen section, immunohistochemistry, or the like. The QC machine learning model can detect the presence, absence, or extent of artifacts, including scan results or errors (e.g., blurring, missing tissue, lines, etc.) and/or from tissue preparation (e.g., missing or folded fabric, bubbles, cracks in glass, scratches, dust, pen, excess dye, etc.).

[0036] According to one implementation, a QA machine learning model may be generated to generate disease designations based on digitized pathology images received as inputs to the model. Disease designations may be one or more of cancer detection, cancer grade, cancer origin, diagnosis, presence or absence of a microorganism, specimen type, cancer type, cancer status, tumor size, level risk and/or degree of injury. Disease assignments from the QA machine learning model can be compared with an external assignment, and the result of the comparison can be a match or mismatch assignment (for example, if no match is detected). If a discrepancy designation is determined, a discrepancy level can be determined and, based on the discrepancy level, either a warning or a trigger can be generated. The external assignment may be based on an assignment made by a pathologist, a third-party institution, and/or a system other than the QA machine learning model.

[0037] A trigger based on a level of discrepancy may initiate a manual reassignment of the digital pathology image or generation of an alert that informs applicable individuals or entities of the discrepancy.

[0038] The QA machine learning model can be trained based on supervised, partially supervised, or unsupervised training images, biomarkers, patient information, and applicable assignment outputs. Inputs to a generated QA machine learning mode can be digital pathology images from one or more pathology categories such as histology, cytology, frozen section, or immunohistochemistry.

[0039] Notifications, visual indicators and/or reports may be generated based on the output of QC and/or QA machine learning models as described in this document. Reports may be based on an individual scanned image or a plurality of scanned images either over a certain period of time or generally retrospectively.

[0040] The systems described in this document can be implemented locally (e.g., on premises) and/or remotely (e.g., cloud-based). Systems may or may not have user interface(s) and workflows that pathologists can access directly (e.g., a downstream oncologist may be flagged that there has been a discrepancy, notifications to the care team, etc.). Consequently, the implementations described in this document can be used as stand-alone operations, or used within a digital workflow. In use, the systems described can perform QA/QC analysis and delivery before or after a diagnosis is received (e.g., by a pathologist). The implementations described in this document can be performed in real time (for example, if slides are immediately sent from the scanner to a QA or QC machine learning model) or can be performed in batch mode where discrepancies can be recorded and/or reported.

[0041] Although the subject matter described is described as implemented based on oncology applications, these implementations may be used for other forms of disease (e.g., infectious diseases, detection processes, or the like that use digital images for diagnostic purposes. In addition In addition to providing QC/QA benefits, the described implementations can be used to train healthcare professionals (e.g., slide technicians, pathologists, etc.) to practice slide generation and/or diagnostic determination, while reducing the risk of harm to the patient.

[0042] Figure 1A illustrates a block diagram of a system and network for determining sample property or image property information pertaining to digital pathology image(s), using machine learning, in accordance with an exemplary embodiment of the present description. As described in this document, the system and network in Figure 1A may include a machine learning model with QA/QC tools to evaluate the quality of digital images and/or determine disease diagnosis to ensure the quality of an external diagnosis .

[0043] Specifically, Figure 1A illustrates an electronic network 120 that can be connected to servers in hospitals, laboratories and/or doctors' offices, etc. For example, medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124 and/or laboratory information systems 125, etc., may be connected to an electronic network 120, such as the Internet, via of one or more computers, servers and/or portable mobile devices. According to one implementation, the electronic network 120 may also be connected to server systems 110, which may include processing devices that are configured to implement a machine learning model 100, in accordance with an exemplary embodiment of the subject matter described.

[0044] Medical servers 121, hospital servers 122, clinical trials servers 123, research laboratory servers 124, and/or laboratory information systems 125 may create or obtain images of one or more categories of pathology specimens, including sample ( patient cytology(s), histopathology sample(s), slide(s) of cytology sample(s), histology, immunohistochemistry, digitized images of slide(s) ) of the histopathological sample(s), or any combination thereof. Medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124, and/or laboratory information systems 125 may also obtain any combination of patient-specific information, such as age, medical history, treatment history of cancer, family history, information from previous biopsy or cytology, etc. Medical servers 121, hospital servers 122, clinical trial servers 123, research laboratory servers 124, and/or laboratory information systems 125 may transmit scanned slide images and/or patient-specific information to server systems 110 over the network electronics 120. The server system(s) 110 may include one or more storage devices 109 for storing images and data received from at least one of the medical servers 121, hospital servers 122, assay servers clinical servers 123, research laboratory servers 124, and/or laboratory information systems 125. Server systems 110 may also include processing devices for processing images and data stored in storage devices 109. Server systems 110 may further include a or more machine learning tools or capabilities through the machine learning model 100. For example, the processing devices may include QA tools and/or QC tools (collectively described as QA/QC tools 101), as per shown as machine learning model 100, in accordance with one embodiment. Alternatively or in addition, the present disclosure (or portions of the system and methods of the present disclosure) may be performed on a local processing device (e.g., a laptop).

[0045] Medical servers 121, hospital servers 122, clinical trials servers 123, research laboratory servers 124, and/or laboratory information systems 125 refer to systems used by pathologists to review slide images. In hospital settings, tissue type information can be stored on a LIS 125.

[0046] Figure 1B illustrates an exemplary block diagram of a machine learning model 100 for determining sample property or image property information regarding digital pathology image(s) using machine learning.

[0047] Specifically, Figure 1B depicts components of the machine learning model 100, according to one embodiment. For example, the machine learning model 100 may include a QA/QC tool 101, a data ingestion tool 102, a slide input tool 103, a slide digitizer 104, a slide manager 105, a slide storage 106 and a visualization application tool 108. For clarification, the machine learning model 100 shown in Figures 1A and 1B is a previously trained and generated machine learning model (e.g., a QA machine learning model, a QC machine learning model, etc.). Additional description is provided in this document for training and generating different types of machine learning models that can be used as machine learning model 100.

[0048] The QA/QC tool 101, as described herein, refers to a process and system for determining a characteristic (e.g., artifact, biomarker, etc.) to determine image quality and/or designations of diseases pertaining to the digital pathology image(s) using a machine learning model, such as a QA machine learning model or a QC machine learning model. The QA/QC tool 101 may include a plurality of machine learning models or may load one machine learning model at a time.

[0049] Data ingestion tool 102 refers to a process and system for facilitating the transfer of digital pathology images to the various tools, modules, components and devices of machine learning module 100 that are used to characterize and processing digital pathology images, according to an exemplary embodiment.

[0050] Slide input tool 103 refers to a process and system for digitizing pathology images and converting them into a digital form, in accordance with an exemplary embodiment. The slides can be digitized with the slide digitizer 104 , and the slide manager 105 can process the images on the slides into digitized pathology images and store the digitized images in the store 106 .

[0051] Visualization application tool 108 refers to a process and system for providing a user (e.g., pathologist) with characterization or image property information pertaining to the digital pathology image(s), according to an exemplary embodiment. Information can be provided through various output interfaces (e.g. a screen, a monitor, a storage device and/or a web browser, etc.). As an example, the visualization application tool 108 may apply an overlay layer over the digital pathology image(s), and the overlay layer may highlight key areas of consideration. The overlay layer may be or may be based on the output of the QA/QC tool 101 of the machine learning module 100. As discussed herein, the visualization application tool 108 may be used to show artifacts based on the output of a QC machine learning model and/or can be used to show biomarkers based on the output of a QA machine learning model.

[0052] The QA/QC tool 101, and each of its components, may transmit and/or receive scanned slide images and/or patient information to server systems 110, medical servers 121, hospital servers 122, clinical trials 123, research laboratory servers 124, and/or laboratory information systems 125 over a network 120. Additionally, server systems 110 may include storage devices for storing images and data received from at least one QA/QC tool 101, data ingestion tool 102, slide input tool 103, slide digitizer 104, slide manager 105, and visualization application tool 108. Server systems 110 may also include processing devices for process images and data stored on storage devices. Server systems 110 may further include one or more machine learning tools or resources, for example due to processing devices. Alternatively or additionally, the present disclosure (or portions of the system and methods of the present disclosure) may be performed on a local processing device (e.g., a laptop).

[0053] The QA/QC tool 101 may provide output from the machine learning module 100 (e.g., a quality designation, disease designation, comparison result, etc.). As an example, the slide input tool 103 and the data ingestion tool 102 may receive inputs to the machine learning module 100 and the QA/QC tool 101 may identify artifacts and/or biomarkers in the slides based on the data, and output an image highlighting the artifacts and/or biomarkers via the visualization application tool 108.

[0054] Any of the above devices, tools, and modules may be located on a device that may be connected to an electronic network 120, such as the Internet or a cloud service provider, through one or more computers, servers, and / or portable mobile devices.

[0055] Figure 2A shows a flowchart 200 for issuing quality designations for digital images, according to exemplary implementations of the subject matter described. At 202 of Figure 2A, a digital image corresponding to a target sample associated with a pathology category can be received. The digital image may be a digital pathology image captured using the slide input tool 103 of Figure 1B. At 204, a QC machine learning model may be received (e.g., at machine learning module 100). The QC machine learning model may be trained by processing a plurality of training images that are of the same pathology category as the digital image received in 202. The pathology categories may include, but are not limited to, histology, cytology, section frozen or immunohistochemistry. At 206, the digital image of 202 can be provided to the QC machine learning model as an input to the model. One or more other attributes may also be provided as an input to the QC machine learning model. The one or more other attributes may include, but are not limited to, a pathology category, a slide type, a glass type, a tissue type, a tissue region, a chemical used, an amount of dye, time applied, type of scanner, date or similar. At 208, the QC machine learning model can produce a quality designation for the digital image. The quality designation may be a pass or a reject and may also include a scale such as a pass scale and/or a reject scale that provides a more accurate indication of the quality of the digital image provided at 202 and as input to the model. of QC machine learning at 206. At 210, the quality designation may be output as a data signal, a report, a notification, an alert, a visual output (e.g., via visualization application tool 108), or similar.

[0056] As shown in Figure 2A, a digital image corresponding to a target sample associated with a pathology category may be received at 202. The target sample may be a tissue sample from a biopsy or otherwise recovered from a patient. . The target sample may be retrieved during a surgical procedure in which a portion of a patient's tissue is retrieved from the patient's body for analysis. The target sample can be a portion or subset of the total amount of tissue extracted from patients, such that multiple sample slides can be generated from tissue extracted from a single procedure.

[0057] The target sample may be associated with at least one category of pathology, such as histology, cytology, frozen section or immunohistochemistry, as described herein. According to one implementation, the pathology category and other image information about the target digital image or sample may also be received. Image information may include, but is not limited to, a slide type, a glass type, a tissue type, a tissue region, a chemical used, and an amount of dye.

[0058] At 204, a QC machine learning model may be received. The QC machine learning model may be trained and generated in the machine learning module 100 or may be externally trained and generated and received in the module machine learning model 100. The QC machine learning model may be trained by processing a plurality of training images that are of the same pathology category as the digital image received in 202. Pathology categories may include, but are not limited to histology, cytology, frozen section or immunohistochemistry. The QC machine learning model can be instantiated using deep learning. The QC machine learning model can use information about an entire digital pathology image, for example, the type of sample, the overall quality of the sample section, the overall quality of the glass slide itself, or characteristics of the tissue morphology , and determine a general quality designation for digital pathology images.

[0059] The QC machine learning model can detect artifacts associated with a digital image based on an analysis of the properties of the digital image. Artifacts can be detected based on the application of one or more digital or mathematical filters, scanning techniques (e.g. pixel-by-pixel scanning), pixel comparison technique (e.g. comparing one set of pixels with another) or similar. Artifacts that the QC machine learning model can be trained to identify may include, but are not limited to, artifacts including scan results or errors (e.g., blurring, missing tissue, lines, etc.) and/or from the preparation of fabric (e.g. missing or folded fabric, bubbles, cracks in glass, scratches, dust, pen, excess dye, etc.).

[0060] To generate the QC machine learning model in 204, a training data set including a large plurality of digital pathology images of pathology samples (e.g., histology, cytology, immunohistochemistry, etc.) can be received. Digital pathology images may be digital images generated from physical biopsy specimens as described herein, or they may be images that are algorithmically generated to replicate human tissue, for example, by a rendering system or a generative adversarial model. Information associated with the image or sample (e.g., slide type, glass type, tissue type, tissue region, chemical used, amount of dye, application time, scanner type, date, etc.) can be also received as part of the training data set. Additionally, as part of training the QC machine learning model, each image can be paired with output information about the known or assumed quality of the corresponding digital image or sample. This output information may include known or assumed artifact information, expected quality information, or the like. For example, a certain image may be provided as a training image and may be known to have folded fabric as part of the digital image. The image and an indication of the presence, location and/or extent of the fold can be provided and paired with the image. The QC machine learning model may learn from a plurality of such training images and associated information, so that the parameters of the QC machine learning model are trained (e.g., tuned) to be able to detect the presence, absence, or degree of an artifact detected in each training image. Although supervised training is provided as an example, it is understood that QC machine learning model training can be partially supervised or unsupervised.

[0061] The training data set including the digital pathology images, the information associated with the image or sample, and/or the output information, may be generated and/or provided by one or more of the systems 110, medical servers 121 , hospital servers 122, clinical trial servers 123, research laboratory servers 124, and/or laboratory information systems 125. Images used for training may come from real sources (e.g., humans, animals, etc.) or may come from from synthetic sources (e.g. graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with a variety of stains, such as (but not limited to) H&E, hematoxylin alone, IHC, molecular pathology, etc.; and/or (b) tissue samples scanned from a 3D imaging device such as microCT.

[0062] The QC machine learning model can be generated based on the application of the digital pathology images with, optionally, the associated information paired with the output information applied by a machine learning algorithm. The machine learning algorithm may accept, as inputs, the pathology samples, associated information, and output information and implement training using one or more techniques. For example, the QC machine learning model can be trained on one or more convolutional neural networks (CNN), multi-instance learning CNN or multi-label multi-instance learning, recurrent neural networks (RNN), memory RNN long term (LSTM), gate recurrent unit (GRU) RNN, graph convolution networks or similar or a combination thereof. Convolutional neural networks can directly learn the image feature representations needed to discriminate between features, which can work extremely well when there are large amounts of data to train for each sample, while the other methods can be used with traditional computer vision capabilities , for example SURF or SIFT, or with learned associations (e.g. descriptors) produced by a trained convolutional neural network, which can have advantages when there are only small amounts of data to train. The trained QC machine learning model can be configured to provide quality assignments for the digital pathology images.

[0063] Figure 3 shows an example of training module 300 for training a QC machine learning model. As shown in Figure 3, training data 302 may include one or more of pathology images 304 (e.g., digital representation of biopsy images), input data 306 (e.g., a digital pathology image dataset ) and known results 308 (e.g., quality designations) related to input data 306. Training data 302 and a training algorithm 310 may be provided to a training component 320 that may apply the training data 302 to the algorithm training module 310 to generate a QC machine learning model.

[0064] At 206, the QC machine learning model may be provided with a patient-based digital pathology image (e.g., a digital pathology sample image (e.g., histology, cytology, immunohistochemistry, etc.) ), as well as, optionally, associated information. After training, the QC machine learning mode can be applied to the digital pathology image and associated information to determine a quality designation for the patient-based digital pathology image at 208.

[0065] The quality designation may be an approval or rejection of the digital pathology image based on predetermined or dynamically determined thresholds. A dynamically determined threshold can be established based on the volume of digital images provided to the QC machine learning model. For example, the QC machine learning model may be set to reject the worst 10% of digital pathology images, and consequently the number of digital pathology images that are rejected may change based on the number of digital pathology images inserted into the model. As a result of the quality designation (e.g., a rejection), the target sample may be prepared again (e.g., in a laboratory) and/or the existing target sample may be rescanned. The specific action may be based on the artifact identified by the QC machine learning model, such that a blur may indicate a scan error and a new scan may be initiated, while missing tissue may indicate a pathology sampling error and a new sample can be prepared again in a laboratory environment.

[0066] According to one implementation, the quality designation may be a quality index determination. The quality score can be indicative of QC issues (e.g., artifacts such as errors or imperfections) for digital pathology images at a global or local level and this can greatly affect the usability of a digital pathology image. The quality index can be determined as a function of the number or extent of artifacts found in a given image. The quality score can be determined based on the training dataset used to train the QC machine learning model or can be determined relative to other digital pathology images provided as input to the QC machine learning model .

[0067] The output of the QC machine learning model (i.e., quality assignment) at 210 may be provided to a storage device 109 of Figure 1A (e.g., cloud storage, hard drive, network, etc.). The output of the QC machine learning model can be or can also be a notification based on the quality characteristic. The notification may include information about the artifact, such as the presence of one or more artifacts, the absence of some specific artifact or artifacts, and/or a grade of one or more artifacts present in the digital pathology sample. Notification may be provided through any applicable technique, such as a notification signal, a report, a message via an application, a notification via a device, or the like. Notification may be provided to any applicable device or professional (e.g., histology technician, scanner operator, pathologist, registrar, etc.). As an example, the output of the QC machine learning model may be integrated with the history of the corresponding target sample, for example, in laboratory information systems 125 that store a record of the patient and the associated target sample.

[0068] According to one implementation, the output of the QC machine learning model may be a report based on the quality designation or process for obtaining, using, and/or evaluating the digital pathology image. The report can be in any applicable format, such as PDF format, HTML format, application format or similar. The report may include, for example, types of artifacts (e.g., errors) found, a number of each type of artifact, clinical impact of the artifacts, time to rectify artifacts, potential patterns or correlations in artifacts (e.g., a particular scanner that tends to generate artifacts, a particular color type that tends to generate artifacts, a particular tissue source that tends to generate artifacts, etc.). The report may also include a workflow analysis, such as overall response times and/or quality of work. The report can also provide historically relevant information based on a plurality of digital pathology samples analyzed by the QC machine learning model. Historically relevant information may be generated based on factoring in the outputs of the plurality of digital pathology samples and the corresponding report may include a change rate, agreement rates, or the like.

[0069] According to one implementation, the output of the QC machine learning model, at 210, may be or may include a visual indicator. The visual indicator may be provided, for example, through the visualization application tool 108 of Figure 1B. The visual indicator can highlight a particular artifact as it relates to the digital pathology image. As an example, the visual indicator may be an overlay on the digital pathology image that highlights the location of the artifact, the type of artifact, the degree of presence of the artifact, or the like.

[0070] According to one implementation, the QC machine learning algorithm may also be trained based on and/or receive as inputs clinical information (e.g., patient information, surgical information, diagnostic information, etc.), laboratory information (e.g. processing times, professionals, tests, etc.). The QC machine learning algorithm can provide a quality assignment based on such inputs. Additionally, the QC machine learning algorithm may provide alternative or additional outputs, such as a processing score, a practitioner rating, or the like.

[0071] As described herein, the QC machine learning algorithm can be integrated within a clinical workflow system to generate, analyze and/or review digital pathology images.

[0072] Figure 2B shows a flowchart 220 for generating disease designations based on digital pathology images and outputting comparison results based on comparing disease designations with external designations in accordance with exemplary implementations of the subject matter described. At 222 of Figure 2B, a digital image corresponding to a target sample associated with a pathology category can be received. The digital image may be a digital pathology image captured using the slide input tool 103 of Figure 1B. At 224, a QA machine learning model may be received (e.g., at machine learning module 100). The QA machine learning model may be trained by processing a plurality of training images that are of the same pathology category as the digital image received in 222. Pathology categories may include, but are not limited to, histology, cytology, section frozen or immunohistochemistry. In 226, the digital image of 222 can be provided to the QA machine learning model as an input to the model. One or more other attributes may also be provided as an input to the QC machine learning model. The one or more other attributes may include, but are not limited to, slide-based attributes, patient attributes, tissue attributes, or the like. At 228, the QA machine learning model can issue a disease designation for the digital image. The disease designation may be generated based on the detection of one or more biomarkers and may include one or more of a cancer detection, cancer grade, origin of the cancer, diagnosis, presence or absence of a microorganism, sample type , type of cancer, cancer status, tumor size, injury risk level, grade or similar. At 230, an external assignment for the target sample may be received. The external designation may be based on the digital image received at 222 or the target sample upon which the digital image was based. At 232, the disease designation can be compared with the external designation and at 234, the comparison result based on comparing the disease designation with the external designation can be output. The result of the comparison may be output as a data signal, a report, a notification, an alert, a visual output (e.g., via visualization application tool 108), or the like.

[0073] As shown in Figure 2B, a digital image corresponding to a target sample associated with a pathology category may be received at 222. The target sample may be a biopsy or tissue sample recovered from a patient. The target sample may be retrieved during a surgical procedure in which a portion of a patient's tissue is retrieved from the patient's body for analysis. The target sample can be a portion or subset of the total amount of tissue extracted from patients, such that multiple sample slides can be generated from tissue extracted from a single procedure.

[0074] The target sample may be associated with at least one category of pathology, such as histology, cytology, frozen section or immunohistochemistry, as described herein. According to one implementation, the pathology category and other image information about the target digital image or sample may also be received. Image information may include, but is not limited to, a slide type, a glass type, a tissue type, a tissue region, a chemical used, and an amount of dye.

[0075] At 224, a QA machine learning model may be received. The QA machine learning model may be trained and generated in the machine learning module 100 or may be trained and generated externally and received in the machine learning module 100. The QC machine learning model may be trained by processing a plurality of training images that are of the same pathology category as the digital image received in 202. The QA machine learning model can detect the presence or absence of cancer in more than one tissue type (e.g., breast cancer prostate, breast cancer, bladder cancer, etc.). It can also detect additional biomarkers or information important for staging. For example, for bladder cancer, the generalized machine learning model can generate the presence or absence of muscularis propria, a muscle that needs to be detected for bladder cancer staging. The QA machine learning model can be trained with large amounts of data to predict diseases, biomarkers, and other attributes relevant to cancer treatment from multiple tissue types. Through this process, it can detect the presence of cancer and/or biomarkers across a wide range of different tissue types so that its layers are constructed based on an understanding of tumor characteristics as well as normal and abnormal morphology. of the fabric.

[0076] The QA machine learning model can be generated based on a patient data set including a large plurality of digital images of pathology samples (e.g., histology, cytology, immunohistochemistry, etc.). Pathology samples may be digital images generated from physical biopsy samples as described in this document, or they may be images that are algorithmically generated to replicate human tissue, for example, by a rendering system or a generative adversarial model. Patient-associated information (genomic information, laboratory tests, radiology, patient characteristics, patient information, treatment information, etc.) can also be received as part of the patient dataset to train the patient's machine learning model. QA. Additionally, as part of training the QA machine learning model, each patient dataset can be paired with information or indications about cancer characteristic outputs (e.g., biomarkers), such as presence/absence of disease, presence of staging variables (e.g., muscularis propria for bladder cancer), cancer form classification (e.g., lobular or ductal for breast cancer), and other variables relevant to different cancer types, outcome status (e.g., response , recurrence, etc.) and/or the presence of any biomarkers.

[0077] The patient data set, patient-associated information, and/or cancer feature outputs may be received from any one or any combination of server systems 110, medical servers 121, hospital servers 122, clinical trials 123, research laboratory servers 124, and/or laboratory information systems 125. Images used for training may come from real sources (e.g., humans, animals, etc.) or may come from synthetic sources (e.g. , graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with a variety of stains, such as (but not limited to) H&E, hematoxylin alone, IHC, molecular pathology, etc.; and/or (b) tissue samples scanned from a 3D imaging device such as microCT.

[0078] The QA machine learning model can be generated based on applying the patient dataset and patient-associated information paired with cancer characteristic output to a machine learning algorithm. The machine learning algorithm may accept, as inputs, the pathology samples, patient-associated information, and cancer feature outputs and implement training using one or more techniques. For example, the generalized machine learning model can be trained on one or more convolutional neural networks (CNN), multi-instance learning CNN or multi-label multi-instance learning, recurrent neural networks (RNN), long-memory RNN term (LSTM), gate recurrent unit (GRU) RNN, graph convolution networks or similar or a combination thereof. Convolutional neural networks can directly learn the image feature representations needed to discriminate between features, which can work extremely well when there are large amounts of data to train for each sample, while the other methods can be used with traditional computer vision capabilities , for example SURF or SIFT, or with learned associations (e.g. descriptors) produced by a trained convolutional neural network, which can have advantages when there are only small amounts of data to train. The trained QA machine learning model can be configured to provide disease designations (cancer characteristics) as outputs based on patient data and patient-associated information.

[0079] Figure 3 shows an exemplary training module 300 for training the QA machine learning model. As shown in Figure 3, training data 302 may include one or more pathology images 304 (e.g., digital representation of biopsy images), input data 306 (e.g., a set of patient data, associated information patient, etc.) and known outcomes 308 (e.g., cancer characteristics) related to input data 306. Training data 302 and a training algorithm 310 may be provided to a training component 320 that may apply the data training algorithm 302 to training algorithm 310 to generate a machine learning model.

[0080] The QA machine learning model may receive a set of patient data, such as one or more digital pathology images (e.g., histology, cytology, immunohistochemistry, etc.) at 226. According to In an implementation, the QA machine learning model can also receive information associated with the patient (e.g., genomics, laboratory tests, radiology, patient characteristics, etc.). After training, the QA machine learning model may be applied to the patient dataset and patient-associated information to determine one or more disease designations in 228. A disease designation may be one or more detection cancer, grade of cancer, origin of the cancer, diagnosis, a presence or absence of a microorganism, type of sample, type of cancer, status of the cancer, size of the tumor, level of risk of injury, grade or the like. Disease designations may not be cancer-specific, so the QA machine learning model can provide disease designations across all cancer types, if any.

[0081] The output of the QA machine learning model (i.e., one or more disease designations) may be provided to a storage device 109 of Figure 1A (e.g., cloud storage, hard drive, network drive , etc.). The storage device 109 may store one or more other disease designations from previous digital pathology image inputs.

[0082] At 230, an external designation associated with the digital image can be received. The external designation may be one or more of cancer detection, cancer grade, cancer origin, diagnosis, presence or absence of a microorganism, specimen type, cancer type, cancer status, tumor size, level of risk of injuries, degree or similar. The external designation can be based on the digital image or the target sample based on which the digital image is generated.

[0083] The external designation may be a manual designation provided by a healthcare professional (e.g., a pathologist) based on a review of the digital image or target sample associated with the digital image. Alternatively or in addition, the external designation may be provided by a third party (e.g., a third-party healthcare facility, a third-party system, a system not associated with the QA machine learning model, a different machine learning model, or similar) and, consequently, can be a manual designation or an automated designation.

[0084] The external designation may be in a format that is comparable to the disease designation received at 228, such that the external designation may be compared to the disease designation at 232. The comparison may be conducted by comparing data points, comparing final results, text analysis, optical character recognition (OCR) or similar. Although a single comparison is discussed in this document, it is understood that multiple comparisons of a plurality of different attributes of the disease designation and/or the external designation may be made. For example, the disease designation and external designation may provide a type of cancer and the severity of the cancer. Consequently, both the type of cancer and the severity of the cancer, as provided by the respective disease designation and the external designation, can be compared.

[0085] The comparison result can be output at 234 (i.e., the designation of the comparison result). The output of the comparison result may be provided to a storage device 109 of Figure 1A (e.g., cloud storage, hard drive, network drive, etc.). The storage device 109 may also store previous comparison results between previous disease designations and external designations. The result of the comparison can be the difference between the output of the QA machine learning model and a healthcare professional's (e.g., pathologist) subjective assessment of the digital image or associated target sample and can confirm findings, flag errors, identify problems or cause a review based on the comparison result.

[0086] The comparison results output at 234 may be or may also be a notification. The notification may include information about similarities between the disease designation (i.e., output from the QA machine learning model) and the external designation, differences between the disease designation and the external designation, degrees of differences or similarities between the disease designation and the external designation, or similar. Notification may be provided through any applicable technique, such as a notification signal, a report, a message via an application, a notification via a device, or the like. Notification may be provided to any applicable device or professional (e.g., histology technician, scanner operator, pathologist, registrar, etc.). As an example, the result of the comparison may be integrated into the history of the corresponding target sample, for example, in laboratory information systems 125 that store a record of the patient and the associated target sample.

[0087] According to one implementation, the result of the comparison may be presented as a report based on the similarities, differences and/or degree of similarity or differences between the disease designation and the external designation. The report can be in any applicable format, such as PDF format, HTML format, application format, or similar. The report may include, for example, types of results found, overlapping results between disease designation and external designation, contradictory results between disease designation and external designation, clinical impact of similarities or differences, time to rectify differences, potential patterns or correlations in the comparison, or similar. The report may also include a workflow analysis, such as overall response times and/or quality of work. The report can also provide historically relevant information based on a plurality of digital pathology samples analyzed by the QA machine learning model and/or the historical comparison results.

[0088] According to one implementation, the output of the comparison results at 234 may be or may include a visual indicator. The visual indicator may be provided, for example, through the visualization application tool 108 of Figure 1B. The visual indicator may identify a particular difference, for example, by highlighting biomarkers used to generate the disease designation, slide attributes that contribute to the disease designation, or the like.

[0089] According to one implementation, the QA machine learning algorithm may also be trained based on and/or receive as inputs clinical information (e.g., patient information, surgical information, diagnostic information, etc.), laboratory information (e.g. processing times, professionals, tests, etc.). The QA machine learning algorithm can provide a quality assignment based on such inputs. Additionally, the QA machine learning algorithm may provide alternative or additional outputs, such as a processing score, a practitioner rating, or the like.

[0090] As described herein, the QA machine learning algorithm and/or comparison results may be integrated within a clinical workflow system to generate, analyze, and/or review digital pathology images.

[0091] Figure 4 is a diagram showing a clinical workflow of a detection-based technique. In accordance with implementations of the subject matter described, a QA and/or QC analysis (QA/QC analysis 402) in accordance with the techniques of Figure 2A (i.e., QC analysis) and/or Figure 2B (i.e., QA analysis) can be implemented at one or a plurality of points during the workflow. As shown, a patient registration may occur at the beginning of the clinical workflow of the detection-based technique, at 404. The patient registration may include generating a physical or digital patient file that includes or receives patient identifying information, information patient's medical information and/or other information related to the patient.

[0092] At 406, surgery on the patient is performed. The surgery on the patient can be a biopsy or cytology that extracts the patient's tissue from the patient's body. As shown, QA/QC analysis 402 can occur during or after surgery on the patient in 406. For example, an image of a biopsy tissue can be captured and evaluated through the QC analysis of Figure 2A. The QC analysis in Figure 2A can provide a quality designation of the biopsy tissue sample so that the system can determine whether a new sample is needed based on the quality designation.

[0093] A frozen section preparation and review step 408 may be performed based on tissue extracted from surgery on patient 406. Similar to QA/QC analysis 402 following surgery on patient 406, QA/QC analysis 402 may occur on or after preparation and review of the frozen section 408. For example, an image of a frozen section may be captured and evaluated through the QC analysis of Figure 2A. The QC analysis in Figure 2A can provide a quality designation of the frozen section so that the system can determine whether a new frozen section is needed based on the quality designation.

[0094] Preparation of pathology material 410 (e.g., staining a sample) may occur based on surgery on the patient in 406 and/or frozen section preparation and review in 408. Similar to QA/QC analysis 402 After surgery on patient 406, QA/QC analysis 402 may occur on or after preparation of pathology material 410. For example, an image of a prepared pathology material may be captured and evaluated through the QC analysis of Figure 2A . The QC analysis of Figure 2A can provide a quality designation of the prepared pathology material so that the system can determine whether new prepared pathology material is needed based on the quality designation.

[0095] According to one implementation, processing of external material 412 may occur as a substitute for surgery on patient 406, preparation and review of frozen section 408, and preparation of pathology material 410. Processing of external material 412 may be based on a digital or physical sample that may be previously obtained or provided by a third party. Similar to QA/QC analysis 402 following surgery on patient 406, QA/QC analysis 402 may occur based on processing of external material 412.

[0096] Digitized material 414 may be generated based on either the preparation of pathology material 410 or the processing of external material 412. Similar to the QA/QC analysis 402 after surgery on patient 406, the QA/QC analysis 402 may occur based on digitized material 414.

[0097] Review material 416 can be generated based on digitized material 414 or processing external material 412. Similar to QA/QC analysis 402 after surgery on patient 406, QA/QC analysis 402 can occur with based on review material 416. For example, an image used for review material 416 can be captured and evaluated through the QC analysis of Figure 2A. The QC analysis in Figure 2A can provide an image quality designation so that the system can determine whether a new frozen section is needed based on the quality designation. As another example, an initial diagnosis may be made based on review material 416 and/or ancillary tests 418. The QA analysis of Figure 2B may provide a comparison result between the initial diagnosis and a disease diagnosis generated by a model. of QA machine learning so that a notification can be generated to indicate an artifact and/or difference indicated by the comparison.

[0098] A diagnosis and report 420 may be generated upon completion of the workflow provided in Figure 4. Similar to the QA/QC analysis in each of the steps in Figure 4, a QA/QC analysis 402 may occur based on the diagnosis and report 420, in accordance with Figures 2A and 2B provided herein.

[0099] Figures 5-7 show exemplary implementations of the QA/QC analysis as described herein with reference to Figures 2A and 2B. Figure 5 shows a flowchart in which the quality of a prepared pathology sample is controlled after digitizing the pathology images. At 502, a QC machine learning model can be trained to identify artifacts. Training can be conducted with images of a pathology specimen (e.g., histology, cytology, frozen section, etc.) and an indication of whether the image contains artifacts or scan errors (e.g., blur, missing tissue, lines, etc.) ) or artifacts from tissue preparation (e.g., missing or folded tissue, bubbles, cracks in glass, scratches, dust, pen, excess dye, etc.) that may negatively affect the quality of the diagnosis.

[0100] At 504, the QC machine learning model may be installed within one or more aspects of a clinical workflow/system, such as on a network drive, workstation, image storage device, virtual machine or integrated directly into the scanner. The installation can be deployed on premises or remotely (e.g. cloud-based). As an example, QC machine learning can be integrated within digitizer software such that all digital scans are passed through QC machine learning and the techniques described in reference to Figure 2A.

[0101] At 506, a digital image of a pathology sample (e.g., histology, cytology, frozen section, etc.) may be received and stored locally or remotely. The 504 QC machine learning model can determine whether an image has scanning or tissue preparation artifacts and, optionally, which type of error.

[0102] At 508, a quality assignment may be made, by the QC machine learning model of 504, regarding whether the image meets a quality threshold. Quality assignment can be made based on one or more inputs provided to the QC machine learning model. If the quality designation meets or exceeds a quality threshold (i.e., the “Yes” branch), then the digital image of 506 may be used for diagnosis. At 510, it may be stored in a file as an image that meets the threshold. quality threshold at 512, and/or the image quality threshold reached by the image may be used in a report at 514. According to one implementation, although not shown in Figure 5, a notification may be generated when a digital image reaches the quality threshold to indicate successful image capture.

[0103] If the quality designation does not meet the quality threshold (i.e., the “No” branch), then a notification may be generated at 516 indicating the same. For example, notification may be provided to an applicable healthcare professional, such as a histology technician, with an indication of which sample needs to be reprepared. Alternatively, or in addition, a scanner operator may be notified to rescan the material. Additionally, an alert may be generated at 518. The alert may be generated, for example, if the degree to which the digital image does not meet the quality threshold is significant. A visual indicator may also be provided to one or more users and may include information about what, where and/or why the digital image did not meet the quality threshold. The digital image can be recaptured at 520.

[0104] The result of 508 on whether or not the digital image reached the quality threshold may be associated with a patient file or sample history, for example, in a laboratory information system.

[0105] A report may be generated based on the result of 508 and may include types of errors found, number of errors found for each type, clinical impact of the errors, time required to rectify the errors, potential patterns or correlations in errors, such such as whether a certain scanner tends to generate errors, a certain type of staining tends to generate errors, a certain tissue source tends to generate errors, or similar. As described in this document, errors can be a type of artifact.

[0106] Figure 6 shows a flowchart for generating and applying comparison results comparing a disease diagnosis and an external diagnosis, as discussed in relation to Figure 2B. The implementation shown in Figure 6 could be, for example, a QA implementation before a final output or diagnosis. Based on the process shown in Figure 6, the quality of a slide evaluation by a pathologist can be assured before the case is closed. At 602, a QA machine learning model may be trained to identify biomarkers for determining cancer detection, cancer grade, origin of cancer, diagnosis, presence or absence of a microorganism, sample type, type of cancer , cancer status, tumor size, injury risk level, grade or similar. Training can be conducted with images of a pathology specimen (e.g., histology, cytology, frozen section, etc.), associated patient information as described herein, and an indication of known biomarkers, or disease designations.

[0107] At 604, the QA machine learning model may be installed within one or more aspects of a clinical workflow/system, such as on a network drive, workstation, image storage device, virtual machine or report analysis. The model can be implemented with logic to manage agreements and disagreements. The installation can be deployed on premises or remotely (e.g. cloud-based). As an example, the QA machine learning model can be integrated with a reporting module so that it receives, as an input, the external designation provided through a report (e.g., a pathologist's report).

[0108] At 606, a digital image of a pathology sample (e.g., histology, cytology, frozen section, etc.) may be received and stored locally or remotely. The QA machine learning model of 604 may generate a disease designation at 608, which may include one or more elements of a diagnostic determination (e.g., detection of cancer, grade of cancer, origin of cancer, diagnosis, presence or absence of a microorganism, type of sample, type of cancer, cancer status, tumor size, injury risk level, grade, etc.). The disease designation can be sent to a local or remote store. At 610, an external designation may be received from a manual designation (e.g., by a pathologist) or an automated designation (e.g., by a third-party system, a different machine learning system, or a computer algorithm).

[0109] At 612, the disease designation of 608 may be compared with the external designation of 610 to produce a “Match”, a “Non-Match”, or a degree of any of the same. As shown in the correspondence branch, if the disease designation and the external designation match, the external designation can be erased and used as an official designation at 614. The digital image can also be stored on file for future use, analysis, or reference in 616. Additionally, the disease designation and/or external designation may be associated with a patient file or sample file and stored accordingly.

[0110] As shown through the Unmatched branch, if the disease designation and the external designation do not match, a notification may be generated at 618. The notification may be provided to any applicable individuals or devices, such as to a primary pathologist before of the respective case being closed. Consequently, the notification may be generated in 618 if the external diagnosis entered is different from the disease diagnosis determined by the QA machine learning model, where the external diagnosis may be entered into a laboratory information system, may be provided by an institution external (e.g. a diagnostic consultation, second opinion, transferred case, etc.), or diagnosis from a pathologist. Based on a mismatch between the disease designation and the external designation, a visual indicator can be provided to show what, where and/or why a discordance was found.

[0111] At 620, an alert may be generated based on the non-match determination if, for example, the degree of difference between the disease diagnosis and the external diagnosis exceeds a threshold, based on the type of mismatch or similar. Additionally, at 622, a reassignment of the corresponding digital image or sample may be initiated based on the incompatibility between the disease diagnosis and the external diagnosis.

[0112] According to one implementation, both disease diagnosis and external diagnosis may be provided via a final diagnosis report. Additionally, periodic retrospective reports for review can be generated. These reports can be generated by healthcare professional, technician, site, institution or similar. Reports may include one or more types of disagreements, types of agreements, number of disagreements rectified to be concordant, disagreements, and any relevant follow-up information available to the patient.

[0113] Figure 7 shows a flowchart for generating and applying comparison results comparing a disease diagnosis and an external diagnosis, as discussed in relation to Figure 2B. The implementation shown in Figure 7 could be, for example, a QA implementation following a final output or diagnosis and could facilitate reporting based on one or more diagnosis. Based on the process shown in Figure 7, the quality of a slide evaluation by a pathologist can be assured after the case is closed. At 702, a QA machine learning model may be trained to identify biomarkers for determining cancer detection, cancer grade, origin of cancer, diagnosis, presence or absence of a microorganism, sample type, type of cancer , cancer status, tumor size, injury risk level, grade or similar. Training can be conducted with images of a pathology specimen (e.g., histology, cytology, frozen section, etc.), associated patient information as described herein, and an indication of known biomarkers, or disease designations.

[0114] At 704, the QA machine learning model may be installed in one or more aspects of a clinical workflow/system, such as on a network drive, workstation, image storage device, machine virtual or report analysis. The model can be implemented with logic to manage agreements and disagreements. The installation can be deployed on premises or remotely (e.g. cloud-based). As an example, the QA machine learning model can be integrated with a reporting module so that it receives, as an input, the external designation provided through a report (e.g., a pathologist's report).

[0115] At 707, a digital image of a pathology sample (e.g., histology, cytology, frozen section, etc.) may be received and stored locally or remotely. The QA machine learning model of 704 may generate a disease designation at 708, which may include one or more elements of a diagnostic determination (e.g., detection of cancer, grade of cancer, origin of cancer, diagnosis, presence or absence of a microorganism, type of sample, type of cancer, cancer status, tumor size, injury risk level, grade, etc.). The disease designation can be sent to a local or remote store. At 710, an external designation may be received from a manual designation (e.g., by a pathologist) or an automated designation (e.g., by a third-party system, a different machine learning system, or a computer algorithm).

[0116] In 712, the disease designation of 708 may be compared with the external designation of 710 to issue a “Match”, a “Non-Match”, or a degree of any of the same. Based on a Match and Non-Match designation, the steps shown in Figure 6 can be implemented. Additionally, a Match designation may be applied to a report at 714 and a Non-Match designation may be applied to a report at 718. The report may include attributes of the Match or Non-Match designations, such as degrees of disagreement or agreement. , specific areas of disagreement or agreement, or similar.

[0117] Furthermore, one or more elements of an external designation for one or more cases may be compared with one or more corresponding elements of the disease designation by the QA machine learning model and comparison of such results may be provided by middle of the report. This comparison can be conducted in several cases. For example, 712 comparison results may be provided for all cases or all cases from a certain tissue type, diagnosed by a certain pathologist, from a certain clinician, patients with a certain insurance, patients with a certain demographic type, from a certain site for the previous period of time (e.g. six months), or similar. The results of one or more of these comparisons can be included in a report to highlight agreements, disagreements, and degrees thereof as categorized by case type, tissue type, individual, patient, insurance, demographic, site, or time period. Consequently, one or more reports may be retrospective and may be generated by pathologist, by technician, by site, by tissue type, by stain, by scanner, by image type, by sample preparation, by diagnosis, by demographic element, by insurer, by referring doctor and/or by institution, or similar. Reports may include one or more types of disagreements, types of agreements, number of disagreements rectified to be concordant, disagreements and any relevant follow-up information available to the patient, and any other individual or aggregate measure of disagreement or discrepancy between the diagnosis of disease by QA machine learning model and external diagnosis.

[0118] The one or more reports may be provided to one or more individuals, administrators, insurance companies, government agencies, physicians, or the like and may include rates (e.g., agreement rates, disagreement rates, etc.) and/or or other quality measures derived from comparing the disease diagnosis by the QA machine learning model and the external diagnosis.

[0119] As shown in Figure 8, device 800 may include a central processing unit (CPU) 820. CPU 820 may be any type of processing device including, for example, any type of special-purpose or purpose-built microprocessor device. general. As will be appreciated by those skilled in the art, the CPU 820 may also be a single processor in a multicore/multiprocessor system, such system operating in isolation, or in a cluster of computing devices operating in a cluster of servers. The CPU 820 may be connected to a data communications infrastructure 810, e.g., a bus, message queue, network, or multi-core message passing scheme.

[0120] Device 800 may also include a main memory 840, e.g., random access memory (RAM), and may also include a secondary memory 830. The secondary memory 830, e.g., a read-only memory (ROM) , it can be, for example, a hard disk drive or a removable storage unit. Such a removable storage unit may comprise, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory or the like. The removable storage drive in this example reads from and/or writes to a removable storage drive in a well-known manner. The removable storage unit may comprise a floppy disk, magnetic tape, optical disc, etc., which is read and written by the removable storage unit. As will be appreciated by those skilled in the art, such a removable storage unit generally includes a computer-usable storage medium having stored the same computer software and/or data.

[0121] In alternative implementations, secondary memory 830 may include other similar means for allowing computer programs or other instructions to be loaded into device 800. Examples of such means may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, and other removable storage drives and interfaces, which allow software and data to be transferred from a removable storage drive to the 800 device.

[0122] Device 800 may also include a communications interface (“COM”) 860. Communications interface 860 allows software and data to be transferred between device 800 and external devices. The communication interface 860 may include a modem, a network interface (such as an Ethernet card), a communication port, a PCMCIA slot and card, or the like. Software and data transferred through communications interface 860 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 860. These signals may be provided to communications interface 860 via a communication path of the device 800, which may be implemented using, for example, wire or cable, optical fiber, a telephone line, a cellular telephone link, an RF link, or other communication channels.

[0123] The hardware elements, operating systems and programming languages of these equipment are conventional in nature, and it is assumed that those skilled in the art are adequately familiar with them. Device 800 may also include input and output ports 850 for connecting with input and output devices such as keyboards, mice, touch screens, monitors, displays, etc. Of course, the various server functions can be implemented in a distributed manner across multiple platforms to distribute the processing load. Alternatively, servers can be implemented by appropriately programming a computer hardware platform.

[0124] Throughout this description, references to components or modules generally refer to items that can logically be grouped together to perform a function or group of related functions. Similar reference numbers are generally intended to refer to the same or similar components. Components and modules can be implemented in software, hardware, or a combination of software and hardware.

[0125] The tools, modules and functions described above can be executed by one or more processors. "Storage" media may include any or all of computers' tangible memory, processors, or the like, or associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, that can provide non-transitory storage to any time for software programming.

[0126] The software may be communicated via the Internet, a cloud service provider, or other telecommunications networks. For example, communications may allow software to be loaded from one computer or processor to another. As used herein, unless restricted to tangible, non-transitory “storage” medium, terms such as “computer- or machine-readable medium” refer to any medium that participates in providing instructions to a processor for execution.

[0127] The previous general description is only exemplary and explanatory, and not restrictive of the description. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention described herein. It is intended that the specification and examples are considered as exemplary only.

Claims (19)

1. Computer-implemented method for processing electronic images using in vitro samples, characterized by the fact that it comprises automatically executed steps of: receiving (222) a digital image corresponding to a target sample associated with a category of pathology, in which the digital image is an image of human or animal tissue and/or an image generated algorithmically to replicate human or animal tissue; determine (204) a quality control (QC) machine learning model, the QC machine learning model being generated by processing a plurality of training images, associated with the pathology category, to predict a designation quality based on one or more artifacts; providing (206) the digital image as an input to the QC machine learning model; receiving (208) the quality designation for the digital image as an output of the QC machine learning model; determine (224) a quality assurance (QA) machine learning model other than the QC machine learning model, the QA machine learning model being generated by processing a plurality of training images, associated with the category of pathology, to predict a disease designation based on one or more biomarkers; providing (226) the digital image as an input to the QA machine learning model based on the quality designation being an approved quality designation; receiving (228) disease designation for the digital image as an output of the QA machine learning model; receive (232) an external designation for one of the target digital image or sample, the external designation generated by a user and comprising a disease property selected from at least one of cancer detection, a cancer grade, an origin of cancer, a diagnosis, a presence or absence of a microorganism, a type of specimen, a type of cancer, cancer status, a tumor size, a level of risk of injury, or a grade; evaluate the external designation by comparing (234) the disease designation with the external designation; reject external designation based on assessment, based on external designation deviating from disease designation; and issuing (236) a comparison result based on the evaluation of the external designation by comparing the disease designation with the external designation. 2. The computer-implemented method of claim 1, wherein the category of pathology is one of histology, cytology, frozen section, or immunohistochemistry. 3. The computer-implemented method of claim 1, wherein the QC machine learning model is trained such that its parameters are set to detect the presence, absence, or degree of an artifact in each image of training. 4. The computer-implemented method of claim 1, further comprising automatically generating a notification based on the quality characteristic, the notification comprising a presence, absence, or artifact grade designation. 5. Computer-implemented method according to claim 1, characterized by the fact that issuing the digital image quality designation additionally comprises a visual indicator corresponding to a digital image artifact. 6. The computer-implemented method of claim 1, wherein issuing the digital image quality designation further comprises generating a report comprising at least one of a type of artifact found, a number of artifacts found, impact clinical of a found artifact, a time to rectify an artifact, pattern potential, or correlation across a plurality of artifacts. 7. The computer-implemented method of claim 1, wherein the one or more artifacts comprise a missing tissue, a glass crack, a bubble, an amount of blur, a folded tissue, a line, a scratch, dust, pen, or an amount of dye. 8. Computer-implemented method for processing electronic images employing in vitro samples, characterized by the fact that it comprises automatically executed steps of: receiving (222) at least one digital image corresponding to a target sample associated with a category of pathology, wherein the digital image is an image of human or animal tissue and/or an image generated algorithmically to replicate human or animal tissue; determine (224) a quality assurance (QA) machine learning model, the QA machine learning model being generated by processing a plurality of training images, associated with the pathology category, to predict a designation of disease based on one or more biomarkers; providing (226) the digital image as an input to the QA machine learning model; receiving (228) disease designation for the digital image as an output of the QA machine learning model; receive (232) an external designation for one of the target digital image or sample, the external designation generated by a user and comprising a disease property selected from at least one of cancer detection, a cancer grade, an origin of cancer, a diagnosis, a presence or absence of a microorganism, a type of specimen, a type of cancer, a cancer status, a tumor size, a level of risk of injury, or a grade; evaluate the external designation by comparing (234) the disease designation with the external designation; reject external designation based on assessment, based on external designation deviating from disease designation; and issuing (236) a comparison result based on the evaluation of the external designation by comparing the disease designation with the external designation. 9. Computer-implemented method according to claim 8, characterized by the fact that the result of the comparison is an indication of discrepancy. 10. The computer-implemented method of claim 9, further comprising: determining a discrepancy level based on the discrepancy indication; generate a warning if the discrepancy level is below a threshold; and generate a trigger if the discrepancy level is above the threshold. 11. The computer-implemented method of claim 10, wherein the trigger comprises at least one of a manual reassignment of the digital image or a generation of an alert. 12. The computer-implemented method of claim 8, wherein the disease designation is at least one of cancer detection, cancer grade, cancer origin, diagnosis, presence or absence of a microorganism , sample type, cancer type, cancer status, tumor size, injury risk level, or grade. 13. The computer-implemented method of claim 12, wherein the QA machine learning model is trained based on the presence, absence, or degree of a disease designation in each training image. 14. The computer-implemented method of claim 8, wherein the external designation is at least one of a manual designation or an automated designation. 15. The computer-implemented method of claim 8, further comprising generating a notification based on the comparison result if the comparison result is a discrepancy designation. 16. The computer-implemented method of claim 8, wherein an external designation is provided by a healthcare professional, a third-party entity, or a secondary system. 17. Computer-implemented method according to claim 8, characterized by the fact that it additionally comprises emitting a visual indicator corresponding to the result of the comparison. 18. The computer-implemented method of claim 8, further comprising: receiving a plurality of additional comparison results; and generating a report based on the comparison result and additional comparison results, the report comprising at least one of a disagreement type, agreement type, number of disagreements rectified to be concordant, or follow-up disagreement. 19. System for processing electronic images, characterized by the fact that it comprises: at least one memory storing instructions; and at least one processor executing instructions to perform operations comprising: receiving at least one digital image corresponding to a target sample associated with a category of pathology, wherein the digital image is an image of human or animal tissue and/or a algorithmically generated image to replicate human or animal tissue; determine a quality assurance (QA) machine learning model, the QA machine learning model being generated by processing a plurality of training images, associated with the pathology category, to predict a disease designation based on one or more biomarkers; provide the digital image as an input to the QA machine learning model; receive disease designation for the digital image as an output from the QA machine learning model; receiving an external designation for one of the target digital image or sample, the external designation generated by a user and comprising a disease property selected from at least one of cancer detection, a cancer grade, a cancer origin, a diagnosis, a presence or absence of a microorganism, a sample type, a cancer type, cancer status, a tumor size, a level of risk of injury, or a grade; evaluate the external designation by comparing the disease designation with the external designation; reject external designation based on assessment, based on external designation deviating from disease designation; and issue a comparison result based on the evaluation of the external designation by comparing the disease designation with the external designation.