WO2022221712A1 - Detecting, scoring and predicting disease risk using multiple medical-imaging modalities - Google Patents

Detecting, scoring and predicting disease risk using multiple medical-imaging modalities Download PDF

Info

Publication number
WO2022221712A1
WO2022221712A1 PCT/US2022/025093 US2022025093W WO2022221712A1 WO 2022221712 A1 WO2022221712 A1 WO 2022221712A1 US 2022025093 W US2022025093 W US 2022025093W WO 2022221712 A1 WO2022221712 A1 WO 2022221712A1
Authority
WO
WIPO (PCT)
Prior art keywords
contours
tissue
computer
cancer
medical image
Prior art date
Application number
PCT/US2022/025093
Other languages
French (fr)
Inventor
Homayoun KARIMABADI
William Scott Daughton
Hoanh X. Vu
Kevin Harris
Original Assignee
Curemetrix, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Curemetrix, Inc. filed Critical Curemetrix, Inc.
Publication of WO2022221712A1 publication Critical patent/WO2022221712A1/en

Links

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/20—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
    • G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment

Definitions

  • Non-invasive medical imaging is often used by health professionals to detect and diagnose illness, and to guide medical interventions. For example, when cancer is suspected, medical information about the tissue, such as a medical image, may be gathered for the affected tissue, where a physician reviews the medical information to identify possible areas in the tissue that may have cancer cells. This analysis typically leads to an all-clear diagnosis (if no areas are identified by the physician) or a recommendation for a biopsy of the tissue to confirm that any possible areas of cancer cells are in fact cancerous cells. In the context of breast cancer, the medical image is typically a mammogram. This existing approach results in an about 60% cumulative risk of a false positive and an about 20% average false negative rate.
  • a false positive may result in a patient who did not have cancer having to endure a painful, intrusive, and unnecessary biopsy.
  • a false negative may result in not detecting cancer as early as it could have otherwise been detected.
  • CHD coronary heart disease
  • ASCVD Atherosclerotic Cardiovascular Disease
  • CAC Coronary Artery Calcium
  • a calcium coronary scan involves a cardiac computerized tomography (CT) scan performed in a CT scanner. Based on the extent of coronary artery calcification detected by the unenhanced low-dose CT scan, a number called an Agatston score is calculated.
  • the calculation is based on the weighted density score given to the highest attenuation value (HU) multiplied by the area of a calcification speck, and the score of every calcified speck is summed up to give the total CAC score.
  • the total calcium score allows for grading of coronary artery disease and early stratification of the risk for CHD and the severity of atherosclerosis in patients, which can be used to guide subsequent interventions, such as for high blood pressure or raised cholesterol.
  • This system includes: a computing device including a network interface for communications over a data network; and a clinical indication score engine having a processor and a memory, and including a network interface for communications over the data network.
  • the memory stores the medical image
  • the processor analyzes the medical image, generates a clinical score for at least a portion of the medical image, and transmits an instruction to a measurement device to a acquire a second medical image associated with a second non- invasive medical imaging technique based at least in part on the clinical score.
  • analysis of the medical image includes: determining contours in the medical image satisfying one or more criterion; analyzing one or more geometric attributes and one or more contrast attributes of contours included in the first subset of contours to identify a second subset of contours based at least in part on the contours satisfying one or more predetermined geometric and contrast attributes; selecting a third subset of contours from the second subset of contours that corresponds to one or more potential instances of a type of tissue, the third subset selected based at least in part on contours within the third subset satisfying first criteria associated with the type of tissue; ranking contours included in the third subset of contours based at least in part on a selection metric, the selection metric accounting for a combination of contrast and intensity; grouping the contours included in the third subset of contours into nested structures; selecting one or more instances of the type of tissue from the nested structures satisfying second criteria associated with the type of tissue; grouping the selected one or more instances of the type of tissue into clusters based on neighboring instances of the type of
  • the memory stores the second medical image, and the processor revises the scale of suspiciousness for the clinical indication based at least in part on the second medical image.
  • the clinical indication may include a type of cancer, such as neurological, lung, prostate or breast cancer.
  • the clinical indication may include: a type of cardiovascular disease, or a type of Thyroid disease.
  • the type of tissue may include calcifications.
  • the medical image includes one or more of an x-ray image or a CT scan.
  • the second medical image includes one or more of a magnetic resonance (MRI) image or an ultrasound image.
  • MRI magnetic resonance
  • the non-invasive medical imaging technique and/or the second non-invasive medical imaging technique may involve the use of an injected contrast.
  • the medical information may include clinical data (such as an age of the individual) and/or a medical history, e.g., of a family of the individual.
  • the generated clinical score for at least a portion of the medical image may include a clinical score for: the individual, for an examination of the individual, and/or a lesion or a feature in the medical image.
  • the instructions may be provided while non-invasive imaging of patient is being performed or while the patient is available for the second medical image to be acquired following acquisition of the medical image.
  • the patient may still be at a facility that includes the non-invasive medical imaging technique and the second non-invasive medical imaging technique.
  • the processor may extract tagged data from the medical image, where the medical image is included in a computer file.
  • the tagged data may include one or more of a side, a pixel spacing, an orientation, a protocol, or a date.
  • the tagged data is included in a Digital Imaging and Communications in Medicine (DICOM) header.
  • the processor may convert the medical image to a real array of intensities for contouring, such as a 4-byte real array of intensities.
  • the processor may select intensity levels for determining contours in the medical image.
  • the one or more criterion may include that each contour in the first subset of contours is (i) closed and (ii) includes a contour value larger than a surrounding area external to the contour. [0019] In some embodiments, contours not satisfying the one or more criterion are discarded. [0020] Note that the one or more geometric attributes of contours may include at least one of: a centroid, an area, a perimeter, a circle ratio, and an interior flag. [0021] Moreover, the one or more contrast attributes of contours may include at least one of: an intensity, an inward contrast, an outward contrast, or a gradient scale. [0022] Furthermore, the processor may detect an object in the medical image for exclusion from further analysis.
  • the object may be an external object or a foreign object (such as a biopsy clip, a breast implant, a clamp, etc.).
  • the object may be detected through the object having at least one of: an area greater than a predetermined area, an intensity greater than a predetermined intensity, or a circle ratio greater than a predetermined circle ratio.
  • selecting the third subset of contours may include excluding contours located within a predetermined distance from at least one of an edge of the medical image and an edge of tissue.
  • the first criteria may include contours having a predetermined area and a predetermined gradient scale.
  • the predetermined area may be between 0.003 mm2 and 800 mm2 and the predetermined gradient scale may be less than 1.3 mm.
  • the first criteria may include contours having a predetermined intensity, a predetermined circle ratio, a predetermined inward contrast, or a predetermined outward contrast.
  • the predetermined intensity may be greater than 0.67 times a maximum intensity
  • the predetermined circle ratio is greater than 0.65
  • the predetermined inward contrast is greater than 1.06
  • the predetermined outward contrast is greater than 1.22.
  • the first criteria may include contours having a predetermined area, a predetermined circle ratio, or at least one of a predetermined inward contrast and a predetermined gradient scale.
  • the predetermined area may be less than 0.30 mm2
  • the predetermined circle ratio may be greater than 0.65
  • the predetermined inward contrast may be greater than 1.04
  • the predetermined gradient scale may be greater than 0.3 mm.
  • the first criteria may include contours having a predetermined area, a predetermined circle ratio, or a predetermined intensity.
  • the memory may save the third subset of contours.
  • the processor may identify instances of the type of tissue for each nested structure based on at least one of: a contour derivative or a grouping parameter computed for each nested structure.
  • the contour derivative may measure how rapidly intensity varies across a nested structure.
  • the processor may identify outer contours in each nested structure representing a contour shape and inner contours in each nested structure providing data on internal gradients.
  • the second criteria may include a threshold on a contour derivate and a threshold on a grouping parameter.
  • the processor may computer cluster properties.
  • the cluster properties may include one or more of: a cluster centroid, a cluster half-length, a cluster half-width, an aspect ratio, a principal axis, or a packing fraction.
  • the system does not provide the instruction to the measurement device. Instead, the medical image and the second medical image are acquired concurrently by the measurement device, and then are provided to the system via the computing device.
  • the system diagnoses the clinical indication.
  • the system may compute a classification associated with the clinical indication based at least in part on the medical image and the second medical image.
  • the measurement device may include non-invasive medical imaging technique and the second non-invasive medical imaging technique.
  • Other embodiments provide the computing device.
  • the computing device includes the measurement device.
  • Other embodiments provide a computer-readable storage medium for use with the system. When program instructions stored in the computer-readable storage medium are executed by the system, the program instructions may cause the system to perform at least some of the aforementioned operations of the system.
  • Figure 1 illustrates a networked cancer detection and quantification system.
  • Figure 2 is a flow chart of a method for determining a cancer score.
  • Figure 3 is a flow chart of a method for detecting and quantifying cancer.
  • Figure 4 depicts a medical image of calcifications in a patient’s tissue.
  • Figure 5 illustrates an example of a selection process for a malignant cluster of micro-calcifications.
  • Figure 6 illustrates an example of the identification of a cluster of calcifications.
  • Figure 7 is a graph showing an example of a packing fraction as a function of ⁇ Ai>Amax for exemplary clusters.
  • Figure 8 is a drawing showing an example of an updated worklist.
  • Figures 9A, 9B and 9C provide example images and the associated BAC.
  • Figure 10 is a drawing showing an example of the Pearson correlation between the two BAC scores.
  • Figure 11 provides example images comparing the Bradley and WLS scores.
  • Figure 12 is a drawing showing an example of a comparison of BAC to CAC as a predictor of CHD.
  • Figure 13A is a drawing showing an example of a ROC for the cmAngio model.
  • Figure 13B is a drawing showing an example of positive and negative probability densities for the cmAngio model.
  • Figure 13C is a drawing showing an example of a probability of event for the cmAngio model.
  • Figure 14 provides example normal and breast cancer images showing micro-calcifications in situ, and inserted into a normal image.
  • Figure 15 provides example normal and breast cancer images showing micro-calcifications in situ, and inserted into a normal image, with modification.
  • Figure 16 provides example micro-calcifications inserted into a malignant mass image.
  • Figure 17 provides example lucent calcifications inserted into normal image.
  • Figure 18 provides an example malignant mass image, and extracted mass including buffer pixels.
  • Figure 19 provides an example synthetic mass image including the extracted malignant mass of Figure 18.
  • Figures 20A and 20B provides drawings showing an example the improvement in resultant AUC using the same algorithm with and without training using synthetics.
  • Figure 21 provides an example of AUC improvement using synthetics.
  • Figure 22 provides an example of AUC improvement using synthetics.
  • Figure 23 provides an example of AUC improvement using synthetics.
  • Figure 24 provides an example of the general improvement of AUC using synthetics.
  • Figure 25 provides an example of improvements of Image-Based ROC and FROC using synthetics.
  • Figure 26 depicts an exemplary triage system.
  • Figure 27A depicts a raw image of corrosion on an aircraft.
  • Figure 27B depicts ROIs which can be highlighted using a suspicion code.
  • Figure 28 depicts several processing steps using a medical image according to the Neural Network of the present invention.
  • Figure 29 depicts optional processing of a medical image.
  • Figure 30 is a graph having an X-axis of Percent of Exams Identified as Suspicious and a Y-axis of Sensitivity. Percent of exams identified as suspicious in test set.
  • Figure 31 is a graph having an X-axis of False Positive Rate and a Y-axis of Sensitivity. ROC curve based on test set.
  • Figure 32 is a graph having an X-axis of False Positive Rate and a Y-axis of Sensitivity, and two threshold points 1 and 2 as described herein. ROC curve with suspicion score threshold.
  • Figure 33 depicts an exemplary computing environment for determining a score and uncertainty level of medical images.
  • Figure 34 depicts a flowchart for determining a score and uncertainty level of medical images.
  • Figure 35 depicts images analyzed and evaluated by the systems and methods herein when training the model.
  • Figures 36A-D depict images for training the model and determining uncertainty.
  • Figures 36E and 36F depict images in the training set with high aleatoric uncertainty.
  • Figure 37 depicts a larger region of a surrounding breast for context when training a model.
  • Figure 38 depicts the inclusion of larger surrounding breast tissue in the crop when training a model.
  • Figures 39A-C depict images in the training with high epistemic uncertainty.
  • Figure 40 depicts the increased crop size to improve a model.
  • Figures 41A-C depicts the efficacy of a Bayesian NN used with the systems and methods herein for cases of high and low levels of uncertainty.
  • Anomaly refers to an outlier, novelty, noise, deviation, or exception. In the healthcare context, the term can refer to a lesion or feature associated with a physiological disease or disorder that is structurally and/or compositionally distinct from surrounding healthy and/or normal tissue.
  • BI-RADS® As used herein, the term “BI-RADS®” refers to the standardized quality control system for interpreting mammograms which was developed by the American College of Radiology.
  • BI-RADS® Assessment Categories are: 0: Incomplete; 1: Negative; 2: Benign; 3: Probably benign; 4: Suspicious; 5: Highly suggestive of malignancy; and 6: Known biopsy – proven malignancy.
  • the BI-RADS® atlas is available at http://www.acr.org.
  • Breast Density refers to categories which a radiologist uses for describing a patient’s mammogram. Breast density categories include Class A (or 1): Fatty; Class B (or 2): Scattered fibroglandular density; Class C (or 3): Heterogeneously dense; and Class D (or 4): Extremely dense.
  • CADe refers to Computer-Aided Detection -- the identification of a location in data that a CADe system, in accordance with a CAD algorithm operating on the data, highlights for attention by a technician.
  • the identification can be a mark on a medical image, or a more general indication such as a visual cue, sound, or other perceptible indicator such as an email, text, score, and the like.
  • CADx refers to Computer-Aided Diagnosis – the use of systems to evaluate and associate a medical indication, or conclusion of the presence or absence of a condition or disease, to conspicuous structures identified in a medical image. For example, in mammography, CADx highlights microcalcification clusters and dense structures in soft tissue, and allows a radiologist to draw conclusions about the condition of the pathology. Both CADe and CADx may jointly be referred to as “CAD” throughout the Specification.
  • CAD algorithm means a computer implemented program for detecting and quantifying anomalies in Data. Preferably, the CAD algorithm is the Neural Network.
  • Deep learning As used herein, the term “deep learning” refers to information generated by a sensor which is used to train the Neural Network described herein. Data can also include a collection of information, whether or not generated by a sensor, such as manually generated information, including digital text, handwriting, numerical tables, and the like.
  • Deep learning As used herein, the term “deep learning” is broadly defined to include machine learning which can be supervised, semi-supervised, or unsupervised. Architectures of deep learning include deep neural networks, deep belief networks, recurrent neural networks, and convolutional neural networks.
  • DNN Deep neural network
  • Deep neural network is an artificial neural network with multiple layers between input and output layers. DNN model linear and non-linear relationships.
  • DICOM® refers to the international standard to transmit, store, retrieve, print, process, and display medical imaging information.
  • DICOM® refers to the international standard to transmit, store, retrieve, print, process, and display medical imaging information.
  • a full description of the DICOM® standard and the association of Structured Reports with DICOM® images is provided in D. Clunie, “DICOM Structured Reporting,” PixelMed Publishing, Bangor, Pennsylvania (2000); see also http://www.dicomstandard.org. Medical information associated with DICOM® and Structured Reports can be burned in, overlaid, or provided separate from the original image.
  • Medical image generally refers to X-ray imaging, CT imaging, MRI, positron emission tomography (PET), Digital Two- Dimensional (2D) imaging, Three-Dimensional (3D) Tomosynthesis, single-photon emission computed tomography (SPECT), ultrasound (US), endoscopy, thermography, medical photography, nuclear medicine functional imaging, elastography, photoacoustic imaging, echocardiography, functional near-infrared imaging, magnetic particle imaging, and the like.
  • Neural Network refers to the systems and algorithms described in the section “Neural Network Architecture” below.
  • Recall Exams As used herein, the term “Recall Exam” is defined as those for which further evaluation (e.g., additional mammographic views, breast ultrasound, etc.) were recommended during the interpretation of the screening mammograms.
  • Sensitivity As used herein, the term “sensitivity” is the ability of a test to correctly identify those with a physiological anomaly (true positive rate). In certain aspects, “sensitivity” can be defined as the number of true positive exams with at least one true positive mark provided by the CAD algorithm described herein divided by the number of True Positive Exams.
  • Specificity As used herein, the term “specificity” is the ability of the test to correctly identify those without the physiological anomaly (true negative rate).
  • the term “specificity” can be defined as the number of true negative exams without any marks divided by the number of True Negative Exams.
  • Suspicion / Suspiciousness As used herein, the terms “suspicion” or “suspiciousness” broadly refer to observable conditions which can be classified as an anomaly. For example, in the field of mammography, a suspicious feature of a mammogram could include the presence of a microcalcification cluster or mass. In addition, a suspicious feature could be defined to include the complexity of soft tissue within a breast, breast density, breast size, breast volume, and the like.
  • True Positive Exams As used herein, the term “True Positive Exams” are biopsy confirmed cancer exams.
  • the disease may include: cancer, a type of ASCVD (such as CHD), a neurological disease, etc.
  • cancer is used as an illustrative example.
  • “detection” of a disease may include uncovering, to a particular degree or range of certainty (which may be a predetermined degree/range, or a degree/range following standard industry practice), whether the disease (such as cancerous cells) is present (or not present) in a sample of tissue.
  • detection may include discovering, affirming, finding, uncovering, unearthing, revealing, exposing, etc., the existence or absence of the disease (such as cancer cells) in a sample, which can be depicted in a medical image.
  • the cancer cells may include malignant or benign cells.
  • “quantification” of cancer may include determining, indicating, or expressing the quantity of cancer cells in a sample.
  • the quantity of cancer cells may include a specific number, range, or threshold of cells, the size of cells or groupings of cells, and so forth.
  • Quantification of cancer may also or instead include generating a “score” or “indication” as described herein.
  • Quantification of cancer may also or instead include generating a “grade” or “stage” of cancer.
  • “detection” of cancer may be included in the “quantification” of cancer and vice-versa.
  • a quantity of cancer cells is determined (i.e., a quantification)
  • cancer is detected.
  • a certain cancer score is determined, cancer is detected.
  • the devices, systems, and methods discussed herein can be adapted to detect and quantify other cancers including without limitation brain, lung, liver, prostate, bone, cervical, colon, leukemia, Hodgkin disease, kidney, lymphoma, oral, skin, stomach, testicular, thyroid, and so forth.
  • embodiments generally described herein are detecting and quantifying cancer in medical images of human tissue, the embodiments may also or instead be applicable to cancer in animals, for example.
  • the disclosed medical-imaging analysis techniques may include the triage of anomalous images or anomalous studies (which may include anomalies within an individual image within a study comprising multiple images) generated by a variety of imaging modalities.
  • anomaly detection and suspiciousness categorization methods discussed herein can be adapted for other anomaly detection including in data streams and other data generated by a variety of sensors or found in a variety of databases.
  • embodiments generally described herein relate to medical images, the embodiments may also or instead be applicable more broadly to image categorization for use in other fields such as in facial recognition, optical character recognition, landmark detection, drone videography, industrial equipment inspection and maintenance, among many other examples.
  • the devices, systems, and methods discussed herein may utilize medical image analysis, which may be automated through the use of various hardware and software as described herein.
  • the medical image analysis techniques discussed herein may thus be used quantify cancer (e.g., breast cancer) and/or generate a cancer quantification.
  • cancer e.g., breast cancer
  • the implementations discussed herein may also or instead generate a cancer quantification based on other pieces of medical information about tissue other than images as described herein and may be implemented in other ways than those described herein that are within the scope of the disclosure.
  • the computer-based cancer quantification system and method may be used for detecting and quantifying breast cancer in humans where the medical images are mammograms.
  • an accurate quantification of cancer may be used for an accurate detection of cancer in a piece of medical information, such as a medical image, an early detection of cancer, the growth rate of cancer, or a prediction of the likelihood of cancer.
  • An accurate quantification of cancer may also or instead be used to reduce the number of unnecessary biopsies (i.e., reduce false positives) and reduce the number of undiagnosed cancers (i.e., reduce false negatives).
  • An accurate quantification of cancer may also or instead be used to determine a tumor “grade,” e.g., a measure of the aggression of a specific form of cancer, whether the cancer is changing or is it staying localized (in some cases one may want to leave the cancer alone rather than operate based on the tumor grade), and so forth.
  • An accurate quantification of cancer may also or instead be used to determine how a treatment is affecting the cancer cells or is producing new cancer cells.
  • the devices, systems, and methods discussed herein may be used to generate a “score” that quantifies any tissue anomalies.
  • the score may also be referred to herein as a “Q score,” “Q factor,” or the like.
  • the cancer score may be expressed in any suitable or useful level of granularity such as with discrete categories (e.g., cancerous, non-cancerous, benign, malignant, cancer-free, tumor-free, and so on), or with a numerical score, alphabetic score/grade, or other quantitative indicator.
  • discrete categories e.g., cancerous, non-cancerous, benign, malignant, cancer-free, tumor-free, and so on
  • numerical score e.g., alphabetic score/grade, or other quantitative indicator.
  • the cancer score may be a two-state score (e.g., cancer detected or cancer-free), a three-state score (e.g., cancer detected, cancer-free, unknown), a five-state score (e.g., unknown, cancer detected, cancer-free, benign, malignant), a range-bounded quantity (e.g., a score from 0–10, 0-100, or 0–1,000), or any other suitable score for quantifying cancer with any desired degree of granularity.
  • the cancer score may also or instead be scaled.
  • tissue abnormalities may be associated with a score or the like, which may be based on a predetermined scale, e.g., 0–100, where certain known benign abnormalities would have a score close to or equal to 0 and certain known malignant abnormalities detected in advanced stages would have a score close to or equal to 100 (or vice-versa).
  • cancer information may be multi-dimensional, so that multiple aspects may be independently scored. It shall be understood that the cancer score may change to indicate that cancer is more likely as the cancer cells/tumor grows and the cancer score may also change to indicate the opposite when the cancer cells/tumor shrinks. As discussed above, in one implementation, a smaller cancer score indicates a benign tumor and a larger cancer score indicates cancer.
  • the devices, systems, and methods discussed herein may be used to guide a radiologist analyzing a medical image, or to pre-screen, supplement, verify, or replace a radiologist’s review.
  • a radiologist typically reviews each mammogram. It has been shown that for every 100 screening mammograms performed, 10% are recalled for subsequent procedures, and of those, only 5% have cancer. This indicates that the prevalence of cancer in all mammograms is only 0.5%. Thus, 99.5% of the time, there is no cancer shown in the mammogram and yet the radiologist typically reviews the mammogram.
  • the devices, systems, and methods discussed herein may be used to pre-screen mammograms, score the mammograms according to a cancer score as discussed herein, and/or identify mammograms that show no anomalies or show only known benign anomalies (no cancer) and thus detect the absence of cancer so that these can be ignored for further analysis.
  • implementations may generate an indication of the absence of cancer in certain medical images and the radiologist need not review those medical images in detail based on the indication of the absence of cancer for the particular mammogram.
  • the radiologist may not need to analyze a large percentage of mammograms, thus significantly reducing a radiologist’s workload.
  • implementations may be used to generate an assessment or prediction of the activity of a cancer for a patient (e.g., implementations can determine that, over a particular time period, a cancer will not grow significantly), which may be used to determine a treatment for the particular patient.
  • a patient with prostate cancer may receive an assessment that the cancer is not going to grow significantly in the next six months and the patient may then opt for a less invasive treatment plan.
  • a retrospective study can be conducted whereby the present systems and methods are used to analyze a radiologist’s previous findings to determine whether the radiologist failed to detect cancer in a medical image, and whether a cancer score as described herein is different from the radiologist assessment.
  • Figure 1 illustrates a networked cancer (and, more generally, an anomaly) detection and quantification system.
  • the system 100 may be used to generate synthetic 2D images, triage a workflow and/or determine uncertainty levels of analysis.
  • the system 100 may include a client server implementation of a cancer quantification system.
  • the system 100 may include one or more computing devices 102 that are each used by a user or an administrator to couple to and interact with, over a network 104, a backend component 106.
  • a client server/web implementation of the system 100 is shown, the system 100 may also be implemented using a software as a service (SaaS) model, a standalone computer, and other computer architectures.
  • SaaS software as a service
  • the one or more computing devices 102 may include a processor-based computing device that has at least one processor, memory, persistent storage, a display, and communication circuits (such as a network interface and/or one or more input/output interfaces) so that each computing device 102 can communicate with the backend component 106, display a generated cancer score, submit pieces of medical information to the backend component 106, or otherwise interact with the backend component 106 or another component of the system 100.
  • the computing device 102 may include without limitation a smartphone device, a tablet computer, a personal computer, a workstation, a laptop computer, a server, a terminal device, a cellular phone, a wearable computer, a television, a set-top box, and the like.
  • the computing device 102 may execute an application, such as a known browser application or mobile application, that facilitates the interaction of the computing device 102 with the backend component 106.
  • the one or more computing devices 102 may also or instead include an endpoint, for example including client devices such as a computer or computer system, a Personal Digital Assistant, a mobile phone, or any other mobile or fixed computing device.
  • client devices such as a computer or computer system, a Personal Digital Assistant, a mobile phone, or any other mobile or fixed computing device.
  • the computing device 102 may be used for any of the entities described herein.
  • the computing device 102 may be implemented using hardware (e.g., in a desktop computer), software (e.g., in a virtual machine or the like), or a combination of software and hardware.
  • the computing device 102 may be a standalone device, a device integrated into another entity or device, a platform distributed across multiple entities, or a virtualized device executing in a virtualization environment.
  • the processor in computing device 102 may be any processor or other processing circuitry capable of processing instructions for execution within the computing device 102 or system 100. Examples of such processors are CPUs and GPUs.
  • the processor may include a single-threaded processor, a multi-threaded processor, a multi-core processor and so forth.
  • the processor may be capable of processing instructions stored in the memory or a data store.
  • the memory may store information within the computing device 102.
  • the memory may include any volatile or non-volatile memory or other computer-readable medium, including without limitation a Random Access Memory (RAM), a flash memory, a Read Only Memory (ROM), a Programmable Read-only Memory (PROM), an Erasable PROM (EPROM), registers, and so forth.
  • RAM Random Access Memory
  • ROM Read Only Memory
  • PROM Programmable Read-only Memory
  • EPROM Erasable PROM
  • registers and so forth.
  • the memory may store program instructions, program data, executables, and other software and data useful for controlling operation of the computing device 102 and configuring the computing device 102 to perform functions for a user.
  • the memory may include a number of different stages and types of memory for different aspects of operation of the computing device 102.
  • a processor may include on-board memory and/or cache for faster access to certain data or instructions, and a separate, main memory or the like may be included to expand memory capacity as desired.
  • the memory may, in general, include a non-volatile computer readable medium containing computer code that, when executed by the computing device 102 creates an execution environment for a computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of the foregoing, and/or code that performs some or all of the steps set forth in the algorithmic descriptions set forth herein.
  • code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of the foregoing, and/or code that performs some or all of the steps set forth in the algorithmic descriptions set forth herein.
  • any number of memories may be usefully incorporated into the computing device 102.
  • a first memory may provide non-volatile storage such as a disk drive for permanent or long-term storage of files and code even when the computing device 102 is powered down.
  • a second memory such as a random access memory may provide volatile (but higher speed) memory for storing instructions and data for executing processes.
  • a third memory may be used to improve performance by providing higher speed memory physically adjacent to the processor for registers, caching, and so forth.
  • the processor and the memory can be supplemented by, or incorporated in, logic circuitry.
  • the network 104 may include a communications path such as a wired or wireless network that uses a communications protocol and a data protocol, such as HTTP or HTTPS and HTML or JSON or REST, to allow each computing device 102 to interact with the backend component 106.
  • the network104 may be a wired network, a wireless computer network, a wireless digital data network, a cellular wireless digital data network, or a combination of these networks that form a pathway between each computing device 102 and the backend component 106.
  • the network 104 may also or instead include any data network(s) or internetwork(s) suitable for communicating data and control information among participants in the system 100. This may include public networks such as the Internet, private networks, and telecommunications networks such as the Public Switched Telephone Network or cellular networks using third generation cellular technology (e.g., 3G or IMT-2000), fourth generation cellular technology (e.g., 4G, LTE.
  • third generation cellular technology e.g., 3G or IMT-2000
  • fourth generation cellular technology e.g., 4G, LTE.
  • the network 104 may also include a combination of data networks, and need not be limited to a strictly public or private network.
  • the participants in the system 100 may each be configured with a network interface 124 for communications over the network.
  • a user 108 of the system 100 may be a technician, patient, a doctor, a radiologist, a health care organization, an image analyst, and the like.
  • the user 108 may, using the computing device 102, submit one or more pieces of medical information 110 for analysis or quantification by the system 100 and/or receive, from the backend component 106, a cancer quantification score based on the received pieces of medical information 110.
  • the backend component 106 may include storage 112 coupled to the backend component 106 (e.g., a memory, a database, and the like) that may store various data associated with the system 100 including a plurality of pieces of medical information 110 that may be used to generate one or more cancer quantification scores, user data associated with the system, and the like.
  • the storage 112 may be implemented using a known software-based or hardware- based storage system.
  • the backend component 106 may be implemented using one or more computing resources including without limitation a processor 114, a memory 116, persistent memory/storage, and the like.
  • each computing resource may be a blade server, a server computer, an application server, a database server, a cloud computing resource and the like.
  • the backend component 106 may have a web server 118 or the like that manages the connections and interactions with each computing device 102, generates HTML code to send to each computing device 102, receives data from each computing device 102, and the like.
  • the web server 118 may be implemented in hardware or software.
  • the backend component 106 may include a cancer score engine 120 that analyze pieces of medical information 110 about tissue.
  • the cancer score engine 120 may generate any indications of cancer in any regions of the tissue and may generate a cancer score 122 for any regions of the tissue in which there is an indication of cancer.
  • the cancer score engine 120 may receive/obtain the pieces of medical information 110 about tissue from a computing device 102, over a computer network from a third-party, or from the storage 112 of the system 100.
  • the cancer score 122 may be transmitted through the network 104, e.g., for display on the one or more computing devices 102.
  • the cancer score engine 120 may be implemented in software or hardware.
  • the cancer score engine 120 may comprise a plurality of lines of computer code that may be stored in a memory 116 and executed by a processor 114 of the backend component 106 so that the processor 114 is configured to perform the processes of the cancer score engine 120 (and its components) as described herein.
  • the cancer score engine 120 may comprise a microcontroller, a programmable logic device, an application specific integrated circuit, or other hardware device in which the hardware device performs the processes of the cancer score engine 120 (and its components) as described herein.
  • the cancer score engine 120 may include an algorithm or series of algorithms that assist in generating the cancer score 122 as discussed herein.
  • the backend component 106 may include an image analysis engine that analyze pieces of medical information 110.
  • the image analysis engine may generate quantitative scores related to whole images and/or ROIs in images, suspicion code, and may optionally generate an output which may be a general classification, for example, as to whether the image, or study comprising multiple images, is “normal” or “not normal.”
  • the image analysis engine can be described under the Neural Network Architecture section below.
  • the image analysis engine may receive/obtain pieces of medical information 110 from a computing device 102, over a computer network from a third-party, or from the storage 112 of the system 100.
  • the output such as a suspicion code, may be transmitted through the network 104, e.g., for display or intake on the one or more computing devices 102.
  • the image analysis engine may be implemented in software or hardware.
  • the image analysis engine (and its components) may comprise a plurality of lines of computer code that may be stored in a memory and executed by a processor of the backend component 106 so that the processor is configured to perform the processes of the image analysis engine (and its components) as described herein.
  • the image analysis engine may comprise a microcontroller, a programmable logic device, an application specific integrated circuit, or other hardware device in which the hardware device performs the processes of the image analysis engine (and its components) as described herein.
  • the image analysis engine may include an algorithm or series of algorithms that assist in generating the output as discussed more fully below as the Neural Network Architecture.
  • the one or more pieces of medical information 110 may include a medical image.
  • the medical image may include an x-ray image, e.g., a mammogram and the like.
  • the medical image may also or instead include magnetic resonance (MRI) images, CT scan images, ultrasound images, and so on.
  • MRI magnetic resonance
  • the system 100 may instead be implemented as part of a standalone computer implementation of a cancer detection and quantification system.
  • the cancer score engine 120 may be executed on one of the computing devices 102, e.g., by a processor, based on one or more pieces of medical information 110 stored in the computing device 102 or input into the computing device 102.
  • the computing device 102 may have a display 126 and any other additional hardware including without limitation input/output devices such as a keyboard and a mouse as shown.
  • the display 126 may include a user interface, e.g., a graphical user interface.
  • the computing device 102 may also include the processor, and a persistent storage device such as flash memory or a hard disk drive and memory, such as DRAM or SRAM, that are connected to each other.
  • a persistent storage device such as flash memory or a hard disk drive and memory, such as DRAM or SRAM, that are connected to each other.
  • the memory may store the cancer score engine 120 and an operating system and the processor of the system may execute a plurality of lines of computer code that implement the cancer score engine 120 so that the processor of the computer system is configured to perform the processes of the cancer score engine 120 as described herein.
  • the cancer score engine 120 may, in general, receive one or more pieces of medical information 110 about a piece of tissue of a patient and, for each piece of tissue for the patient, generate one or more cancer scores 122 about one or more regions in the piece of tissue.
  • the piece of tissue may include without limitation any piece of human tissue or any piece of animal tissue that may have cancer cells.
  • the image analysis engine may, in general, receive one or more pieces of medical information 110 and, for each anatomical feature of the patient, generate information related thereto.
  • a worklist management processor may be configured to implement the worklist processing functionalities described herein.
  • the system 100 includes one or more non-invasive medical imaging or measurement devices 128.
  • the measurement devices 128 may concurrently, sequentially, or a combination of both, perform measurements on at least a portion of a person or an animal, such as: X-ray imaging, CT imaging, MRI, PET, Digital 2D imaging, 3D Tomosynthesis, SPECT, ultrasound (US), endoscopy, thermography, medical photography, nuclear medicine functional imaging, elastography, photoacoustic imaging, echocardiography, functional near-infrared imaging, magnetic particle imaging, and/or type of another medical imaging technique.
  • the X-ray imaging, the CT imaging, Digital 2D imaging, 3D Tomosynthesis, etc. may use a low-energy, non-thermionic cathode (which is sometimes referred to as a ‘cold cathode’).
  • the non-thermionic cathode may include a field-emission cathode that selectively provides X-rays to at least the portion of the person or the animal.
  • the measurement devices 128 may be arranged in an array (such as a circular or donut-shaped array) around and/or along (e.g., along a longitudinal direction) at least the portion of the person or the animal.
  • the system 100 may be used for retrospective analysis of medical information.
  • the analysis of the medical information by the system 100 may be performed concurrently or while one or more medical imaging techniques are performed, or may be performed immediately after a given medical imaging technique is performed but before a subsequent medical imaging technique in a series of medical imaging techniques is performed.
  • the one or more medical imaging techniques may be adapted based at least in part on at least a portion of the analysis or information that is learned during one or more medical imaging techniques.
  • the availability of different medical imaging techniques in the measurement devices 128 may allow the system 100 to acquire medical images or information using the medical imaging technique that is best suited for an anomaly and/or a type of tissue.
  • CT imaging may be used for a mammography of a breast, while ultrasound may be subsequently used for an identified cyst.
  • ultrasound may be used assess a potential false positive.
  • multiple medical imaging techniques are used to acquire medical images of the same tissue.
  • different medical imaging techniques may be used to acquire medical images of different (non-overlapping) types of tissue.
  • the system 100 may address real-time performance requirements when acquiring medical images, such as time- varying or non-static performance requirements associated with hand-held ultrasound measurements.
  • the system 100 may improve the signal- to-noise ratio of medical images that are acquired using low-energy X-rays.
  • a pretrained neural network such as generative adversarial network or GAN
  • GAN generative adversarial network
  • the GAN may be trained using a training dataset with medical images or synthetic medical images at a range of tilt angles relative to a plane of the medical images in order to improve the accuracy of the reconstructed medical images and to increase the robustness of the reconstruction process to the effect of tilt (and, more generally, planar misregistration.
  • the medical images acquired using the medical imaging techniques may or may not be registered relative to at least the portion of the person or the animal. For example, a medical image of a breast may not be registered, while a medical image of a lung may be registered relative to the ribs, or a medical image of an eye may be registered relative to an eye socket.
  • FIG. 2 is a flow chart of a method for determining a cancer score.
  • the cancer score may be generated or determined by the cancer score engine.
  • the method 200 may involve processing and analyzing one or more pieces of medical information for generating, for one or more regions of the piece of tissue, an indication of cancer.
  • the method 200 may gather variables that are pertinent to the cancer, programmatically analyze the piece(s) of medical information to determine values of the variables, transform these variables for use in generating a cancer indication or score, and then generate an indication of cancer based on these variables or a transformation of these variables.
  • a cancer score generator component may receive the indications of cancer in the one or more regions of the tissue and generate a cancer score for at least one region of the tissue.
  • the cancer score may have a value that increases as a cancer tumor grows and decreases as a cancer tumor shrinks.
  • the cancer score may be normalized and have threshold levels so that, for example, a normalized cancer score of 1–3 indicates a benign tumor, a normalized cancer score of 4–6 indicates suspicious cells, and a normalized cancer score of 7–10 indicates that cancer is present in particular region(s) of tissue.
  • the method 200 may include receiving one or more pieces of medical information for processing and analysis.
  • the medical information may include information about a patient’s tissue, e.g., medical images of the tissue.
  • the medical information may include preprocessed or raw data, which is then processed and analyzed by the systems or methods described herein.
  • the cancer score engine may include a medical information analysis component that receives one or more pieces of medical information, where the cancer score engine then processes and analyzes this information.
  • the medical information may be automatically streamed to the cancer score engine by an uneven length preprocessed time series input.
  • the header of a DICOM file may contain information on the image contained within it including, but not limited to, the pixel resolution in physical units, criteria for interpreting the pixel intensity, etc.
  • the method 200 may include analyzing the one or more pieces of medical information about the tissue. This may include gathering variables values about the medical information (e.g., a mammogram), where generating the indication of cancer may be based on the gathered variable values.
  • the variables may include an intensity value for contours of any calcifications, a gradient of the calcifications, one or more characteristics about each calcification, and a hierarchical structure of the calcifications in a cluster.
  • the method 200 may include generating an indication of cancer.
  • the indication of cancer may be generated for one or more regions of the tissue in the medical images.
  • the method 200 may include generating a cancer quantification score.
  • generating a cancer quantification score may include generating a cancer score for each region of the tissue based on the indication of cancer in each region of the tissue.
  • the cancer quantification score may indicate an absence of cancer in the region of the tissue.
  • the method 200 may include generating guidance for a medical professional based on one or more of the indications of cancer and the cancer quantification score.
  • the guidance may include, e.g., guidance for a radiologist based on the presence or absence of cancer in the region of the tissue.
  • the guidance may be generated by applying rules based on the analysis of the medical information, the indication of cancer, or the cancer quantification score.
  • Implementations may utilize one or more algorithms for detecting and quantifying cancer from medical information supplied to the system. For example, for detecting and quantifying breast cancer, the algorithm may detect and quantify micro- calcifications in mammogram images.
  • the algorithm may in general include (1) detecting and grouping calcifications into clusters, (2) classifying types of benign clusters, (3) quantifying clusters that are potentially malignant with a ‘Q factor’ as discussed herein, and (4) saving output quantities to evaluate performance.
  • a first algorithm generates an indication of cancer and a second algorithm generates a cancer score.
  • Figure 3 is a flow chart of a method for detecting and quantifying cancer. The method 300 may be performed using one or more algorithms as described herein, or with assistance from an algorithm.
  • the method 300 may be performed by a computer program product comprising non-transitory computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of the method 300.
  • the method 300 may in general be performed on one or more pieces of medical information, e.g., one or more images, for detecting an object, area, region, feature, data point, piece of information, etc., of interest (e.g., a calcification, lesion, mass, tumor, and the like in a medical image of tissue).
  • the method 300 may include initializing an algorithm.
  • memory structures may be declared and various free parameters for a model may be set.
  • the parameters and model choices are spread throughout code of a computing device.
  • all model parameters are set in a single place during the initialization of the algorithm, along with a clear description of each parameter including the specific section of the code/algorithm where it is utilized. This may be provided through an interactive feature for a user, e.g., a graphical user interface of a user device. In this manner, the parameters may be adjusted or inputted by a user of the method 300. In a non-interactive version, a piece of medical information may simply be received, e.g., a single image to analyze. [0146] As shown in step 304, the method 300 may include reading data.
  • the data may include a DICOM header and image data.
  • the DICOM header may contain a range of useful information including without limitation, the side (i.e., left or right), orientation, view, protocol, date of procedure, and so forth, many of which may be listed in a filename convention. This information may be extracted for use by the algorithm—for example, in order to compare results from multiple views, or from a time series of images.
  • DICOM tags include without limitation: (1) pixel spacing (e.g., hex tag - (0028x,0030x)), which may be useful to scale the image in terms of real physical dimensions (e.g., mm), which can compute a ‘Q factor’ consistently; (2) diagnostic vs screening (e.g., hex tag - (0032x,1060x)), which may allow for inclusion or exclusion of diagnostic images from studies; and (3) patient orientation (e.g., hex tag - (0020x,0020x)), which may allow for displaying the images in a consistent manner, i.e., in the same orientation as used by radiologists in typical computer-aided design (CAD) systems, which can be advantageous when contour data is returned for display and/or analysis.
  • pixel spacing e.g., hex tag - (0028x,0030x)
  • diagnostic vs screening e.g., hex tag - (0032x,1060x)
  • patient orientation e.g
  • a predetermined orientation may be assigned (e.g., for mammograms – where the nipple points to the left in all images as is the industry standard).
  • an orientation where burned-in lettering is displayed/oriented correctly may be utilized.
  • the image data may be read in with ⁇ 1 x ⁇ 2 elements and converted to a 4- byte real array of intensities I( ⁇ 1, ⁇ 2) for contouring as a final step for reading data.
  • the method 300 may include computing contours for the image. For this step 306, the intensity levels for contouring may first be selected, where an example will now be described.
  • the side and view information are burned into an image at 100% of the maximum possible intensity, while the intensity levels within tissue in the image can be significantly less than this peak value.
  • contouring algorithms may return all contours within a given domain, here, an implementation may only be interested in keeping a subset of contours that include contours that are (a) closed and (b) where the contour value is larger than the surrounding area outside.
  • This may be the first contour selection criteria identified in method 300.
  • the closed loops that are found can be of two possible types: (1) the contour value is larger than the surrounding values in the image (i.e. such contours enclose a bright spot, and potential calcifications). (2) the contour value is below the surrounding values in the images (i.e., such contours enclose a darker region, which may be ruled out)
  • the algorithm may only select the subset of contours of the first type.
  • the method 300 may then analyze the data to determine whether there are closed contours and/or whether the contour value is larger than the surrounding area outside. If contours do not meet these criteria, then they may be discarded as explained below. [0150] As shown in step 310, the method 300 may include discarding contours that do not meet desired criterion, e.g., contours that are not closed. [0151] As shown in step 312, the method 300 may include analyzing the geometry and contrast of the contours, e.g., the closed contours that were not discarded by the previous step.
  • Contouring an image e.g., a mammogram
  • the method 300 may include detecting an object, e.g., detecting an object in the image.
  • the object may be an external object, or other regions where detection may be beneficial, e.g., for exclusion in an analysis by the algorithm.
  • an image may include external objects, such as implants or diagnostic clamps.
  • some images may include regions with exceptionally poor contrast. Often there are small scale contours within the interior of these regions, which can appear as calcifications to the algorithm, and thus trigger false positives.
  • the algorithm can be configured to find one or more such regions in each image, e.g., based on the following contour selection criteria: A ⁇ 800 mm 2 and I ⁇ 0.5I scale and C ratio > 0.22 , (Eq.2) corresponding to large bright regions with fairly smooth boundaries.
  • contour selection criteria A ⁇ 800 mm 2 and I ⁇ 0.5I scale and C ratio > 0.22 , (Eq.2) corresponding to large bright regions with fairly smooth boundaries.
  • contour selection criteria A ⁇ 800 mm 2 and I ⁇ 0.5I scale and C ratio > 0.22 , (Eq.2) corresponding to large bright regions with fairly smooth boundaries.
  • contour selection criteria For images that have objects, a number of contours may satisfy this criterion and these will typically be nested inside one another.
  • the contour that maximizes the triple product AIC ratio of these selection criteria may be selected. In most cases, finding the precise boundary may not be necessary, since the method 300 may just be attempting to exclude the interior area where false positives can form.
  • the method 300 may then select contours for discarding (step 318) or keeping (step 320).
  • the contours may be the contours computed above, which are then searched through for identifying potential calcifications or other features of interest.
  • the micro-calcifications of interest typically occur for a fairly narrow range of sizes (contour areas).
  • the micro-calcifications can feature a range of contour shapes, intensity levels, and contrasts (i.e., spatial gradients).
  • Contours may be excluded that are within the interior of an object identified above in step 314. Also, contours may be excluded that are within a specified distance from the edge of the tissue or the edge of the image using the interior flag variable computed in step 312. [0156] 2.
  • Contours may be included that are within the following range of areas and gradient scale: 0.003 mm 2 ⁇ A ⁇ 800 mm 2 and L g ⁇ 1.3 mm , (Eq.3) and that also meet one of the following criteria (a)–(e), which are provided again by way of example: [0157]
  • (a) Contours may be kept that enclose relatively bright regions with relatively desirable contrast values (these values may be selected by a user/administrator) and that are within a range of shapes that are not too highly deformed. This criterion may capture many of the most obvious calcifications.
  • contours may be kept that satisfy the following criteria: I o > 0.67I scale and C ratio > 0.65 and C in > 1.06 and C out > 1.22 [0158]
  • Contours may be kept that have relatively weak contrast if the area is within the correct range for the smaller (weak) calcifications, and if the contours are more nearly circular or have shorter gradient scales.
  • contours may be kept that satisfy the following: A ⁇ 0.30 mm 2 and [(C ratio > 0.80 and C in > 1.04) or (C ratio > 0.65 and L g ⁇ 0.3 mm)] [0159]
  • Contours may be kept that are relatively large and bright.
  • contours may be kept that satisfy the following: (I o > 0.75I scale and C ratio > 0.69 and A > 1.2 mm 2 ) or (I o > 0.90I scale and C ratio > 0.90 and A > 4.0 mm 2 ) While these may be too large to be cancerous, these types of contours may be markers of type-2 benign clusters (e.g., fatty necrosis, etc.). These benign clusters may be ignored entirely in the analysis. However, the calcifications within these benign clusters may also have a range of shapes and sizes, some of which overlap with the selection criteria in (a)–(b) above. Thus, the method 300 may find all of the members of the type-2 clusters, and group their smaller members with these larger shapes.
  • contours may be kept that help reduce false-positives.
  • contours may be kept that satisfy the following: (I o > 0.62I scale and C ratio > 0.67 and C out > 2 and A > 0.2 mm 2 ) or (I o > 0.60I scale and C ratio > 0.50 and 3 mm 2 > A > 1.5 mm 2 ) or (L g ⁇ 0.4 mm and C ratio > 0.67 and 3 mm 2 > A > 1.3 mm 2 )
  • Including these types of contours may allow the method 300 to reduce some common types of false-positives when contours are grouped into nested structures, as described below. For example, some larger calcifications are hollow in the center, resulting in a ring-like structure.
  • (e) Contours may be kept that include relatively high central intensity, even if the contrast is relatively poor.
  • contours may be kept that satisfy the following: (I o > 0.90I scale and C ratio > 0.50)
  • the method 300 may include cataloging the contours, e.g., cataloging the contours that are kept by the steps listed above.
  • the method 300 may include determining whether to discard the contours, in which case the method proceeds to step 324, or whether to order and select the contours, in which case the method proceeds to step 326.
  • the number of contours selected by (a)–(e) above into the full catalog is still quite large ⁇ 2 x 104. The more contours that are kept can increase the overall sensitivity, but can also lead to much longer analysis times in following steps of the method 300.
  • the method 300 is scoring the contours on a relative scale for each image.
  • the method 300 may include grouping the contours into nested structures/hierarchies.
  • N max 6000 contours stored within the primary catalog (or whatever number is selected above). However, in most images, only a small fraction of these contours will correspond to true calcifications. Furthermore, as illustrated in Figures 4 and 5 described below, there may be at least several nested contours associated with each, and up to ten or more nested contours for calcifications with a strong intensity contrast. For any given calcification, it is desirous to identify a contour that characterizes the shape of the structure. To accomplish this, the primary library may be sorted through and the contours may be grouped into nested hierarchical structures.
  • the outer most contour (parent) in each nested series may correspond to the shape, while the inner nested contours (children) can be used to precisely measure variations in contrast across the structure, as described below when selecting calcifications from the nested contours.
  • the Nmax contours are first sorted according to the area enclosed by each. Next, starting with the largest contour (call top level the “parent”), the contour library is searched to find the next smallest contour in the list that exists inside the area enclosed by the top level (parent). This would be the first “child”, which is grouped as part of this nested structure, and exclude it from our subsequent searches below.
  • the library is searched again to find the next largest contour that is inside the top-level parent (this one will have a smaller area than the first child). Normally, in simply nested structures, this contour would also be inside the 1st child, but that is not always the case. One could have multiple “peaks” inside the overall parent contour, and can be useful for looking at the internal structure of masses.
  • the library is searched until no more contours that are inside the top-level parent. Then, the next largest contour in the library, which has not yet been grouped, is searched to repeat this process. [0167] After completing this step, there may be a list of the outer contours for each nested series, and a list of pointers to the inner nested contours for each of these structures.
  • Some fraction of these nested structures may correspond to calcifications, but others may not.
  • the following properties for each nested series may be computed, which are provided by way of example and not of limitation: [0168] 1. Contour Derivatives – To identify calcifications, it may be desirous to precisely characterize how rapidly the intensity varies across the structure. Already, the method 300 may have computed several quantities that characterize this same general idea in an average sense (i.e., the inner Cin and outer Cout contrast described above) and for a local gradient scale – Lg. Once the contours are grouped into nested structures, the method 300 may compute the fractional change in area and/or intensity between any two nested contours in the structure.
  • ⁇ A may be the minimum fractional area change between any two nested contours in the structure
  • ⁇ I corresponds to maximum fractional intensity change between the inner nested contours and the outermost contour that defines the shape.
  • Small values of ⁇ A ⁇ 1 may correspond to tightly nested contours, where the local gradient in intensity is large, while values of ⁇ I > 1 measures the fractional intensity variation across the set of nested contours (very similar to the inner contrast C in discussed above).
  • F(c max , c min , x) C norm exp[–(x/c max ) 2 ](1 – exp[–(x/c min ) 2 ]) (Eq.7) where is a normalization constant , and c min , c max are constants that set the minimum and maximum scales of interest for any given quantity.
  • the selection function is applied to spatial separation, and the constants (r min , r max ) are used to select a relevant range of separations.
  • the function ⁇ may be constructed to reach a maximum value of unity between this range of scales, and then to fall off exponentially for separations outside this specified range. Likewise, when applied to Eq.
  • the selection function may maximize within the specified range of areas (a min , a max ) and fall off rapidly outside this range.
  • dQ i may have desired properties.
  • the value of dQ i may remain small.
  • the value of dQ i may increase quadratically with the number of neighbors, if they are within the right range of separations, and have the right range of sizes to be of concern.
  • the value may increase with the central intensity of the potential calcifications, and with the number of nested contours within each, both of which may correlate with visibility to the human eye.
  • the method 300 may include selecting calcifications from the nested contours, where, if the calcifications are not selected the method 300 discards the calcifications as shown by step 332, and where, if the calcifications are selected, the method 300 proceeds to step 334. [0171] At this point in the method 300, all contours may have been found and characterized, contours may have been eliminated that occur inside objects, the most interesting contours may have been selected, and the contours may have been grouped into nested structures, with the outermost contour representing the shape and the inner contours providing additional information on the internal gradients.
  • the final selection for calcifications may be made based on the following two criteria, which are provided by way of example and not of limitation: [0172] 1. Strong Calcifications ⁇ ⁇ A/ ⁇ I ⁇ 0.15 [0173] This threshold on the contour derivate (see Eq.4) may capture most of the clear calcifications with sharp boundaries. This selection may be made regardless of whether the calcification has any close neighbors. [0174] 2. Weaker Grouped Calcifications ⁇ ⁇ Qi ⁇ 3 [0175] This threshold on the grouping parameter (see Eq.5) may select weaker calcifications that are grouped together appropriately (as discussed above). Note that the threshold value of ⁇ Qi may be dependent on the scaling parameters chosen in Eqs.5–7.
  • the method 300 may include grouping calcifications into clusters. After identifying all calcifications within the image, next, they may be grouped into clusters according to the following procedure.
  • the method 300 may start with the calcification with the largest number of neighbors, which is used to form the first cluster.
  • New calcifications may be recursively added to this cluster, until there are no remaining calcifications within a distance Rc of any member.
  • the method 300 may proceed to the next unassigned calcification and repeat this process until all calcifications that should be grouped into a cluster have been assigned.
  • only calcifications with at least two neighbors i.e., three members
  • Calcifications that are not assigned to a cluster may be ignored completely for the rest of the method 300.
  • This approach for forming clusters may be advantageous, and may depend only on the scale Rc. In most images, the actual clusters in the tissue (e.g., breast) are well separated, and this approach works well.
  • the method 300 may include computing cluster properties. To aid in the classification process, it may be useful to characterize the distribution of calcifications within the cluster.
  • the cluster centroid can be defined as: , (Eq.8) where w n is the weight for the nth calcification and N c is the number of calcifications within the cluster.
  • w n the weight for the nth calcification
  • N c the number of calcifications within the cluster.
  • a displacement matrix for each cluster may be defined: , (Eq.9) where again the contrasts for the weights may be employed.
  • This symmetric positive-definite matrix may have two real eigenvalues (e 1 , e 2 ) and two eigenvectors (d 1 ,d 2 ), which can be used to define the following quantities, which are provided by way of example and not of limitation: 1.
  • Cluster Half-Length ⁇ where e 1 is the maximum eigenvalue of D ij 2.
  • Cluster Half-Width ⁇ where e 2 is the maximum eigenvalue of ⁇ ij 3.
  • Aspect Ratio ⁇ A w/L 4.
  • the method 300 may also compute the mean and standard deviation of the geometric and contrast properties described above, including, e.g., intensity, contrast, area, etc. [0179] As shown in step 338, the method 300 may include classifying clusters as benign, in which the method 300 proceeds to step 340, or classifying clusters as possible cancer or cancerous in which the method proceeds to step 342. [0180] Calcifications may form within tissue over a wide range of scales and for a variety of reasons. Calcifications may be of benign origin, or clusters of micro-calcifications may be indicative of cancer. Typically, benign calcifications are more common.
  • the large majority of clusters identified by the method 300 are expected to be of benign origin.
  • the strategy of the method 300 may thus be to identify and exclude the most common types of benign clusters, and then to score the remaining clusters with the ‘Q factor’ as described below.
  • the method 300 may classify a type for each the cluster/calcification, which is illustrated by step 340.
  • Some types of benign clusters are provided below by way of example and not of limitation.
  • Type-1 Vascular
  • a common type of benign cluster is associated with vascular calcifications. While these are of potential interest in studies of cardiovascular disease, these clusters may not be relevant to cancer (e.g., breast cancer).
  • the method 300 may identify a large number of calcifications organized along the vessel wall.
  • the range of spatial scales and separation distances for these vascular calcifications often overlaps with the micro-calcifications relevant to cancer (e.g., breast cancer).
  • cancer e.g., breast cancer
  • other approaches to exclude these from consideration may be utilized.
  • Vascular calcifications are usually easy to spot visually, since they are well- organized along the wall of the tubular vessel. As such, at least two strategies may be used to automatically identify these vascular calcifications, i.e., using an algorithm or the like.
  • the high-degree of spatial correlation can be measured, e.g., by performing a regression analysis on the positions of the calcifications.
  • An alternative and potentially complementary approach is to employ edge detection techniques to identify the vessel walls, and then to exclude calcifications that are located along these structures.
  • the approach is based on performing a regression analysis to a polynomial of specified order.
  • the steps in this vascular detection subroutine may include without limitation: [0186] 1. Only accepting clusters having between a certain number of members (e.g., between 3 and 500 members). Depending on the number of members, there may be a look-up table to specify (1) the order of the polynomial, (2) the threshold tolerance in the fit, and (3) the number of points that can be dropped.
  • first order and second order polynomials are used, and the tolerance allowed varies from a range of values, e.g., 0.01 to 0.036. These tolerances may correspond to a normalized chi-squared of the fit (i.e., normalized to the length of the polynomial curve).
  • the cluster may be rotated into a frame where the x-axis is aligned with the principal axis of the cluster computed above.
  • a polynomial least-squares regression may be performed, and the chi-squared fit parameter can be computed and normalized by the length of the curve.
  • the cluster may be identified as vascular (type-1), otherwise the specified number of outlier points may be dropped, and the fit may be recomputed to see if the method 300 can find one within the tolerance specification.
  • the method 300 may attempt to split them into smaller subgroups, and then apply the polynomial fitting procedure to the subgroups.
  • the method 300 has two different strategies for splitting and fitting, and the algorithm is set to employ one or both of these (i.e., apply the second if the first fails). If the routine finds any portion of the cluster that is well fit by ‘q polynomial,’ then the entire cluster may be classified as vascular. [0189] 4. Even with the above variations, it may be difficult to pick a tolerance for the fitting threshold that identifies all of the vascular clusters, while excluding ones that are potentially malignant. Thus, the method 300 may include a final check that applies to clusters that have a fitting tolerance somewhat above the threshold (and thus would not be identified as type-1), but where the principal axis aligns with a clear vascular cluster.
  • Type-2 Large Calcifications and Fatty Necrosis
  • Another common type of benign clusters is associated with larger calcifications and fatty necrosis. These clusters may include larger members, with areas that may be significantly larger than micro-calcifications associated with malignancy.
  • the ‘Q score’ described below may be relatively small. However, in other cases, there may be an overlap in the relevant range of areas with malignant clusters. Furthermore, the method 300 may find a number of smaller structures in the vicinity of the larger calcifications, which can then give rise to false-positives as described below. [0192] In terms of geometric properties, these benign clusters may be characterized by relatively larger areas and by their fairly dense grouping. To this end, a cluster library may be created in which the geometric and contrast properties described above are extracted for interesting clusters for use and evaluation by the method 300.
  • the cluster library may show that malignant clusters tend to be more dispersed (lower Pf ) with a smaller range of areas, while the benign clusters are more densely packed (larger Pf ) and/or larger areas.
  • Type-3 Diffuse Round Calcifications
  • Another type of false-positive may be clusters that are characterized by diffuse, nearly circular calcifications. These are often fairly bright and have relatively good contrast, and thus many calcifications are often identified.
  • the range of calcification sizes may be relatively similar to malignant micro-calcifications, but they tend to be spread over broader areas of the tissue (e.g., breast tissue), and also they may often appear on both sides in a similar manner.
  • a technique for identifying these clusters uses the Cratio and Pf. Another technique may compare different sides of an image, e.g., comparing the left and right sides.
  • the method 300 may include quantifying clusters with a ‘Q score.’
  • the Q score as discussed herein may refer to a measurement, e.g., a number that quantifies the likelihood of malignancy for each cluster.
  • the Q score may include an analytic function of the geometric and contrast properties of the calcifications within each cluster, as well as their detailed spatial arrangements.
  • the Q score may quantify aspects of the calcifications more quickly, accurately, and consistently than is possible for a human. In comparison to the black box approach of a neural net, the Q score enables a clear explanation in physical terms about how the method 300 is scoring any given cluster.
  • Some features built into the functional form for the Q score have already been discussed above in association with the grouping parameter dQi. It is well-established that clusters of micro-calcifications associated with cancer occur for a fairly limited range of spatial scales and separation distances.
  • the function may increase monotonically with these features, and allow sufficient flexibility to adjust free scaling parameters in order to optimize the overall performance.
  • An example of this function is: , (Eq.11) , where M o is a free parameter, M is the number of calcifications in the clusters, and all other symbols have been defined herein.
  • the Q parameter may be roughly analogous to a potential energy or the like for the cluster (assuming a particular form for the pair-wise interactions), where the free scaling parameters have been adjusted to maximize the energy for malignant clusters.
  • the Q parameter defined in Eq.11 may be a sum over the clustering parameter for each calcification (see Eq.5). From this point of view, the approach for selecting calcifications may be connected with the overall strategy for scoring the significance of the final clusters. [0198]
  • a program performing a multi- dimensional optimization may be used in order to find values that maximize the area under the curve (AUC) for the receiver operating characteristic (ROC) curve.
  • AUC area under the curve
  • ROC receiver operating characteristic
  • the method 300 may include saving results. For example, after completing the analysis for each image, the following results may be saved, which are provided by way of example and not of limitation: 1. List of clusters identified in the image, including the Q score and the cluster properties; 2. Outer contours for each of the calcifications within each cluster, along with the geometric and contrast properties for each of these shapes; and 3. Information to generate the ROC curves. This information may be extracted and saved along with the Q score for each cluster. [0200] Then, the method 300 may terminate in step 346. [0201] It will be understood that any values recited above with respect to the method 300 (or otherwise herein) are provided by way of example only, and are not meant to limit the embodiments described herein.
  • Figure 4 depicts a medical image of calcifications in a patient’s tissue.
  • the image 400 in the figure may represent an original medical image of a patient’s tissue, e.g., a mammogram or the like.
  • the image 400 may include calcifications 402, which may need to be identified for detecting whether cancer is present.
  • Figure 5 illustrates an example of a selection process for a malignant cluster of micro-calcifications.
  • the figure includes a first image 510 and a second image 520.
  • the first image 510 may represent the contours 512 selected by any of the criteria outlined above. More specifically, these may be the contours 512 before they are grouped into nested hierarchies.
  • the calcifications most noticeable to the human eye may have many nested contours 512, which are grouped together into nested structures as described above (these nested structures are shown, for example by the bubbled area 514 in the figure).
  • FIG. 6 illustrates an example of the identification of a cluster of calcifications.
  • the figure shows the identification of type-2 clusters with fatty necrosis using techniques described herein.
  • the figure includes a first image 610 and a second image 620.
  • FIG. 7 is a graph showing an example of a packing fraction as a function of ⁇ Ai>Amax for exemplary clusters.
  • the x-axis is the ⁇ Ai>Amax, where ⁇ Ai> is the average area of the calcifications within the cluster and Amax is the area of the largest calcification in the cluster.
  • the y-axis is the Pf, i.e., the packing fraction (see discussion above).
  • the lighter lineweight points 702 (x’s) correspond to malignant clusters, while the heavier lineweight points 704 (o’s) correspond to various type-2 clusters.
  • this specific graph 700 only clusters with less than 30 calcifications are included for clarity and by way of example, but clusters with more calcifications may also or instead be used.
  • An approximate boundary separating the benign from malignant clusters is given by the line 706.
  • the figure may represent the derivation of criteria for identifying type-2 clusters that are used in the techniques described herein, where each point in the graph 700 corresponds to a single cluster in an exemplary study, which was used to refine the techniques described herein.
  • an implementation may include a method for determining a cancer score. Determining a cancer score may be accomplished through the user of a cancer score engine (and its components) or the like as described herein. In the method, an event of interest may be defined. An event of interest may be any object of interest to be identified from data.
  • Examples of events may be cancerous lesions, masses, physiological anomalies, and the like.
  • the method may gather variables for the events of interest, such as x1, x2...xn.
  • the variables may be a minimum number of variables that allow the event to be predicted and a score to be generated, e.g., a cancer score.
  • the variables may be gathered by identifying a number of variables and then discarding variables that are not predictive and/or show the wrong behavior for the event of interest.
  • the event of interest may be breast cancer and the variables may include closed intensity contours of calcifications in mammogram images, gradients of the calcifications, one or more characteristics about each calcification, such as perimeter, contrast and/or a number of neighbors, a texture and shape of each calcification and/or a hierarchical structure of the calcifications in a cluster, such as how tightly the calcifications are nested, if there are nested levels of calcifications and the like.
  • the variables for mammogram images may also or instead include other variables. This method may also include a clustering of individual calcifications with their neighbors and then grouping into prototype clusters which may be ordered based on a number of neighbors.
  • the values of these variables may be determined by the computer analysis of the mammogram images.
  • the method may calculate Q0 for each cluster of calcifications in which Q0 is a function of the variables calculated over each cluster.
  • Q0 may include use of any of the functions described herein.
  • the method may calculate a Q1, where Q1 is equal to (Q0) x (a penalty function). The penalty function may selected be such that Q1 incorporates a classification scheme.
  • the penalty function may discard calcifications that are spaced too far apart and thus are unlikely to be suspicious cells.
  • the method may then normalize Q1 to generate a cancer score.
  • the parameters of Q0 and Q1 may be optimized to maximize the area under a well-known ROC or free receiver operating characteristic (FROC) curve.
  • FROC free receiver operating characteristic
  • the cancer score may have thresholds and classify the clusters of calcifications into Type-1, Type-2 and Type-3, where Type-1 identifies a linear or curvilinear cluster (benign lesion), Type-2 identifies a cluster that has one or more calcification members that are exceptionally large and/or bright in the mammogram image, and Type-3 identifies a cluster that is likely malignant.
  • the method may display the cancer score in some form. For example, as shown in figures included herein of medical images, a cluster of calcifications may be classified as cancerous thereby warranting a biopsy.
  • the present invention includes patients' age in the predictive model described above. [0213] Specifically, in addition to the physical features of each calcification described in previous steps, as an individual structure and as part of a cluster of micro- calcifications, patients' age is supplied as an additional feature, to be used to construct a random-forest model.
  • This random-forest model taking as input all of the physical features, including the Q-score provided by the Q-algorithm described above, and a patient's age, provides an overall final (random-forest) score for each region of interest identified by the Q- algorithm.
  • the regions of interests are ranked on the scale of suspiciousness based on this random-forest score.
  • This additional age feature led to a significant increase in the AUC of the ROC, from 0.927 to 0.959. Performance was improved at all operating points.
  • each calcification as an individual structure and as part of a cluster of micro-calcifications, together with clinical data, are used to construct a random-forest model.
  • This random-forest model taking as input all of the physical features, including the Q-score provided by the original Q-algorithm, provides an overall final (random-forest) score for each region of interest identified by the Q-algorithm. The regions of interests are ranked on the scale of suspiciousness based on this random-forest score.
  • Detecting, Scoring, and Predicting Cardiovascular Disease Risk Using Medical Imaging [0216] Any calcification in the artery, regardless of where the artery is, indicates presence/onset of artery disease.
  • BAC Breast arterial calcification
  • ASCVD which includes carotid artery disease, CHD, and peripheral arterial disease
  • CKD Chronic Kidney Disease
  • a risk calculator can be developed to relate the BAC score to the risk for an event.
  • the BAC score may also be incorporated into the patient’s medical record. This can then be monitored and when the BAC score is in the range that signals an elevated risk for ASCVD and CKD the radiologist can alert the patient’s physician.
  • their BAC score is relatively high or increasing rapidly, further diagnostics and possibly treatment can be initiated, expectedly preventing ASCVD and/or CKD.
  • BAC score (especially when documented over time) has the potential to aid substantially in the prevention/early detection of ASCVD and CKD.
  • Mammography on the other hand is already widely used as a screening tool for breast cancer, so to measure the BAC score no additional imaging or radiation is required. Therefore, these disadvantages mentioned for measuring the CAC score would not apply to measuring the BAC score and using this to evaluate the risk for ASCVD.
  • One of the important benefits of a score evaluated on screening imaging instead of diagnostic imaging, is that the BAC score over time can be measured and documented. Another important benefit is that the BAC score is measured not only for symptomatic women as with a diagnostic tool, but also for asymptomatic women.
  • the physician of the patient can then use the (development over time of) the BAC score in addition to other clinical data to decide if further diagnostics and possibly treatment is necessary.
  • the BAC score (especially when documented over time) has the potential to aid substantially in the prevention/early detection of ASCVD and CKD.
  • CHD is the most common ASCVD and CAC score is an established diagnostic tool for CHD
  • the first step was to look at the correlation between BAC and CAC. If such a correlation exists, then one can use BAC as a proxy for CAC and immediately open up the use of BAC as a preventive and/or diagnostic tool for CHD. This has led to a number of studies to assess whether there is a strong enough correlation between BAC and CAC.
  • the suspicion codes may also be forwarded to other healthcare functional units, such as the Full-Field Digital Mammography (“FFDM”) system, Radiology Information System (“RIS”), Clinical Information System (“CIS”), or any Hospital Information System (“HIS”) which can receive a suspicion code which may indicate the need for treatment by a physician or other healthcare worker.
  • FFDM Full-Field Digital Mammography
  • RIS Radiology Information System
  • CIS Clinical Information System
  • HIS Hospital Information System
  • the suspicion codes may be forwarded in real-time such as through a text message, email, web app, phone app, or other methods.
  • a suspicion code may highlight a particular slice within a multi-slice environment, or a particular frame in a multi-frame video, for example.
  • a notification result file may be generated to include the suspicion code.
  • such notification result file can be in the form of one of the file formats or communication formats discussed previously.
  • the worklist may be updated to display for further review.
  • An example of an updated worklist is depicted in Figure 8.
  • the worklist can highlight images or studies within a listing of all pending reviews, can sort the listing of all pending reviews in order of suspicious and not suspicious, or can list only the suspicious images or studies for priority review.
  • An individual image or study within a workflow may also be highlighted with an indication that the image or study comprises a suspicion code, and an image or study may also include markings such as ROI, scoring associated with an ROI, and the like.
  • Neural Network Architecture Data generated by industrial systems can be analyzed for the presence of one or more anomalies by comparing a normal state to a changed state. However, better precision and accuracy of such analysis has been the subject of ongoing research and development, and there continues to be a need for even more precision and accuracy in anomaly detection. [0234] In the healthcare field, data is often analyzed with respect to the sensitivity and specificity of the analysis. The terms “sensitivity” is the ability of a test to correctly identify those with a physiological anomaly (true positive rate), whereas the term “specificity” is the ability of the test to correctly identify those without the physiological anomaly (true negative rate).
  • sensors examples include acceleration sensors, acoustic and sound sensors, automotive sensors, capacitance sensors, chemical sensors, digital component sensors, electric current sensors, magnetic sensors, flow sensors, fluid property sensors, force sensors, humidity sensors, ionizing radiation sensors, mass air flow sensors, photo optic sensors, piezo film sensors, position sensors, pressure sensors, rate and inertial sensors, speed sensors, temperature sensors, torque sensors, traffic sensors, ultrasonic sensors, vibration sensors, and water-level sensors.
  • the data may be provided by manual entry, such as by a physician or other professional.
  • the data which can be analyzed for the presence of an anomaly can be static or real-time.
  • anomaly detection under supervised anomaly detection, two static images can be categorized as “normal” and “anomaly” respectively, and that categorization can be used to train an algorithm to detect anomalies in other images.
  • a continuous series of data can be analyzed unsupervised to detect rare occurrences or bursts of unexpected activity.
  • a system designed to detect anomalies can alert technical staff each time an anomaly is detected so that they can more rapidly review the condition. Therefore, anomaly detection is particularly suitable for companies in the manufacturing, oil and gas, transportation and logistics, aviation, automotive, and energy and utilities, and healthcare fields.
  • a detected anomaly can be found in a computer network environment, manufacturing plant, assembly line, industrial control system, maintenance cycle review, energy production, hospital system work flow, and the like.
  • data which can be analyzed for the presence of an anomaly can be generated by a sensor using one of or a combination of different modalities including X-ray imaging, CT scan or imaging, MRI, PET, SPECT, US, endoscopy, thermography, medical photography, nuclear medicine functional imaging, elastography, photoacoustic imaging, echocardiography, functional near-infrared imaging, magnetic particle imaging, and the like.
  • the resultant data may be captured in the form of a medical image which can be used for the detection of lesions and other physiological anomalies in various parts of the body, and include both two-dimensional (“2D”, e.g., X-ray) and three- dimensional (“3D”, e.g., 3D tomography) imaging.
  • 2D two-dimensional
  • 3D three- dimensional
  • a preferred medical imaging format is DICOM® (Digital Imaging and Communications in Medicine) which is the international standard to transmit, store, retrieve, print, process, and display medical imaging information.
  • DICOM® Digital Imaging and Communications in Medicine
  • Those of skill in the art will also recognize various other image formats which can be used for anomaly detection, including JPEG, PNG, TIFF, GIF, and the like.
  • Medical images can also include structured reporting (SR) which is used for the transmission and storage of clinical documents which can accompany the medical image.
  • SR structured reporting
  • Examples of lesions and other physiological anomalies which can be imaged using any one, or combination, of the modalities above include Abdominal Aortic Aneurysm, Abnormal Vaginal Bleeding, Alzheimer's Disease, Anal Cancer, Angina Pectoris, Appendicitis, Arterial Calcifications, Arthritis, Benign Prostatic Hyperplasia, Blood Clots, Bone Fracture, Brain Tumors, Breast Cancer, Breast Lumps, Carotid Artery Stenosis and Restenosis, Cervical Cancer, Cholecystitis, Chronic Obstructive Pulmonary Disease, Cirrhosis of the Liver, Colorectal Cancer, Crohn’s Disease, Croup, Cystic Fibrosis, Dementia, Dense Breasts, Diffuse Interstitial Lung Disease, Diverticulitis, Endometrial Cancer, Epi
  • any lesion characterizing a disease that is structurally and/or compositionally distinct from surrounding healthy tissue can be highlighted by an appropriate imaging modality.
  • medical image properties e.g., size, contrast, presentation of normal tissue and anomalies, etc.
  • 2D mammographic images are typically about 3300 x 4000 pixels
  • histopathology image size can be 60,000 x 60,000 pixels (typical 15mm x 15mm tissue slice, scanned at 40X magnification, or 0.25 microns/pixel).
  • Anomalies in medical images are also depicted with different brightness depending on the imaging modality. For example, bleeding or hemorrhage is typically shown bright in MR, X ray, and CT images.
  • a myocardial infarction region appears dark in X ray images, CT images, and in T1 MR, but bright in T2 MR.
  • tumors are dark in X ray and CT and T1 MR images, unless calcified, but bright in T2 MR.
  • multiple sclerosis plaques can appear dark in CT images and T1 MR, but appear bright in T2 MR.
  • anomaly detection has historically taken several algorithmic approaches including the classification approach, the segmentation technique, and content-based image retrieval.
  • algorithmic approaches including the classification approach, the segmentation technique, and content-based image retrieval.
  • LVQ Learning Vector Quantization
  • PNN Probabilistic Neural Networks
  • Hopfield networks Support Vector Machines (SVM), Synchronized Oscillator Network, and Adaptive Resonance Theory (ART).
  • CNNs Convolutional Neural Networks
  • Table 1 provides an Outline of Neural Network Architecture.
  • Table 1 provides an Outline of Neural Network Architecture.
  • This new technique (a) enables targeted data augmentation which addresses the sparsity of examples in the data set, and (b) can be used to quickly calibrate the network for new types of images that the algorithm has not been trained on.
  • Enhanced Objection Detection [0249]
  • a new neural network has been developed which builds on state-of-the-art object detection strategies and includes the following combination of features for optimal performance on anomaly detection in industrial systems, including in medical imaging: [0250] • Integrated Pre-Processing [0251]
  • the new neural network includes a built-in filter in which the parameters of the filters are learned during training simultaneously with the parameters of an object detector and a classifier.
  • the new neural network is designed so that as new images are entered into the system for analysis, it can optionally re-train and update the models automatically.
  • the new neural network combines pre-processing, region proposal, and classification into one learning system. This feature enables the neural network to learn an optimized strategy for proposing ROIs and classifying them (see, e.g., Figure 28).
  • the power of deep learning is that it learns features/strategies through examples as opposed to previous techniques which were based on hand-engineered features/strategies. This approach outperforms, both in terms of accuracy and speed, certain prior algorithms where one is required to hand design a region proposal method and then train a classifier on the proposed regions.
  • the new neural network is configured to address the problem that anomalies can occur on a variety of scales, and to increase the neural network efficacy in early detection of such anomalies, including lesions and other physiological anomalies, the new neural network calculates and combines feature maps at multiple spatial scales.
  • the new neural network includes an ROI classification system which takes into account local features (extracted from the ROI) as well as global context (features from the entire image).
  • the new neural network uses high resolution images, and each region uses global features, and the training is performed within the same network (i.e., not a random forest process).
  • Adaptive training [0261] The new neural network is configured to focus attention on the most difficult cases because detection datasets contain an overwhelming number of easy examples and a small number of hard examples. Automatic selection of the hard examples can make training more effective and efficient.
  • the new neural network is configured to be integrated into different backbone architectures of various alternative neural networks including, but not limited to, Perception Neural Network, Feed Forward Neural Network, Artificial Neuron, Deep Feed Forward Neural Network, Radial Basis Function Neural Network, Recurrent Neural Network, Long/Short Term Memory, Gated Recurrent Unit, Auto Encoder Neural Network, Variational AE Neural Network, Denoising AE Neural Network, Sparse AE Neural Network, Marakov Chain Neural Network, Modular Neural Network, Hopfield Network, Boltzmann Machine, Restricted BM Neural Network, Deep Belief Network, Deep Convolutional Network, Deconvolutional Network, Deep Convolutional Inverse Graphics Network, Generative Adversarial Network, Liquid State Machine Neural Network, Extreme Learning Machine Neural Network, Echo State Network, Deep Residual Network, Kohonen Self Organizing Neural Network, Support Vector Machine Neural Network, Neural Turing Machine Neural Network, Convolution
  • the new neural network is configured to achieve optimal performance, achieved in part by processing the images at high resolution. For example, in the radiological field, some detected microcalcifications are barely above the resolution limit.
  • the new neural network comprises a modified receptive field (the region in the input space that a particular neural network’s feature is analyzing) of the networks to make them better suited for high resolution images that are processed.
  • the new neural network uses loss functions appropriate for anomaly detection where positive cases (for example, cancers or other lesions when analyzing human physiology) are much more rare than negative cases.
  • Fusion layer In the context of detecting physiological anomalies in humans and other animals, the new neural network can associate clinical data (e.g., age, family history, genomics, etc.) with the neural network system.
  • Extendable In the context of detecting physiological anomalies in humans and other animals, the new neural network can associate clinical data (e.g., age, family history, genomics, etc.) with the neural network system.
  • the new neural network can treat each image separately or in combination with other views (for example, LLC versus RCC in mammography studies) or data or images created prior to the present analysis. It is also designed to be extendable from 2D to 3D image analysis.
  • the new neural network code can utilize any number of GPUs for distributed training which results in enhanced neural network training speeds.
  • the new network can scale up to much larger problems such as 3D tomosynthesis.
  • Ensemble Modeling and Cascading Classifiers [0275]
  • the new neural network offers the capability to include cascading classifiers (e.g., random forest, CNN) as well as ensemble modeling where outputs of different object detectors and classifiers can be combined to achieve optimal performance.
  • cascading classifiers e.g., random forest, CNN
  • ensemble modeling outputs of different object detectors and classifiers can be combined to achieve optimal performance.
  • High performance The new neural network processes high resolution images at 3 seconds/image on a standard GPU.
  • the new neural network can optionally detect and provide outlines of even weak micro-calcifications (see, e.g., Figure 29). This can be accomplished by a Fully Convolutional Network (“FCN”) segmentation process which can provide masks to highlight individual calcifications.
  • FCN Fully Convolutional Network
  • Optional Rigorous benchmarks [0281] The new neural network provides fine scale segmentation of the algorithm to enable rigorous efficacy checking. For example, in case of micro-calcifications in mammographic images, the system can assess how many detected micro-calcifications match the ground truth micro-calcifications.
  • Weighted Length Score is defined as: where N is the number of images for this patient, s is the severity and l the length of the calcification.
  • a second method, that we refer to as Bradley Score is a physics-based score that takes into account the morphology of individual calcifications which make up BAC (International Publication PCT/US2016/054074).
  • Bradley Score is a physics-based score that takes into account the morphology of individual calcifications which make up BAC (International Publication PCT/US2016/054074).
  • ROC the ROC of BAC as a predictor of CHD. This enables us to judge the viability of one method over another. When comparing two methods for creating a BAC score, the method that yields the highest accuracy of prediction of CHD is the optimal choice.
  • Figure 10 shows the Pearson correlation between our two BAC scores. While there is significant correlation between the two, there are cases where WLS is small but have high Bradley score. This can happen since Bradley score takes into account morphology of calcifications making up BAC. If the BAC is short but has intense groupings of calcifications, the Bradley score would be high but the WLS score would be low. This is illustrated in Figure 11. [0295] Gold standard for fine tuning and selection of BAC score models [0296] As mentioned, our gold standard is the efficacy of BAC as a predictor of CHD. To evaluate this efficacy Random Forest was trained and tested using cross-validation, the positive label being the diagnosis of CHD.
  • BAC may be an indicator of both calcified and noncalcified plaque (not detectable by the CAC test) in the coronary artery because of this appear to be comparable or possibly even better as a predictor for CHD. [0300] It makes sense that this could also hold for the carotid artery and peripheral arteries.
  • Examples of indications which can be utilized in this system include, but are not limited to: • I50.9 - Heart Failure • I50.30 - Congestive Heart Failure • I50.32 - Congestive Heart Failure - Chronic diastolic (congestive) heart Chronic • I25.9 - Chronic ischemic heart disease, Chronic • I50.22 - Chronic systolic (congestive) heart Chronic • I50.20 - Congestive Heart Failure - Unspecified systolic (congestive) heart Unspecified • I50.33 - Acute on chronic diastolic Acute heart failure • I50.23 - Acute on chronic systolic Acute heart failure • I50.810 - Right Heart Failure • I50.1 - Left ventricular failure, unspecified Left • I50.42 - Chronic combined systolic (congestive) Chronic diastolic (congestive) heart failure • I50.43 - Acute on chronic combined Acute (congestive
  • anomaly detection and classification there is a large general class of problems called anomaly detection and classification.
  • the anomalous nature of such events implies that they are rare compared to normal events.
  • the degree of difficulty for event detection and classification can vary significantly across a given class.
  • the number of examples across that range of difficulty is often not uniform, resulting in a paucity of examples for the difficult cases.
  • the lack of sufficient number of training examples in machine learning is especially challenging. This issue is exacerbated for deep learning algorithms which require many more examples than traditional machine learning techniques such as random forest.
  • Various strategies have been proposed to augment the training set with additional examples which rely on mathematical operations on the original examples.
  • the auxiliary data set can include geometrical transformations (e.g., rotation, translation, flipping), distortions, blurring, and contrast adjustments which, while useful, offer limited variations in the data set and may offer only modest boost in performance.
  • geometrical transformations e.g., rotation, translation, flipping
  • distortions e.g., blurring
  • contrast adjustments which, while useful, offer limited variations in the data set and may offer only modest boost in performance.
  • synthetic refers to the result of such generation.
  • generation of a two-dimensional medical image can be accomplished through electronics and software processing of one or more medical images to provide a synthetic two-dimensional result. This may be referred to as a synthetic example, or a synthetic case.
  • devices, systems, and methods discussed herein generally describe the generation of synthetic two-dimensional medical images, and the like, they may also or instead be enabled by the devices, systems, and methods discussed herein.
  • the devices, systems, and methods discussed herein can be adapted to generate synthetic two-dimensional medical images of other cancers including without limitation brain, lung, liver, prostate, bone, cervical, colon, leukemia, Hodgkin disease, kidney, lymphoma, oral, skin, stomach, testicular, thyroid, and so forth.
  • the devices, systems and methods discussed herein can be adapted to generate synthetic two-dimensional images for other physiological anomalies which are not cancers, e.g., bone fractures, arterial aneurysms, vascular blockage, cysts, polyps, congenital and genetic anomalies, and the like.
  • embodiments generally described herein are generating two-dimensional medical images of human tissue, the embodiments may also or instead be applicable to cancer in animals, for example.
  • Enhanced Modeling Techniques [0327] To address the lack of sufficient examples when a machine learning model is not performing well due to lack of sufficient rare event examples, a technique has been developed to create an unlimited number of examples of rare events. The creation of such synthetic cases which augment the original data set leads to increases in sensitivity and specificity in detection and classification problems. For example, in the two-class classification problem, one can perform targeted generation of synthetic examples for the parts of the ROC curve where the model is not performing as well due to lack of sufficient examples. [0328] Therefore, what is provided are devices, systems, and methods to create synthetic examples of the events of interest.
  • the creation of synthetic cases can be done in several ways. We have recently used deep learning for this purpose (See, e.g., international patent application PCT/US2016/054074, which is incorporated herein by reference in its entirety). The creation can also be done in a more manual process as we describe herein. [0330] To illustrate a new approach, the specific problem of breast cancer was considered in which screening is performed with X-ray mammography.
  • the first visible indication of cancer in the X-ray image is from a cluster of micro-calcifications.
  • the visible number of calcifications can range from a few, to a large number (100+).
  • the detectability of these small features depends on a variety of factors, including the size of the cluster, the location of calcifications within the breast, and the mean density of the breast tissue. Larger calcifications within fatty tissue have very good contrast, while the smallest clusters within dense tissue are extremely difficult to detect.
  • these small clusters may occur in the proximity of various types of benign calcifications, making accurate classification more difficult.
  • the approach as it is related to mammography and breast cancer, may be summarized in the following steps: [0333] 1.
  • the first step is to detect the individual micro-calcifications within a biopsy-proven cancer image (e.g., a ground truth image). This can be done, for example, in the following two ways.
  • a biopsy-proven cancer image e.g., a ground truth image.
  • an algorithm for detecting, classifying, and quantifying micro- calcifications such as that described in international patent application PCT/US2016/054074, automatically computes the small regions in each image corresponding to micro-calcification.
  • each micro-calcification will be in the form of a contour (ordered x,y pairs) that describes the outer shape, and a set of pixels within the contour, that describe variations of intensity within the shape.
  • a specified number of buffer pixels are extracted which are immediately outside the contour. The intensity within those buffer pixels is also stored and is used for blending the micro-calcifications into the new images.
  • the mathematical features of the cluster are stored in a library.
  • the above process is repeated for each biopsy-proven cancer, focusing more on the more difficult or unusual cases.
  • the end result is a library of malignant clusters with a precise mathematical description of each individual calcification, as well as their arrangement within a cluster.
  • the above process could also be repeated for most types of false-positives (clusters that existing algorithms incorrectly identify of cancer).
  • the resulting false-positive library could then be used to construct new synthetic examples in the same manner as described below, allowing specific targeting of the most difficult types.
  • the next step is to insert the extracted cancers into new images.
  • a predetermined Breast Imaging Reporting and Data System category 1 or 2 image (negative or benign findings; commonly known as BI-RADS 1 or BI-RADS 2) may be used which is an indication that the image displays a low risk of cancer.
  • the human would proceed as follows: [0336] A. Select a malignant cluster from the library with desired features (number and size of calcifications) [0337] B. Choose a normal image with desired features (mean breast density, complicated density structures, or other complications such as benign calcifications) [0338] C. Pick an appropriate point within the image to insert the cluster [0339] D. Choose any mathematical transformation to apply to cluster.
  • Examples include rotating the cluster by a specified angle, reducing the total number of calcifications to insert (e.g., reducing the size of the cluster), re-scaling the size of individual micro- calcifications, and so on.
  • the cluster is inserted into the image at the specified location.
  • the pixel intensity stored for each micro-calcification is used to over-write the pixel intensity within the normal image.
  • a linear interpolation method may be used to blend the pixel intensity between the inserted cancer pixels and the normal image. This allows micro-calcifications to blend more smoothly into the new image.
  • each individual calcification will depend upon the pixel intensity within the calcification, relative to mean pixel intensity in the region where it is inserted.
  • An acceptable synthetic example should blend fairly smoothly, and look natural, or substantially identical to the normal image but for the blended insertion.
  • the resultant synthetic two-dimensional images are indistinguishable from images of actual cancers, even to trained radiologists.
  • Example 2 [0350] The original cancer of Figure 15 is again shown in panel (a), along with a close-up of the micro-calcifications (bottom). The original cluster consists of approximately 30+ micro-calcifications with a range of sizes, and across a significant variation in tissue density. This full cluster is inserted into a much denser normal breast in panel (b).
  • Example 3 In certain cancers, micro-calcifications are visible within a mass. This class is particularly dangerous, so it is important for algorithms to accurately recognize both features (mass and calcifications).
  • Example 5 The same basic technique can be applied for generating synthetic masses.
  • Figure.18 panel (a) a small malignant mass within fatty tissue is shown.
  • a contour (darkened line) identifies the approximate boundary of the mass.
  • the extracted mass is shown corresponding to the interior pixels inside the contour, and three buffer regions outside the contour to permit smooth blending within the new image. For masses, these buffer regions are preferably somewhat larger.
  • Figure 19 provides an example of a synthetic mass created using the extracted cancer from Figure 18. The mass is inserted into dense breast tissue.
  • Figures.20A and 20B demonstrate the significant improvement in image-based ROC and FROC curves.
  • Figure 20A shows an increase in the AUC from 0.947 to 0.964.
  • Figure 20B shows an increase in the AUC from 0.938 to 0.958.
  • Figures 21, 22 and 23 Three such examples are shown in Figures 21, 22 and 23, which include an old score (no synthetics) and new score (synthetics used to retrain the model).
  • Figures 21 provides that the new model improves AUC from 0.567 to 0.646.
  • Figure 22 provides that the new model improves AUC from 0.377 to 0.762.
  • Figure 23 provides that the new model improves AUC from 0.642 to 0.963.
  • Protocol 2 Targeted generation of synthetics
  • the systems and methods above were used to generate examples to improve performance of an algorithm for specific parts of the resultant ROC. The efficacy of this approach is illustrated in Figure 24 where the effects on image level ROC were compared when adding 91 new cancer cases, versus adding synthetic cases.
  • CADe may be used to identify and analyze data for anomalies in industrial systems, whether independently of human review, or for review by a technician.
  • a CAD algorithm such as that described in WO Pub. No.
  • WO2017058848 which is incorporated herein by reference in its entirety, can be implemented to detect anomalies in data, and in the healthcare context, lesions in medical images.
  • Detected physiological anomalies are typically digitally marked in an image (including in a DICOM® image) which can be burned in, overlaid, or provided separate from the original image.
  • CADe results may support the technician’s analysis, or may suggest further analysis of the image is required.
  • CADe results may also be displayed in a “concurrent reading” mode in which the technician analyzes the CADe results while they look at the data.
  • Standard mammography CAD algorithms analyze digital or digitized images of standard mammographic views (e.g., Cranial-Caudal (“CC”) and mediolateral-oblique (“MLO”)) for characteristics commonly associated with breast cancer, such as microcalcifications and masses.
  • CC Cranial-Caudal
  • MLO mediolateral-oblique
  • the median time in days between screening and diagnostic mammogram was 6.5 days; between diagnostic mammogram and needle biopsy was 6.0 days; between needle biopsy and surgery was 14.0 days (Kaufman CS, Shockney L, Rabinowitz B, Coleman C, Beard C, Landercasper J, et al.
  • NQMBC National Quality Measures for Breast Centers
  • FIG. 26 depicts an exemplary triage system. While the following discussion illustrates the use of the triage system to prioritize or segment medical images for evaluation, in other embodiments the triage system may be used to prioritize or segment non- medical images for evaluation.
  • Figure 27A depicts a raw image of corrosion on an aircraft
  • Figure 27B depicts ROIs which can be highlighted using a suspicion code.
  • EXAMPLES [0378] Aspects of the present teachings may be further understood in light of the following examples, which should not be construed as limiting the scope of the present teachings in any way.
  • Example 1. Stand-Alone Clinical Efficacy [0380] The Breast Cancer Surveillance Consortium (BCSC) collected the results of 1,838,372 screening mammograms from 2004 to 2008 from eight mammography registries across the country (http://www.bcsc- research.org/statistics/performance/screening/2009/perf_age.html).
  • the sensitivity of radiologists was 84.4%, and the specificity was 90.8% with a recall rate of 9.6%.
  • the radiologists of the study above considered 9.6% of the exams suspicious enough to indicate a follow-up for the patient.
  • the stand-alone performance of the systems and methods of the present invention were assessed to determine the sensitivity and specificity of the image analysis algorithm and to determine what the notification rate would be at different sensitivities.
  • the population composition of the quarantine test set is shown in the Table 4. This test set was quarantined and not made available to the researchers when training or validating the algorithm. Table 4.
  • Normal cases are those that are either biopsy benign or contain 2 years of subsequent normal evaluations, and cancer cases are those that are biopsy confirmed cancer taken less than 270 prior to a cancer positive tissue biopsy.
  • the CAD algorithm is set to a sensitivity of 84.4%, in direct comparison to the BCSC study results, the algorithm has a stand-alone specificity of 90.55% which indicates that 6.1% of the cases are suspicious.
  • the systems and methods of the present invention identify approximately 36% ( (9.6%-6.1%) / 9.6% ) fewer exams as suspicious.
  • the algorithm can be set to notify the radiologist that 9.6% of the cases are suspicious, the sensitivity can be shifted up to 89% thus allowing for a 5.5% improvement in sensitivity over the radiologists of the BCSC study. This increase in sensitivity would identify as suspicious approximately 2.7 more cancers per 10,000 patients screened for breast cancer.
  • the algorithm could be set to a 96% sensitivity (or higher). Although this would increase the percent of cases marked as suspicious to 23% it would still be only a fraction of the total cases reviewed by a radiologist. This sensitivity would represent a 13.7% improvement over radiologists and would identify as suspicious approximately 6.8 more cancers per 10,000 patients screened.
  • TP True Positives are defined as exams that contain biopsy confirmed cancer which the software correctly identified as “Suspicious”
  • FN False Negatives are defined as exams that contain biopsy confirmed cancer which the software did not identify as Suspicious but instead identified as “” (blank)
  • TN True Negatives are defined as exams that are biopsy confirmed benign or normal which the software correctly did not identify as Suspicious but instead identified as “” (blank)
  • FP False Positives are defined as exams that are biopsy confirmed benign or normal which the software identified as “Suspicious” [0392]
  • each identified anomaly is scored on a scale of 0-100 and anomaly scores that are over a preset threshold are considered suspicious.
  • the threshold is generated from the ROC and represents an operating point on the ROC tied to a specific sensitivity and specificity.
  • Figure 32 is a ROC with the anomaly scores displayed in orange.
  • Table 5 provides sample data points on the ROC showing a score threshold. In order to understand how to interpret Table 5, two data points (1 and 2), annotated with boxes, are shown in Figure 32. Table 5.
  • Stand-alone Time-based performance [0395] In order to establish a baseline for the time-based performance of the systems and methods of the present invention, a set of test cases was processed through the full operational flow from first receipt of the exam by the image forwarding software to receipt of the returned notification file at the viewing workstation.
  • the parallel workflow created by the device via a notification has the potential to positively impact the standard of care. Specifically, in the event of a True Positive (TP) study identified by the device, the parallel workflow of the systems and methods of the present invention allows a radiologist to more quickly identify and prioritize the assessment of suspicious exams.
  • TP True Positive
  • the device is intended to send notifications so as to alert a specialist as to the timely existence of a case that may potentially benefit from that specialist’s attention, who would have reviewed them at a later time, had the device not been available.
  • Example 2 AI-based Triage Software Used to Improved Clinical Efficiency and Patient Experience
  • Case Population [0409] An exemplary population of cases (exams) below in Tables 6 and 7 can be selected to represent a representative cross-section of patients based on age and density typically seen in a screening population. Notably, Table 6 provides a population with/without cancer for testing, and Table 7 provides a population of varying breast density for testing. The population can be enriched with biopsy confirmed cancers to more accurately measure the efficacy of the output software. Table 6. Table 7.
  • Inclusion Criteria Female, and have a standard 4-view screening mammogram, and met none of the exclusion criteria.
  • Exclusion Criteria Significant existing breast trauma, previous surgical biopsy, previous breast cancer, and inadequate technical image quality
  • Study Design [0413] Step 1: Stand-alone performance assessment [0414] A. Each exam is processed as if it was being processed in a clinic: 1. The exam is exported from the PACS. 2. Each exam can optionally be run through image forwarding software that can perform anonymization and de-identification processes. 3. Each exam can optionally be encrypted and transmitted to a cloud-based server over a standard corporate network typical of a hospital network. 4. On the cloud based-server, the CAD algorithm will be used to process the exam. 5.
  • Each suspicious exam will be assigned a suspicion code.
  • This code will label the exam as “suspicious” if is at least one anomaly on the exam that has an anomaly score greater than or equal to the operating point threshold used in the study.
  • a notification result file in the form of a DICOM® SR is generated by the CAD algorithm and transmitted back to the PACS in DICOM® format.
  • the DICOM® SR can be decrypted when necessary, and re-associated with the original exam by image receiving software.
  • the PACS ingests the notification file and acquires the suspicion code.
  • the worklist in the PACS is updated to reflect acquisition of the DICOM® SR indicating completion of the process.
  • Efficacy Results are measured in terms of case-level sensitivity, case-level specificity, and case-level area under the curve (AUC) measured on a reader operator characteristic (ROC) curve.
  • Performance Results are measured in terms of the total time for Step 1: stand-alone performance assessment above; and from exporting the exam from the PACS until the PACS registers receipt of the DICOM® SR. Performance is measured in minutes and seconds.
  • Step 2 Clinical data collection
  • TAT audit data is collected from a study site for a 60-day window that tracks the following metrics: [0419] 1.
  • Secondary metrics are collected during the same period including the following: [0422] 1. Percent of patients recalled (assigned a BI-RADS® 0). [0423] 2. Percent of patients who return for their follow-up based on the recall. [0424] 3. Average amount of time between initial screening exam and the follow-up exam.
  • screening mammograms are read in a serial fashion, often through the use of “batch” screening, where exams are read by the radiologist at designated times when there are fewer interruptions or distractions. They may be sorted by study date and time, patient name, or other pieces of information that may be known about the patient or exam at the time of reading. However, this means that difficult or very abnormal cases are read in the mix with no means to prioritize them to be read earlier in the batch. In sites where there is a large backlog of exams to read, there may be serious cases which may need to recalled which may not be reviewed for days or weeks.
  • AI artificial intelligence
  • Case Population [0437] The population of cases (exams) in Table 8 (which provides a population with/without cancer for testing) was selected to be a representative cross-section of patients based on age and density typically seen in a screening population. The population was enriched with biopsy confirmed cancers to more accurately measure the efficacy of the CAD algorithm. An exemplary population having diverse or varying breast densities for testing can also be measured (which could be as provided in Table 9). Table 8. Table 9.
  • Inclusion Criteria Female, and have a standard 4-view screening mammogram, and met none of the exclusion criteria [0439] Exclusion Criteria: Significant existing breast trauma, previous surgical biopsy, previous breast cancer, and inadequate technical image quality. [0440] Reader Population: 6 MQSA certified radiologists with varying years of experience as shown below: [0441] Study Design [0442] A. The case population is divided in half maintaining all ratios with an effort to make the two datasets consistent in terms of complexity. This is overseen by 2 MQSA radiologists who are not participating in the study. [0443] B. Half of the cases are used in the Unassisted Read (“Non Triage Cases”) [0444] C.
  • Table 10 [0453] In Table 11, the products are compared. Notably, in Table 11, OP 1 refers to the Operating Point setting for the software. The leading software has 3 operating points (0, 1, and 2) that dictate the sensitivity and specificity settings for the CADe software. A higher operating point corresponds to a higher sensitivity. Table 11. [0454] In Table 11, exemplary “Threshold 60” is the lesion score threshold used in this study. The threshold can range from 0-100 and is an anomaly score that indicates the level of suspicion tied to a specific region of interest. Higher scores are more suspicious. The set threshold corresponds to a sensitivity and specificity setting for the CADe software.
  • Tables 12 through 16 provide the results of testing, and the degree to which the systems and methods of the present invention compare to the present market-leading CADe software. Notably, Table 12 compares false positives for mass and calcifications, Table 13 compares false positives for density, Table 14 compares false positives for BI- RADS® 0, Table 15 compares the reading time reduction, and Table 16 compares the increased efficacy.
  • Example 5 – TAT/Efficiency Determination [0457]
  • Objectives To investigate the effectiveness of the novel AI software on mammography interpretation, the new Neural Network can be compared to current CADe systems. Similar to the Examples above, several factors can be measured including: 90% case level AUC; Average / Min / Max full cycle time to process study (from capture to delivered triage result / notification file); Typical time between study capture and determination to recall patient; Typical time for patient to be notified of recall; Typical time for patient to return for recall; and Percentage of notified patients who return for recall.
  • Results It is expected that the systems and methods of the present invention will provide that the CAD algorithm allows for a faster notification to the technician of the potential need to analyze potentially suspicious data, including the recall of a patient, when compared to waiting for the current clinical process to resolve. In addition, it is predicted that by providing faster notification, technicians and/or treating physicians could adjust their clinical process to have patients wait in the treatment room before being released to a waiting room. This revised action will likely decrease the amount of time it takes to notify a patient, thus reducing patient anxiety and increasing the likelihood of notifications actually reaching the patient. This revised action will also likely increase the percentage of women who participate in a recall because the feedback is immediate and they are already in the office.
  • Example 6 – Efficacy Determination [0460] Objective: What is needed to measure and demonstrate, similar to the Examples above, includes: 90% case level AUC; Improved reader efficacy; and Improved reader performance (efficiency). [0461] Results: What is expected is that the systems and methods of the present invention allow technicians and radiologist to read medical images faster with the same or greater efficacy compared with current modalities. [0462] Prediction of Probability Distribution Function of Classifiers [0463] Rapid advances in technology have led to a proliferation of data. Techniques used to generate, collect and process data involve machine learning (ML)/AI- based systems and methods.
  • ML machine learning
  • ML/AI-based systems apply DNN for: (i) computer vision (e.g., anomaly detection, classification, segmentation); (ii) time series prediction and forecasting; (iii) speech recognition; and (iv) natural language processing (NLP).
  • DNN computer vision
  • ii time series prediction and forecasting
  • iii speech recognition
  • NLP natural language processing
  • a shortcoming of DNN is that when faced with examples coming from a different distribution than the training data, the DNN generates the wrong predictions, with high confidence. This is attributed to the inability of the DNN-derived models to quantify and output its uncertainty in each instance.
  • An illustrative example is when a DNN is: (i) trained to identify cancerous lesions in mammography and (ii) given an image with microcalcifications.
  • the DNN-derived model is forced to classify the image as either cancerous lesion or normal, while having no way of expressing the uncertainty due to the fact that the training set of DNN-derived model has not seen examples of microcalcifications. High uncertainty output is akin to saying “I am not sure”. [0465]
  • the above is an example of uncertainty which arises from limitations in the training set.
  • Another source of uncertainty is from: (i) variance in the data or (ii) the underlying process. An instance of this in mammography is where even images taken of the same patient a few minutes apart are not identical due to positioning of the patient and the compression of the breast.
  • Deep neural networks in the analysis of medical images and other complex data sets are used in the system and methods herein for prediction of probability distribution function of classifiers and expressing the results to the user.
  • the model outputs other parameters of the distribution function such as standard deviation in addition to the mean of the distribution (score) at each case.
  • the standard deviation can be interpreted as the measure of uncertainty or confidence level. The determination of the uncertainty or confidence level is not limited to the distribution function.
  • cases in the bucket of capabilities (c) may indicate more difficult cases which require additional diagnostics or follow up.
  • the systems and methods herein perform the capabilities for: d) increasing the efficacy of the model by separating cases where the model is confident from cases which are uncertain; and e) creating a single score that takes into account other metrics of the distribution function rather than just its mean.
  • a score can include a combination of mean and variance such that: (i) the score for small variance is similar to the mean; and (ii) the score is reduced significantly for high variance.
  • Network 3320 is a digital telecommunications network for sharing resources between nodes (i.e., computing devices). Data transmission between nodes is supported over physical connections (twisted pair and fiber-optic cables) and/or wireless connections (Wi-Fi, microwave transmission, and free-space optical communication).
  • Device 3305 may be any machine which is instructed to carry out sequences of arithmetic or logical operations via computer programming.
  • Device 3305 may include without limitation a Smartphone device, a tablet computer, a personal computer, a laptop computer, a terminal device, a cellular phone, and the like.
  • UI 3310 and program 3315 reside on device 3305.
  • UI 3310 facilitates human-computer interaction which as a graphical user interface (GUI), which is composed of a tactile user interface and visual interface.
  • GUI graphical user interface
  • Program 3315 receives data from training data set 3325 and medical information 3330.
  • program 3315 Based on the training data set 3325, program 3315 receives cases comprising medical images and other associated information (diagnosis, salient features of the medical images which lead to the diagnosis, and DICOM header).
  • the DICOM header may contain a range of useful information including without limitation, the side (i.e., left or right), orientation, view, protocol, date of procedure, and so forth, many of which may be listed in a filename convention. This information may be extracted for use by the algorithm— for example, in order to compare results from multiple views, or from a time series of images.
  • DICOM tags include without limitation: (a) pixel spacing (e.g., hex tag— (0028x,0030x)), which may be useful to scale the image in terms of real physical dimensions (e.g., mm), which can compute a ‘Q factor’ consistently; (b) diagnostic vs. screening (e.g., hex tag—(0032x,1060x)), which may allow for inclusion or exclusion of diagnostic images from studies; and (c) patient orientation (e.g., hex tag—(0020x,0020x)), which may allow for displaying the images in a consistent manner. Stated another way, the images are displayed in the same orientation as used by radiologists in typical computer-aided design (CAD) systems.
  • CAD computer-aided design
  • a predetermined orientation may be assigned (e.g., for mammograms—where the nipple points to the left in all images as is the industry standard).
  • parameters are set for program 3315 to create a model.
  • the model implements an assessment protocol on data sets. More specifically, program 3315 generates models by using training techniques: (A) ensemble learning; or (B) deep neural net architectures which use point estimates.
  • the implemented assessment protocol can generate a score and determine a level of uncertainty by: (i) focusing on particular parameters; (ii) ignoring other parameters; and (iii) evaluating the significance (i.e., weight) of each parameter when.
  • the score is associated with a probability of cancer or the degree of suspiciousness of a lesion in medical information 3330.
  • Images with high levels of uncertainty can be sent to buckets for retraining.
  • the assessment protocol can be modified. In turn, this can increase the confidence level (i.e., reduce the uncertainty).
  • program 3315 focuses on the perimeter of the organ captured and the overall appearance of the organs in the known medical images. The perimeter is focused on because certain spots are implicated with cancer risks. Medical images A and B do not have spots on the perimeter but the overall appearance of images A and B are noticeably different from known medical images used to train the model. Image A has surface ridges in the interior while the colorations are not obscuring the perimeter.
  • Image B has colorations which obscure the perimeter, while being absent of the interior.
  • the surface ridge in the interior and colorations are factors which are not initially understood and therefore images A and B are deemed as having high uncertainty.
  • program 3315 sends images A and B to a bucket for retraining. [0475] By determining the level of uncertainty and retraining images with high levels of uncertainty, the confidence in the score is evaluated and thereby program 3315 is providing a level of granularity when analyzing images and other data sets, while improving the models and implementing assessment protocol.
  • program 3315 receives image data from medical information 3330, wherein the image data are, for example, mammograms of a plurality of patients.
  • program 3315 In response to program 3315 applying the implemented assessment protocol on the medical information 3330, program 3315 generates a score and level of uncertainty.
  • the generated score and level of uncertainty are outputted to UI 3310.
  • the generated score on the medical image can be thought of as the mean of the probability distribution function (PDF).
  • PDF probability distribution function
  • PDF probability distribution function
  • a normal distribution is uniquely characterized by its mean and standard deviation. Standard deviation can be thought of as a measure of uncertainty of model output.
  • Program 3315 applies machine learning techniques when assessing uncertainties in mammography. More specifically, program 3315 accounts for aleatoric and epistemic parameters. Aleatoric parameters assess statistical uncertainty (i.e., stochastic variability in data), which is always present and thus not possible to eliminate with more data. Examples of aleatoric parameters as applied to mammography in the systems and methods herein include: (1) positioning / compression of breast; (2) variations in sensor efficiency or X-ray calibration; and (3) random seeds used to train or test models.
  • Epistemic parameters assesses systematic uncertainty (i.e., missing knowledge due to limited data), which should decreases with more data and more precise instruments.
  • Examples of epistemic parameters as applied to mammography in the systems and methods herein include: (4) spatial resolution of sensors; (5) limited spatial views (typically 4 for 2D mammography and prior visits); (6) Image processing algorithms (presentation view) - different for each vendor; (7) architecture of neural network; (8) labels which are incorrect or missing; (9) rare cases with limited examples in training set; (10) random selection of women based on age, genetics, and breast density; and (11) inherent feature similarities between cancer and benign instances. Even seemingly “perfect” images (i.e., high resolutions images with well defined features) have limited information content.
  • Program 3315 can use epistemic parameters, which may reduce uncertainty by considering prior images and carefully comparing right and left views. Otherwise, other modalities (ultra-sound, MRI) - or biopsy are needed to complement the mammograms when program 3315 assesses uncertainty. Epistemic uncertainty associated with examples 4-10 decreases as the quality of images increases, and program 3315 acquires more data. However, uncertainties associated with example 10 may be large and are not possible to eliminate.
  • Program 3315 uses machine learning techniques during anomaly detection during mammography and quantification of breast arterial calcifications (BAC). Regardless of the location of calcification, any calcification in the artery indicates the presence/onset of artery disease. As such, observation of BAC in mammogram has direct impact on the risk factor for, but not limited to, coronary heart disease (CHD), kidney disease, and stroke. Leading cause of CHD is due to plaque buildup, which can rupture or narrow the coronary artery, regardless of whether the plaque is calcified. Calcification is the last stage of plaque development. BAC may be an indicator of both calcified and non-calcified plaque in the coronary artery.
  • CHD coronary heart disease
  • BAC may be an indicator of both calcified and non-calcified plaque in the coronary artery.
  • program 3315 performs the steps in flowchart 3400. These steps determine a measure of uncertainty/confidence level as an output of the systems and methods herein.
  • program 3315 generates models in response to receiving the contents of training data set 3325. Techniques A and B are machine learning techniques used by program 3315 to generate models.
  • Technique A is based on ensemble learning and relies on the creation of several models that are maximally de-correlated but with similar efficacy. In practice, the models have some degree of correlation.
  • program 3315 runs the models for each instance and generates the corresponding probability distribution function which at the minimum may be the mean and variance. While Bayesian models used to generate probability distribution function are in technique A, the systems and methods herein can also apply to non-Bayesian models. When program 3315 uses technique A, the variance in models are minimized such that the generated score is incorporated into model outputs which is accessible to an end user of program 3315 in an actionable way.
  • Some non-Bayesian modeling techniques to create different models can be, but not limited to, one or combination of the following: 1.
  • Program 3315 can also use deep neural networks (DNN) of technique B to generate models.
  • DNN frameworks use point estimate for the weights in every node and also use non-probabilistic loss function.
  • program 3315 can use the probability distribution over the weights, and/or loss function.
  • the Bayesian neural nets have been partially incorporated into DNN frameworks, such as Pytorch and Tensorflow 2.0.
  • program 3315 receives data sets in medical information 3330.
  • a score and uncertainty levels of the data sets in medical information 3330 are determined by the models applied by program 3315.
  • program 3315 applies models on data sets in medical information 3330.
  • the applied models of program 3315 use an assessment protocol to determine a similarity level to known examples of cancer.
  • the assessment protocol involves the application of autoencoder or tangent kernel of the classifier on medical information 3330.
  • the assessment protocol establishes a baseline for diagnoses (e.g., protrusions which are deemed as cancerous lesions or benign abnormalities) to aid in the evaluation of medical images.
  • diagnoses e.g., protrusions which are deemed as cancerous lesions or benign abnormalities
  • Program 3315 can compare the incoming data sets from medical information 3330 to the baseline and thereby finding similarities and differences. This is the basis for a similarity level. However, the evaluation does not end with similarities and differences.
  • program 3315 can focus on certain factors or ignore other factors to obtain a more granular, comprehensive, and accurate evaluation of the incoming data sets. Thus, the assessment protocol does not end the analysis with a yes or no answer (i.e., binary classification).
  • program 3315 determines uncertainty estimates for any ensemble classifier. In random forest (used as a binary classifier), each tree classifies the test cases as either belonging to class 0 or class 1.
  • the binary prediction for tree I is xi.
  • the usual output that is further processed is the average of xi, which is the averaging over all trees. This is the fraction of the trees which classified the case as being in class 1, and thus referred to as p.
  • program 3315 can be used for binary classification and non-binary classification.
  • the model may be trained for, but not limited to, density classification and biopsy.
  • Breast tissue comprises milk glands, ducts, and dense and non- dense (fatty) breast tissue.
  • radiologists grade breast based on the density, using the BI-RADS® reporting system.
  • BI-RADS® system is based on the proportion of dense to non-dense tissue, with score of 1 representing almost entirely fatty to 4 being extremely dense.
  • there is significant intra-reader and inter-reader variability in assigning a BI-RADS® score This poses an issue with training neural net models for density classification since unlike biopsy confirmed cancer cases, there is no established convention for the density label.
  • a breast originally assigned a density of 2 by radiologist A may be considered density of 3 by radiologist B.
  • the images in mammography may have devices, markers, and other artifacts in them.
  • a trained model may exhibit degraded performance if program 3315 is not presented with enough such examples in the training set.
  • Program 3315 may determine uncertainty such that variability and artifacts are addressed. More specifically, program 3315 consider two uncertainties - aleatoric (irreducible) and epistemic (reducible). While epistemic uncertainty can be reduced with additional data, aleatoric uncertainty is due to inherent variation in the system such as reader dependent variations in the density BI-RADS® score.
  • program 3315 uses a threshold above 99.5 percentile of aleatoric uncertainty to flag cases where the density labels may be: (i) less clear cut and (ii) borderline between neighboring density classes.
  • program 3315 uses a threshold above 99.5 percentile of epistemic uncertainty to flag cases where either the model does not have enough examples similar to it in the training set and/or cases that should not be included in the training set.
  • program 3315 implements the solution based on the uncertainty level (as determined in step 3420).
  • program 3315 devises new resulting metrics which are (i) beyond the score that have not been incorporated into the model outputs or (ii) in a way to make it accessible to the user in an actionable way.
  • the incorporation of the probability distribution over the loss function can be implemented. If the probability distribution is over the weights, program 3315 runs the model multiple times to construct the probability distribution function.
  • program 3315 can use the uncertainty and other metrics of the probability distribution function as listed below. This list is not exhaustive and is meant only as illustration of diverse types of deployment: • Risk assessment - risk mitigation strategies • Lesion detection and classification • Biopsy classifier • Triage - create a separate bucket for cases with high uncertainty.
  • step 3430 program 3315 determines if the uncertainty level is above a threshold associated with an acceptable level of uncertainty. Accordingly, if the threshold is exceeded, then program 3315 proceeds to retrain the models in step 3430. Stated another way, there may be models generated that do not have a high enough confidence level for program 3315 to make accurate evaluations of medical information 3330. More specifically, cases with high uncertainty levels are sent to a bucket and subsequently retrained.
  • program 3315 displays the output with the score to include other metrics (e.g., uncertainty and confidence level) that become available through techniques A and B in step 3435. This indicates that the generated models have a high enough confidence level for program 3315 to make accurate evaluations of medical information 3330. Otherwise, the models are retrained in step 3440.
  • program 3315 generates models using techniques A or B. Based on the models, the images are deemed to have a low uncertainty or high uncertainty.
  • program 3315 receives images and accompanying information. Images 315 and 320 are cases of a high score and low score. Image 315 is absent of surface lesions that are clearly visible in image 320.
  • images 3505 and 310 are not as straightforward for program 3315 to analyze when training the model.
  • Image 3505 is deemed to have a high uncertainty
  • image 3510 is deemed to have low uncertainty.
  • Program 3315 can ascertain that: (i) images 315 and 320 have elements which decrease the probability of accurate and precise binary classification regions by noticing that: (a) image 3505 appears obscure whereas (b) none of the regions of image 3510 appear obscure; and (ii) images 315 and 320 are absent of elements which decrease the probability of accurate and precise binary classification. [0498] Referring to Figures 36A-D, program 3315 can identify cases with high uncertainty of one type, which had a low uncertainty of the other type.
  • Figures 36A-D depicts examples of cases where the model predicted low uncertainty and examples with high uncertainty are depicted. Cases with high uncertainty are seen to be cases where the cancer is not as clearly evident in the image.
  • One possible application of this new information about the model uncertainty is that one can devise a separate protocol for flagged cases where the model has high uncertainty (e.g., step 3425).
  • Figure 36A shows an example from the training set where the original radiologist marked the image as density 2 while the model scored the image as density 3. Normally this case would count as a false positive by the model.
  • Program 3315 accounts for review A of a panel of radiologists of image 3600A, whereby the consensus of review A is: (a) 50% confidence for a density of 2 and (b) 50% confidence for a density of 3.
  • Program 3315 applies the protocol, which accounts for review A of the panel, the model score, and original radiologist marking, on image 3500 and thus deemed as having high aleatoric uncertainty (99.51 percentile).
  • program 3315 successfully flags image 3600A as a borderline case, i.e., a case with high uncertainty, based on the scores of the original radiologist, model, and panel.
  • Figure 36B shows an example from the training set where the original radiologist marked image 3600B as a density of 3. However, program 3315 scores image 3600B with a density of 2. Normally, this case is counted as a false positive by the model. Program 3315 accounts for review B of a panel of radiologists of image 3600B, whereby the consensus of review B is classified with 100% confidence for a density of 2.
  • Program 3315 applies the protocol, which accounts for review B of the panel, the model score, the original radiologist marking, on image 3600B and thus deemed as having high aleatoric uncertainty in this case (99.84 percentile). Stated another way, program 3315 successfully flags image 3600B as the wrong label. Accordingly, program 3315 identifies wrongly labeled cases and subsequently corrects the label or removes the wrongly labeled cases from the training set. This improves the efficacy of the model and subsequently implemented protocol. In a specific instance, the subsequently implemented protocol by program 3315 leads to improvement of the 4 class kappa from 0.82 to 0.85 and binary kappa from 0.92 to 0.96.
  • Figure 36C depicts a nonstandard, magnification diagnostic view in image 3600C. Due to the high epistemic uncertainty of image 3600C, program 3315 is flagged. The protocol, as applied on image 3600C, program 3315 determines image 3600C is overly saturated. Such cases can be removed from the training set. Another application of this uncertainty is in running the model live at a client’s site. Density classification by radiologists is based on examination of all of the screening images and then assigning one breast density score to the patient. The model can consider all images of a patient and form a consensus breast density score by discarding images that have high epistemic uncertainty. Thus, program 3315 excludes images, such as image 3600C, that should not be considered in the density classification.
  • Figure 36D depicts another example in the training set which has an embedded artifact in image 3600D.
  • Program 3315 analyzes image 3600D and subsequently deemed as having high epistemic uncertainty.
  • Program 3315 flags image 3600D as the implemented protocol notes there are very few of such images in the training set. In such instances, program 3315 either: (i) finds similar examples to add to the training set, or (ii) eliminates the case from the training set.
  • FIGs 36E and 36F from aleatoric and epistemic uncertainty from a crop-level CNN training model for lesions program 3315 can leverage uncertainty to improve the training of the models.
  • High aleatoric uncertainty indicates cases where the model is not sure about the classification score since there are other examples in the data set that appear similar but have an opposite class.
  • High epistemic uncertainty indicates cases where the model has not seen sufficient representation/examples in the training set.
  • program 3315 examines cases in the training set with high aleatoric uncertainty.
  • Figures 36E and 36F show two cases in the training set with high aleatoric uncertainty.
  • Figure 36E is a positive class that is misclassified as a negative class by the trained model
  • Figure 36F panel is a negative class that is correctly classified by the trained model.
  • the visual similarities of the two crop level images in Figures 36E and 36F, one normal and one cancer is indicative of the difficulty in distinguishing the correct class with high confidence for these two crops.
  • This additional context enables the model, as generated by program 3315, to: (i) get the correct class in contrast to the model prediction of the wrong class in Figure 36E, while (ii) significantly reducing aleatoric uncertainty also from 98.42 percentile to 81.83 percentile. This uncertainty can be further reduced if the models are trained by program 3315 with crops at different crop size levels. This improves the training, whereby program 3315 creates crops at different crop size levels and combines the scores of the crops, such methods such as majority voting or averaging the scores. This approach can be performed even without retraining the model. Another way of improving the model is to use different crop size levels as part of data augmentation.
  • program 3315 can run the model for different crop sizes and then creates a final score through such methods, but not limited to, majority voting or averaging the scores.
  • program 3315 can feed crops at different crop size levels to a multi-scale convolution neural net.
  • the epistemic uncertainty has increased in Figure 37 as compared to that in Figure 36E. This is attributed to the fact that the model was trained on tight crops and it has not seen many examples that contain as much surrounding breast tissue.
  • FIG 38 the effect of program 3315 trained model on the crop in Figure 36F with class 0 but with the inclusion of a larger surrounding breast tissue in the crop is depicted.
  • the aleatoric percentile is significantly reduced from 99.06 percentile to 35.95 percentile.
  • the confidence level in score of the model is higher since it has more context for assessing the suspiciousness of the lesion.
  • the epistemic percentile has increased, as expected since the model is trained on tight crops and the high epistemic uncertainty points to the fact that there are not many cases in the training set with such large segment of the surrounding tissue around a lesion like structure.
  • Figures 39A-C several cases with high epistemic uncertainty in the training set are depicted. These are cases that despite providing the label in the training set, the model has difficulty learning the correct classification.
  • Figures 39A-C are very unusual cases with scant representation in the training set.
  • the high epistemic uncertainty indicates the need to include more such examples for the training set to improve the model.
  • program 3315 increases the crop size to include more of the surrounding breast tissue, and this results in: (i) a decreases the aleatoric uncertainty from 58.19 to 30.14 percentile; and (ii) a match between the predicted class and the actual mass. This is consistent with program 3315 improving the performance of the model when providing more context, as described with respect to Figures 36E and 36F. However, the epistemic uncertainty has slightly increased from 97.37 to 99.92 percentile.
  • program 3315 can use crops of cancer and normal/benign images extracted from 2D mammography to train a CNN one-class classifier.
  • Program 3315 has also used the same images to train a Bayesian NN. The advantage of the Bayesian NN in this case is that one gets a prediction of the uncertainty.
  • Figures 41A-C show the efficacy of the Bayesian NN for (a) including all cases independent of their uncertainty; and (b) including only cases where the model is confident about its classification score. Stated another way, cases with high uncertainty are not included in the evaluation of medical information 3330. Thus, the model efficacy is improved when cases with high uncertainty are flagged and not included in the evaluation.
  • Three ROCs are shown where the change in AUC is compared to the full set of data as program 3315 filters out cases based on uncertainty values. Different measures of uncertainty are applied by program 3315. In Graph 4100A, the ROC is under a 40-percent cutoff where program 3315 keeps 40% of the cases for each uncertainty measure (each representing a separate curve).
  • the ROC is under a 60-percent quantile cutoff where program 3315 keeps 60% of the cases for each uncertainty measure (each representing a separate curve).
  • the ROC is under an 80-percent cutoff where program 3315 keeps 80% of the cases for each uncertainty measure (each representing a separate curve).
  • AUC area under the curve
  • Classification probability sigma refers to the sample standard deviation of classification probability score inferred by the model and is denoted as sigma in Figures 41A, 41B, and 41C. It is also possible to convert classification probability to threshold-based classification. As an example, using a threshold of 0.5 for binary classification, program 3315 can convert classification probability to threshold-based classification and then acquire the corresponding sample standard deviation, referred to as sigma_0_1 in Figures 41A, 41B and 41C.
  • a ranking for different types of uncertainty measures can be acquired within each uncertainty estimation, respectively. For example, the ranking of entropy can be computed across all the images.
  • a “quantile filter” can then be deployed based on the ranking of specific uncertainty to enhance the performance of program 3315.
  • program 3315 will exclude images with this specific kind of uncertainty above X ⁇ 100 percentile. For example, a 0.4 quantile filter on entropy will let program 3315 exclude images with entropy values above the 40 percentile.
  • Figure 41A which shows application to detection of suspicious mass in breast mammography, demonstrates that when program 3315 applies 0.4 quantile filter based on the 5 types of uncertainty separately, entropy, sigma_0_1, sigma, aleatoric uncertainty, and epistemic uncertainty, the cutoffs for each uncertainty measure are 0.65, 0.32, 0.12, 0.21, and 0.21, respectively. This results in AUCs of 0.896, 0.910, 0.905, 0.887, and 0.905, respectively which are all higher than the AUC of 0.824 for the inference on the full set of images. Note that different numbers of cancer cases are eliminated for each uncertainty type as indicated in the figure caption.
  • Figure 41B shows the resulting AUCs of AUC of 0.878, 0.881, 0.886, 0.866, and 0.866 if the quantile cut off is set to 0.6 which is still higher than the AUC of 0.824 for the full set but somewhat lower than the case when the quantile cutoff is set to 0.4.
  • Figure 41C demonstrates that when program 3315 applies 0.8 quantile filter on the 5 types of uncertainty separately with the resulting AUCs of 0.850, 0.846, 0.843, 0.850, 0.843, respectively.
  • the improvement in AUC over the full set of images is still significant but is the smallest among all three quantile filter. However, the number of cancer cases eliminated is also smaller than the other two quantiles.
  • program 3315 demonstrates the viability of all 5 types of uncertainties in enhancing the efficacy of the model as measured by the AUC.
  • program 3315 measures uncertainty is most useful for particular applications where the goal is to have the highest AUC while removing as few cases as possible.
  • Example 1 In the first example, the systems and methods herein present a case in medical imaging where a prior from certain probabilistic continuous distribution is put on the weights of the traditional pointwise neural network model and thus make it a variational dense layer. A fully Bayesian convolutional network may also be possible to deploy but can be more difficult to train. While the full model pipeline for computer assisted design (CAD) in medical imaging can consist of several neural networks including an object detector network, the final output of CAD is currently a score. The focus here is to show how changing the model output from score (mean) to one where both the mean as well as the standard deviation (i.e., an uncertainty level) is provided improves the results.
  • CAD computer assisted design
  • Program 3315 can also explain how this additional information can be useful for the user (e.g., radiologist using device 3305). As such, program 3315 is not limited to CNN here is not limited to CNN. Thus, the systems and methods herein are generally applicable to all CAD applications and can be incorporated into all CAD pipelines. [0517]
  • Example 2 [0518] In the second example, program 3315 uses an object detection neural network to assess the uncertainty of the model prediction. Unlike the previous example where the weights of the neural net are Bayesian, here program 3315 uses a standard faster RCNN object detection network with pointwise weights where the loss function is replaced with the quantile regression loss function.
  • program 3315 creates different models from those of technique A and B by changing the alpha parameter which refers to the desired quantile.
  • a quantile of 0.5 is equivalent to the median which is value obtained in the standard approach, based on minimizing the Mean Absolute Error.
  • Figures 9A- 9C shows the ROC for three values of alpha. The result shows the uncertainty of the model in different parts of the ROC.
  • program 3315 supports a triage system connected to a Picture Archiving Retrieval System. It is impractical for a radiologist or program 3315 to review each case in the training set, which can consist of tens or hundreds of thousands or even more images.
  • Program 3315 uses: (i) epistemic uncertainty to enable identification of all of the above cases; and (ii) retraining buckets (as described above). Thus program 3315 may identify which: (i) images are indicative of abnormal medical conditions requiring immediate medical care (i.e., low uncertainty); (ii) images that require further analysis (i.e., high uncertainty); and (iii) retraining.
  • an anomaly in or on a tissue surface can be analyzed using program 3315, which provides a suspicion code. That code can, in real-time, populate a user interface to alert the radiologist or technician of a suspicious anomaly in or on the tissue surface, thereby leading to an auto-populating setting. In the auto-populating setting, a notification result file may be reported and thus generated to include the suspicion code.
  • the notification result file can be in the form of a portable document format (PDF), DICOM® SR, JSON, HTTP, HTTPS, HTML, REST, or other communication format in which a “Suspicious” or “” (blank) code may be transmitted in a FFDM, CIS, RIS, HIS, or the like.
  • PDF portable document format
  • DICOM® SR JSON
  • HTTP HyperText Transfer Protocol
  • HTTPS HyperText Markup Language
  • HTML HyperText Markup Language
  • REST or other communication format in which a “Suspicious” or “” (blank) code may be transmitted in a FFDM, CIS, RIS, HIS, or the like.
  • Figure 42 is a flow chart of a method for determining a clinical score.
  • the clinical score may be generated or determined by a clinical score engine (such as the cancer score engine) in a system.
  • the method 420 may involve processing and analyzing one or more pieces of medical information for generating, for one or more regions of
  • the method 4200 may gather variables that are pertinent to the clinical indication, programmatically analyze the piece(s) of medical information to determine values of the variables, transform these variables for use in generating a clinical indication or score, and then generate an indication of the clinical indication based on these variables or a transformation of these variables.
  • a clinical score generator component may receive the indications of the clinical indication in the one or more regions of the tissue and generate a clinical score for at least one region of the tissue.
  • the clinical score may have a value that increases as a cancer tumor grows and decreases as a cancer tumor shrinks.
  • the clinical score may be normalized and have threshold levels so that, for example, a normalized clinical score of 1–3 indicates a benign tumor, a normalized clinical score of 4–6 indicates suspicious cells, and a normalized clinical score of 7–10 indicates that cancer is present in particular region(s) of tissue.
  • the method 4200 may include receiving one or more pieces of medical information for processing and analysis.
  • the medical information may include information about a patient’s tissue, e.g., medical images of the tissue.
  • the medical information may include preprocessed or raw data, which is then processed and analyzed by the systems or methods described herein.
  • the clinical score engine may include a medical information analysis component that receives one or more pieces of medical information, where the clinical score engine then processes and analyzes this information.
  • the medical information may be automatically streamed to the clinical score engine by an uneven length preprocessed time series input.
  • the header of a DICOM file may contain information on the image contained within it including, but not limited to, the pixel resolution in physical units, criteria for interpreting the pixel intensity, etc.
  • the method 4200 may include analyzing the one or more pieces of medical information about the tissue. This may include gathering variables values about the medical information (e.g., a mammogram), where generating the indication of the clinical indication may be based on the gathered variable values.
  • the variables may include an intensity value for contours of instances of a type of tissue, a gradient of the instances of the type of tissue, one or more characteristics about each of the instances of the type of tissue, and a hierarchical structure of the instances of the type of tissue in a cluster.
  • the method 4200 may include generating an indication of the clinical indication (such as a biomarker).
  • the indication of the clinical indication may be generated for one or more regions of the tissue in the medical images.
  • the method 4200 may include generating a clinical quantification score.
  • generating a clinical quantification score may include generating a clinical score for each region of the tissue based on the indication of the clinical indication in each region of the tissue.
  • the clinical quantification score may indicate an absence of the clinical indication in the region of the tissue.
  • the method 4200 may include generating guidance for a medical professional (such as a physician or a technician performing non-invasive medical imaging) based on one or more of the indications of the clinical indication and the clinical quantification score.
  • the guidance may include, e.g., guidance for a radiologist based on the presence or absence of the clinical indication in the region of the tissue.
  • Implementations may utilize one or more algorithms for detecting and quantifying the clinical from medical information supplied to the system.
  • the algorithm may detect and quantify micro-instances of the type of tissue (such as calcifications) in mammogram images.
  • the algorithm may in general include (1) detecting and grouping the instances of the type of tissue into clusters, (2) classifying types of benign clusters, (3) quantifying clusters that are potentially instances of the clinical indication with a ‘Q factor’ as discussed herein, and (4) saving output quantities to evaluate performance.
  • a first algorithm generates an indication of the clinical indication and a second algorithm generates a clinical score.
  • a system than implements method 4200 may receive a medical image associated with a non-invasive medical imaging technique. Then, the system may: analyze the medical image, generate a clinical score for the medical image; and transmit an instruction to a measurement device to a acquire a second medical image associated with a second non-invasive medical imaging technique based at least in part on the clinical score.
  • the analysis of the medical image may include: determining contours in the medical image satisfying one or more criterion; analyzing one or more geometric attributes and one or more contrast attributes of contours included in the first subset of contours to identify a second subset of contours based at least in part on the contours satisfying one or more predetermined geometric and contrast attributes; selecting a third subset of contours from the second subset of contours that corresponds to one or more potential instances of a type of tissue, the third subset selected based at least in part on contours within the third subset satisfying first criteria associated with the type of tissue; ranking contours included in the third subset of contours based at least in part on a selection metric, the selection metric accounting for a combination of contrast and intensity; grouping the contours included in the third subset of contours into nested structures; selecting one or more instances of the type of tissue from the nested structures satisfying second criteria associated with the type of tissue; grouping the selected one or more instances of the type of tissue into clusters based on neighbor
  • the system may: receive a second medical image associated with a second non-invasive medical imaging technique; and revise the scale of suspiciousness for the clinical indication with the processor based at least in part on the second medical image. For example, the system may reduce a false positive rate for detecting the clinical indication by using information provided by different non-invasive imaging techniques.
  • the second medical image may provide additional information that corrects for artifacts associated with a quality of the medical image (such as operator motion or patient motion) and/or improves the clinical score.
  • the non- invasive medical imaging techniques may be different or may be the same non-invasive medical imaging technique.
  • the system does not provide the instruction to the measurement device. Instead, the medical image and the second medical image are acquired concurrently by the measurement device, and then are provided to the system.
  • the system diagnoses the clinical indication. For example, the system may compute a classification associated with the clinical indication based at least in part on the medical image and the second medical image.
  • the system may compute a classification associated with the clinical indication based at least in part on the medical image and the second medical image.
  • the systems and methods disclosed herein may be implemented via one or more components, systems, servers, appliances, other subcomponents, or distributed between such elements.
  • such systems When implemented as a system, such systems may include an/or involve, inter alia, components such as software modules, general-purpose CPU, GPUs, RAM, etc., found in general-purpose computers.
  • a server In implementations where the innovations reside on a server, such a server may include or involve components such as CPU, GPUs, RAM, etc., such as those found in general-purpose computers.
  • the systems and methods herein may be achieved via implementations with disparate or entirely different software, hardware and/or firmware components, beyond that set forth above.
  • aspects of the innovations herein may be implemented consistent with numerous general purpose or special purpose computing systems or configurations.
  • Various exemplary computing systems, environments, and/or configurations that may be suitable for use with the innovations herein may include, but are not limited to: software or other components within or embodied on personal computers, servers or server computing devices such as routing/connectivity components, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, consumer electronic devices, network PCs, other existing computer platforms, distributed computing environments that include one or more of the above systems or devices, etc.
  • aspects of the systems and methods may be achieved via or performed by logic and/or logic instructions including program modules, executed in association with such components or circuitry, for example.
  • program modules may include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular instructions herein.
  • the embodiments may also be practiced in the context of distributed software, computer, or circuit settings where circuitry is connected via communication buses, circuitry or links. In distributed settings, control/instructions may occur from both local and remote computer storage media including memory storage devices.
  • the software, circuitry and components herein may also include and/or utilize one or more type of computer readable media.
  • Computer readable media can be any available media that is resident on, associable with, or can be accessed by such circuits and/or computing components.
  • Computer readable media may comprise computer storage media and communication media.
  • Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and can accessed by computing component.
  • Communication media may comprise computer readable instructions, data structures, program modules and/or other components. Further, communication media may include wired media such as a wired network or direct-wired connection, where media of any type herein does not include transitory media. Combinations of the any of the above are also included within the scope of computer readable media.
  • the terms component, module, device, etc. may refer to any type of logical or functional software elements, circuits, blocks and/or processes that may be implemented in a variety of ways. For example, the functions of various circuits and/or blocks can be combined with one another into any other number of modules.
  • Each module may even be implemented as a software program stored on a tangible memory (e.g., random access memory, read only memory, CD-ROM memory, hard disk drive, etc.) to be read by a central processing unit to implement the functions of the innovations herein.
  • the modules can comprise programming instructions transmitted to a general-purpose computer or to processing/graphics hardware via a transmission carrier wave.
  • the modules can be implemented as hardware logic circuitry implementing the functions encompassed by the innovations herein.
  • the modules can be implemented using special purpose instructions (SIMD instructions), field programmable logic arrays or any mix thereof which provides the desired level performance and cost.
  • SIMD instructions special purpose instructions
  • features consistent with the disclosure may be implemented via computer-hardware, software and/or firmware.
  • the systems and methods disclosed herein may be embodied in various forms including, for example, a data processor, such as a computer that also includes a database, digital electronic circuitry, firmware, software, or in combinations of them.
  • a data processor such as a computer that also includes a database, digital electronic circuitry, firmware, software, or in combinations of them.
  • firmware firmware
  • software software
  • the above-noted features and other aspects and principles of the innovations herein may be implemented in various environments. Such environments and related applications may be specially constructed for performing the various routines, processes and/or operations according to the implementations described herein or they may include a general-purpose computer or computing platform selectively activated or reconfigured by code to provide the necessary functionality.
  • aspects of the method and system described herein, such as the logic may also be implemented as functionality programmed into any of a variety of circuitry, including programmable logic devices (“PLDs”), such as field programmable gate arrays (“FPGAs”), programmable array logic (“PAL”) devices, electrically programmable logic and memory devices and standard cell-based devices, as well as application specific integrated circuits.
  • PLDs programmable logic devices
  • FPGAs field programmable gate arrays
  • PAL programmable array logic
  • electrically programmable logic and memory devices and standard cell-based devices as well as application specific integrated circuits.
  • Some other possibilities for implementing aspects include: memory devices, microcontrollers with memory (such as EEPROM), embedded microprocessors, firmware, software, etc.
  • aspects may be embodied in microprocessors having software-based circuit emulation, discrete logic (sequential and combinatorial), custom devices, fuzzy (neural) logic, quantum devices, and hybrids of any of the above device types.
  • the underlying device technologies may be provided in a variety of component types, e.g., metal-oxide semiconductor field-effect transistor (“MOSFET”) technologies like complementary metal- oxide semiconductor (“CMOS”), bipolar technologies like emitter-coupled logic (“ECL”), polymer technologies (e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures), mixed analog and digital, and so on.
  • MOSFET metal-oxide semiconductor field-effect transistor
  • CMOS complementary metal- oxide semiconductor
  • ECL emitter-coupled logic
  • polymer technologies e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures
  • mixed analog and digital and so on.
  • the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in a sense of “including, but not limited to.” Additionally, the words “herein,” “hereunder,” “above,” “below,” and words of similar import refer to this application as a whole and not to any particular portions of this application. [0546] Moreover, the above systems, devices, methods, processes, and the like may be realized in hardware, software, or any combination of these suitable for a particular application.
  • the hardware may include a general-purpose computer and/or dedicated computing device.
  • microprocessors microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices or processing circuitry, along with internal and/or external memory.
  • This may also, or instead, include one or more application specific integrated circuits, programmable gate arrays, programmable array logic components, or any other device or devices that may be configured to process electronic signals.
  • a realization of the processes or devices described above may include computer-executable code created using a structured programming language such as C, an object oriented programming language such as C++, or any other high-level or low-level programming language (including assembly languages, hardware description languages, and database programming languages and technologies) that may be stored, compiled or interpreted to run on one of the above devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software.
  • the methods may be embodied in systems that perform the steps thereof, and may be distributed across devices in a number of ways. At the same time, processing may be distributed across devices such as the various systems described above, or all of the functionality may be integrated into a dedicated, standalone device or other hardware.
  • Embodiments disclosed herein may include computer program products comprising computer-executable code or computer-usable code that, when executing on one or more computing devices, performs any and/or all of the steps thereof.
  • the code may be stored in a non-transitory fashion in a computer memory, which may be a memory from which the program executes (such as random access memory associated with a processor), or a storage device such as a disk drive, flash memory or any other optical, electromagnetic, magnetic, infrared or other device or combination of devices.
  • any of the systems and methods described above may be embodied in any suitable transmission or propagation medium carrying computer-executable code and/or any inputs or outputs from same.
  • the devices, systems, and methods described above are set forth by way of example and not of limitation. Absent an explicit indication to the contrary, the disclosed steps may be modified, supplemented, omitted, and/or re-ordered without departing from the scope of this disclosure. Numerous variations, additions, omissions, and other modifications will be apparent to one of ordinary skill in the art. In addition, the order or presentation of method steps in the description and drawings above is not intended to require this order of performing the recited steps unless a particular order is expressly required or otherwise clear from the context.
  • performing the step of X includes any suitable method for causing another party such as a remote user, a remote processing resource (e.g., a server or cloud computer) or a machine to perform the step of X.
  • performing steps X, Y and Z may include any method of directing or controlling any combination of such other individuals or resources to perform steps X, Y and Z to obtain the benefit of such steps.
  • Coronary artery calcium score current status. Radiologia brasileira, 50(3), 182-189. [0563] Schiffrin, E. L., Lipman, M. L., & Mann, J. F. (2007). Chronic kidney disease: effects on the cardiovascular system. Circulation, 116(1), 85-97. [0564] World Health Organization (2017). Cardiovascular diseases (CVDs). Fact sheet. Updated May, 2017. [0565] Zaman, A. G., Helft, G., Worthley, S. G., & Badimon, J. J. (2000). The role of plaque rupture and thrombosis in coronary artery disease. Atherosclerosis, 149(2), 251- 266.

Abstract

A piece of medical information, e.g., a medical image of tissue, may be received for processing and analysis on a computing device or system. A region of the medical image may be analyzed to determine a presence of one or more contours in the region. One or more properties of the one or more contours may be extracted, where the one or more properties are inputted into a first algorithm to determine an indication of a clinical indication for the region. The indication of cancer may be inputted into a second algorithm to generate a clinical score for the region. Moreover, based at least in part on the clinical score, an instruction to acquire a second medical image of the tissue is provided. Then, in response to receiving the second medical image, the clinical score is revised based at least in part on the second medical image.

Description

DETECTING, SCORING AND PREDICTING DISEASE RISK USING MULTIPLE MEDICAL-IMAGING MODALITIES CROSS-REFERENCE TO RELATED APPLICATIONS [0001] This application claims priority from U.S. Provisional Application Serial No. 63/175,535 filed on April 15, 2021, the entirety of which is incorporated herein by reference. STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT [0002] Not Applicable. FIELD [0003] The present disclosure generally relates to devices, systems, and methods for detecting and scoring disease risk using multiple medical-imaging modalities. INTRODUCTION [0004] Non-invasive medical imaging is often used by health professionals to detect and diagnose illness, and to guide medical interventions. For example, when cancer is suspected, medical information about the tissue, such as a medical image, may be gathered for the affected tissue, where a physician reviews the medical information to identify possible areas in the tissue that may have cancer cells. This analysis typically leads to an all-clear diagnosis (if no areas are identified by the physician) or a recommendation for a biopsy of the tissue to confirm that any possible areas of cancer cells are in fact cancerous cells. In the context of breast cancer, the medical image is typically a mammogram. This existing approach results in an about 60% cumulative risk of a false positive and an about 20% average false negative rate. A false positive may result in a patient who did not have cancer having to endure a painful, intrusive, and unnecessary biopsy. A false negative may result in not detecting cancer as early as it could have otherwise been detected. [0005] Similarly, coronary heart disease (CHD), which is the most-common Atherosclerotic Cardiovascular Disease (ASCVD) and a significant source of annual mortality, is currently assessed using a calcium coronary scan to determine a Coronary Artery Calcium (CAC) score. A calcium coronary scan involves a cardiac computerized tomography (CT) scan performed in a CT scanner. Based on the extent of coronary artery calcification detected by the unenhanced low-dose CT scan, a number called an Agatston score is calculated. Notably, the calculation is based on the weighted density score given to the highest attenuation value (HU) multiplied by the area of a calcification speck, and the score of every calcified speck is summed up to give the total CAC score. The total calcium score allows for grading of coronary artery disease and early stratification of the risk for CHD and the severity of atherosclerosis in patients, which can be used to guide subsequent interventions, such as for high blood pressure or raised cholesterol. [0006] However, there are often limitations associated with a particular non- invasive medical imaging technique. For example, it is often difficult to accurately assess dense breast tissue or cysts using mammography. Moreover, an important disadvantage of a CAC score is the exposure of the patient to additional radiation. Consequently, it is not done in a screening/prevention scenario, but it is used as a diagnostic tool for symptomatic patients. Furthermore, because of the additional radiation, multiple CAC score measurements per patient are not a viable option, and there is no acceptable way to measure the CAC score over time. [0007] Accordingly, what is needed are devices, systems, and methods for more accurately and precisely detecting, scoring, and predicting disease risk. SUMMARY [0008] A system for determining a risk of a clinical indication (e.g., by detecting, scoring and predicting the risk of the clinical indication) is described. This system includes: a computing device including a network interface for communications over a data network; and a clinical indication score engine having a processor and a memory, and including a network interface for communications over the data network. Moreover, in response to receiving medical information associated with an individual including a medical image associated with a non-invasive medical imaging technique from the computing device, the memory stores the medical image, and the processor analyzes the medical image, generates a clinical score for at least a portion of the medical image, and transmits an instruction to a measurement device to a acquire a second medical image associated with a second non- invasive medical imaging technique based at least in part on the clinical score. Furthermore, analysis of the medical image includes: determining contours in the medical image satisfying one or more criterion; analyzing one or more geometric attributes and one or more contrast attributes of contours included in the first subset of contours to identify a second subset of contours based at least in part on the contours satisfying one or more predetermined geometric and contrast attributes; selecting a third subset of contours from the second subset of contours that corresponds to one or more potential instances of a type of tissue, the third subset selected based at least in part on contours within the third subset satisfying first criteria associated with the type of tissue; ranking contours included in the third subset of contours based at least in part on a selection metric, the selection metric accounting for a combination of contrast and intensity; grouping the contours included in the third subset of contours into nested structures; selecting one or more instances of the type of tissue from the nested structures satisfying second criteria associated with the type of tissue; grouping the selected one or more instances of the type of tissue into clusters based on neighboring instances of the type of tissue and a spatial cluster scale; classifying the clusters as benign or possibly associated with the clinical indication by performing one or more of: a regression analysis on the one or more instances of the type of tissue within the clusters, edge detection, a density analysis of the clusters, or a circularity analysis of the clusters; scoring the clusters using an analytic function to generate the clinical score; and combining physical features of each of the one or more instances of the type of tissue, as an individual structure and as part of a cluster of micro-instances of the type of tissue, together with clinical data, to construct a predictive model (such as a machine-learning model or a neural network) and provide a scale of suspiciousness for the clinical indication. Additionally, in response to receiving the second medical image associated with the second non-invasive medical imaging technique from the computing device, the memory stores the second medical image, and the processor revises the scale of suspiciousness for the clinical indication based at least in part on the second medical image. [0009] Note that the clinical indication may include a type of cancer, such as neurological, lung, prostate or breast cancer. Alternatively or additionally, the clinical indication may include: a type of cardiovascular disease, or a type of Thyroid disease. [0010] Moreover, the type of tissue may include calcifications. [0011] Furthermore, the medical image includes one or more of an x-ray image or a CT scan. Additionally, the second medical image includes one or more of a magnetic resonance (MRI) image or an ultrasound image. Note that the non-invasive medical imaging technique and/or the second non-invasive medical imaging technique may involve the use of an injected contrast. [0012] The medical information may include clinical data (such as an age of the individual) and/or a medical history, e.g., of a family of the individual. [0013] Moreover, the generated clinical score for at least a portion of the medical image may include a clinical score for: the individual, for an examination of the individual, and/or a lesion or a feature in the medical image. [0014] In some embodiments, the instructions may be provided while non-invasive imaging of patient is being performed or while the patient is available for the second medical image to be acquired following acquisition of the medical image. For example, the patient may still be at a facility that includes the non-invasive medical imaging technique and the second non-invasive medical imaging technique. [0015] Note that the processor may extract tagged data from the medical image, where the medical image is included in a computer file. For example, the tagged data may include one or more of a side, a pixel spacing, an orientation, a protocol, or a date. In some embodiments, the tagged data is included in a Digital Imaging and Communications in Medicine (DICOM) header. [0016] Moreover, the processor may convert the medical image to a real array of intensities for contouring, such as a 4-byte real array of intensities. [0017] Furthermore, the processor may select intensity levels for determining contours in the medical image. [0018] Additionally, the one or more criterion may include that each contour in the first subset of contours is (i) closed and (ii) includes a contour value larger than a surrounding area external to the contour. [0019] In some embodiments, contours not satisfying the one or more criterion are discarded. [0020] Note that the one or more geometric attributes of contours may include at least one of: a centroid, an area, a perimeter, a circle ratio, and an interior flag. [0021] Moreover, the one or more contrast attributes of contours may include at least one of: an intensity, an inward contrast, an outward contrast, or a gradient scale. [0022] Furthermore, the processor may detect an object in the medical image for exclusion from further analysis. For example, the object may be an external object or a foreign object (such as a biopsy clip, a breast implant, a clamp, etc.). In some embodiments, the object may be detected through the object having at least one of: an area greater than a predetermined area, an intensity greater than a predetermined intensity, or a circle ratio greater than a predetermined circle ratio. [0023] Additionally, selecting the third subset of contours may include excluding contours located within a predetermined distance from at least one of an edge of the medical image and an edge of tissue. [0024] Note that the first criteria may include contours having a predetermined area and a predetermined gradient scale. For example, the predetermined area may be between 0.003 mm2 and 800 mm2 and the predetermined gradient scale may be less than 1.3 mm. [0025] Moreover, the first criteria may include contours having a predetermined intensity, a predetermined circle ratio, a predetermined inward contrast, or a predetermined outward contrast. For example, the predetermined intensity may be greater than 0.67 times a maximum intensity, the predetermined circle ratio is greater than 0.65, the predetermined inward contrast is greater than 1.06, or the predetermined outward contrast is greater than 1.22. [0026] Furthermore, the first criteria may include contours having a predetermined area, a predetermined circle ratio, or at least one of a predetermined inward contrast and a predetermined gradient scale. For example, the predetermined area may be less than 0.30 mm2, the predetermined circle ratio may be greater than 0.65, the predetermined inward contrast may be greater than 1.04, and/or the predetermined gradient scale may be greater than 0.3 mm. [0027] Additionally, the first criteria may include contours having a predetermined area, a predetermined circle ratio, or a predetermined intensity. [0028] In some embodiments, the memory may save the third subset of contours. [0029] Note that the processor may identify instances of the type of tissue for each nested structure based on at least one of: a contour derivative or a grouping parameter computed for each nested structure. For example, the contour derivative may measure how rapidly intensity varies across a nested structure. [0030] Moreover, the processor may identify outer contours in each nested structure representing a contour shape and inner contours in each nested structure providing data on internal gradients. [0031] Furthermore, the second criteria may include a threshold on a contour derivate and a threshold on a grouping parameter. [0032] Additionally, the processor may computer cluster properties. For example, the cluster properties may include one or more of: a cluster centroid, a cluster half-length, a cluster half-width, an aspect ratio, a principal axis, or a packing fraction. [0033] In some embodiments, the system does not provide the instruction to the measurement device. Instead, the medical image and the second medical image are acquired concurrently by the measurement device, and then are provided to the system via the computing device. [0034] Note that instead of or in addition to detecting the clinical indication, in other embodiments the system diagnoses the clinical indication. For example, the system may compute a classification associated with the clinical indication based at least in part on the medical image and the second medical image. [0035] Moreover, the measurement device may include non-invasive medical imaging technique and the second non-invasive medical imaging technique. [0036] Other embodiments provide the computing device. In some embodiments, the computing device includes the measurement device. [0037] Other embodiments provide a computer-readable storage medium for use with the system. When program instructions stored in the computer-readable storage medium are executed by the system, the program instructions may cause the system to perform at least some of the aforementioned operations of the system. [0038] Other embodiments provide a method for computing a scale of suspiciousness for the clinical indication. The method includes at least some of the aforementioned operations performed by the system. [0039] This Summary is provided for purposes of illustrating some exemplary embodiments, so as to provide a basic understanding of some aspects of the subject matter described herein. Accordingly, it will be appreciated that the above-described features are only examples and should not be construed to narrow the scope or spirit of the subject matter described herein in any way. Other features, aspects, and advantages of the subject matter described herein will become apparent from the following Detailed Description, Figures, and Claims. BRIEF DESCRIPTION OF THE DRAWINGS [0040] The foregoing and other objects, features and advantages of the devices, systems, and methods described herein will be apparent from the following description of particular embodiments thereof, as illustrated in the accompanying drawings. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the devices, systems, and methods described herein [0041] Figure 1 illustrates a networked cancer detection and quantification system. [0042] Figure 2 is a flow chart of a method for determining a cancer score. [0043] Figure 3 is a flow chart of a method for detecting and quantifying cancer. [0044] Figure 4 depicts a medical image of calcifications in a patient’s tissue. [0045] Figure 5 illustrates an example of a selection process for a malignant cluster of micro-calcifications. [0046] Figure 6 illustrates an example of the identification of a cluster of calcifications. [0047] Figure 7 is a graph showing an example of a packing fraction as a function of <Ai>Amax for exemplary clusters. [0048] Figure 8 is a drawing showing an example of an updated worklist. [0049] Figures 9A, 9B and 9C provide example images and the associated BAC. [0050] Figure 10 is a drawing showing an example of the Pearson correlation between the two BAC scores. [0051] Figure 11 provides example images comparing the Bradley and WLS scores. [0052] Figure 12 is a drawing showing an example of a comparison of BAC to CAC as a predictor of CHD. [0053] Figure 13A is a drawing showing an example of a ROC for the cmAngio model. [0054] Figure 13B is a drawing showing an example of positive and negative probability densities for the cmAngio model. [0055] Figure 13C is a drawing showing an example of a probability of event for the cmAngio model. [0056] Figure 14 provides example normal and breast cancer images showing micro-calcifications in situ, and inserted into a normal image. [0057] Figure 15 provides example normal and breast cancer images showing micro-calcifications in situ, and inserted into a normal image, with modification. [0058] Figure 16 provides example micro-calcifications inserted into a malignant mass image. [0059] Figure 17 provides example lucent calcifications inserted into normal image. [0060] Figure 18 provides an example malignant mass image, and extracted mass including buffer pixels. [0061] Figure 19 provides an example synthetic mass image including the extracted malignant mass of Figure 18. [0062] Figures 20A and 20B provides drawings showing an example the improvement in resultant AUC using the same algorithm with and without training using synthetics. [0063] Figure 21 provides an example of AUC improvement using synthetics. [0064] Figure 22 provides an example of AUC improvement using synthetics. [0065] Figure 23 provides an example of AUC improvement using synthetics. [0066] Figure 24 provides an example of the general improvement of AUC using synthetics. [0067] Figure 25 provides an example of improvements of Image-Based ROC and FROC using synthetics. [0068] Figure 26 depicts an exemplary triage system. [0069] Figure 27A depicts a raw image of corrosion on an aircraft. [0070] Figure 27B depicts ROIs which can be highlighted using a suspicion code. [0071] Figure 28 depicts several processing steps using a medical image according to the Neural Network of the present invention. [0072] Figure 29 depicts optional processing of a medical image. [0073] Figure 30 is a graph having an X-axis of Percent of Exams Identified as Suspicious and a Y-axis of Sensitivity. Percent of exams identified as suspicious in test set. [0074] Figure 31 is a graph having an X-axis of False Positive Rate and a Y-axis of Sensitivity. ROC curve based on test set. [0075] Figure 32 is a graph having an X-axis of False Positive Rate and a Y-axis of Sensitivity, and two threshold points 1 and 2 as described herein. ROC curve with suspicion score threshold. [0076] Figure 33 depicts an exemplary computing environment for determining a score and uncertainty level of medical images. [0077] Figure 34 depicts a flowchart for determining a score and uncertainty level of medical images. [0078] Figure 35 depicts images analyzed and evaluated by the systems and methods herein when training the model. [0079] Figures 36A-D depict images for training the model and determining uncertainty. [0080] Figures 36E and 36F depict images in the training set with high aleatoric uncertainty. [0081] Figure 37 depicts a larger region of a surrounding breast for context when training a model. [0082] Figure 38 depicts the inclusion of larger surrounding breast tissue in the crop when training a model. [0083] Figures 39A-C depict images in the training with high epistemic uncertainty. [0084] Figure 40 depicts the increased crop size to improve a model. [0085] Figures 41A-C depicts the efficacy of a Bayesian NN used with the systems and methods herein for cases of high and low levels of uncertainty. DETAILED DESCRIPTION [0086] The embodiments will now be described more fully hereinafter with reference to the accompanying figures, in which preferred embodiments are shown. The foregoing may, however, be embodied in many different forms and should not be construed as limited to the illustrated embodiments set forth herein. Rather, these illustrated embodiments are provided so that this disclosure will convey the scope to those skilled in the art. [0087] Anomaly: As used herein, the term “anomaly” refers to an outlier, novelty, noise, deviation, or exception. In the healthcare context, the term can refer to a lesion or feature associated with a physiological disease or disorder that is structurally and/or compositionally distinct from surrounding healthy and/or normal tissue. Those of skill in the art will recognize what anomalies would be present with respect to the diseases and disorders disclosed herein. For example, a cancerous mass would be detectable in a mammogram and indicate breast cancer. In another example, a calcification would be detectable in an echocardiogram and indicate heart disease. Those of skill in the art will recognize myriad other detectable anomalies which are indicative of a disease or disorder. [0088] BI-RADS®: As used herein, the term “BI-RADS®” refers to the standardized quality control system for interpreting mammograms which was developed by the American College of Radiology. BI-RADS® Assessment Categories are: 0: Incomplete; 1: Negative; 2: Benign; 3: Probably benign; 4: Suspicious; 5: Highly suggestive of malignancy; and 6: Known biopsy – proven malignancy. The BI-RADS® atlas is available at http://www.acr.org. [0089] Breast Density: As used herein, the term “breast density” refers to categories which a radiologist uses for describing a patient’s mammogram. Breast density categories include Class A (or 1): Fatty; Class B (or 2): Scattered fibroglandular density; Class C (or 3): Heterogeneously dense; and Class D (or 4): Extremely dense. [0090] CADe: As used herein, the term “CADe” refers to Computer-Aided Detection -- the identification of a location in data that a CADe system, in accordance with a CAD algorithm operating on the data, highlights for attention by a technician. For example, the identification can be a mark on a medical image, or a more general indication such as a visual cue, sound, or other perceptible indicator such as an email, text, score, and the like. [0091] CADx: As used herein, the term “CADx” refers to Computer-Aided Diagnosis – the use of systems to evaluate and associate a medical indication, or conclusion of the presence or absence of a condition or disease, to conspicuous structures identified in a medical image. For example, in mammography, CADx highlights microcalcification clusters and dense structures in soft tissue, and allows a radiologist to draw conclusions about the condition of the pathology. Both CADe and CADx may jointly be referred to as “CAD” throughout the Specification. [0092] CAD algorithm: As used herein, the term “CAD algorithm” means a computer implemented program for detecting and quantifying anomalies in Data. Preferably, the CAD algorithm is the Neural Network. [0093] Data: As used herein, the term “data” refers to information generated by a sensor which is used to train the Neural Network described herein. Data can also include a collection of information, whether or not generated by a sensor, such as manually generated information, including digital text, handwriting, numerical tables, and the like. [0094] Deep learning: As used herein, the term “deep learning” is broadly defined to include machine learning which can be supervised, semi-supervised, or unsupervised. Architectures of deep learning include deep neural networks, deep belief networks, recurrent neural networks, and convolutional neural networks. [0095] Deep neural network (DNN): As used herein, the term “deep neural network” is an artificial neural network with multiple layers between input and output layers. DNN model linear and non-linear relationships. [0096] DICOM®: As used herein, the term “DICOM®” refers to the international standard to transmit, store, retrieve, print, process, and display medical imaging information. A full description of the DICOM® standard and the association of Structured Reports with DICOM® images is provided in D. Clunie, “DICOM Structured Reporting,” PixelMed Publishing, Bangor, Pennsylvania (2000); see also http://www.dicomstandard.org. Medical information associated with DICOM® and Structured Reports can be burned in, overlaid, or provided separate from the original image. [0097] Medical image: As used herein, the terms “medical image” generally refers to X-ray imaging, CT imaging, MRI, positron emission tomography (PET), Digital Two- Dimensional (2D) imaging, Three-Dimensional (3D) Tomosynthesis, single-photon emission computed tomography (SPECT), ultrasound (US), endoscopy, thermography, medical photography, nuclear medicine functional imaging, elastography, photoacoustic imaging, echocardiography, functional near-infrared imaging, magnetic particle imaging, and the like. [0098] Neural Network: As used herein, the term “neural network” refers to the systems and algorithms described in the section “Neural Network Architecture” below. [0099] Recall Exams: As used herein, the term “Recall Exam” is defined as those for which further evaluation (e.g., additional mammographic views, breast ultrasound, etc.) were recommended during the interpretation of the screening mammograms. [0100] Sensitivity: As used herein, the term “sensitivity” is the ability of a test to correctly identify those with a physiological anomaly (true positive rate). In certain aspects, “sensitivity” can be defined as the number of true positive exams with at least one true positive mark provided by the CAD algorithm described herein divided by the number of True Positive Exams. [0101] Specificity: As used herein, the term “specificity” is the ability of the test to correctly identify those without the physiological anomaly (true negative rate). In certain aspects, the term “specificity” can be defined as the number of true negative exams without any marks divided by the number of True Negative Exams. [0102] Suspicion / Suspiciousness: As used herein, the terms “suspicion” or “suspiciousness” broadly refer to observable conditions which can be classified as an anomaly. For example, in the field of mammography, a suspicious feature of a mammogram could include the presence of a microcalcification cluster or mass. In addition, a suspicious feature could be defined to include the complexity of soft tissue within a breast, breast density, breast size, breast volume, and the like. [0103] True Positive Exams: As used herein, the term “True Positive Exams” are biopsy confirmed cancer exams. [0104] True Negative / Normal Exams: As used herein, the term “True Negative Exams” or “Normal Exams” are either biopsy confirmed negative exams or exams with at least 1 year of negative findings (BI-RADS® 1 or 2). [0105] All documents mentioned herein are hereby incorporated by reference in their entirety. References to items in the singular should be understood to include items in the plural, and vice versa, unless explicitly stated otherwise or clear from the text. Grammatical conjunctions are intended to express any and all disjunctive and conjunctive combinations of conjoined clauses, sentences, words, and the like, unless otherwise stated or clear from the context. Thus, the term “or” should generally be understood to mean “and/or” and so forth. [0106] Recitation of ranges of values herein are not intended to be limiting, referring instead individually to any and all values falling within the range, unless otherwise indicated herein, and each separate value within such a range is incorporated into the specification as if it were individually recited herein. The words “about,” “approximately,” or the like, when accompanying a numerical value, are to be construed as indicating a deviation as would be appreciated by one of ordinary skill in the art to operate satisfactorily for an intended purpose. Ranges of values and/or numeric values are provided herein as examples only, and do not constitute a limitation on the scope of the described embodiments. The use of any and all examples, or exemplary language (“e.g.,” “such as,” or the like) provided herein, is intended merely to better illuminate the embodiments and does not pose a limitation on the scope of the embodiments. No language in the specification should be construed as indicating any unclaimed element as essential to the practice of the embodiments. [0107] In the following description, it is understood that terms such as “first,” “second,” “top,” “bottom,” “up,” “down,” and the like, are words of convenience and are not to be construed as limiting terms. [0108] In general, described herein are devices, systems, and methods for computer- based disease detection and quantification. For example, the disease may include: cancer, a type of ASCVD (such as CHD), a neurological disease, etc. In the discussion that follows, cancer is used as an illustrative example. As used throughout this disclosure, “detection” of a disease may include uncovering, to a particular degree or range of certainty (which may be a predetermined degree/range, or a degree/range following standard industry practice), whether the disease (such as cancerous cells) is present (or not present) in a sample of tissue. Thus, detection may include discovering, affirming, finding, uncovering, unearthing, revealing, exposing, etc., the existence or absence of the disease (such as cancer cells) in a sample, which can be depicted in a medical image. The cancer cells may include malignant or benign cells. As used throughout this disclosure, “quantification” of cancer may include determining, indicating, or expressing the quantity of cancer cells in a sample. The quantity of cancer cells may include a specific number, range, or threshold of cells, the size of cells or groupings of cells, and so forth. Quantification of cancer may also or instead include generating a “score” or “indication” as described herein. Quantification of cancer may also or instead include generating a “grade” or “stage” of cancer. In general, and unless explicitly stated or otherwise apparent from the context, “detection” of cancer may be included in the “quantification” of cancer and vice-versa. For instance, in an aspect, if a quantity of cancer cells is determined (i.e., a quantification), then cancer is detected. In another aspect, if a certain cancer score is determined, cancer is detected. [0109] Although devices, systems, and methods discussed herein generally describe the detection and quantification of cancer, detection and quantification of other diseases, cells, physiological anomalies, and the like, may also or instead be enabled or automated by the devices, systems, and methods discussed herein. Although certain embodiments discussed herein describe the detection and quantification of cancer for the specific use case of breast cancer, the devices, systems, and methods discussed herein can be adapted to detect and quantify other cancers including without limitation brain, lung, liver, prostate, bone, cervical, colon, leukemia, Hodgkin disease, kidney, lymphoma, oral, skin, stomach, testicular, thyroid, and so forth. Furthermore, although embodiments generally described herein are detecting and quantifying cancer in medical images of human tissue, the embodiments may also or instead be applicable to cancer in animals, for example. More generally, in the healthcare context, the disclosed medical-imaging analysis techniques may include the triage of anomalous images or anomalous studies (which may include anomalies within an individual image within a study comprising multiple images) generated by a variety of imaging modalities. Although certain embodiments discussed herein relate to anomaly detection and suspiciousness categorization, methods discussed herein can be adapted for other anomaly detection including in data streams and other data generated by a variety of sensors or found in a variety of databases. Although embodiments generally described herein relate to medical images, the embodiments may also or instead be applicable more broadly to image categorization for use in other fields such as in facial recognition, optical character recognition, landmark detection, drone videography, industrial equipment inspection and maintenance, among many other examples. [0110] In general, the devices, systems, and methods discussed herein may utilize medical image analysis, which may be automated through the use of various hardware and software as described herein. The medical image analysis techniques discussed herein may thus be used quantify cancer (e.g., breast cancer) and/or generate a cancer quantification. It will be appreciated, however, that the implementations discussed herein may also or instead generate a cancer quantification based on other pieces of medical information about tissue other than images as described herein and may be implemented in other ways than those described herein that are within the scope of the disclosure. In one embodiment described below, the computer-based cancer quantification system and method may be used for detecting and quantifying breast cancer in humans where the medical images are mammograms. However, in other embodiments the medical image analysis techniques discussed herein may be used for the detection of non-medical anomalies and categorizing images and other data for review by a technician. [0111] Implementations may provide an accurate quantification of cancer that can be utilized in a number of different ways. For example, an accurate quantification of cancer may be used for an accurate detection of cancer in a piece of medical information, such as a medical image, an early detection of cancer, the growth rate of cancer, or a prediction of the likelihood of cancer. An accurate quantification of cancer may also or instead be used to reduce the number of unnecessary biopsies (i.e., reduce false positives) and reduce the number of undiagnosed cancers (i.e., reduce false negatives). An accurate quantification of cancer may also or instead be used to determine a tumor “grade,” e.g., a measure of the aggression of a specific form of cancer, whether the cancer is changing or is it staying localized (in some cases one may want to leave the cancer alone rather than operate based on the tumor grade), and so forth. An accurate quantification of cancer may also or instead be used to determine how a treatment is affecting the cancer cells or is producing new cancer cells. [0112] The devices, systems, and methods discussed herein may be used to generate a “score” that quantifies any tissue anomalies. The score may also be referred to herein as a “Q score,” “Q factor,” or the like. With respect to cancer, the cancer score may be expressed in any suitable or useful level of granularity such as with discrete categories (e.g., cancerous, non-cancerous, benign, malignant, cancer-free, tumor-free, and so on), or with a numerical score, alphabetic score/grade, or other quantitative indicator. For example, the cancer score may be a two-state score (e.g., cancer detected or cancer-free), a three-state score (e.g., cancer detected, cancer-free, unknown), a five-state score (e.g., unknown, cancer detected, cancer-free, benign, malignant), a range-bounded quantity (e.g., a score from 0–10, 0-100, or 0–1,000), or any other suitable score for quantifying cancer with any desired degree of granularity. The cancer score may also or instead be scaled. By way of example, tissue abnormalities may be associated with a score or the like, which may be based on a predetermined scale, e.g., 0–100, where certain known benign abnormalities would have a score close to or equal to 0 and certain known malignant abnormalities detected in advanced stages would have a score close to or equal to 100 (or vice-versa). In another aspect, cancer information may be multi-dimensional, so that multiple aspects may be independently scored. It shall be understood that the cancer score may change to indicate that cancer is more likely as the cancer cells/tumor grows and the cancer score may also change to indicate the opposite when the cancer cells/tumor shrinks. As discussed above, in one implementation, a smaller cancer score indicates a benign tumor and a larger cancer score indicates cancer. [0113] As another example, the devices, systems, and methods discussed herein may be used to guide a radiologist analyzing a medical image, or to pre-screen, supplement, verify, or replace a radiologist’s review. For example, in the context of breast cancer, a radiologist typically reviews each mammogram. It has been shown that for every 100 screening mammograms performed, 10% are recalled for subsequent procedures, and of those, only 5% have cancer. This indicates that the prevalence of cancer in all mammograms is only 0.5%. Thus, 99.5% of the time, there is no cancer shown in the mammogram and yet the radiologist typically reviews the mammogram. Thus, the devices, systems, and methods discussed herein may be used to pre-screen mammograms, score the mammograms according to a cancer score as discussed herein, and/or identify mammograms that show no anomalies or show only known benign anomalies (no cancer) and thus detect the absence of cancer so that these can be ignored for further analysis. Thus, implementations may generate an indication of the absence of cancer in certain medical images and the radiologist need not review those medical images in detail based on the indication of the absence of cancer for the particular mammogram. With this pre-screening, particular by automated systems contemplated herein, the radiologist may not need to analyze a large percentage of mammograms, thus significantly reducing a radiologist’s workload. [0114] As yet another example, implementations may be used to generate an assessment or prediction of the activity of a cancer for a patient (e.g., implementations can determine that, over a particular time period, a cancer will not grow significantly), which may be used to determine a treatment for the particular patient. By way of example, a patient with prostate cancer may receive an assessment that the cancer is not going to grow significantly in the next six months and the patient may then opt for a less invasive treatment plan. [0115] In another example, a retrospective study can be conducted whereby the present systems and methods are used to analyze a radiologist’s previous findings to determine whether the radiologist failed to detect cancer in a medical image, and whether a cancer score as described herein is different from the radiologist assessment. [0116] Figure 1 illustrates a networked cancer (and, more generally, an anomaly) detection and quantification system. In some embodiments, the system 100 may be used to generate synthetic 2D images, triage a workflow and/or determine uncertainty levels of analysis. As shown in the figure, the system 100 may include a client server implementation of a cancer quantification system. The system 100 may include one or more computing devices 102 that are each used by a user or an administrator to couple to and interact with, over a network 104, a backend component 106. Although a client server/web implementation of the system 100 is shown, the system 100 may also be implemented using a software as a service (SaaS) model, a standalone computer, and other computer architectures. [0117] The one or more computing devices 102 may include a processor-based computing device that has at least one processor, memory, persistent storage, a display, and communication circuits (such as a network interface and/or one or more input/output interfaces) so that each computing device 102 can communicate with the backend component 106, display a generated cancer score, submit pieces of medical information to the backend component 106, or otherwise interact with the backend component 106 or another component of the system 100. For example, the computing device 102 may include without limitation a smartphone device, a tablet computer, a personal computer, a workstation, a laptop computer, a server, a terminal device, a cellular phone, a wearable computer, a television, a set-top box, and the like. In some embodiments, the computing device 102 may execute an application, such as a known browser application or mobile application, that facilitates the interaction of the computing device 102 with the backend component 106. The one or more computing devices 102 may also or instead include an endpoint, for example including client devices such as a computer or computer system, a Personal Digital Assistant, a mobile phone, or any other mobile or fixed computing device. [0118] The computing device 102 may be used for any of the entities described herein. In certain aspects, the computing device 102 may be implemented using hardware (e.g., in a desktop computer), software (e.g., in a virtual machine or the like), or a combination of software and hardware. The computing device 102 may be a standalone device, a device integrated into another entity or device, a platform distributed across multiple entities, or a virtualized device executing in a virtualization environment. [0119] The processor in computing device 102 may be any processor or other processing circuitry capable of processing instructions for execution within the computing device 102 or system 100. Examples of such processors are CPUs and GPUs. The processor may include a single-threaded processor, a multi-threaded processor, a multi-core processor and so forth. The processor may be capable of processing instructions stored in the memory or a data store. [0120] The memory may store information within the computing device 102. The memory may include any volatile or non-volatile memory or other computer-readable medium, including without limitation a Random Access Memory (RAM), a flash memory, a Read Only Memory (ROM), a Programmable Read-only Memory (PROM), an Erasable PROM (EPROM), registers, and so forth. The memory may store program instructions, program data, executables, and other software and data useful for controlling operation of the computing device 102 and configuring the computing device 102 to perform functions for a user. The memory may include a number of different stages and types of memory for different aspects of operation of the computing device 102. For example, a processor may include on-board memory and/or cache for faster access to certain data or instructions, and a separate, main memory or the like may be included to expand memory capacity as desired. All such memory types may be a part of the memory as contemplated herein. [0121] The memory may, in general, include a non-volatile computer readable medium containing computer code that, when executed by the computing device 102 creates an execution environment for a computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of the foregoing, and/or code that performs some or all of the steps set forth in the algorithmic descriptions set forth herein. In some embodiments, any number of memories may be usefully incorporated into the computing device 102. For example, a first memory may provide non-volatile storage such as a disk drive for permanent or long-term storage of files and code even when the computing device 102 is powered down. A second memory such as a random access memory may provide volatile (but higher speed) memory for storing instructions and data for executing processes. A third memory may be used to improve performance by providing higher speed memory physically adjacent to the processor for registers, caching, and so forth. The processor and the memory can be supplemented by, or incorporated in, logic circuitry. [0122] The network 104 may include a communications path such as a wired or wireless network that uses a communications protocol and a data protocol, such as HTTP or HTTPS and HTML or JSON or REST, to allow each computing device 102 to interact with the backend component 106. The network104 may be a wired network, a wireless computer network, a wireless digital data network, a cellular wireless digital data network, or a combination of these networks that form a pathway between each computing device 102 and the backend component 106. [0123] The network 104 may also or instead include any data network(s) or internetwork(s) suitable for communicating data and control information among participants in the system 100. This may include public networks such as the Internet, private networks, and telecommunications networks such as the Public Switched Telephone Network or cellular networks using third generation cellular technology (e.g., 3G or IMT-2000), fourth generation cellular technology (e.g., 4G, LTE. MT-Advanced, E-UTRA, etc.) or WiMax- Advanced (IEEE 802.16m), fifth generation cellular technology e.g., 5G) and/or other technologies, as well as any of a variety of corporate area, metropolitan area, campus or other local area networks or enterprise networks, along with any switches, routers, hubs, gateways, and the like that might be used to carry data among participants in the system 100. The network 104 may also include a combination of data networks, and need not be limited to a strictly public or private network. The participants in the system 100 may each be configured with a network interface 124 for communications over the network. [0124] A user 108 of the system 100 may be a technician, patient, a doctor, a radiologist, a health care organization, an image analyst, and the like. The user 108 may, using the computing device 102, submit one or more pieces of medical information 110 for analysis or quantification by the system 100 and/or receive, from the backend component 106, a cancer quantification score based on the received pieces of medical information 110. The backend component 106 may include storage 112 coupled to the backend component 106 (e.g., a memory, a database, and the like) that may store various data associated with the system 100 including a plurality of pieces of medical information 110 that may be used to generate one or more cancer quantification scores, user data associated with the system, and the like. The storage 112 may be implemented using a known software-based or hardware- based storage system. [0125] The backend component 106 may be implemented using one or more computing resources including without limitation a processor 114, a memory 116, persistent memory/storage, and the like. By way of example, each computing resource may be a blade server, a server computer, an application server, a database server, a cloud computing resource and the like. When the system 100 is implemented as the client server architecture as shown in the figure, the backend component 106 may have a web server 118 or the like that manages the connections and interactions with each computing device 102, generates HTML code to send to each computing device 102, receives data from each computing device 102, and the like. The web server 118 may be implemented in hardware or software. [0126] The backend component 106 may include a cancer score engine 120 that analyze pieces of medical information 110 about tissue. The cancer score engine 120 may generate any indications of cancer in any regions of the tissue and may generate a cancer score 122 for any regions of the tissue in which there is an indication of cancer. The cancer score engine 120 may receive/obtain the pieces of medical information 110 about tissue from a computing device 102, over a computer network from a third-party, or from the storage 112 of the system 100. The cancer score 122 may be transmitted through the network 104, e.g., for display on the one or more computing devices 102. The cancer score engine 120 may be implemented in software or hardware. When the cancer score engine 120 is implemented in software, the cancer score engine 120 (and its components) may comprise a plurality of lines of computer code that may be stored in a memory 116 and executed by a processor 114 of the backend component 106 so that the processor 114 is configured to perform the processes of the cancer score engine 120 (and its components) as described herein. When the cancer score engine 120 is implemented in hardware, the cancer score engine 120 (and its components) may comprise a microcontroller, a programmable logic device, an application specific integrated circuit, or other hardware device in which the hardware device performs the processes of the cancer score engine 120 (and its components) as described herein. The cancer score engine 120 may include an algorithm or series of algorithms that assist in generating the cancer score 122 as discussed herein. [0127] In some embodiments, the backend component 106 (and/or each computing device 102) may include an image analysis engine that analyze pieces of medical information 110. The image analysis engine may generate quantitative scores related to whole images and/or ROIs in images, suspicion code, and may optionally generate an output which may be a general classification, for example, as to whether the image, or study comprising multiple images, is “normal” or “not normal.” In one aspect, the image analysis engine can be described under the Neural Network Architecture section below. The image analysis engine may receive/obtain pieces of medical information 110 from a computing device 102, over a computer network from a third-party, or from the storage 112 of the system 100. The output, such as a suspicion code, may be transmitted through the network 104, e.g., for display or intake on the one or more computing devices 102. The image analysis engine may be implemented in software or hardware. When the image analysis engine is implemented in software, the image analysis engine (and its components) may comprise a plurality of lines of computer code that may be stored in a memory and executed by a processor of the backend component 106 so that the processor is configured to perform the processes of the image analysis engine (and its components) as described herein. When the image analysis engine is implemented in hardware, the image analysis engine (and its components) may comprise a microcontroller, a programmable logic device, an application specific integrated circuit, or other hardware device in which the hardware device performs the processes of the image analysis engine (and its components) as described herein. The image analysis engine may include an algorithm or series of algorithms that assist in generating the output as discussed more fully below as the Neural Network Architecture. [0128] The one or more pieces of medical information 110 may include a medical image. The medical image may include an x-ray image, e.g., a mammogram and the like. The medical image may also or instead include magnetic resonance (MRI) images, CT scan images, ultrasound images, and so on. As discussed further below, multiple medical-imaging modalities may be used in the analysis. [0129] The system 100 may instead be implemented as part of a standalone computer implementation of a cancer detection and quantification system. In this implementation, the cancer score engine 120 may be executed on one of the computing devices 102, e.g., by a processor, based on one or more pieces of medical information 110 stored in the computing device 102 or input into the computing device 102. The computing device 102 may have a display 126 and any other additional hardware including without limitation input/output devices such as a keyboard and a mouse as shown. The display 126 may include a user interface, e.g., a graphical user interface. The computing device 102 may also include the processor, and a persistent storage device such as flash memory or a hard disk drive and memory, such as DRAM or SRAM, that are connected to each other. When the computing device 102 is used to implement the cancer (or anomaly) quantification system and the cancer score engine 120 (or the image analysis engine) is implemented in software, the memory may store the cancer score engine 120 and an operating system and the processor of the system may execute a plurality of lines of computer code that implement the cancer score engine 120 so that the processor of the computer system is configured to perform the processes of the cancer score engine 120 as described herein. [0130] The cancer score engine 120 may, in general, receive one or more pieces of medical information 110 about a piece of tissue of a patient and, for each piece of tissue for the patient, generate one or more cancer scores 122 about one or more regions in the piece of tissue. The piece of tissue may include without limitation any piece of human tissue or any piece of animal tissue that may have cancer cells. [0131] Alternatively or additionally, the image analysis engine may, in general, receive one or more pieces of medical information 110 and, for each anatomical feature of the patient, generate information related thereto. In addition, a worklist management processor may be configured to implement the worklist processing functionalities described herein. One of skill in the art will recognize, however, that such worklist processing can be performed by any combination of the processors described herein, or by any other processor or combination of processors (such as CIS/HIS/RIS processors) coupled to the network without departing from the scope of the preferred embodiments. [0132] In some embodiments, the system 100 includes one or more non-invasive medical imaging or measurement devices 128. The measurement devices 128 may concurrently, sequentially, or a combination of both, perform measurements on at least a portion of a person or an animal, such as: X-ray imaging, CT imaging, MRI, PET, Digital 2D imaging, 3D Tomosynthesis, SPECT, ultrasound (US), endoscopy, thermography, medical photography, nuclear medicine functional imaging, elastography, photoacoustic imaging, echocardiography, functional near-infrared imaging, magnetic particle imaging, and/or type of another medical imaging technique. Note that the X-ray imaging, the CT imaging, Digital 2D imaging, 3D Tomosynthesis, etc. may use a low-energy, non-thermionic cathode (which is sometimes referred to as a ‘cold cathode’). The non-thermionic cathode may include a field-emission cathode that selectively provides X-rays to at least the portion of the person or the animal. Moreover, note that the measurement devices 128 may be arranged in an array (such as a circular or donut-shaped array) around and/or along (e.g., along a longitudinal direction) at least the portion of the person or the animal. [0133] Thus, in some embodiments the system 100 may be used for retrospective analysis of medical information. However, in other embodiments, the analysis of the medical information by the system 100 may be performed concurrently or while one or more medical imaging techniques are performed, or may be performed immediately after a given medical imaging technique is performed but before a subsequent medical imaging technique in a series of medical imaging techniques is performed. In this way, the one or more medical imaging techniques may be adapted based at least in part on at least a portion of the analysis or information that is learned during one or more medical imaging techniques. [0134] Moreover, the availability of different medical imaging techniques in the measurement devices 128 may allow the system 100 to acquire medical images or information using the medical imaging technique that is best suited for an anomaly and/or a type of tissue. For example, CT imaging may be used for a mammography of a breast, while ultrasound may be subsequently used for an identified cyst. Thus, ultrasound may be used assess a potential false positive. In some embodiments, multiple medical imaging techniques are used to acquire medical images of the same tissue. Alternatively, different medical imaging techniques may be used to acquire medical images of different (non-overlapping) types of tissue. The use of multiple medical imaging techniques may allow the system 100 to address real-time performance requirements when acquiring medical images, such as time- varying or non-static performance requirements associated with hand-held ultrasound measurements. [0135] Furthermore, in some embodiments the system 100 may improve the signal- to-noise ratio of medical images that are acquired using low-energy X-rays. For example, a pretrained neural network (such as generative adversarial network or GAN) may be used to correct for artifacts or missing pixels in a medical image that is acquired using low-energy X- rays. In some embodiments, the GAN may be trained using a training dataset with medical images or synthetic medical images at a range of tilt angles relative to a plane of the medical images in order to improve the accuracy of the reconstructed medical images and to increase the robustness of the reconstruction process to the effect of tilt (and, more generally, planar misregistration. [0136] Additionally, the medical images acquired using the medical imaging techniques may or may not be registered relative to at least the portion of the person or the animal. For example, a medical image of a breast may not be registered, while a medical image of a lung may be registered relative to the ribs, or a medical image of an eye may be registered relative to an eye socket. Consequently, in some embodiments, at least some of the medical images may be registered relative to one or more anatomical structures or landmarks in at least the portion of the person or the animal. [0137] Figure 2 is a flow chart of a method for determining a cancer score. The cancer score may be generated or determined by the cancer score engine. In general, the method 200 may involve processing and analyzing one or more pieces of medical information for generating, for one or more regions of the piece of tissue, an indication of cancer. For example, as described in more detail below, the method 200 may gather variables that are pertinent to the cancer, programmatically analyze the piece(s) of medical information to determine values of the variables, transform these variables for use in generating a cancer indication or score, and then generate an indication of cancer based on these variables or a transformation of these variables. A cancer score generator component may receive the indications of cancer in the one or more regions of the tissue and generate a cancer score for at least one region of the tissue. The cancer score may have a value that increases as a cancer tumor grows and decreases as a cancer tumor shrinks. In some embodiments, the cancer score may be normalized and have threshold levels so that, for example, a normalized cancer score of 1–3 indicates a benign tumor, a normalized cancer score of 4–6 indicates suspicious cells, and a normalized cancer score of 7–10 indicates that cancer is present in particular region(s) of tissue. [0138] As shown in step 202, the method 200 may include receiving one or more pieces of medical information for processing and analysis. The medical information may include information about a patient’s tissue, e.g., medical images of the tissue. The medical information may include preprocessed or raw data, which is then processed and analyzed by the systems or methods described herein. In an aspect, the cancer score engine may include a medical information analysis component that receives one or more pieces of medical information, where the cancer score engine then processes and analyzes this information. The medical information may be automatically streamed to the cancer score engine by an uneven length preprocessed time series input. For example, the header of a DICOM file may contain information on the image contained within it including, but not limited to, the pixel resolution in physical units, criteria for interpreting the pixel intensity, etc. [0139] As shown in step 204, the method 200 may include analyzing the one or more pieces of medical information about the tissue. This may include gathering variables values about the medical information (e.g., a mammogram), where generating the indication of cancer may be based on the gathered variable values. The variables may include an intensity value for contours of any calcifications, a gradient of the calcifications, one or more characteristics about each calcification, and a hierarchical structure of the calcifications in a cluster. [0140] As shown in step 206, the method 200 may include generating an indication of cancer. By way of example, the indication of cancer may be generated for one or more regions of the tissue in the medical images. [0141] As shown in step 208, the method 200 may include generating a cancer quantification score. By way of example, generating a cancer quantification score may include generating a cancer score for each region of the tissue based on the indication of cancer in each region of the tissue. The cancer quantification score may indicate an absence of cancer in the region of the tissue. [0142] As shown in step 210, the method 200 may include generating guidance for a medical professional based on one or more of the indications of cancer and the cancer quantification score. The guidance may include, e.g., guidance for a radiologist based on the presence or absence of cancer in the region of the tissue. The guidance may be generated by applying rules based on the analysis of the medical information, the indication of cancer, or the cancer quantification score. [0143] Implementations may utilize one or more algorithms for detecting and quantifying cancer from medical information supplied to the system. For example, for detecting and quantifying breast cancer, the algorithm may detect and quantify micro- calcifications in mammogram images. The algorithm may in general include (1) detecting and grouping calcifications into clusters, (2) classifying types of benign clusters, (3) quantifying clusters that are potentially malignant with a ‘Q factor’ as discussed herein, and (4) saving output quantities to evaluate performance. In an implementation, a first algorithm generates an indication of cancer and a second algorithm generates a cancer score. [0144] Figure 3 is a flow chart of a method for detecting and quantifying cancer. The method 300 may be performed using one or more algorithms as described herein, or with assistance from an algorithm. Thus, the method 300 may be performed by a computer program product comprising non-transitory computer executable code embodied in a non- transitory computer readable medium that, when executing on one or more computing devices, performs the steps of the method 300. The method 300 may in general be performed on one or more pieces of medical information, e.g., one or more images, for detecting an object, area, region, feature, data point, piece of information, etc., of interest (e.g., a calcification, lesion, mass, tumor, and the like in a medical image of tissue). [0145] As shown in step 302, the method 300 may include initializing an algorithm. In this step 302, memory structures may be declared and various free parameters for a model may be set. In an aspect, the parameters and model choices are spread throughout code of a computing device. In another aspect, all model parameters are set in a single place during the initialization of the algorithm, along with a clear description of each parameter including the specific section of the code/algorithm where it is utilized. This may be provided through an interactive feature for a user, e.g., a graphical user interface of a user device. In this manner, the parameters may be adjusted or inputted by a user of the method 300. In a non-interactive version, a piece of medical information may simply be received, e.g., a single image to analyze. [0146] As shown in step 304, the method 300 may include reading data. The data may include a DICOM header and image data. The DICOM header may contain a range of useful information including without limitation, the side (i.e., left or right), orientation, view, protocol, date of procedure, and so forth, many of which may be listed in a filename convention. This information may be extracted for use by the algorithm—for example, in order to compare results from multiple views, or from a time series of images. Examples of DICOM tags include without limitation: (1) pixel spacing (e.g., hex tag - (0028x,0030x)), which may be useful to scale the image in terms of real physical dimensions (e.g., mm), which can compute a ‘Q factor’ consistently; (2) diagnostic vs screening (e.g., hex tag - (0032x,1060x)), which may allow for inclusion or exclusion of diagnostic images from studies; and (3) patient orientation (e.g., hex tag - (0020x,0020x)), which may allow for displaying the images in a consistent manner, i.e., in the same orientation as used by radiologists in typical computer-aided design (CAD) systems, which can be advantageous when contour data is returned for display and/or analysis. For consistency in analysis, a predetermined orientation may be assigned (e.g., for mammograms – where the nipple points to the left in all images as is the industry standard). Alternatively, an orientation where burned-in lettering is displayed/oriented correctly may be utilized. [0147] The image data may be read in with η1 x η2 elements and converted to a 4- byte real array of intensities I(η1, η2) for contouring as a final step for reading data. [0148] As shown in step 306, the method 300 may include computing contours for the image. For this step 306, the intensity levels for contouring may first be selected, where an example will now be described. Typically, the side and view information are burned into an image at 100% of the maximum possible intensity, while the intensity levels within tissue in the image can be significantly less than this peak value. In order to scale the contours in a consistent manner, the maximum intensity scale may be defined as Iscale = max[I(x1, x2)] within the tissue (i.e., excluding the burned in region). By way of example, in an aspect, the following set of contours may be selected: I = (0.05, 0.075, 0.10, … → 0.70)Iscale → 27 levels , (0.71, 0.72, 0.73, …→ 0.99)Iscale → 29 levels , (Eq.1) for a total of 56 levels. This set may provide a sufficient number of contours to work with most medical images. While contouring algorithms may return all contours within a given domain, here, an implementation may only be interested in keeping a subset of contours that include contours that are (a) closed and (b) where the contour value is larger than the surrounding area outside. This may be the first contour selection criteria identified in method 300. For example, after contouring the image, the closed loops that are found can be of two possible types: (1) the contour value is larger than the surrounding values in the image (i.e. such contours enclose a bright spot, and potential calcifications). (2) the contour value is below the surrounding values in the images (i.e., such contours enclose a darker region, which may be ruled out) The algorithm may only select the subset of contours of the first type. [0149] As shown in step 308, the method 300 may then analyze the data to determine whether there are closed contours and/or whether the contour value is larger than the surrounding area outside. If contours do not meet these criteria, then they may be discarded as explained below. [0150] As shown in step 310, the method 300 may include discarding contours that do not meet desired criterion, e.g., contours that are not closed. [0151] As shown in step 312, the method 300 may include analyzing the geometry and contrast of the contours, e.g., the closed contours that were not discarded by the previous step. Contouring an image (e.g., a mammogram) with the intensity levels provided above can result in 105 → 106 closed contours, most of which do not correspond to clear structures of interest. To identify the contours that correspond to calcifications, masses, or external objects, the method 300 may evaluate the following geometric and contrast characteristics for each of the closed contours computed above, where each of the following are provided by way of example and not of limitation: 1. Centroid → x1, x2 2. Area → A 3. Perimeter →
Figure imgf000028_0001
4. Circle Ratio → Cratio → 1 for a perfect circle
Figure imgf000028_0002
5. Intensity → Io at the centroid location 6. Inward contrast → Cin = Io / I where I is the contour value 7. Outward contrast → Cout = Io / Iout where Iout is the average intensity outside the contour 9. Gradient scale →
Figure imgf000028_0003
10. Interior flag – ignore contours that are too close to the edge of the tissue in the image. [0152] As shown in step 314, the method 300 may include detecting an object, e.g., detecting an object in the image. The object may be an external object, or other regions where detection may be beneficial, e.g., for exclusion in an analysis by the algorithm. For example, an image may include external objects, such as implants or diagnostic clamps. Also, some images may include regions with exceptionally poor contrast. Often there are small scale contours within the interior of these regions, which can appear as calcifications to the algorithm, and thus trigger false positives. Thus, these regions may be detected and excluded from consideration. Thus, in an aspect, the algorithm can be configured to find one or more such regions in each image, e.g., based on the following contour selection criteria: A ≥ 800 mm2 and I ≥ 0.5Iscale and Cratio > 0.22 , (Eq.2) corresponding to large bright regions with fairly smooth boundaries. For images that have objects, a number of contours may satisfy this criterion and these will typically be nested inside one another. In order to find the contour that best approximates the shape of the object, the contour that maximizes the triple product AICratio of these selection criteria may be selected. In most cases, finding the precise boundary may not be necessary, since the method 300 may just be attempting to exclude the interior area where false positives can form. In some images, false positives form just outside of the object, and thus a buffer region to exclude pixels immediately around the object may be added. [0153] As shown in step 316, the method 300 may then select contours for discarding (step 318) or keeping (step 320). [0154] The contours may be the contours computed above, which are then searched through for identifying potential calcifications or other features of interest. For breast cancer, the micro-calcifications of interest typically occur for a fairly narrow range of sizes (contour areas). However, depending on a particular patient, as well as the type and stage of cancer, the micro-calcifications can feature a range of contour shapes, intensity levels, and contrasts (i.e., spatial gradients). By way of example and not of limitation, the following selection can be used for most images: [0155] 1. Contours may be excluded that are within the interior of an object identified above in step 314. Also, contours may be excluded that are within a specified distance from the edge of the tissue or the edge of the image using the interior flag variable computed in step 312. [0156] 2. Contours may be included that are within the following range of areas and gradient scale: 0.003 mm2 < A < 800 mm2 and Lg < 1.3 mm , (Eq.3) and that also meet one of the following criteria (a)–(e), which are provided again by way of example: [0157] (a) Contours may be kept that enclose relatively bright regions with relatively desirable contrast values (these values may be selected by a user/administrator) and that are within a range of shapes that are not too highly deformed. This criterion may capture many of the most obvious calcifications. For example, contours may be kept that satisfy the following criteria: Io > 0.67Iscale and Cratio > 0.65 and Cin > 1.06 and Cout > 1.22 [0158] (b) Contours may be kept that have relatively weak contrast if the area is within the correct range for the smaller (weak) calcifications, and if the contours are more nearly circular or have shorter gradient scales. For example, contours may be kept that satisfy the following: A < 0.30 mm2 and [(Cratio > 0.80 and Cin > 1.04) or (Cratio > 0.65 and Lg < 0.3 mm)] [0159] (c) Contours may be kept that are relatively large and bright. For example, contours may be kept that satisfy the following: (Io > 0.75Iscale and Cratio > 0.69 and A > 1.2 mm2) or (Io > 0.90Iscale and Cratio > 0.90 and A > 4.0 mm2) While these may be too large to be cancerous, these types of contours may be markers of type-2 benign clusters (e.g., fatty necrosis, etc.). These benign clusters may be ignored entirely in the analysis. However, the calcifications within these benign clusters may also have a range of shapes and sizes, some of which overlap with the selection criteria in (a)–(b) above. Thus, the method 300 may find all of the members of the type-2 clusters, and group their smaller members with these larger shapes. [0160] (d) Certain classes of contours may be kept that help reduce false-positives. For example, contours may be kept that satisfy the following: (Io > 0.62Iscale and Cratio > 0.67 and Cout > 2 and A > 0.2 mm2) or (Io > 0.60Iscale and Cratio > 0.50 and 3 mm2 > A > 1.5 mm2) or (Lg < 0.4 mm and Cratio > 0.67 and 3 mm2 > A > 1.3 mm2) Including these types of contours may allow the method 300 to reduce some common types of false-positives when contours are grouped into nested structures, as described below. For example, some larger calcifications are hollow in the center, resulting in a ring-like structure. The choice in (a)–(b) often results in these rings being broken up into many smaller apparent calcifications. However, by including the contours that encompass the entire structure, these small nested contours may be grouped with their outer parent, and thus allow the algorithm to understand these as a single composite structure (and not multiple distinct calcifications). [0161] (e) Contours may be kept that include relatively high central intensity, even if the contrast is relatively poor. For example, contours may be kept that satisfy the following: (Io > 0.90Iscale and Cratio > 0.50) [0162] 3. Full Contour Catalog. As shown in step 320, the method 300 may include cataloging the contours, e.g., cataloging the contours that are kept by the steps listed above. This may include developing a full contour catalog. This may be accomplished by saving a pointer to any contour that passes the above selection criteria. This may allow the method 300 to easily refer back to the contour at any later stage, including all of the associated geometric and contrast properties described above. [0163] 4. Order and Select. As shown in step 322, the method 300 may include determining whether to discard the contours, in which case the method proceeds to step 324, or whether to order and select the contours, in which case the method proceeds to step 326. In many images, the number of contours selected by (a)–(e) above into the full catalog is still quite large ~ 2 x 104. The more contours that are kept can increase the overall sensitivity, but can also lead to much longer analysis times in following steps of the method 300. Furthermore, keeping too many contours at this phase may lead to more false-positives. For these reasons, another selection process may be used that restricts the total number of contours that are considered to a predetermined number, e.g., Nmax = 6000. In any given image, the method 300 is typically looking for the contours with the best overall combination of intensity and contrast, since these are the ones that are most apparent to a human eye. However, the absolute value of the intensity and contrasts might be different in various images, even for the same patient. Thus, the contours in the primary library may be rank ordered based on the following: Selection Metric = 4(Cout – 1) + I/Io , In this manner, the contours with the relative best combination of contrast and intensity may be found at the top of list. In essence, in an aspect, the method 300 is scoring the contours on a relative scale for each image. [0164] 5. Primary Contour Catalog. As shown in step 326, the method 300 may include cataloging the contours, e.g., into a primary contour catalog or the like. This may be accomplished through saving a list of pointers to the first Nmax = 6000 contours identified by the above ordering (or whatever number is selected). This allows the method 300 to easily refer back to the best contours, including all of the associated geometric and contrast properties described above. [0165] As shown in step 328, the method 300 may include grouping the contours into nested structures/hierarchies. After completing the selection processes described above, there may only be Nmax = 6000 contours stored within the primary catalog (or whatever number is selected above). However, in most images, only a small fraction of these contours will correspond to true calcifications. Furthermore, as illustrated in Figures 4 and 5 described below, there may be at least several nested contours associated with each, and up to ten or more nested contours for calcifications with a strong intensity contrast. For any given calcification, it is desirous to identify a contour that characterizes the shape of the structure. To accomplish this, the primary library may be sorted through and the contours may be grouped into nested hierarchical structures. The outer most contour (parent) in each nested series may correspond to the shape, while the inner nested contours (children) can be used to precisely measure variations in contrast across the structure, as described below when selecting calcifications from the nested contours. [0166] For example, the Nmax contours are first sorted according to the area enclosed by each. Next, starting with the largest contour (call top level the “parent”), the contour library is searched to find the next smallest contour in the list that exists inside the area enclosed by the top level (parent). This would be the first “child”, which is grouped as part of this nested structure, and exclude it from our subsequent searches below. Next, the library is searched again to find the next largest contour that is inside the top-level parent (this one will have a smaller area than the first child). Normally, in simply nested structures, this contour would also be inside the 1st child, but that is not always the case. One could have multiple “peaks” inside the overall parent contour, and can be useful for looking at the internal structure of masses. The library is searched until no more contours that are inside the top-level parent. Then, the next largest contour in the library, which has not yet been grouped, is searched to repeat this process. [0167] After completing this step, there may be a list of the outer contours for each nested series, and a list of pointers to the inner nested contours for each of these structures. Some fraction of these nested structures may correspond to calcifications, but others may not. In order to aid in a final selection, the following properties for each nested series may be computed, which are provided by way of example and not of limitation: [0168] 1. Contour Derivatives – To identify calcifications, it may be desirous to precisely characterize how rapidly the intensity varies across the structure. Already, the method 300 may have computed several quantities that characterize this same general idea in an average sense (i.e., the inner Cin and outer Cout contrast described above) and for a local gradient scale – Lg. Once the contours are grouped into nested structures, the method 300 may compute the fractional change in area and/or intensity between any two nested contours in the structure. After trying a range of possibilities, the following two parameters may be defined: (Eq.4)
Figure imgf000032_0001
where the nested contours may be indexed from i = 1 → Ɲ, with i = 1 corresponding to the outer contour and Ɲ the innermost. Here, δA may be the minimum fractional area change between any two nested contours in the structure, and δI corresponds to maximum fractional intensity change between the inner nested contours and the outermost contour that defines the shape. Small values of δA << 1 may correspond to tightly nested contours, where the local gradient in intensity is large, while values of δI > 1 measures the fractional intensity variation across the set of nested contours (very similar to the inner contrast Cin discussed above). [0169] 2. Grouping Parameter. As will be discussed below with reference to grouping the contours into nested structures, one can accurately identify most of the clear calcifications using simple thresholds on δA and δI. However, a reasonable threshold on these parameters may often miss weaker calcifications and/or in some cases entire clusters if the calcifications are less distinct. Furthermore, if one lowers the thresholds to capture these missing clusters, it may result in an unacceptable increase in false-positives in other images. The basic problem may include the following: if these weaker calcifications are judged purely by themselves, it may often be unclear (even to the human eye) whether they are truly a calcification or not. However, these weaker structures are often far more interesting if they are grouped together, with the right size and right spatial separation. To put this idea into practice, a new collective parameter that characterizes these groupings may be used. To proceed, each of the nested structures identified in as a potential calcification may be viewed and the following parameter for the ith structure may be evaluated: (Eq.5)
Figure imgf000033_0001
where the summation is over the M other nested structures in the image, rij is the separation distance, and qi, qj are statistical weights defined by: qi = (Ioi/ Iscale) min[Ɲi, Ɲmax] F(amin, amax, Ai) (Eq.6) where Ioi is the central intensity, Ai is the area of the ith calcification, Ɲi is the number of nested contours, and Ɲmax is a limit placed on the importance of nesting in the weight. Finally, the following selection function may be defined: F(cmax, cmin, x) = Cnormexp[–(x/cmax)2](1 – exp[–(x/cmin)2]) (Eq.7) where is a normalization constant , and cmin, cmax are
Figure imgf000033_0002
Figure imgf000033_0003
constants that set the minimum and maximum scales of interest for any given quantity. For example, in Eq.5 the selection function is applied to spatial separation, and the constants (rmin, rmax) are used to select a relevant range of separations. The function Ƒ may be constructed to reach a maximum value of unity between this range of scales, and then to fall off exponentially for separations outside this specified range. Likewise, when applied to Eq. 6, the selection function may maximize within the specified range of areas (amin, amax) and fall off rapidly outside this range. Thus, dQi may have desired properties. In the absence of neighbors (i.e., within a few rmax), the value of dQi may remain small. However, the value of dQi may increase quadratically with the number of neighbors, if they are within the right range of separations, and have the right range of sizes to be of concern. Furthermore, the value may increase with the central intensity of the potential calcifications, and with the number of nested contours within each, both of which may correlate with visibility to the human eye. [0170] As shown in step 330, the method 300 may include selecting calcifications from the nested contours, where, if the calcifications are not selected the method 300 discards the calcifications as shown by step 332, and where, if the calcifications are selected, the method 300 proceeds to step 334. [0171] At this point in the method 300, all contours may have been found and characterized, contours may have been eliminated that occur inside objects, the most interesting contours may have been selected, and the contours may have been grouped into nested structures, with the outermost contour representing the shape and the inner contours providing additional information on the internal gradients. The final selection for calcifications may be made based on the following two criteria, which are provided by way of example and not of limitation: [0172] 1. Strong Calcifications → δA/δI < 0.15 [0173] This threshold on the contour derivate (see Eq.4) may capture most of the clear calcifications with sharp boundaries. This selection may be made regardless of whether the calcification has any close neighbors. [0174] 2. Weaker Grouped Calcifications → δQi < 3 [0175] This threshold on the grouping parameter (see Eq.5) may select weaker calcifications that are grouped together appropriately (as discussed above). Note that the threshold value of δQi may be dependent on the scaling parameters chosen in Eqs.5–7. Ultimately, it may be desirable for these choices to be consistent with the scaling parameters chosen below, which are used to compute a ‘Q score’ for each cluster. [0176] As shown in step 334, the method 300 may include grouping calcifications into clusters. After identifying all calcifications within the image, next, they may be grouped into clusters according to the following procedure. A spatial cluster scale (e.g., Rc = 7 mm) may be defined, and for each calcification the number of neighbors within this range is counted. In addition, a minimum number of calcifications to consider as a cluster (e.g., Nmin = 3) may be specified. Next, the method 300 may start with the calcification with the largest number of neighbors, which is used to form the first cluster. New calcifications may be recursively added to this cluster, until there are no remaining calcifications within a distance Rc of any member. After finding all members of first cluster, the method 300 may proceed to the next unassigned calcification and repeat this process until all calcifications that should be grouped into a cluster have been assigned. In an aspect, only calcifications with at least two neighbors (i.e., three members) are grouped into clusters. Calcifications that are not assigned to a cluster may be ignored completely for the rest of the method 300. [0177] This approach for forming clusters may be advantageous, and may depend only on the scale Rc. In most images, the actual clusters in the tissue (e.g., breast) are well separated, and this approach works well. However, in images with many vascular clusters and/or other types of benign calcifications, it may become difficult to separate out new (potentially cancerous) clusters from the pre-existing background of benign clusters. Indeed, the cancer may appear next to a vascular cluster. In this case, the clustering approach may group the new cancer together with the vascular, which may result in misclassification, where prevention/accounting for this is discussed below. [0178] As shown in step 336, the method 300 may include computing cluster properties. To aid in the classification process, it may be useful to characterize the distribution of calcifications within the cluster. If the centroid is identified for a calcification by the ordered pair (x1, x2), then the cluster centroid can be defined as:
Figure imgf000035_0001
, (Eq.8) where wn is the weight for the nth calcification and Nc is the number of calcifications within the cluster. In an aspect, the outward contrast of each calcification for the weights wn = Cout is used. Next, a displacement matrix for each cluster may be defined: , (Eq.9)
Figure imgf000035_0002
where again the contrasts for the weights may be employed. This symmetric positive-definite matrix may have two real eigenvalues (e1, e2) and two eigenvectors (d1,d2), which can be used to define the following quantities, which are provided by way of example and not of limitation: 1. Cluster Half-Length → , where e1 is the maximum eigenvalue of Dij
Figure imgf000036_0001
2. Cluster Half-Width → , where e2 is the maximum eigenvalue of Ɗij
Figure imgf000036_0002
3. Aspect Ratio → A = w/L 4. Principal Axis → d1 vector aligned with long direction in the cluster 5. Packing Fraction → – (area inside calcifications)/(approximate cluster
Figure imgf000036_0003
area) For each cluster, the method 300 may also compute the mean and standard deviation of the geometric and contrast properties described above, including, e.g., intensity, contrast, area, etc. [0179] As shown in step 338, the method 300 may include classifying clusters as benign, in which the method 300 proceeds to step 340, or classifying clusters as possible cancer or cancerous in which the method proceeds to step 342. [0180] Calcifications may form within tissue over a wide range of scales and for a variety of reasons. Calcifications may be of benign origin, or clusters of micro-calcifications may be indicative of cancer. Typically, benign calcifications are more common. Thus, when used as a screening tool, the large majority of clusters identified by the method 300 are expected to be of benign origin. The strategy of the method 300 may thus be to identify and exclude the most common types of benign clusters, and then to score the remaining clusters with the ‘Q factor’ as described below. [0181] If the clusters are classified as benign, the method 300 may classify a type for each the cluster/calcification, which is illustrated by step 340. Some types of benign clusters are provided below by way of example and not of limitation. [0182] Type-1: Vascular [0183] A common type of benign cluster is associated with vascular calcifications. While these are of potential interest in studies of cardiovascular disease, these clusters may not be relevant to cancer (e.g., breast cancer). However, if vascular calcifications are present within a given image, the method 300 may identify a large number of calcifications organized along the vessel wall. Unfortunately, the range of spatial scales and separation distances for these vascular calcifications often overlaps with the micro-calcifications relevant to cancer (e.g., breast cancer). Thus, one often cannot differentiate based on the Q factor discussed below. Instead, other approaches to exclude these from consideration may be utilized. [0184] Vascular calcifications are usually easy to spot visually, since they are well- organized along the wall of the tubular vessel. As such, at least two strategies may be used to automatically identify these vascular calcifications, i.e., using an algorithm or the like. First, the high-degree of spatial correlation can be measured, e.g., by performing a regression analysis on the positions of the calcifications. An alternative and potentially complementary approach is to employ edge detection techniques to identify the vessel walls, and then to exclude calcifications that are located along these structures. [0185] In an aspect, the approach is based on performing a regression analysis to a polynomial of specified order. The steps in this vascular detection subroutine may include without limitation: [0186] 1. Only accepting clusters having between a certain number of members (e.g., between 3 and 500 members). Depending on the number of members, there may be a look-up table to specify (1) the order of the polynomial, (2) the threshold tolerance in the fit, and (3) the number of points that can be dropped. This may allow a higher-order fit and/or slightly larger tolerances for clusters with more members. In an aspect, only first order (linear) and second order polynomials are used, and the tolerance allowed varies from a range of values, e.g., 0.01 to 0.036. These tolerances may correspond to a normalized chi-squared of the fit (i.e., normalized to the length of the polynomial curve). [0187] 2. Next, the cluster may be rotated into a frame where the x-axis is aligned with the principal axis of the cluster computed above. A polynomial least-squares regression may be performed, and the chi-squared fit parameter can be computed and normalized by the length of the curve. If the fit is within tolerance, the cluster may be identified as vascular (type-1), otherwise the specified number of outlier points may be dropped, and the fit may be recomputed to see if the method 300 can find one within the tolerance specification. [0188] 3. For large vascular clusters, it may be difficult to fit with a second-order polynomial, especially if the cluster has multiple tree-like branches, or many outlier contours that are not well-aligned along the tubular structure. Thus, for large clusters the method 300 may attempt to split them into smaller subgroups, and then apply the polynomial fitting procedure to the subgroups. In an aspect, the method 300 has two different strategies for splitting and fitting, and the algorithm is set to employ one or both of these (i.e., apply the second if the first fails). If the routine finds any portion of the cluster that is well fit by ‘q polynomial,’ then the entire cluster may be classified as vascular. [0189] 4. Even with the above variations, it may be difficult to pick a tolerance for the fitting threshold that identifies all of the vascular clusters, while excluding ones that are potentially malignant. Thus, the method 300 may include a final check that applies to clusters that have a fitting tolerance somewhat above the threshold (and thus would not be identified as type-1), but where the principal axis aligns with a clear vascular cluster. Often a series of vascular clusters will form along the same vessel, or along a neighboring vessel. As a result, the principal axes of these two clusters may be well-aligned, they may be in the same proximity, and they often have a high aspect ratio (see cluster properties described above in step 336). Thus, by introducing a final check on these other factors, the method 300 may be able to identify and exclude additional vascular clusters. [0190] Type-2: Large Calcifications and Fatty Necrosis [0191] Another common type of benign clusters is associated with larger calcifications and fatty necrosis. These clusters may include larger members, with areas that may be significantly larger than micro-calcifications associated with malignancy. For clusters comprised entirely of larger calcifications, the ‘Q score’ described below may be relatively small. However, in other cases, there may be an overlap in the relevant range of areas with malignant clusters. Furthermore, the method 300 may find a number of smaller structures in the vicinity of the larger calcifications, which can then give rise to false-positives as described below. [0192] In terms of geometric properties, these benign clusters may be characterized by relatively larger areas and by their fairly dense grouping. To this end, a cluster library may be created in which the geometric and contrast properties described above are extracted for interesting clusters for use and evaluation by the method 300. The cluster library may show that malignant clusters tend to be more dispersed (lower Pf ) with a smaller range of areas, while the benign clusters are more densely packed (larger Pf ) and/or larger areas. An approximate threshold curve to identify type-2 clusters may be provided as a line such as: Pf = 0.85 – 0.6<Ai>Amax , (Eq.10) where <Ai> is the average area of the calcifications within the cluster, Amax is the area of the largest calcification in the cluster, and Pf is the packing fraction (see description above). In an aspect, this criterion is used to identify and exclude these benign clusters. While this may lead to some misclassifications, it may not greatly impact scoring metrics, since the malignant clusters are often correctly identified in other images/views, and for the rare times they are misclassified, this only occurs in one of the images/views. In another aspect, active contouring techniques are used to remedy misclassifications. [0193] Type-3: Diffuse Round Calcifications [0194] Another type of false-positive may be clusters that are characterized by diffuse, nearly circular calcifications. These are often fairly bright and have relatively good contrast, and thus many calcifications are often identified. The range of calcification sizes may be relatively similar to malignant micro-calcifications, but they tend to be spread over broader areas of the tissue (e.g., breast tissue), and also they may often appear on both sides in a similar manner. A technique for identifying these clusters uses the Cratio and Pf. Another technique may compare different sides of an image, e.g., comparing the left and right sides. [0195] As shown in step 342, the method 300 may include quantifying clusters with a ‘Q score.’ The Q score as discussed herein may refer to a measurement, e.g., a number that quantifies the likelihood of malignancy for each cluster. [0196] The Q score may include an analytic function of the geometric and contrast properties of the calcifications within each cluster, as well as their detailed spatial arrangements. The Q score may quantify aspects of the calcifications more quickly, accurately, and consistently than is possible for a human. In comparison to the black box approach of a neural net, the Q score enables a clear explanation in physical terms about how the method 300 is scoring any given cluster. [0197] Some features built into the functional form for the Q score have already been discussed above in association with the grouping parameter dQi. It is well-established that clusters of micro-calcifications associated with cancer occur for a fairly limited range of spatial scales and separation distances. As the cancer develops from an early phase, the number of visible micro-calcifications will increase, along with the intensity and contrast of each visible calcification. To this end, the function may increase monotonically with these features, and allow sufficient flexibility to adjust free scaling parameters in order to optimize the overall performance. An example of this function is: , (Eq.11)
Figure imgf000039_0001
,
Figure imgf000039_0002
where Mo is a free parameter, M is the number of calcifications in the clusters, and all other symbols have been defined herein. The Q parameter may be roughly analogous to a potential energy or the like for the cluster (assuming a particular form for the pair-wise interactions), where the free scaling parameters have been adjusted to maximize the energy for malignant clusters. Aside from a normalization factor, the Q parameter defined in Eq.11 may be a sum over the clustering parameter for each calcification (see Eq.5). From this point of view, the approach for selecting calcifications may be connected with the overall strategy for scoring the significance of the final clusters. [0198] To determine the free scaling parameters, a program performing a multi- dimensional optimization may be used in order to find values that maximize the area under the curve (AUC) for the receiver operating characteristic (ROC) curve. Example values for these parameters are as follows: amin = 0.054 mm2, amax = 0.42 mm2, rmin = 0.0 mm, rmax = 2.69 mm, Ɲmax = 5, Mo = 10. [0199] As shown in step 344, the method 300 may include saving results. For example, after completing the analysis for each image, the following results may be saved, which are provided by way of example and not of limitation: 1. List of clusters identified in the image, including the Q score and the cluster properties; 2. Outer contours for each of the calcifications within each cluster, along with the geometric and contrast properties for each of these shapes; and 3. Information to generate the ROC curves. This information may be extracted and saved along with the Q score for each cluster. [0200] Then, the method 300 may terminate in step 346. [0201] It will be understood that any values recited above with respect to the method 300 (or otherwise herein) are provided by way of example only, and are not meant to limit the embodiments described herein. These values may also or instead include predetermined (e.g., “best practice”) values, e.g., discovered through a trial-and-error process. These values may be varied by a user or administrator, e.g., using a graphical user interface that includes fields for inputting the values. [0202] Figure 4 depicts a medical image of calcifications in a patient’s tissue. The image 400 in the figure may represent an original medical image of a patient’s tissue, e.g., a mammogram or the like. The image 400 may include calcifications 402, which may need to be identified for detecting whether cancer is present. In other words, the image 400 may be used in the devices, systems, and methods described herein for detecting and quantifying cancer in a patient’s tissue shown therein. [0203] Figure 5 illustrates an example of a selection process for a malignant cluster of micro-calcifications. Specifically, the figure includes a first image 510 and a second image 520. The first image 510 may represent the contours 512 selected by any of the criteria outlined above. More specifically, these may be the contours 512 before they are grouped into nested hierarchies. The calcifications most noticeable to the human eye may have many nested contours 512, which are grouped together into nested structures as described above (these nested structures are shown, for example by the bubbled area 514 in the figure). These internal nested contours 512 may be used to evaluate gradients in the intensity across the structure (see δA and δI parameters defined in the equation above), and the parameter dQi may be used to characterize the groupings (as explained above). The final calcifications 522 shown in the second image 520 may be selected according to the criteria described above with respect to step 330 of the method 300. Specifically, the second image 520 shows the outer boundaries of these final selected calcifications 522. [0204] Figure 6 illustrates an example of the identification of a cluster of calcifications. For example, the figure shows the identification of type-2 clusters with fatty necrosis using techniques described herein. The figure includes a first image 610 and a second image 620. The first image 610 shows an original medical image of tissue without any overlaid contours. The second image 612 shows contours 622 identified by the techniques described herein, e.g., the method described with reference to Figure 3, which may utilize one or more algorithms. [0205] Figure 7 is a graph showing an example of a packing fraction as a function of <Ai>Amax for exemplary clusters. Specifically, in the graph 700, the x-axis is the <Ai>Amax, where <Ai> is the average area of the calcifications within the cluster and Amax is the area of the largest calcification in the cluster. In the graph 700, the y-axis is the Pf, i.e., the packing fraction (see discussion above). [0206] In the graph 700, the lighter lineweight points 702 (x’s) correspond to malignant clusters, while the heavier lineweight points 704 (o’s) correspond to various type-2 clusters. In this specific graph 700, only clusters with less than 30 calcifications are included for clarity and by way of example, but clusters with more calcifications may also or instead be used. An approximate boundary separating the benign from malignant clusters is given by the line 706. In general, the figure may represent the derivation of criteria for identifying type-2 clusters that are used in the techniques described herein, where each point in the graph 700 corresponds to a single cluster in an exemplary study, which was used to refine the techniques described herein. The results show that malignant clusters tend to be more dispersed (lower Pf) with a smaller range of areas, while the benign clusters are more densely packed (larger Pf) and/or larger areas. An approximate threshold curve to identify type-2 clusters may be provided by the line 706, which may take the form: Pf = 0.85 – 0.6<Ai>Amax [0207] In an implementation, this is the criteria used to identify and exclude benign clusters. As shown in the graph 700, the malignant points to the right of the line 706 may be misclassified as benign type-2. While this may be undesirable, it also may not impact the scoring metrics described herein, as discussed above. As further shown, it appears that the outer boundaries corresponding to the calcifications may be too large and/or encompasses multiple smaller calcifications. This may cause the measured areas to be larger than they should be, and thus may move the cluster into the benign region of the parameters space in the figure. This can be improved upon by utilizing advancements in the contouring techniques, e.g., active contouring techniques. [0208] As discussed herein, an implementation may include a method for determining a cancer score. Determining a cancer score may be accomplished through the user of a cancer score engine (and its components) or the like as described herein. In the method, an event of interest may be defined. An event of interest may be any object of interest to be identified from data. Examples of events may be cancerous lesions, masses, physiological anomalies, and the like. The method may gather variables for the events of interest, such as x1, x2...xn. The variables may be a minimum number of variables that allow the event to be predicted and a score to be generated, e.g., a cancer score. The variables may be gathered by identifying a number of variables and then discarding variables that are not predictive and/or show the wrong behavior for the event of interest. In one implementation, the event of interest may be breast cancer and the variables may include closed intensity contours of calcifications in mammogram images, gradients of the calcifications, one or more characteristics about each calcification, such as perimeter, contrast and/or a number of neighbors, a texture and shape of each calcification and/or a hierarchical structure of the calcifications in a cluster, such as how tightly the calcifications are nested, if there are nested levels of calcifications and the like. The variables for mammogram images may also or instead include other variables. This method may also include a clustering of individual calcifications with their neighbors and then grouping into prototype clusters which may be ordered based on a number of neighbors. In the breast cancer example, the values of these variables may be determined by the computer analysis of the mammogram images. [0209] The method may calculate a Q0 based on the values of the variables, where Q0 is an analytical function of the variables so that Q0 = F(x1, x2, .., xn). Thus, in the breast cancer implementation, the method may calculate Q0 for each cluster of calcifications in which Q0 is a function of the variables calculated over each cluster. For example, Q0 may include use of any of the functions described herein. [0210] The method may calculate a Q1, where Q1 is equal to (Q0) x (a penalty function). The penalty function may selected be such that Q1 incorporates a classification scheme. In the breast cancer implementation, the penalty function may discard calcifications that are spaced too far apart and thus are unlikely to be suspicious cells. The method may then normalize Q1 to generate a cancer score. During the normalization, the parameters of Q0 and Q1 may be optimized to maximize the area under a well-known ROC or free receiver operating characteristic (FROC) curve. By way of example, the FROC curves are described in Bornefalk et al., “On the Comparison of FROC curves in Mammography CAD Systems”, Med. Phys.32, pp.412–17 (2005), which is hereby incorporated by reference in its entirety. Thus, based on the above curves, the cancer score may have thresholds and classify the clusters of calcifications into Type-1, Type-2 and Type-3, where Type-1 identifies a linear or curvilinear cluster (benign lesion), Type-2 identifies a cluster that has one or more calcification members that are exceptionally large and/or bright in the mammogram image, and Type-3 identifies a cluster that is likely malignant. The method may display the cancer score in some form. For example, as shown in figures included herein of medical images, a cluster of calcifications may be classified as cancerous thereby warranting a biopsy. [0211] Random-Forest Model [0212] As women age, coronary artery micro-calcifications become more prevalent (Laurie Margolies et al. Digital Mammography and Screening for Coronary Artery Disease, JACC Cardiovasc Imaging, 2016.9(4): p.350-60). In addition, life-style habits, genetics, surgical history, and medications are factors in prevalence of coronary artery calcifications (Robyn L. McClelland, Hyoju Chung, Robert Detrano, Wendy Post, Richard A. Kronmal. Distribution of Coronary Artery Calcium By Race, Gender, and Age: Results from the Multi- Ethnic Study of Atherosclerosis (MESA), Circulation, 113(1), 30-37). What has not been recognized, however, is the association of age, life-style habits, genetics, surgical history, and medications, among other physical factors, in the detection, quantitation, and/or prediction of breast cancer via analysis of breast calcifications. The present invention includes patients' age in the predictive model described above. [0213] Specifically, in addition to the physical features of each calcification described in previous steps, as an individual structure and as part of a cluster of micro- calcifications, patients' age is supplied as an additional feature, to be used to construct a random-forest model. This random-forest model, taking as input all of the physical features, including the Q-score provided by the Q-algorithm described above, and a patient's age, provides an overall final (random-forest) score for each region of interest identified by the Q- algorithm. The regions of interests are ranked on the scale of suspiciousness based on this random-forest score. This additional age feature led to a significant increase in the AUC of the ROC, from 0.927 to 0.959. Performance was improved at all operating points. It is anticipated that including more clinical data as additional random-forest features, which includes but is not limited to the following sets: (1) Family history and genetics (e.g., https://www.cancer.org/cancer/breast-cancer/risk-and-prevention/breast-cancer-risk-factors- you-cannot-change.html), and (2) Lifestyle habits (e.g., https://www.cancer.org/cancer/breast-cancer/risk-and-prevention/lifestyle-related-breast- cancer-risk-factors.html). [0214] Thus, the physical features of each calcification, as an individual structure and as part of a cluster of micro-calcifications, together with clinical data, are used to construct a random-forest model. This random-forest model, taking as input all of the physical features, including the Q-score provided by the original Q-algorithm, provides an overall final (random-forest) score for each region of interest identified by the Q-algorithm. The regions of interests are ranked on the scale of suspiciousness based on this random-forest score. [0215] Detecting, Scoring, and Predicting Cardiovascular Disease Risk Using Medical Imaging [0216] Any calcification in the artery, regardless of where the artery is, indicates presence/onset of artery disease. Any information about artery calcification in any location like in the breast is therefore useful information. [0217] Furthermore, if you have plaque at a certain location such as in the breast (see, e.g., Figures 9A-9C), the chance that a plaque will form, or is present, at another location like the heart is higher. Calcification is the last stage of plaque development. Breast arterial calcification (BAC) may be an indicator of both calcified and noncalcified plaque in the coronary and/or carotid artery. [0218] The presence and quantification of BAC is therefore likely to be very valuable for evaluating a person's risk for ASCVD (which includes carotid artery disease, CHD, and peripheral arterial disease) and Chronic Kidney Disease (CKD). [0219] Here, we have developed a system for automated detection and scoring of BAC. A risk calculator can be developed to relate the BAC score to the risk for an event. The BAC score may also be incorporated into the patient’s medical record. This can then be monitored and when the BAC score is in the range that signals an elevated risk for ASCVD and CKD the radiologist can alert the patient’s physician. [0220] This would be especially useful for asymptomatic individuals that have a high risk without knowing it and as a consequence do not get treated. When their BAC score is relatively high or increasing rapidly, further diagnostics and possibly treatment can be initiated, expectedly preventing ASCVD and/or CKD. As such we believe that the BAC score (especially when documented over time) has the potential to aid substantially in the prevention/early detection of ASCVD and CKD. [0221] Mammography on the other hand is already widely used as a screening tool for breast cancer, so to measure the BAC score no additional imaging or radiation is required. Therefore, these disadvantages mentioned for measuring the CAC score would not apply to measuring the BAC score and using this to evaluate the risk for ASCVD. One of the important benefits of a score evaluated on screening imaging instead of diagnostic imaging, is that the BAC score over time can be measured and documented. Another important benefit is that the BAC score is measured not only for symptomatic women as with a diagnostic tool, but also for asymptomatic women. Asymptomatic women that are at risk without knowing it, are currently not treated because without symptoms there is no incentive to get their CAC score measured. So they do not know they are at risk and will not find out until they experience symptoms/an event. However, these women will have a nonzero BAC score or will most likely develop a nonzero BAC score over time. So their yearly BAC score measurement will show for at least part of these women that they are at risk and further action can be taken, expectedly preventing symptoms/events for these women. [0222] The BAC score could be incorporated into the patient’s medical record. This could then be monitored over time and when the BAC score is in the range that signals an elevated risk for ASCVD and CKD the radiologist can alert the patient’s physician. The physician of the patient can then use the (development over time of) the BAC score in addition to other clinical data to decide if further diagnostics and possibly treatment is necessary. As such we believe that the BAC score (especially when documented over time) has the potential to aid substantially in the prevention/early detection of ASCVD and CKD. [0223] Because CHD is the most common ASCVD and CAC score is an established diagnostic tool for CHD, the first step was to look at the correlation between BAC and CAC. If such a correlation exists, then one can use BAC as a proxy for CAC and immediately open up the use of BAC as a preventive and/or diagnostic tool for CHD. This has led to a number of studies to assess whether there is a strong enough correlation between BAC and CAC. However, the most extensive study to date has not found the correlation to be sufficient for this. Using a larger data set we have found similar result. This is not unexpected for the following reasons. [0224] Artery calcification in any artery indicates presence of artery disease and if you have a plaque at a certain location, the chance that a plaque will form at another location is higher. With increasing age and development of atherosclerosis, the chances of having plaque at multiple locations increases. As artery calcification does not necessarily start in the heart, this denotes that with increasing age the chance that if you have BAC you will have CAC increases and the other way around. Furthermore, calcification is the last stage of plaque development, so even with a zero CAC score a person can have a plaque in the coronary artery. [0225] For these reasons it makes sense that if there is BAC there is not necessarily a nonzero CAC score and vice versa, which explains why there is not a high correlation between BAC and CAC. However, it also implies that when a person has BAC, the chance that this person will develop a nonzero CAC score is increased and vice versa. This indicates that having BAC could predict development of CAC, and having CAC can lead to development of BAC. So using this reasoning, the fact that a person has BAC could either indicate currently present CAC in a person or predict CAC in the near future. Knowing CAC is an established predictor of CHD, this would indicate that even without showing a clear correlation with CAC, BAC could play an important role in the prediction and prevention of CHD and more generally for ASCVD. [0226] Based on this reasoning, we took a different approach. We reasoned that observation of BAC should impact the risk factor for CHD, kidney disease, stroke, among others. [0227] To test our hypothesis, we considered the efficacy of BAC vs CAC as a predictor of CHD. If both show similar efficacy, this would be a strong indicator that BAC has comparable power to CAC in relation to CHD. This opens the possibility of creating a risk calculator based on BAC. [0228] This led us to the development of the proposed system which consists of the following modules: a) module for automated detection of BAC at ROI level, b) module for automated segmentation of BAC within the ROI, c) module for automated scoring of BAC, and d) a module for reporting and storing the BAC score in the PAC system. Since women over 40 are recommended to have annual mammograms, our system would retain the BAC score over time. The time development of BAC score may increase the efficacy of BAC as a predictor of ASCVD and CKD. [0229] There are numerous other examples of abnormalities within anatomical features which can be detected and categorized using the systems and methods of the present invention. It will be appreciated by those of skill in the art that the suspicion codes may also be forwarded to other healthcare functional units, such as the Full-Field Digital Mammography (“FFDM”) system, Radiology Information System (“RIS”), Clinical Information System (“CIS”), or any Hospital Information System (“HIS”) which can receive a suspicion code which may indicate the need for treatment by a physician or other healthcare worker. In addition, the suspicion codes may be forwarded in real-time such as through a text message, email, web app, phone app, or other methods. In yet another aspect, in a three- dimensional (3D) environment, a suspicion code may highlight a particular slice within a multi-slice environment, or a particular frame in a multi-frame video, for example. [0230] A notification result file may be generated to include the suspicion code. In non-limiting examples, such notification result file can be in the form of one of the file formats or communication formats discussed previously. [0231] Immediately upon, or after, the notification result file is associated with the medical exam, the worklist may be updated to display for further review. An example of an updated worklist is depicted in Figure 8. The worklist can highlight images or studies within a listing of all pending reviews, can sort the listing of all pending reviews in order of suspicious and not suspicious, or can list only the suspicious images or studies for priority review. An individual image or study within a workflow may also be highlighted with an indication that the image or study comprises a suspicion code, and an image or study may also include markings such as ROI, scoring associated with an ROI, and the like. [0232] Neural Network Architecture [0233] Data generated by industrial systems can be analyzed for the presence of one or more anomalies by comparing a normal state to a changed state. However, better precision and accuracy of such analysis has been the subject of ongoing research and development, and there continues to be a need for even more precision and accuracy in anomaly detection. [0234] In the healthcare field, data is often analyzed with respect to the sensitivity and specificity of the analysis. The terms “sensitivity” is the ability of a test to correctly identify those with a physiological anomaly (true positive rate), whereas the term “specificity” is the ability of the test to correctly identify those without the physiological anomaly (true negative rate). There continues to be a great need for systems and methods of data analysis which result in both higher rates of true positive detection and lower false positive generation. Such systems and methods could result in better outcomes for patients suffering from physiological anomalies which ideally could be treated earlier than current state of the art technologies allow. [0235] Those of skill in the art will recognize that there are a multitude of sensors which can generate data, and that the data can be analyzed to detect an anomaly. Examples of such sensors include acceleration sensors, acoustic and sound sensors, automotive sensors, capacitance sensors, chemical sensors, digital component sensors, electric current sensors, magnetic sensors, flow sensors, fluid property sensors, force sensors, humidity sensors, ionizing radiation sensors, mass air flow sensors, photo optic sensors, piezo film sensors, position sensors, pressure sensors, rate and inertial sensors, speed sensors, temperature sensors, torque sensors, traffic sensors, ultrasonic sensors, vibration sensors, and water-level sensors. In an alternative, the data may be provided by manual entry, such as by a physician or other professional. [0236] The data which can be analyzed for the presence of an anomaly can be static or real-time. For instance, under supervised anomaly detection, two static images can be categorized as “normal” and “anomaly” respectively, and that categorization can be used to train an algorithm to detect anomalies in other images. In another instance, a continuous series of data can be analyzed unsupervised to detect rare occurrences or bursts of unexpected activity. [0237] In general, a system designed to detect anomalies can alert technical staff each time an anomaly is detected so that they can more rapidly review the condition. Therefore, anomaly detection is particularly suitable for companies in the manufacturing, oil and gas, transportation and logistics, aviation, automotive, and energy and utilities, and healthcare fields. In those industries, a detected anomaly can be found in a computer network environment, manufacturing plant, assembly line, industrial control system, maintenance cycle review, energy production, hospital system work flow, and the like. [0238] In the healthcare context, data which can be analyzed for the presence of an anomaly can be generated by a sensor using one of or a combination of different modalities including X-ray imaging, CT scan or imaging, MRI, PET, SPECT, US, endoscopy, thermography, medical photography, nuclear medicine functional imaging, elastography, photoacoustic imaging, echocardiography, functional near-infrared imaging, magnetic particle imaging, and the like. The resultant data may be captured in the form of a medical image which can be used for the detection of lesions and other physiological anomalies in various parts of the body, and include both two-dimensional (“2D”, e.g., X-ray) and three- dimensional (“3D”, e.g., 3D tomography) imaging. [0239] A preferred medical imaging format is DICOM® (Digital Imaging and Communications in Medicine) which is the international standard to transmit, store, retrieve, print, process, and display medical imaging information. Those of skill in the art will also recognize various other image formats which can be used for anomaly detection, including JPEG, PNG, TIFF, GIF, and the like. Medical images can also include structured reporting (SR) which is used for the transmission and storage of clinical documents which can accompany the medical image. [0240] Examples of lesions and other physiological anomalies which can be imaged using any one, or combination, of the modalities above include Abdominal Aortic Aneurysm, Abnormal Vaginal Bleeding, Alzheimer's Disease, Anal Cancer, Angina Pectoris, Appendicitis, Arterial Calcifications, Arthritis, Benign Prostatic Hyperplasia, Blood Clots, Bone Fracture, Brain Tumors, Breast Cancer, Breast Lumps, Carotid Artery Stenosis and Restenosis, Cervical Cancer, Cholecystitis, Chronic Obstructive Pulmonary Disease, Cirrhosis of the Liver, Colorectal Cancer, Crohn’s Disease, Croup, Cystic Fibrosis, Dementia, Dense Breasts, Diffuse Interstitial Lung Disease, Diverticulitis, Endometrial Cancer, Epilepsy, Esophageal Cancer, Fatty Liver Disease and Liver Fibrosis, Embryo and Fetal Abnormalities in the Womb, Gallstones, Head and Neck Cancer, Head Injury, Hematuria or Blood in Urine, Kidney and Bladder Stones, Kidney Failure, Lung Cancer, Lymphoma, Osteoporosis, Ovarian Cancer, Peripheral Artery Disease, Pneumonia, Prostate Cancer, Renal Cysts, Stroke, and Venous Insufficiency (Varicose Veins), among other physiological anomalies. Any lesion characterizing a disease that is structurally and/or compositionally distinct from surrounding healthy tissue can be highlighted by an appropriate imaging modality. [0241] There are also variations in certain medical image properties (e.g., size, contrast, presentation of normal tissue and anomalies, etc.) across different medical imaging applications. For example, 2D mammographic images are typically about 3300 x 4000 pixels, whereas histopathology image size can be 60,000 x 60,000 pixels (typical 15mm x 15mm tissue slice, scanned at 40X magnification, or 0.25 microns/pixel). [0242] Anomalies in medical images are also depicted with different brightness depending on the imaging modality. For example, bleeding or hemorrhage is typically shown bright in MR, X ray, and CT images. A myocardial infarction region appears dark in X ray images, CT images, and in T1 MR, but bright in T2 MR. Likewise, tumors are dark in X ray and CT and T1 MR images, unless calcified, but bright in T2 MR. In addition, multiple sclerosis plaques can appear dark in CT images and T1 MR, but appear bright in T2 MR. [0243] Those of skill in the art have taken a variety of different approaches to detecting anomalies in data generated by industrial systems. In the healthcare context, significant research has recently been conducted on detecting lesions and other physiological anomalies that may be present in an animal. Considering the wide variety of data generated, and the various ways of interpreting the data, anomaly detection has historically taken several algorithmic approaches including the classification approach, the segmentation technique, and content-based image retrieval. [0244] With the advent of neural networks, however, it was discovered that algorithms can be optimized for training with a very small number of parameters and do not need to make a priori assumptions on the properties of the underlying data. In the recent past, a variety of neural networks (both supervised and unsupervised) have been useful in detecting anomalies in medical images including Radial Basis Functions (RBF), Learning Vector Quantization (LVQ), Probabilistic Neural Networks (PNN), Hopfield networks, Support Vector Machines (SVM), Synchronized Oscillator Network, and Adaptive Resonance Theory (ART). More recently, Convolutional Neural Networks (CNNs) such as those described herein have become preferred models for object detection (including anomalies) in images. [0245] Unlike the previous attempts to increase accuracy, precision, specificity, and sensitivity (including lower false positive generation) in anomaly detection, and to address variations in medical imaging, a new architecture and algorithm pipeline has been developed that is adaptable and can be tuned for deployment across anomaly detection, including the entire medical imaging domain. Some of the unique aspects of this solution are provided in Table 1, which provides an Outline of Neural Network Architecture.
Figure imgf000050_0001
Table 1. [0246] Data augmentation [0247] Two types of data augmentation techniques have been developed. First, real time data augmentation is used in the training modules. This includes rotational, flipping, filters that affect the appearance of the image such as contrast variations, opacity variation, and filters that change the shape of the image such as distortion, warping, and blurring. Secondly, A new technique has been developed, as provided in US App. Ser. No.62/548,894, filed on August 22, 2017, that enables the creation of a large number of "synthetic" examples for training. As will be recognized by those of skill in the art, neural networks aren’t “smart” to begin with – they require a relatively large number of data samples to be adequately trained. This new technique (a) enables targeted data augmentation which addresses the sparsity of examples in the data set, and (b) can be used to quickly calibrate the network for new types of images that the algorithm has not been trained on. [0248] Enhanced Objection Detection [0249] A new neural network has been developed which builds on state-of-the-art object detection strategies and includes the following combination of features for optimal performance on anomaly detection in industrial systems, including in medical imaging: [0250] • Integrated Pre-Processing [0251] In addition to pre-filtering of images such as applying Gaussian filters and/or histogram-based adjustments, the new neural network includes a built-in filter in which the parameters of the filters are learned during training simultaneously with the parameters of an object detector and a classifier. [0252] • Self-learning [0253] The new neural network is designed so that as new images are entered into the system for analysis, it can optionally re-train and update the models automatically. [0254] • Integrated region proposal and classification architecture [0255] The new neural network combines pre-processing, region proposal, and classification into one learning system. This feature enables the neural network to learn an optimized strategy for proposing ROIs and classifying them (see, e.g., Figure 28). The power of deep learning is that it learns features/strategies through examples as opposed to previous techniques which were based on hand-engineered features/strategies. This approach outperforms, both in terms of accuracy and speed, certain prior algorithms where one is required to hand design a region proposal method and then train a classifier on the proposed regions. [0256] • Multi-scale anomaly detection [0257] The new neural network is configured to address the problem that anomalies can occur on a variety of scales, and to increase the neural network efficacy in early detection of such anomalies, including lesions and other physiological anomalies, the new neural network calculates and combines feature maps at multiple spatial scales. [0258] • Global context [0259] The new neural network includes an ROI classification system which takes into account local features (extracted from the ROI) as well as global context (features from the entire image). Unlike prior systems which uses down-sampled images, collects features from the downscaled images, and along with features from ROIs are fed into random forest, the new neural network uses high resolution images, and each region uses global features, and the training is performed within the same network (i.e., not a random forest process). [0260] • Adaptive training [0261] The new neural network is configured to focus attention on the most difficult cases because detection datasets contain an overwhelming number of easy examples and a small number of hard examples. Automatic selection of the hard examples can make training more effective and efficient. [0262] • Modular design [0263] The new neural network is configured to be integrated into different backbone architectures of various alternative neural networks including, but not limited to, Perception Neural Network, Feed Forward Neural Network, Artificial Neuron, Deep Feed Forward Neural Network, Radial Basis Function Neural Network, Recurrent Neural Network, Long/Short Term Memory, Gated Recurrent Unit, Auto Encoder Neural Network, Variational AE Neural Network, Denoising AE Neural Network, Sparse AE Neural Network, Marakov Chain Neural Network, Modular Neural Network, Hopfield Network, Boltzmann Machine, Restricted BM Neural Network, Deep Belief Network, Deep Convolutional Network, Deconvolutional Network, Deep Convolutional Inverse Graphics Network, Generative Adversarial Network, Liquid State Machine Neural Network, Extreme Learning Machine Neural Network, Echo State Network, Deep Residual Network, Kohonen Self Organizing Neural Network, Support Vector Machine Neural Network, Neural Turing Machine Neural Network, Convolutional Neural Networks such as LeNet, AlexNet, ZF Net, GoogLeNet, VGG Net, Microsoft ResNet, and Region Based CNNs including, but not limited to, Fast R- CNN, Faster R-CNN, R-FCN, Multibox, SSD, and YOLO. This also enables ensemble modeling, a proven strategy in machine learning, where results of different networks are combined for optimal performance (e.g., anomalies missed by one network may be detected by another network). [0264] • Novel receptive field [0265] The new neural network is configured to achieve optimal performance, achieved in part by processing the images at high resolution. For example, in the radiological field, some detected microcalcifications are barely above the resolution limit. The new neural network comprises a modified receptive field (the region in the input space that a particular neural network’s feature is analyzing) of the networks to make them better suited for high resolution images that are processed. [0266] • Novel loss functions [0267] The new neural network uses loss functions appropriate for anomaly detection where positive cases (for example, cancers or other lesions when analyzing human physiology) are much more rare than negative cases. [0268] • Fusion layer [0269] In the context of detecting physiological anomalies in humans and other animals, the new neural network can associate clinical data (e.g., age, family history, genomics, etc.) with the neural network system. [0270] • Extendable [0271] The new neural network can treat each image separately or in combination with other views (for example, LLC versus RCC in mammography studies) or data or images created prior to the present analysis. It is also designed to be extendable from 2D to 3D image analysis. [0272] • Scalable [0273] The new neural network code can utilize any number of GPUs for distributed training which results in enhanced neural network training speeds. The new network can scale up to much larger problems such as 3D tomosynthesis. [0274] • Ensemble Modeling and Cascading Classifiers [0275] The new neural network offers the capability to include cascading classifiers (e.g., random forest, CNN) as well as ensemble modeling where outputs of different object detectors and classifiers can be combined to achieve optimal performance. [0276] • High performance [0277] The new neural network processes high resolution images at 3 seconds/image on a standard GPU. [0278] • Custom fine scale segmentation [0279] In the healthcare context, the new neural network can optionally detect and provide outlines of even weak micro-calcifications (see, e.g., Figure 29). This can be accomplished by a Fully Convolutional Network (“FCN”) segmentation process which can provide masks to highlight individual calcifications. [0280] • Optional Rigorous benchmarks [0281] The new neural network provides fine scale segmentation of the algorithm to enable rigorous efficacy checking. For example, in case of micro-calcifications in mammographic images, the system can assess how many detected micro-calcifications match the ground truth micro-calcifications. [0282] EXAMPLES [0283] Aspects of the present teachings may be further understood in light of the following examples, which should not be construed as limiting the scope of the present teachings in any way. [0284] Module 1 - Automated detection of BAC [0285] We have developed a learning technique for automated detection of BAC. The following three figures show the output of this module for three different mammography cases. It is evident from these figures that the algorithm catches even the faintest BACs and can even detect BACs that deviate from a linear structure. [0286] Module 2 - Automated segmentation of BAC [0287] Figures 9A-9C present examples of original versus “zoomed in” images and the associated BAC. [0288] Module 3 - Automated scoring of BAC [0289] There have been several attempts to create a BAC score. These attempts have generally been based on approaches to mimic how the CAC score is created and use measurements of macroscopic property of the BAC (e.g., length or area). One of the major deficiencies of such approaches is that they require hand measurements, both in terms of detection as well as scoring, and are not automated. Another issue is that they are done in vacuum, without a gold standard to compare the viability of one method over another. [0290] We have developed two different techniques for scoring BAC that address these issues. [0291] First, we have created two automated techniques for scoring BAC. One, called Weighted Length Score (WLS), is defined as:
Figure imgf000056_0001
where N is the number of images for this patient, s is the severity and l the length of the calcification. [0292] A second method, that we refer to as Bradley Score, is a physics-based score that takes into account the morphology of individual calcifications which make up BAC (International Publication PCT/US2016/054074). [0293] Secondly, we have established a gold standard for comparing different BAC scores. This gold standard is the ROC of BAC as a predictor of CHD. This enables us to judge the viability of one method over another. When comparing two methods for creating a BAC score, the method that yields the highest accuracy of prediction of CHD is the optimal choice. [0294] Figure 10 shows the Pearson correlation between our two BAC scores. While there is significant correlation between the two, there are cases where WLS is small but have high Bradley score. This can happen since Bradley score takes into account morphology of calcifications making up BAC. If the BAC is short but has intense groupings of calcifications, the Bradley score would be high but the WLS score would be low. This is illustrated in Figure 11. [0295] Gold standard for fine tuning and selection of BAC score models [0296] As mentioned, our gold standard is the efficacy of BAC as a predictor of CHD. To evaluate this efficacy Random Forest was trained and tested using cross-validation, the positive label being the diagnosis of CHD. As features for the Random Forest combinations of BAC or CAC were made with the age of the patient and diagnosis of hypertension because both of these are correlated with CHD and are readily accessible. Age and hypertension as well as age by itself were also included as combinations of features to improve the visibility of the effect of BAC and CAC. An ROC was created to compare BAC to CAC as predictors of CHD. Figure 12 shows this comparison. [0297] To compare to what is used in practice, we looked at treatment thresholds used in practice. The Society of Cardiovascular Computed Tomography (SCCT) consensus from 2017 recommends Table 2 for treatment (Hecht, 2017):
Figure imgf000057_0001
Table 2. [0298] As shown in Figure 12, the relevant parts of the ROC are those corresponding to the true positive rates of these treatment thresholds, which are shown in dashed lines. As can be seen, BAC is comparable, if not better, than CAC as a predictor of CHD at the 3 threshold marks. [0299] This result is surprising, but can be explained as follows. CHD is possible with CAC of zero. Leading cause of CHD is due to plaque build up, which can rupture or narrow the coronary artery, regardless of whether the plaque is calcified (Zaman, 2000). Plaques can develop at multiple locations, at different time points and can be in different stages of development (Hong, 2010). If you have plaque at a certain location such as in the breast, the chance that a plaque will form, or is present, at another location like the heart is higher (Allison et al., 2004). Calcification is the last stage of plaque development (Zaman, 2000). So it is possible that BAC is measurable before CAC. BAC may be an indicator of both calcified and noncalcified plaque (not detectable by the CAC test) in the coronary artery because of this appear to be comparable or possibly even better as a predictor for CHD. [0300] It makes sense that this could also hold for the carotid artery and peripheral arteries. This data was limited by the necessity of a CAC score measurement and did not include enough cases that did not have CHD but did have ASCVD to explore the relation between BAC and ASCVD (CHD, peripheral arterial disease and carotid artery disease). New data is being acquired to further explore this relation. Especially considering CHD being the most common ASCVD and the previously mentioned facts about plaques (e.g. that if there is a measured calcification somewhere, the chance that there is a plaque elsewhere is higher especially with increasing age) we have a clear indication that the observed relation between BAC and CHD will also hold for BAC and ASCVD. [0301] Risk Calculator [0302] We have also developed a technique to relate the ROC and the classifier score to the probability of an event. This enables development of a risk calculator based on BAC with the availability of the longitudinal data. This risk calculator could be used to decide whether further diagnostics is necessary and/or treatment. [0303] 1. Relationship between classifier score and probability of event (risk) [0304] A. Goal [0305] We would like to find a relationship between the output of a classifier (the score) and the probability of event occurring within some time interval T. [0306] The classifier is trained using feature vectors paired with binary labels which simply say whether or not an event occurred with the time interval T. The math shown here is very general, but for example the feature vector could be x = (age, blood pressure, and BAC score) and the event could be a CVD event. The features are all measured at time t and then the patient is observed up to time t +T and the label y is either 0 or 1 depending on whether the patient had an event within the time interval or not. [0307] After training the classifier we get a function that maps from feature vector x to a scalar value s (the score). The goal is to convert the score into a probability of having an event with the time interval T. This is more meaningful than the raw score s. [0308] B. Theory [0309] Once we have trained the classifier, we can make a validation ROC by calculating TPR(s) and FPR(s). From TPR(s) and FPR(s) we can find the PDFs for the positive class (event) and the negative class (no event). [0310] Then we can use Bayes law to calculate the probability that a patient has an event (within some fixed time window) based in part on p(+) and p(-), which are the prior probabilities of the patient having an event or not having an event. We can approximate these values using the N_+ and N_-, the number of patients with an event and the number of patients without an event in the validation set. [0311] This tells us what the risk of the patient having an event is, given that the patient was assigned score s by the classifier. [0312] 1. Relationship between risk and the ROC tangent slope [0313] Another interesting thing to compute is the ratio of probability of event p(+|s) to the probability of no event p(-|s). Note that p(+|s) + p(-|s) = 1. [0314] The probability depends in part on the slope of the ROC. The slope is small in the upper part of the ROC (low score, low risk) and the slope is large in the lower part of the ROC (high score, high risk). For a normal ROC the slope should increase monotonically as you move up in score (moving from the top to the bottom). When this occurs, it means that the probability also increases monotonically as the score increases. [0315] A. Example [0316] As an example, we will calculate the probability of CAD event as a function of the RF score based on the cmAngio model which uses CAC, age, and hypertension. This is shown in Figures 13A-13C. [0317] The ROC was generated by averaging over 40 random test/train splits. In each split there are 249 negative cases and 41 positive cases, so the prior Npos/Nneg = 0.16 based on the test set (this is a biased sample so it does not apply to the general population). Table 3 summarizes characteristics of the ROC.
Figure imgf000059_0001
Table 3. [0318] Note that, while the cmAngio data was used just to illustrate the idea, there are a few problems with this example: • • The number of patients is small. • • We don't have a well-defined time window over which the event / no event determination was made, so it is not clear over what time period the 'probability of event' applies to. Ideally all patients should have observed for a fixed window of time (for example in the MESA study it was 10 years) after the measurements are taken. • • The probability of event does not apply to the general population, but to the biased sample of people in our test set (not necessarily a problem, but something to keep in mind). • • This analysis does not account for people dying. [0319] Examples of indications which can be utilized in this system include, but are not limited to: • I50.9 - Heart Failure • I50.30 - Congestive Heart Failure • I50.32 - Congestive Heart Failure - Chronic diastolic (congestive) heart Chronic • I25.9 - Chronic ischemic heart disease, Chronic • I50.22 - Chronic systolic (congestive) heart Chronic • I50.20 - Congestive Heart Failure - Unspecified systolic (congestive) heart Unspecified • I50.33 - Acute on chronic diastolic Acute heart failure • I50.23 - Acute on chronic systolic Acute heart failure • I50.810 - Right Heart Failure • I50.1 - Left ventricular failure, unspecified Left • I50.42 - Chronic combined systolic (congestive) Chronic diastolic (congestive) heart failure • I50.43 - Acute on chronic combined Acute (congestive) and diastolic (congestive) heart failure • I50.40 - Unspecified combined systolic (congestive) Unspecified diastolic (congestive) heart failure • I50.812 - Chronic right heart failure Chronic • I50.811 - Acute right heart failure Acute • I63.9 - Ischemic stroke • I25.10 - Atherosclerotic Cardiovascular Disease • I65.29 - Occlusion and stenosis of Occlusion carotid artery • I25.84 - Coronary atherosclerosis due to Coronary coronary lesion • I25.810 - Atherosclerosis of coronary artery Atherosclerosis graft(s) without angina pectoris • I25.83 - Coronary atherosclerosis due to Coronary rich plaque • I25.119 - Atherosclerotic heart disease of Atherosclerotic coronary artery with unspecified angina pectoris • I65.22 - Occlusion and stenosis of Occlusion carotid artery • I25.118 - Atherosclerotic heart disease of Atherosclerotic coronary artery with other forms of angina pectoris • I70.209 - Unspecified atherosclerosis of native Unspecified of extremities, unspecified extremity • I65.23 - Occlusion and stenosis of Occlusion carotid artery • I66.9 - Occlusion and stenosis of Occlusion cerebral artery • I70.219 - Atherosclerosis of native arteries Atherosclerosis extremities with intermittent claudication, unspecified extremity • I25.110 - Atherosclerotic heart disease of Atherosclerotic coronary artery with unstable angina pectoris • I70.229 - Atherosclerosis of native arteries Atherosclerosis extremities with rest pain, unspecified extremity • I65.09 - Occlusion and stenosis of Occlusion vertebral artery • I66.09 - Occlusion and stenosis of Occlusion middle cerebral artery • I25.111 - Atherosclerotic heart disease of Atherosclerotic coronary artery with angina pectoris with documented spasm • I70.8 - Atherosclerosis of other arteries Atherosclerosis • I25.89 - Other forms of chronic Other heart disease • I25.708 - Atherosclerosis of coronary artery Atherosclerosis graft(s), unspecified, with other forms of angina pectoris • I65.8 - Occlusion and stenosis of Occlusion precerebral arteries • •I25.758 - Atherosclerosis of native coronary Atherosclerosis of transplanted heart with other forms of angina pectoris • I25.701 - Atherosclerosis of coronary artery Atherosclerosis graft(s), unspecified, with angina pectoris with documented spasm • I70.213 - Atherosclerosis of native arteries Atherosclerosis extremities with intermittent claudication, bilateral legs • I65.02 - Occlusion and stenosis of Occlusion vertebral artery • I66.29 - Occlusion and stenosis of Occlusion posterior cerebral artery • I70.299 - Other atherosclerosis of native Other of extremities, unspecified extremity • I25.700 - Atherosclerosis of coronary artery Atherosclerosis graft(s), unspecified, with unstable angina pectoris • I25.709 - Atherosclerosis of coronary artery Atherosclerosis graft(s), unspecified, with unspecified angina pectoris • I25.719 - Atherosclerosis of autologous vein Atherosclerosis artery bypass graft(s) with unspecified angina pectoris • I70.201 - Unspecified atherosclerosis of native Unspecified of extremities, right leg [0320] Devices, Systems, and Methods for Generating Synthetic 2D Images [0321] A common problem in model development is lack of sufficient examples of rare events. This can manifest in at least two forms: first, there is a large general class of problems called anomaly detection and classification. The anomalous nature of such events implies that they are rare compared to normal events. Second, the degree of difficulty for event detection and classification can vary significantly across a given class. The number of examples across that range of difficulty is often not uniform, resulting in a paucity of examples for the difficult cases. [0322] The lack of sufficient number of training examples in machine learning is especially challenging. This issue is exacerbated for deep learning algorithms which require many more examples than traditional machine learning techniques such as random forest. Various strategies have been proposed to augment the training set with additional examples which rely on mathematical operations on the original examples. However, in the case of image classification, for example, the auxiliary data set can include geometrical transformations (e.g., rotation, translation, flipping), distortions, blurring, and contrast adjustments which, while useful, offer limited variations in the data set and may offer only modest boost in performance. [0323] Therefore, what is needed are devices, systems and methods for generating training examples for model development and enhancing machine learning techniques, and in particular, generating synthetic two-dimensional medical images for such purposes. [0324] In general, described herein are devices, systems, and methods for computer- based generation of synthetic two-dimensional medical images. In this disclosure, “generation” of an image may include synthesizing, producing, processing, creating, and developing an image, example, or case. The term “synthetic” refers to the result of such generation. For instance, in an aspect, such generation of a two-dimensional medical image can be accomplished through electronics and software processing of one or more medical images to provide a synthetic two-dimensional result. This may be referred to as a synthetic example, or a synthetic case. [0325] Although devices, systems, and methods discussed herein generally describe the generation of synthetic two-dimensional medical images, and the like, they may also or instead be enabled by the devices, systems, and methods discussed herein. Although certain embodiments discussed herein describe the generation of two-dimensional medical images for the specific use in breast cancer detection and quantification, the devices, systems, and methods discussed herein can be adapted to generate synthetic two-dimensional medical images of other cancers including without limitation brain, lung, liver, prostate, bone, cervical, colon, leukemia, Hodgkin disease, kidney, lymphoma, oral, skin, stomach, testicular, thyroid, and so forth. Moreover, the devices, systems and methods discussed herein can be adapted to generate synthetic two-dimensional images for other physiological anomalies which are not cancers, e.g., bone fractures, arterial aneurysms, vascular blockage, cysts, polyps, congenital and genetic anomalies, and the like. Furthermore, although embodiments generally described herein are generating two-dimensional medical images of human tissue, the embodiments may also or instead be applicable to cancer in animals, for example. [0326] Enhanced Modeling Techniques [0327] To address the lack of sufficient examples when a machine learning model is not performing well due to lack of sufficient rare event examples, a technique has been developed to create an unlimited number of examples of rare events. The creation of such synthetic cases which augment the original data set leads to increases in sensitivity and specificity in detection and classification problems. For example, in the two-class classification problem, one can perform targeted generation of synthetic examples for the parts of the ROC curve where the model is not performing as well due to lack of sufficient examples. [0328] Therefore, what is provided are devices, systems, and methods to create synthetic examples of the events of interest. This can include synthesizing new examples (i) which are statistically similar to the original examples, (ii) new cases where the examples are transferred to different backgrounds, and (iii) new cases where different types of anomalies are combined to generate hybrid examples which may naturally occur. [0329] The creation of synthetic cases can be done in several ways. We have recently used deep learning for this purpose (See, e.g., international patent application PCT/US2016/054074, which is incorporated herein by reference in its entirety). The creation can also be done in a more manual process as we describe herein. [0330] To illustrate a new approach, the specific problem of breast cancer was considered in which screening is performed with X-ray mammography. In approximately 30% of patients, the first visible indication of cancer in the X-ray image is from a cluster of micro-calcifications. Depending on the progression of the cancer, the visible number of calcifications can range from a few, to a large number (100+). To prevent the further development of cancer, it is crucial to detect the small clusters at the earliest possible time. However, the detectability of these small features depends on a variety of factors, including the size of the cluster, the location of calcifications within the breast, and the mean density of the breast tissue. Larger calcifications within fatty tissue have very good contrast, while the smallest clusters within dense tissue are extremely difficult to detect. Furthermore, these small clusters may occur in the proximity of various types of benign calcifications, making accurate classification more difficult. [0331] Within any given data set of biopsy proven cancers, there will typically be a mix of easy and moderately difficult cases, but only a few of the most difficult, which are near the limits of detectability. Given the rather low incidence of cancer in a screening population (5 cancers out of 1000 people), it is difficult to greatly increase the number of examples, especially the more difficult or unusual cases (e.g., when small cancers appear in conjunction with benign calcifications). However, a new approach allows one to take a limited data set and specifically target the difficult or unusual cases. As described herein, this new approach is demonstrated to significantly improve the performance of machine learning algorithms, and in particular, detection and quantification of cancers, including breast cancer, from the generation and use of synthetic two-dimensional images. [0332] The approach, as it is related to mammography and breast cancer, may be summarized in the following steps: [0333] 1. The first step is to detect the individual micro-calcifications within a biopsy-proven cancer image (e.g., a ground truth image). This can be done, for example, in the following two ways. First, an algorithm for detecting, classifying, and quantifying micro- calcifications, such as that described in international patent application PCT/US2016/054074, automatically computes the small regions in each image corresponding to micro-calcification. Alternatively, it is possible to use the basic machinery provided in that algorithm to allow a human (e.g., a radiologist, or with the assistance of a radiologist) to manually select the final regions for each micro-calcification. For small clusters of greatest interest, this can be done using an interactive interface. In either case, the final representation of each micro- calcification will be in the form of a contour (ordered x,y pairs) that describes the outer shape, and a set of pixels within the contour, that describe variations of intensity within the shape. In addition, a specified number of buffer pixels are extracted which are immediately outside the contour. The intensity within those buffer pixels is also stored and is used for blending the micro-calcifications into the new images. Together, these features represent a complete mathematical description of the malignant cluster. The mathematical features of the cluster (contours, interior pixels, buffer pixels) are stored in a library. [0334] 2. The above process is repeated for each biopsy-proven cancer, focusing more on the more difficult or unusual cases. The end result is a library of malignant clusters with a precise mathematical description of each individual calcification, as well as their arrangement within a cluster. The above process could also be repeated for most types of false-positives (clusters that existing algorithms incorrectly identify of cancer). The resulting false-positive library could then be used to construct new synthetic examples in the same manner as described below, allowing specific targeting of the most difficult types. [0335] 3. The next step is to insert the extracted cancers into new images. For this purpose, a predetermined Breast Imaging Reporting and Data System category 1 or 2 image (negative or benign findings; commonly known as BI-RADS 1 or BI-RADS 2) may be used which is an indication that the image displays a low risk of cancer. To construct synthetic cancer images, the human would proceed as follows: [0336] A. Select a malignant cluster from the library with desired features (number and size of calcifications) [0337] B. Choose a normal image with desired features (mean breast density, complicated density structures, or other complications such as benign calcifications) [0338] C. Pick an appropriate point within the image to insert the cluster [0339] D. Choose any mathematical transformation to apply to cluster. Examples include rotating the cluster by a specified angle, reducing the total number of calcifications to insert (e.g., reducing the size of the cluster), re-scaling the size of individual micro- calcifications, and so on. [0340] Once the above choices are made, the cluster is inserted into the image at the specified location. Within each shape, the pixel intensity stored for each micro-calcification is used to over-write the pixel intensity within the normal image. For the pixels in the buffer region, a linear interpolation method may be used to blend the pixel intensity between the inserted cancer pixels and the normal image. This allows micro-calcifications to blend more smoothly into the new image. One of skill in the art will recognize other techniques for blending a portion of a first image into a second image to make it look “natural”. [0341] Given a cancer library of approximately 100 clusters and a few thousand normal (BI-RADS 1 or BI-RADS 2) images, it is possible to construct a large number of synthetic images. To be most effective, it is important to target the types of cancers which are most difficult for current algorithms to correctly classify. For example, one target corresponds to relatively small clusters within dense breast tissue. To target this specific class, one would focus on the smaller clusters in the cancer library, and choose normal images with higher densities levels (e.g., density levels 2-4). [0342] The final step is to inspect the inserted cancer cluster. The visibility of each individual calcification will depend upon the pixel intensity within the calcification, relative to mean pixel intensity in the region where it is inserted. An acceptable synthetic example should blend fairly smoothly, and look natural, or substantially identical to the normal image but for the blended insertion. The resultant synthetic two-dimensional images are indistinguishable from images of actual cancers, even to trained radiologists. [0343] While the preceding discussion used breast cancer screening as an illustration, a medical exam could relate to a variety of diseases or disorders. [0344] These steps can be performed in devices which are described more fully below. In addition, the methods provided above can be part of a system, which again is described more fully below. The Examples below also provide one of skill in the art specifications for implementing the techniques provided herein. [0345] EXAMPLES [0346] Aspects of the present teachings may be further understood in light of the following examples, which should not be construed as limiting the scope of the present teachings in any way. The systems and methods are illustrated through a series of examples, including (a) micro-calcification clusters, (b) adding calcifications to existing masses, (c) benign lucent calcifications, and (d) synthetic masses: [0347] Example 1 [0348] The original cancer of Figure 14 is shown in panel (a), along with a close-up of the micro-calcifications (bottom). The original cluster consists of approximately a dozen micro-calcifications within a fatty region with very good contrast. The mathematical description for this cluster was extracted with the system described in international patent application PCT/US2016/054074 and saved to a library. The cluster was then inserted (unmodified) into a new normal image in panel (b). The insertion point was chosen in dense region of the breast, resulting in poorer contrast (roughly 7 calcifications clearly visible). [0349] Example 2 [0350] The original cancer of Figure 15 is again shown in panel (a), along with a close-up of the micro-calcifications (bottom). The original cluster consists of approximately 30+ micro-calcifications with a range of sizes, and across a significant variation in tissue density. This full cluster is inserted into a much denser normal breast in panel (b). The number of visible calcifications is markedly reduced due to decreased contrast. In panel (c), the orientation of the cluster is rotated, and the size is reduced by inserting only 20% of the original calcifications into the normal image. This is accomplished by computing the geometric center of original cluster, and specifying an acceptance distance about the center when inserting into the normal image. In this case, the cluster is reduced to approximately 7 calcifications. Using this procedure, it is possible to systematically tailor the difficulty of synthetic cancers, up to limit of detectability. [0351] Example 3 [0352] In certain cancers, micro-calcifications are visible within a mass. This class is particularly dangerous, so it is important for algorithms to accurately recognize both features (mass and calcifications). However, clear examples of this type are less common, so it is desirable to create synthetic examples to better train the algorithms. In Figure 16, micro- calcifications have been inserted into a malignant mass. [0353] Example 4 [0354] Lucent calcifications are benign and often appear has hollowed out rings, half-moon shapes, along with the addition of smooth round calcifications. Despite being benign, machine learning algorithms sometimes mischaracterize these clusters as cancer. Thus, one strategy is to target this class of false-positives by extracting benign lucent clusters as shown in panel (a) of Figure 17, and then inserting these clusters into other normal breast images. Three such examples of synthetic lucent clusters are shown in Figure 17 panel (b). [0355] Example 5 [0356] The same basic technique can be applied for generating synthetic masses. In Figure.18 panel (a), a small malignant mass within fatty tissue is shown. A contour (darkened line) identifies the approximate boundary of the mass. In the right panel, the extracted mass is shown corresponding to the interior pixels inside the contour, and three buffer regions outside the contour to permit smooth blending within the new image. For masses, these buffer regions are preferably somewhat larger. [0357] Example 6 [0358] Figure 19 provides an example of a synthetic mass created using the extracted cancer from Figure 18. The mass is inserted into dense breast tissue. [0359] Example 7 [0360] Use of Synthetic Cases for Targeted Improvements in the ROC/FROC [0361] The use of synthetics enables the generation of examples of cases where the algorithm performance may not be optimal. The following examples provide specifications. [0362] Protocol 1. FRCNN With and Without Micro-Calcifications - Effect of Adding Positives (Synthetic Cancers) [0363] Figures 20A and 20B show the improvement in “Receiver operating characteristic/ Free-response Receiver operating characteristic” (“ROC/FROC”) through addition of difficult (faint and low number of calcifications) synthetic cancer cases in the training set.400 synthetic cancer images were generated using the methods provided herein, and inserted in normal cases. In order to enhance the function of the algorithm, the synthetic cancer cases were made “difficult” by reducing the number of micro-calcifications provided in each cluster and overlaying them in bright normal tissue so as to reduce the contrast. Figures.20A and 20B demonstrate the significant improvement in image-based ROC and FROC curves. For example, Figure 20A shows an increase in the AUC from 0.947 to 0.964. Figure 20B shows an increase in the AUC from 0.938 to 0.958. These improvements can also be seen in specific cancer cases in test sets, which are similar in nature to the class we targeted in the new synthetics (small clusters in dense tissue). Three such examples are shown in Figures 21, 22 and 23, which include an old score (no synthetics) and new score (synthetics used to retrain the model). Figures 21 provides that the new model improves AUC from 0.567 to 0.646. Figure 22 provides that the new model improves AUC from 0.377 to 0.762. Figure 23 provides that the new model improves AUC from 0.642 to 0.963. [0364] Protocol 2. Targeted generation of synthetics [0365] The systems and methods above were used to generate examples to improve performance of an algorithm for specific parts of the resultant ROC. The efficacy of this approach is illustrated in Figure 24 where the effects on image level ROC were compared when adding 91 new cancer cases, versus adding synthetic cases. The ROC is significantly improved with the addition of synthetics to the training set over the case where the algorithm was trained on 91 more cancer cases. See Figure 25. Without being limited to a particular theory, it is believed that this is because the addition of a random set of new cancer cases will include cases where the algorithm has already seen similar examples, whereas with synthetics we can generate targeted examples of cases that the algorithm may have difficulty with. [0366] Protocol 3 - Use of Synthetics as Phantoms for Calibration Purposes [0367] Because there can be a large variation in the quality, pixel resolution, contrast among other image characteristics from different image acquisition systems and vendors, one issue is how to train the algorithms for new sets of images from vendors, and/or systems that the algorithm has not been trained on. Ideally, one would have normal and positive (cancer) cases providing ground truth for each new data set that the algorithm has not been exposed to. In practice, however, this is often not feasible or practical. The use of synthetics in the systems and methods herein enables one to solve this issue where cancer can be extracted from an existing database to generate synthetics, and then reinsert them into the normal cases from any new data set. One can then retrain the algorithms and in the process the algorithm learns to adapt to the new types of images. [0368] Detection and Scoring of Diseases and Conditions Using Medical Imaging [0369] CADe may be used to identify and analyze data for anomalies in industrial systems, whether independently of human review, or for review by a technician. A CAD algorithm, such as that described in WO Pub. No. WO2017058848 which is incorporated herein by reference in its entirety, can be implemented to detect anomalies in data, and in the healthcare context, lesions in medical images. Detected physiological anomalies are typically digitally marked in an image (including in a DICOM® image) which can be burned in, overlaid, or provided separate from the original image. [0370] If a technician first looks at raw data, and then the CADe results, those CADe results may support the technician’s analysis, or may suggest further analysis of the image is required. CADe results may also be displayed in a “concurrent reading” mode in which the technician analyzes the CADe results while they look at the data. [0371] In the healthcare context, current CADe systems and computer-aided diagnostics (“CADx”) systems have not proven to be as reliable and accurate as experienced radiologists. For example, current CADe and CADx systems which are marketed throughout the world display false positives at a rate which interferes with a radiologist’s ability to efficiently analyze medical images. Technicians call this the “Christmas Tree” effect, which renders the CADe and CADx markings harder to interpret. This is especially true when a radiologist is required to analyze a series of medical images in a short period of time. [0372] Current CADe and CADx systems include those described in U.S. Pat. Nos. 5,005,126, 5,622,171, 5,781,442, and 6,621,191, and the like. Standard mammography CAD algorithms analyze digital or digitized images of standard mammographic views (e.g., Cranial-Caudal (“CC”) and mediolateral-oblique (“MLO”)) for characteristics commonly associated with breast cancer, such as microcalcifications and masses. [0373] According to the National Quality Measures for Breast Centers (NQMBC), the median time in days between screening and diagnostic mammogram was 6.5 days; between diagnostic mammogram and needle biopsy was 6.0 days; between needle biopsy and surgery was 14.0 days (Kaufman CS, Shockney L, Rabinowitz B, Coleman C, Beard C, Landercasper J, et al. National Quality Measures for Breast Centers (NQMBC): A robust quality Tool. nn Surg Oncol 2010; 7(2): 377-385). Currently, the time between conducting a screening mammogram and reading of that screening mammogram is driven by factors such as the availability of a radiologist, the scheduling protocols for batch reading, and the volume of exams being conducted at the site. A technician may impact or alter the time by visually identifying a suspicious exam and notifying a radiologist. However, there is no systematic, consistent, objectively efficacious means of triaging all exams and notifying radiologists when there is a suspicious exam. [0374] In recent years, artificial intelligence (“AI”) has improved the ability of CAD algorithms to detect anomalies in data generated by industrial systems, including in medical images. However, the precision, accuracy, sensitivity, and specificity of the analysis has often not been optimized which results in technicians failing to associate actionable intelligence or significant meaning with highlighted data. In addition, the lack of optimization has not resulted in higher efficacy or efficiency with respect to identifying suspicious data which is correlated to a target category. [0375] Accordingly, systems and methods are described for precisely and accurately identifying suspicious data with high sensitivity and specificity such that a technician can confirm whether the identified suspicious data is indicative of the need further analysis, or if the suspicious data is indeed an anomaly. [0376] Figure 26 depicts an exemplary triage system. While the following discussion illustrates the use of the triage system to prioritize or segment medical images for evaluation, in other embodiments the triage system may be used to prioritize or segment non- medical images for evaluation. For example, Figure 27A depicts a raw image of corrosion on an aircraft, and Figure 27B depicts ROIs which can be highlighted using a suspicion code. [0377] EXAMPLES [0378] Aspects of the present teachings may be further understood in light of the following examples, which should not be construed as limiting the scope of the present teachings in any way. [0379] Example 1. Stand-Alone Clinical Efficacy [0380] The Breast Cancer Surveillance Consortium (BCSC) collected the results of 1,838,372 screening mammograms from 2004 to 2008 from eight mammography registries across the country (http://www.bcsc- research.org/statistics/performance/screening/2009/perf_age.html). According to the BCSC study, the sensitivity of radiologists was 84.4%, and the specificity was 90.8% with a recall rate of 9.6%. In other words, the radiologists of the study above considered 9.6% of the exams suspicious enough to indicate a follow-up for the patient. [0381] The stand-alone performance of the systems and methods of the present invention were assessed to determine the sensitivity and specificity of the image analysis algorithm and to determine what the notification rate would be at different sensitivities. The population composition of the quarantine test set is shown in the Table 4. This test set was quarantined and not made available to the researchers when training or validating the algorithm.
Figure imgf000071_0001
Table 4. [0382] Normal cases are those that are either biopsy benign or contain 2 years of subsequent normal evaluations, and cancer cases are those that are biopsy confirmed cancer taken less than 270 prior to a cancer positive tissue biopsy. [0383] When the CAD algorithm is set to a sensitivity of 84.4%, in direct comparison to the BCSC study results, the algorithm has a stand-alone specificity of 90.55% which indicates that 6.1% of the cases are suspicious. Thus, at the same sensitivity and specificity as the radiologists of the BCSC study, the systems and methods of the present invention identify approximately 36% ( (9.6%-6.1%) / 9.6% ) fewer exams as suspicious. Conversely, if the algorithm is set to notify the radiologist that 9.6% of the cases are suspicious, the sensitivity can be shifted up to 89% thus allowing for a 5.5% improvement in sensitivity over the radiologists of the BCSC study. This increase in sensitivity would identify as suspicious approximately 2.7 more cancers per 10,000 patients screened for breast cancer. [0384] Considering that all cases marked as suspicious would be reviewed by radiologists and not automatically recalled, the algorithm could be set to a 96% sensitivity (or higher). Although this would increase the percent of cases marked as suspicious to 23% it would still be only a fraction of the total cases reviewed by a radiologist. This sensitivity would represent a 13.7% improvement over radiologists and would identify as suspicious approximately 6.8 more cancers per 10,000 patients screened. [0385] These performance characteristics are shown in Figures 30 and 31 along a curve that depicts the continuum of performance. The y-axis indicates sensitivity and the x- axis indicates the corresponding percent of patients that would be indicated as suspicious. In this Figures 30 and 31, the curves are calculated as follows: [0386] Sensitivity is calculated as follows: TP / (TP + FN). [0387] Specificity is calculated as follows: TN / (TN+FP). [0388] TP: True Positives are defined as exams that contain biopsy confirmed cancer which the software correctly identified as “Suspicious” [0389] FN: False Negatives are defined as exams that contain biopsy confirmed cancer which the software did not identify as Suspicious but instead identified as “” (blank) [0390] TN: True Negatives are defined as exams that are biopsy confirmed benign or normal which the software correctly did not identify as Suspicious but instead identified as “” (blank) [0391] FP: False Positives are defined as exams that are biopsy confirmed benign or normal which the software identified as “Suspicious” [0392] As noted above, each identified anomaly is scored on a scale of 0-100 and anomaly scores that are over a preset threshold are considered suspicious. The threshold is generated from the ROC and represents an operating point on the ROC tied to a specific sensitivity and specificity. Figure 32 is a ROC with the anomaly scores displayed in orange. [0393] Table 5 provides sample data points on the ROC showing a score threshold. In order to understand how to interpret Table 5, two data points (1 and 2), annotated with boxes, are shown in Figure 32.
Figure imgf000073_0001
Table 5. [0394] Stand-alone Time-based performance [0395] In order to establish a baseline for the time-based performance of the systems and methods of the present invention, a set of test cases was processed through the full operational flow from first receipt of the exam by the image forwarding software to receipt of the returned notification file at the viewing workstation. The average time for this complete process was 3 minutes 54 seconds (3.9 minutes) with a standard deviation of 15 seconds. This means that reading radiologists can be notified in roughly 4 minutes that there is a suspicious exam in their worklist. [0396] Description of triage and notification effects [0397] Depending on the results of image processing, the parallel workflow created by the device via a notification has the potential to positively impact the standard of care. Specifically, in the event of a True Positive (TP) study identified by the device, the parallel workflow of the systems and methods of the present invention allows a radiologist to more quickly identify and prioritize the assessment of suspicious exams. [0398] In the event that the device identifies a False Negative (FN) case and improperly concludes a study does not contain a suspicious anomaly, no notification is sent and the case will be identified through the standard of care workflow without interruption. [0399] In the event that the device identifies a False Positive (FP), the specialist receiving the notification would preview the images and further evaluate the exam on a diagnostic imaging system prior to disregarding the study, leaving the standard of care uninterrupted. This does not affect the patient. [0400] Similarly, if the device identifies a True Negative (TN) case, no notification is sent and the standard of care is uninterrupted. In all scenarios, trained radiologists read all images per standard of care, regardless of the performance of the systems and methods of the present invention. [0401] As noted above, the device is intended to send notifications so as to alert a specialist as to the timely existence of a case that may potentially benefit from that specialist’s attention, who would have reviewed them at a later time, had the device not been available. [0402] Example 2 - AI-based Triage Software Used to Improved Clinical Efficiency and Patient Experience [0403] Objectives: [0404] Primary Endpoint: Determine whether the use of the systems and methods of the present invention can reduce the overall time between study capture and the determination to recall a patient for follow-up [0405] Secondary Endpoint: Determine whether the use of the systems and methods of the present invention can reduce the overall time between identification of a patient with suspicious lesions and the notification of that patient for recall [0406] Background: [0407] According to the National Quality Measures for Breast Centers (NQMBC), the median time in days between screening and diagnostic mammogram was 6.5 days (25%ile 4.0 d, 75%ile 10.5 d); between diagnostic mammogram and needle biopsy was 6.0 d (25%ile 3.9 d, 75%ile 9.0 d); between needle biopsy and surgery was 14.0 d (25%ile 11.0 d, 75%ile 19.5 d) (Kaufman CS, et al., supra). Currently the time between conducting a screening mammogram and reading of that screening mammogram is driven by factors such as the availability of a radiologist, the scheduling protocols for batch reading, and the volume of exams being conducted at the site. A technician may impact or alter the time by visually identifying a suspicious exam and notifying a radiologist. However, there is no systematic, consistent, objectively efficacious means of triaging all exams and notifying radiologists when there is a suspicious exam. With such a system in place the clinic would have the opportunity to alter their standing protocols to prioritize suspicious exams and thus potentially reduce the amount of time between conducting the exam and reading the exam. By creating this prioritization process the turn-around-time (TAT) for these suspicious cases could be reduced, thus improving patient satisfaction and potentially impacting the quality of care provided to patients. Further, by using a system that can be set to a specific sensitivity and specificity, the identification of potentially suspicious cases will be consistent and not rely upon the varied skills of any one specific technologist. [0408] Case Population: [0409] An exemplary population of cases (exams) below in Tables 6 and 7 can be selected to represent a representative cross-section of patients based on age and density typically seen in a screening population. Notably, Table 6 provides a population with/without cancer for testing, and Table 7 provides a population of varying breast density for testing. The population can be enriched with biopsy confirmed cancers to more accurately measure the efficacy of the output software.
Figure imgf000075_0001
Table 6.
Figure imgf000075_0002
Table 7. [0410] Inclusion Criteria: Female, and have a standard 4-view screening mammogram, and met none of the exclusion criteria. [0411] Exclusion Criteria: Significant existing breast trauma, previous surgical biopsy, previous breast cancer, and inadequate technical image quality [0412] Study Design [0413] Step 1: Stand-alone performance assessment [0414] A. Each exam is processed as if it was being processed in a clinic: 1. The exam is exported from the PACS. 2. Each exam can optionally be run through image forwarding software that can perform anonymization and de-identification processes. 3. Each exam can optionally be encrypted and transmitted to a cloud-based server over a standard corporate network typical of a hospital network. 4. On the cloud based-server, the CAD algorithm will be used to process the exam. 5. Each suspicious exam will be assigned a suspicion code. This code will label the exam as “suspicious” if is at least one anomaly on the exam that has an anomaly score greater than or equal to the operating point threshold used in the study. 6. A notification result file in the form of a DICOM® SR is generated by the CAD algorithm and transmitted back to the PACS in DICOM® format. 7. The DICOM® SR can be decrypted when necessary, and re-associated with the original exam by image receiving software. 8. The PACS ingests the notification file and acquires the suspicion code. 9. The worklist in the PACS is updated to reflect acquisition of the DICOM® SR indicating completion of the process. [0415] Efficacy Results are measured in terms of case-level sensitivity, case-level specificity, and case-level area under the curve (AUC) measured on a reader operator characteristic (ROC) curve. [0416] Performance Results are measured in terms of the total time for Step 1: stand-alone performance assessment above; and from exporting the exam from the PACS until the PACS registers receipt of the DICOM® SR. Performance is measured in minutes and seconds. [0417] Step 2: Clinical data collection [0418] TAT audit data is collected from a study site for a 60-day window that tracks the following metrics: [0419] 1. Minimum / maximum / average / median time between completion of image acquisition of all images collected during the screening mammogram and completion of reading of the mammogram by the reading radiologist. [0420] 2. Minimum / maximum / average / median time between the reading radiologist determining that a recall is necessary—as indicated by assigning a BI-RADS® 0 to the exam—and sending of a notification letter to a patient. [0421] Secondary metrics are collected during the same period including the following: [0422] 1. Percent of patients recalled (assigned a BI-RADS® 0). [0423] 2. Percent of patients who return for their follow-up based on the recall. [0424] 3. Average amount of time between initial screening exam and the follow-up exam. [0425] Based on this example, it was determined that the clinical efficiency and efficacy improved, as provided in the results below. [0426] Example 3. AI-based Triage Software Used to Improved Clinical Efficiency and Efficacy [0427] Purpose: To determine whether a CAD algorithm can enhance radiologist productivity and effectiveness with a pre-analysis of screening mammograms prior to presentation to the radiologist for reading. The goal of this study is to quantify the performance benefit of triaging cases using the systems and methods of the present invention prior to interpretation by the radiologist. [0428] Background: Mammography is a widely accepted tool for early detection of breast cancer and is associated with at least 30% reduction in breast cancer mortality globally based on long-term studies. [0429] Typically, screening mammograms are read in a serial fashion, often through the use of “batch” screening, where exams are read by the radiologist at designated times when there are fewer interruptions or distractions. They may be sorted by study date and time, patient name, or other pieces of information that may be known about the patient or exam at the time of reading. However, this means that difficult or very abnormal cases are read in the mix with no means to prioritize them to be read earlier in the batch. In sites where there is a large backlog of exams to read, there may be serious cases which may need to recalled which may not be reviewed for days or weeks. [0430] With the advent of artificial intelligence (AI), there is the opportunity to have imaging studies pre-analyzed by a computer. With workflow automation features in PACS and viewing workstations, it is possible to sort cases prior to interpretation using different filters to expedite more timely recall of patients who have suspicious abnormalities. It is also possible to enhance radiologist productivity by allowing the radiologist to sort a worklist to allow reading of selected cases to accommodate a workday that may be filled with interruptions due to biopsies, diagnostic workups, and conferences. It is also possible to sort batched mammograms to allow a more even distribution of cases to radiologists on a given day or to match complexity of cases with radiologist skill set. [0431] In this study, a CAD algorithm was used to analyze screening mammograms and label individual exams as “suspicious” if there was at least one anomaly on any image of the exam that had an anomaly score in excess of an operating point threshold. Each suspicious exam was assigned a “suspicion code”. This code labeled the exam as “suspicious” if is at least one anomaly on the exam that had an anomaly score greater than or equal to the operating point threshold used in this study. [0432] It has been shown that the mean time to read a mammogram is 112 seconds and that typically CAD (Computer-aided Detection) adds an additional 19 seconds to that time. In this retrospective study, it was shown that the CAD algorithm reduced the overall time it took to read a screening mammogram. Further, the efficacy of reading radiologists was improved by drawing their attention to suspicious exams identified by the CAD algorithm. [0433] Objectives: [0434] Primary Endpoint: Determine whether the use of the systems and methods of the present invention can reduce the average time it takes to read a screening mammogram. [0435] Secondary Endpoint: Determine whether the use of the systems and methods of the present invention can improve the overall efficacy of the reading radiologist [0436] Case Population: [0437] The population of cases (exams) in Table 8 (which provides a population with/without cancer for testing) was selected to be a representative cross-section of patients based on age and density typically seen in a screening population. The population was enriched with biopsy confirmed cancers to more accurately measure the efficacy of the CAD algorithm. An exemplary population having diverse or varying breast densities for testing can also be measured (which could be as provided in Table 9).
Figure imgf000078_0001
Table 8.
Figure imgf000078_0002
Table 9. [0438] Inclusion Criteria: Female, and have a standard 4-view screening mammogram, and met none of the exclusion criteria [0439] Exclusion Criteria: Significant existing breast trauma, previous surgical biopsy, previous breast cancer, and inadequate technical image quality. [0440] Reader Population: 6 MQSA certified radiologists with varying years of experience as shown below: [0441] Study Design [0442] A. The case population is divided in half maintaining all ratios with an effort to make the two datasets consistent in terms of complexity. This is overseen by 2 MQSA radiologists who are not participating in the study. [0443] B. Half of the cases are used in the Unassisted Read (“Non Triage Cases”) [0444] C. The other half of the cases are processed by the CAD algorithm (“Triage Cases”) [0445] Data Analysis [0446] Reader Efficacy is measured independently on each of the data sets in terms of case-level sensitivity, case-level specificity, and case-level area under the curve (AUC) measured on a reader operator characteristic (ROC) curve. [0447] Reader Performance Results is measured independently on each of the data sets in terms of the total time for the reader to process an exam. [0448] Comparisons in terms of efficacy and performance is made across readers and between data sets to determine if the systems and methods of the present invention improves the efficacy and / or performance of readers. [0449] As expected, the efficacy and efficiency of radiologists was improved using the systems and methods of the present invention, as provided in the results below. [0450] Example 4. Efficacy and Efficiency Results [0451] Objectives: To investigate the effectiveness of the novel AI software on mammography interpretation, the new Neural Network was compared to a leading CADe system. It was determined that this new technology is useful in clinical practice to affect efficiency gains. [0452] Patient Population: 2502D, 4-view, Screening Digital Mammograms, typical asymptomatic women of all races and ages. The patients in the population are further described in Table 10:
Figure imgf000080_0001
Table 10. [0453] In Table 11, the products are compared. Notably, in Table 11, OP 1 refers to the Operating Point setting for the software. The leading software has 3 operating points (0, 1, and 2) that dictate the sensitivity and specificity settings for the CADe software. A higher operating point corresponds to a higher sensitivity.
Figure imgf000080_0002
Table 11. [0454] In Table 11, exemplary “Threshold 60” is the lesion score threshold used in this study. The threshold can range from 0-100 and is an anomaly score that indicates the level of suspicion tied to a specific region of interest. Higher scores are more suspicious. The set threshold corresponds to a sensitivity and specificity setting for the CADe software. [0455] Tables 12 through 16 provide the results of testing, and the degree to which the systems and methods of the present invention compare to the present market-leading CADe software. Notably, Table 12 compares false positives for mass and calcifications, Table 13 compares false positives for density, Table 14 compares false positives for BI- RADS® 0, Table 15 compares the reading time reduction, and Table 16 compares the increased efficacy.
Figure imgf000081_0001
Table 12.
Figure imgf000081_0002
Table 13.
Figure imgf000081_0003
Table 14.
Figure imgf000081_0004
Table 15.
Figure imgf000081_0005
Figure imgf000082_0001
Table 16. [0456] Example 5 – TAT/Efficiency Determination [0457] Objectives: To investigate the effectiveness of the novel AI software on mammography interpretation, the new Neural Network can be compared to current CADe systems. Similar to the Examples above, several factors can be measured including: 90% case level AUC; Average / Min / Max full cycle time to process study (from capture to delivered triage result / notification file); Typical time between study capture and determination to recall patient; Typical time for patient to be notified of recall; Typical time for patient to return for recall; and Percentage of notified patients who return for recall. [0458] Results: It is expected that the systems and methods of the present invention will provide that the CAD algorithm allows for a faster notification to the technician of the potential need to analyze potentially suspicious data, including the recall of a patient, when compared to waiting for the current clinical process to resolve. In addition, it is predicted that by providing faster notification, technicians and/or treating physicians could adjust their clinical process to have patients wait in the treatment room before being released to a waiting room. This revised action will likely decrease the amount of time it takes to notify a patient, thus reducing patient anxiety and increasing the likelihood of notifications actually reaching the patient. This revised action will also likely increase the percentage of women who participate in a recall because the feedback is immediate and they are already in the office. Moreover, this revised action will likely decrease the administrative cost of recalls because there will be no need for additional scheduling. [0459] Example 6 – Efficacy Determination [0460] Objective: What is needed to measure and demonstrate, similar to the Examples above, includes: 90% case level AUC; Improved reader efficacy; and Improved reader performance (efficiency). [0461] Results: What is expected is that the systems and methods of the present invention allow technicians and radiologist to read medical images faster with the same or greater efficacy compared with current modalities. [0462] Prediction of Probability Distribution Function of Classifiers [0463] Rapid advances in technology have led to a proliferation of data. Techniques used to generate, collect and process data involve machine learning (ML)/AI- based systems and methods. ML/AI-based systems apply DNN for: (i) computer vision (e.g., anomaly detection, classification, segmentation); (ii) time series prediction and forecasting; (iii) speech recognition; and (iv) natural language processing (NLP). [0464] However, a shortcoming of DNN is that when faced with examples coming from a different distribution than the training data, the DNN generates the wrong predictions, with high confidence. This is attributed to the inability of the DNN-derived models to quantify and output its uncertainty in each instance. An illustrative example is when a DNN is: (i) trained to identify cancerous lesions in mammography and (ii) given an image with microcalcifications. The DNN-derived model is forced to classify the image as either cancerous lesion or normal, while having no way of expressing the uncertainty due to the fact that the training set of DNN-derived model has not seen examples of microcalcifications. High uncertainty output is akin to saying “I am not sure”. [0465] The above is an example of uncertainty which arises from limitations in the training set. Another source of uncertainty is from: (i) variance in the data or (ii) the underlying process. An instance of this in mammography is where even images taken of the same patient a few minutes apart are not identical due to positioning of the patient and the compression of the breast. [0466] Prediction of Probability Distribution Function of Classifiers and Expressing the Results to a User [0467] Deep neural networks in the analysis of medical images and other complex data sets are used in the system and methods herein for prediction of probability distribution function of classifiers and expressing the results to the user. [0468] In the systems and methods herein, the model outputs other parameters of the distribution function such as standard deviation in addition to the mean of the distribution (score) at each case. The standard deviation can be interpreted as the measure of uncertainty or confidence level. The determination of the uncertainty or confidence level is not limited to the distribution function. This enables the following capabilities, including but not limited to: (a) creating different protocols for cases where the model output predicts high uncertainty (low confidence); (b) identification of cases with large uncertainty for further examination and development of suitable training strategy to improve the trained model; and (c) creating a separate bucket for cases with high uncertainty in triage setting. For example, cases in the bucket of capabilities (c) may indicate more difficult cases which require additional diagnostics or follow up. In turn, the systems and methods herein perform the capabilities for: d) increasing the efficacy of the model by separating cases where the model is confident from cases which are uncertain; and e) creating a single score that takes into account other metrics of the distribution function rather than just its mean. For example, a score can include a combination of mean and variance such that: (i) the score for small variance is similar to the mean; and (ii) the score is reduced significantly for high variance. [0469] Reference is made to the figures below to further describe the systems and methods herein. [0470] Referring to Figure 33, device 3305 connects to training data set 3325 and medical information 3330 via network 3320 in computing environment 3300. Network 3320 is a digital telecommunications network for sharing resources between nodes (i.e., computing devices). Data transmission between nodes is supported over physical connections (twisted pair and fiber-optic cables) and/or wireless connections (Wi-Fi, microwave transmission, and free-space optical communication). Device 3305 may be any machine which is instructed to carry out sequences of arithmetic or logical operations via computer programming. Device 3305 may include without limitation a Smartphone device, a tablet computer, a personal computer, a laptop computer, a terminal device, a cellular phone, and the like. [0471] User Interface (UI) 3310 and program 3315 reside on device 3305. UI 3310 facilitates human-computer interaction which as a graphical user interface (GUI), which is composed of a tactile user interface and visual interface. UI 3310 is connected to program 3315 such that user of device 3305 can interact with program 3315 via graphical icons, audio indicators, and text-based user interfaces. [0472] Program 3315 receives data from training data set 3325 and medical information 3330. Based on the training data set 3325, program 3315 receives cases comprising medical images and other associated information (diagnosis, salient features of the medical images which lead to the diagnosis, and DICOM header). The DICOM header may contain a range of useful information including without limitation, the side (i.e., left or right), orientation, view, protocol, date of procedure, and so forth, many of which may be listed in a filename convention. This information may be extracted for use by the algorithm— for example, in order to compare results from multiple views, or from a time series of images. Examples of DICOM tags include without limitation: (a) pixel spacing (e.g., hex tag— (0028x,0030x)), which may be useful to scale the image in terms of real physical dimensions (e.g., mm), which can compute a ‘Q factor’ consistently; (b) diagnostic vs. screening (e.g., hex tag—(0032x,1060x)), which may allow for inclusion or exclusion of diagnostic images from studies; and (c) patient orientation (e.g., hex tag—(0020x,0020x)), which may allow for displaying the images in a consistent manner. Stated another way, the images are displayed in the same orientation as used by radiologists in typical computer-aided design (CAD) systems. This can be advantageous when contour data is returned for display and/or analysis. For consistency in analysis, a predetermined orientation may be assigned (e.g., for mammograms—where the nipple points to the left in all images as is the industry standard). [0473] From these medical images and other associated information, parameters are set for program 3315 to create a model. The model implements an assessment protocol on data sets. More specifically, program 3315 generates models by using training techniques: (A) ensemble learning; or (B) deep neural net architectures which use point estimates. The implemented assessment protocol can generate a score and determine a level of uncertainty by: (i) focusing on particular parameters; (ii) ignoring other parameters; and (iii) evaluating the significance (i.e., weight) of each parameter when. The score is associated with a probability of cancer or the degree of suspiciousness of a lesion in medical information 3330. [0474] Images with high levels of uncertainty can be sent to buckets for retraining. When retraining the model, the assessment protocol can be modified. In turn, this can increase the confidence level (i.e., reduce the uncertainty). When training the model, program 3315 focuses on the perimeter of the organ captured and the overall appearance of the organs in the known medical images. The perimeter is focused on because certain spots are implicated with cancer risks. Medical images A and B do not have spots on the perimeter but the overall appearance of images A and B are noticeably different from known medical images used to train the model. Image A has surface ridges in the interior while the colorations are not obscuring the perimeter. Image B has colorations which obscure the perimeter, while being absent of the interior. The surface ridge in the interior and colorations are factors which are not initially understood and therefore images A and B are deemed as having high uncertainty. To understand the implications (if any) of the surface ridges and colorations, which increase the uncertainty levels, program 3315 sends images A and B to a bucket for retraining. [0475] By determining the level of uncertainty and retraining images with high levels of uncertainty, the confidence in the score is evaluated and thereby program 3315 is providing a level of granularity when analyzing images and other data sets, while improving the models and implementing assessment protocol. [0476] Stated another way, program 3315 receives image data from medical information 3330, wherein the image data are, for example, mammograms of a plurality of patients. In response to program 3315 applying the implemented assessment protocol on the medical information 3330, program 3315 generates a score and level of uncertainty. The generated score and level of uncertainty are outputted to UI 3310. The generated score on the medical image can be thought of as the mean of the probability distribution function (PDF). However, there are other metrics of the distribution function that are useful and provide more granularity into the model output. For example, a normal distribution is uniquely characterized by its mean and standard deviation. Standard deviation can be thought of as a measure of uncertainty of model output. As such, the systems and methods herein can be appended to not only include the score but details of the probability distribution function, including but not limited to a level of uncertainty, for each instance. [0477] Program 3315 applies machine learning techniques when assessing uncertainties in mammography. More specifically, program 3315 accounts for aleatoric and epistemic parameters. Aleatoric parameters assess statistical uncertainty (i.e., stochastic variability in data), which is always present and thus not possible to eliminate with more data. Examples of aleatoric parameters as applied to mammography in the systems and methods herein include: (1) positioning / compression of breast; (2) variations in sensor efficiency or X-ray calibration; and (3) random seeds used to train or test models. Epistemic parameters assesses systematic uncertainty (i.e., missing knowledge due to limited data), which should decreases with more data and more precise instruments. Examples of epistemic parameters as applied to mammography in the systems and methods herein include: (4) spatial resolution of sensors; (5) limited spatial views (typically 4 for 2D mammography and prior visits); (6) Image processing algorithms (presentation view) - different for each vendor; (7) architecture of neural network; (8) labels which are incorrect or missing; (9) rare cases with limited examples in training set; (10) random selection of women based on age, genetics, and breast density; and (11) inherent feature similarities between cancer and benign instances. Even seemingly “perfect” images (i.e., high resolutions images with well defined features) have limited information content. For example, some lesions are highly likely to be cancer (speculated masses), while many other lesions have much lower probability of being cancerous. More images cannot eliminate systematic uncertainty since the information pertaining to cancer versus benign is not available in a single image. Program 3315 can use epistemic parameters, which may reduce uncertainty by considering prior images and carefully comparing right and left views. Otherwise, other modalities (ultra-sound, MRI) - or biopsy are needed to complement the mammograms when program 3315 assesses uncertainty. Epistemic uncertainty associated with examples 4-10 decreases as the quality of images increases, and program 3315 acquires more data. However, uncertainties associated with example 10 may be large and are not possible to eliminate. [0478] Program 3315 uses machine learning techniques during anomaly detection during mammography and quantification of breast arterial calcifications (BAC). Regardless of the location of calcification, any calcification in the artery indicates the presence/onset of artery disease. As such, observation of BAC in mammogram has direct impact on the risk factor for, but not limited to, coronary heart disease (CHD), kidney disease, and stroke. Leading cause of CHD is due to plaque buildup, which can rupture or narrow the coronary artery, regardless of whether the plaque is calcified. Calcification is the last stage of plaque development. BAC may be an indicator of both calcified and non-calcified plaque in the coronary artery. Thus, presence of BAC increases the chance that a plaque will form, or is present, at another location, such as the heart. The use of uncertainty, as determined by program 3315, enables more accurate detection and quantification of BAC which can in turn be used to calculate risk of CHD, stroke, and other diseases. [0479] In Figure 34, program 3315 performs the steps in flowchart 3400. These steps determine a measure of uncertainty/confidence level as an output of the systems and methods herein. [0480] In step 3405, program 3315 generates models in response to receiving the contents of training data set 3325. Techniques A and B are machine learning techniques used by program 3315 to generate models. [0481] Technique A is based on ensemble learning and relies on the creation of several models that are maximally de-correlated but with similar efficacy. In practice, the models have some degree of correlation. At inference, program 3315 runs the models for each instance and generates the corresponding probability distribution function which at the minimum may be the mean and variance. While Bayesian models used to generate probability distribution function are in technique A, the systems and methods herein can also apply to non-Bayesian models. When program 3315 uses technique A, the variance in models are minimized such that the generated score is incorporated into model outputs which is accessible to an end user of program 3315 in an actionable way. [0482] Some non-Bayesian modeling techniques to create different models can be, but not limited to, one or combination of the following: 1. models chosen at different epochs in training; 2. models created based on different neural net architecture; 3. models created by using different initialization of the weights; 4. models created by different ways of combining the predictions; 5. models created by perturbing the trained weights; 6. models created by perturbing the input; 7. models created by using different training sets (e.g., bootstrap aggregating and bagging); and 8. models created by using different test sets (e.g., different cases of normal distribution and bootstrapping techniques for cancer detection). [0483] Program 3315 can also use deep neural networks (DNN) of technique B to generate models. DNN frameworks use point estimate for the weights in every node and also use non-probabilistic loss function. Since an objective of program 3315 is to obtain the output in the form of a probability distribution function in general and estimates of the uncertainty in particular, program 3315 can use the probability distribution over the weights, and/or loss function. The Bayesian neural nets have been partially incorporated into DNN frameworks, such as Pytorch and Tensorflow 2.0. [0484] In step 3410, program 3315 receives data sets in medical information 3330. A score and uncertainty levels of the data sets in medical information 3330 are determined by the models applied by program 3315. [0485] In step 3415, program 3315 applies models on data sets in medical information 3330. The applied models of program 3315 use an assessment protocol to determine a similarity level to known examples of cancer. More specifically, the assessment protocol involves the application of autoencoder or tangent kernel of the classifier on medical information 3330. Stated another way, the assessment protocol establishes a baseline for diagnoses (e.g., protrusions which are deemed as cancerous lesions or benign abnormalities) to aid in the evaluation of medical images. For example, the medical images often may have obscured lesions, which open up the possibility of false positives or false negatives. Program 3315 can compare the incoming data sets from medical information 3330 to the baseline and thereby finding similarities and differences. This is the basis for a similarity level. However, the evaluation does not end with similarities and differences. Based on the assessment protocol, as implemented by generated models, program 3315 can focus on certain factors or ignore other factors to obtain a more granular, comprehensive, and accurate evaluation of the incoming data sets. Thus, the assessment protocol does not end the analysis with a yes or no answer (i.e., binary classification). [0486] In step 3420, program 3315 determines uncertainty estimates for any ensemble classifier. In random forest (used as a binary classifier), each tree classifies the test cases as either belonging to class 0 or class 1. The binary prediction for tree I is xi. The usual output that is further processed is the average of xi, which is the averaging over all trees. This is the fraction of the trees which classified the case as being in class 1, and thus referred to as p. [0487] In step 3420, program 3315 can also calculate the variance of xi. Lower variance means better agreement among individual trees (lower uncertainty). Higher variance means the trees disagree more (high uncertainty). [0488] Generally, program 3315 can simply calculate the variance empirically for each test case. However, for special cases of RF (since each tree is a binary decision), the variance is a function of p where: Var(xi) = p(1-p). [0489] At the extremes p=0 or p=1, the uncertainty is zero since all trees have to agree to reach the extremes. At the midpoint where p=0.5, the uncertainty is a maximum. [0490] Program 3315 determines uncertainty measures via specific use cases when training the model. Thus, program 3315 can be used for binary classification and non-binary classification. For example, the model may be trained for, but not limited to, density classification and biopsy. Breast tissue comprises milk glands, ducts, and dense and non- dense (fatty) breast tissue. In breast mammography, radiologists grade breast based on the density, using the BI-RADS® reporting system. BI-RADS® system is based on the proportion of dense to non-dense tissue, with score of 1 representing almost entirely fatty to 4 being extremely dense. However, there is significant intra-reader and inter-reader variability in assigning a BI-RADS® score. This poses an issue with training neural net models for density classification since unlike biopsy confirmed cancer cases, there is no established convention for the density label. Stated another way, a breast originally assigned a density of 2 by radiologist A may be considered density of 3 by radiologist B. In addition, the images in mammography may have devices, markers, and other artifacts in them. A trained model may exhibit degraded performance if program 3315 is not presented with enough such examples in the training set. [0491] Program 3315 may determine uncertainty such that variability and artifacts are addressed. More specifically, program 3315 consider two uncertainties - aleatoric (irreducible) and epistemic (reducible). While epistemic uncertainty can be reduced with additional data, aleatoric uncertainty is due to inherent variation in the system such as reader dependent variations in the density BI-RADS® score. [0492] In examples of aleatoric uncertainty (fuzzy labels), program 3315 uses a threshold above 99.5 percentile of aleatoric uncertainty to flag cases where the density labels may be: (i) less clear cut and (ii) borderline between neighboring density classes. [0493] In examples of epistemic uncertainty (not having enough representation in the data), program 3315 uses a threshold above 99.5 percentile of epistemic uncertainty to flag cases where either the model does not have enough examples similar to it in the training set and/or cases that should not be included in the training set. [0494] In step 3425, program 3315 implements the solution based on the uncertainty level (as determined in step 3420). Although techniques A and B are used to minimize the variance of the models and improve generalization, program 3315 devises new resulting metrics which are (i) beyond the score that have not been incorporated into the model outputs or (ii) in a way to make it accessible to the user in an actionable way. The incorporation of the probability distribution over the loss function can be implemented. If the probability distribution is over the weights, program 3315 runs the model multiple times to construct the probability distribution function. [0495] In step 3425, program 3315 can use the uncertainty and other metrics of the probability distribution function as listed below. This list is not exhaustive and is meant only as illustration of diverse types of deployment: • Risk assessment - risk mitigation strategies • Lesion detection and classification • Biopsy classifier • Triage - create a separate bucket for cases with high uncertainty. • Self-driving cars • NLP/NLU • Clinical assessment / decision support - care paths • Sensor based system health monitoring – e.g., identification of atrial fibrillation • Targeted retraining of models. [0496] In step 3430, program 3315 determines if the uncertainty level is above a threshold associated with an acceptable level of uncertainty. Accordingly, if the threshold is exceeded, then program 3315 proceeds to retrain the models in step 3430. Stated another way, there may be models generated that do not have a high enough confidence level for program 3315 to make accurate evaluations of medical information 3330. More specifically, cases with high uncertainty levels are sent to a bucket and subsequently retrained. If the threshold is not exceeded, program 3315 displays the output with the score to include other metrics (e.g., uncertainty and confidence level) that become available through techniques A and B in step 3435. This indicates that the generated models have a high enough confidence level for program 3315 to make accurate evaluations of medical information 3330. Otherwise, the models are retrained in step 3440. [0497] Referring to Figure 35, program 3315 generates models using techniques A or B. Based on the models, the images are deemed to have a low uncertainty or high uncertainty. During the training of and generation of the model, program 3315 receives images and accompanying information. Images 315 and 320 are cases of a high score and low score. Image 315 is absent of surface lesions that are clearly visible in image 320. In contrast, images 3505 and 310 are not as straightforward for program 3315 to analyze when training the model. Image 3505 is deemed to have a high uncertainty, whereas image 3510 is deemed to have low uncertainty. Program 3315 can ascertain that: (i) images 315 and 320 have elements which decrease the probability of accurate and precise binary classification regions by noticing that: (a) image 3505 appears obscure whereas (b) none of the regions of image 3510 appear obscure; and (ii) images 315 and 320 are absent of elements which decrease the probability of accurate and precise binary classification. [0498] Referring to Figures 36A-D, program 3315 can identify cases with high uncertainty of one type, which had a low uncertainty of the other type. This further validates the utility of these two types of uncertainty and demonstrates that the two uncertainty measures are distinct, while measuring different types of uncertainty. If the two uncertainties been highly correlated, program 3315 may have not been able distinguish the “fuzziness” of the labels in the training set from not having had sufficient examples of a particular type of image in the training set. [0499] Figures 36A-D depicts examples of cases where the model predicted low uncertainty and examples with high uncertainty are depicted. Cases with high uncertainty are seen to be cases where the cancer is not as clearly evident in the image. One possible application of this new information about the model uncertainty is that one can devise a separate protocol for flagged cases where the model has high uncertainty (e.g., step 3425). For example, such cases may indicate difficult cases that may need additional work ups and can be queued for ordering follow up diagnostic studies. [0500] Figure 36A shows an example from the training set where the original radiologist marked the image as density 2 while the model scored the image as density 3. Normally this case would count as a false positive by the model. Program 3315 accounts for review A of a panel of radiologists of image 3600A, whereby the consensus of review A is: (a) 50% confidence for a density of 2 and (b) 50% confidence for a density of 3. Program 3315 applies the protocol, which accounts for review A of the panel, the model score, and original radiologist marking, on image 3500 and thus deemed as having high aleatoric uncertainty (99.51 percentile). Stated another way, program 3315 successfully flags image 3600A as a borderline case, i.e., a case with high uncertainty, based on the scores of the original radiologist, model, and panel. [0501] Figure 36B shows an example from the training set where the original radiologist marked image 3600B as a density of 3. However, program 3315 scores image 3600B with a density of 2. Normally, this case is counted as a false positive by the model. Program 3315 accounts for review B of a panel of radiologists of image 3600B, whereby the consensus of review B is classified with 100% confidence for a density of 2. Program 3315 applies the protocol, which accounts for review B of the panel, the model score, the original radiologist marking, on image 3600B and thus deemed as having high aleatoric uncertainty in this case (99.84 percentile). Stated another way, program 3315 successfully flags image 3600B as the wrong label. Accordingly, program 3315 identifies wrongly labeled cases and subsequently corrects the label or removes the wrongly labeled cases from the training set. This improves the efficacy of the model and subsequently implemented protocol. In a specific instance, the subsequently implemented protocol by program 3315 leads to improvement of the 4 class kappa from 0.82 to 0.85 and binary kappa from 0.92 to 0.96. [0502] Figure 36C depicts a nonstandard, magnification diagnostic view in image 3600C. Due to the high epistemic uncertainty of image 3600C, program 3315 is flagged. The protocol, as applied on image 3600C, program 3315 determines image 3600C is overly saturated. Such cases can be removed from the training set. Another application of this uncertainty is in running the model live at a client’s site. Density classification by radiologists is based on examination of all of the screening images and then assigning one breast density score to the patient. The model can consider all images of a patient and form a consensus breast density score by discarding images that have high epistemic uncertainty. Thus, program 3315 excludes images, such as image 3600C, that should not be considered in the density classification. [0503] Figure 36D depicts another example in the training set which has an embedded artifact in image 3600D. Program 3315 analyzes image 3600D and subsequently deemed as having high epistemic uncertainty. Program 3315 flags image 3600D as the implemented protocol notes there are very few of such images in the training set. In such instances, program 3315 either: (i) finds similar examples to add to the training set, or (ii) eliminates the case from the training set. [0504] Referring to Figures 36E and 36F, from aleatoric and epistemic uncertainty from a crop-level CNN training model for lesions program 3315 can leverage uncertainty to improve the training of the models. High aleatoric uncertainty indicates cases where the model is not sure about the classification score since there are other examples in the data set that appear similar but have an opposite class. High epistemic uncertainty indicates cases where the model has not seen sufficient representation/examples in the training set. Initially, program 3315 examines cases in the training set with high aleatoric uncertainty. [0505] Figures 36E and 36F show two cases in the training set with high aleatoric uncertainty. Figure 36E is a positive class that is misclassified as a negative class by the trained model, whereas Figure 36F panel is a negative class that is correctly classified by the trained model. The visual similarities of the two crop level images in Figures 36E and 36F, one normal and one cancer, is indicative of the difficulty in distinguishing the correct class with high confidence for these two crops. Assessment of the suspiciousness of the lesions by radiologists relies on the examination of the entire breast and comparing all four views in screening setting in mammography. Assessment of suspiciousness of the lesion is significantly hampered for a human reader at crop level without the context of the entire breast. The tighter the crop, the less of a context the radiologist has to interpret the crop which reduces the probability of correctly classifying the crop. To test whether this is also true for the model, and as a remedy, program 3315 extracts the same lesion but at a crop size that captures more of the surrounding breast area. [0506] Referring to Figure 37, the same crop is determined as the class 1 case in Figure 36E, except Figure 37 has a larger region of the surrounding breast for context. This additional context enables the model, as generated by program 3315, to: (i) get the correct class in contrast to the model prediction of the wrong class in Figure 36E, while (ii) significantly reducing aleatoric uncertainty also from 98.42 percentile to 81.83 percentile. This uncertainty can be further reduced if the models are trained by program 3315 with crops at different crop size levels. This improves the training, whereby program 3315 creates crops at different crop size levels and combines the scores of the crops, such methods such as majority voting or averaging the scores. This approach can be performed even without retraining the model. Another way of improving the model is to use different crop size levels as part of data augmentation. At the test stage, program 3315 can run the model for different crop sizes and then creates a final score through such methods, but not limited to, majority voting or averaging the scores. Another way to improve the models, program 3315 can feed crops at different crop size levels to a multi-scale convolution neural net. The epistemic uncertainty has increased in Figure 37 as compared to that in Figure 36E. This is attributed to the fact that the model was trained on tight crops and it has not seen many examples that contain as much surrounding breast tissue. [0507] Referring to Figure 38, the effect of program 3315 trained model on the crop in Figure 36F with class 0 but with the inclusion of a larger surrounding breast tissue in the crop is depicted. The aleatoric percentile is significantly reduced from 99.06 percentile to 35.95 percentile. Stated another way, the confidence level in score of the model is higher since it has more context for assessing the suspiciousness of the lesion. The epistemic percentile has increased, as expected since the model is trained on tight crops and the high epistemic uncertainty points to the fact that there are not many cases in the training set with such large segment of the surrounding tissue around a lesion like structure. [0508] Referring to Figures 39A-C, several cases with high epistemic uncertainty in the training set are depicted. These are cases that despite providing the label in the training set, the model has difficulty learning the correct classification. Figures 39A-C are very unusual cases with scant representation in the training set. The high epistemic uncertainty indicates the need to include more such examples for the training set to improve the model. In Figure 39A, the presence of large calcifications gives the impression of an oval mass with embedded calcifications. In Figure 39B, there is a region of enhanced density with calcified breast artery. In Figure 39C, there are surgical clips. [0509] Referring to Figure 40, program 3315 increases the crop size to include more of the surrounding breast tissue, and this results in: (i) a decreases the aleatoric uncertainty from 58.19 to 30.14 percentile; and (ii) a match between the predicted class and the actual mass. This is consistent with program 3315 improving the performance of the model when providing more context, as described with respect to Figures 36E and 36F. However, the epistemic uncertainty has slightly increased from 97.37 to 99.92 percentile. This is expected since the larger cropped image includes captures an additional surgical clip as well as the device (the white bar at the bottom of the image) used for magnification view in mammography. There are even fewer such cases in the training set that include both surgical markers and the device, which further explains the higher epistemic uncertainty. [0510] Referring to Figures 41A-C, program 3315 can use crops of cancer and normal/benign images extracted from 2D mammography to train a CNN one-class classifier. Program 3315 has also used the same images to train a Bayesian NN. The advantage of the Bayesian NN in this case is that one gets a prediction of the uncertainty. Figures 41A-C show the efficacy of the Bayesian NN for (a) including all cases independent of their uncertainty; and (b) including only cases where the model is confident about its classification score. Stated another way, cases with high uncertainty are not included in the evaluation of medical information 3330. Thus, the model efficacy is improved when cases with high uncertainty are flagged and not included in the evaluation. [0511] Three ROCs are shown where the change in AUC is compared to the full set of data as program 3315 filters out cases based on uncertainty values. Different measures of uncertainty are applied by program 3315. In Graph 4100A, the ROC is under a 40-percent cutoff where program 3315 keeps 40% of the cases for each uncertainty measure (each representing a separate curve). In Graph 4100B, the ROC is under a 60-percent quantile cutoff where program 3315 keeps 60% of the cases for each uncertainty measure (each representing a separate curve). In Graph 4100C, the ROC is under an 80-percent cutoff where program 3315 keeps 80% of the cases for each uncertainty measure (each representing a separate curve). For each of Graphs 4100A-4100C, 5694 total cases are analyzed without program 3315 and 5290 cases are normal and 404 cases are normal, which results in an area under the curve (AUC) of 0.824. Uncertainty is not accounted for when the AUC of 0.824 is obtained. [0512] Different types of uncertainties are utilized by program 3315: entropy, classification sigma, classification probability sigma, quantification of aleatoric, and epistemic uncertainty. Classification probability sigma refers to the sample standard deviation of classification probability score inferred by the model and is denoted as sigma in Figures 41A, 41B, and 41C. It is also possible to convert classification probability to threshold-based classification. As an example, using a threshold of 0.5 for binary classification, program 3315 can convert classification probability to threshold-based classification and then acquire the corresponding sample standard deviation, referred to as sigma_0_1 in Figures 41A, 41B and 41C. After obtaining different uncertainty measures across all the images in a study, a ranking for different types of uncertainty measures can be acquired within each uncertainty estimation, respectively. For example, the ranking of entropy can be computed across all the images. A “quantile filter” can then be deployed based on the ranking of specific uncertainty to enhance the performance of program 3315. When an X quantile filter is applied for a specific kind of uncertainty, program 3315 will exclude images with this specific kind of uncertainty above X∙100 percentile. For example, a 0.4 quantile filter on entropy will let program 3315 exclude images with entropy values above the 40 percentile. Figure 41A, which shows application to detection of suspicious mass in breast mammography, demonstrates that when program 3315 applies 0.4 quantile filter based on the 5 types of uncertainty separately, entropy, sigma_0_1, sigma, aleatoric uncertainty, and epistemic uncertainty, the cutoffs for each uncertainty measure are 0.65, 0.32, 0.12, 0.21, and 0.21, respectively. This results in AUCs of 0.896, 0.910, 0.905, 0.887, and 0.905, respectively which are all higher than the AUC of 0.824 for the inference on the full set of images. Note that different numbers of cancer cases are eliminated for each uncertainty type as indicated in the figure caption. Figure 41B shows the resulting AUCs of AUC of 0.878, 0.881, 0.886, 0.866, and 0.866 if the quantile cut off is set to 0.6 which is still higher than the AUC of 0.824 for the full set but somewhat lower than the case when the quantile cutoff is set to 0.4. Figure 41C demonstrates that when program 3315 applies 0.8 quantile filter on the 5 types of uncertainty separately with the resulting AUCs of 0.850, 0.846, 0.843, 0.850, 0.843, respectively. The improvement in AUC over the full set of images is still significant but is the smallest among all three quantile filter. However, the number of cancer cases eliminated is also smaller than the other two quantiles. Thus program 3315 demonstrates the viability of all 5 types of uncertainties in enhancing the efficacy of the model as measured by the AUC. The lower the quantile filter, the larger the improvement in the AUC. Thus, program 3315 measures uncertainty is most useful for particular applications where the goal is to have the highest AUC while removing as few cases as possible. [0513] EXAMPLES [0514] Aspects of the present teachings may be further understood in light of the following examples, which should not be construed as limiting the scope of the present teachings in any way. [0515] Example 1 [0516] In the first example, the systems and methods herein present a case in medical imaging where a prior from certain probabilistic continuous distribution is put on the weights of the traditional pointwise neural network model and thus make it a variational dense layer. A fully Bayesian convolutional network may also be possible to deploy but can be more difficult to train. While the full model pipeline for computer assisted design (CAD) in medical imaging can consist of several neural networks including an object detector network, the final output of CAD is currently a score. The focus here is to show how changing the model output from score (mean) to one where both the mean as well as the standard deviation (i.e., an uncertainty level) is provided improves the results. Program 3315 can also explain how this additional information can be useful for the user (e.g., radiologist using device 3305). As such, program 3315 is not limited to CNN here is not limited to CNN. Thus, the systems and methods herein are generally applicable to all CAD applications and can be incorporated into all CAD pipelines. [0517] Example 2 [0518] In the second example, program 3315 uses an object detection neural network to assess the uncertainty of the model prediction. Unlike the previous example where the weights of the neural net are Bayesian, here program 3315 uses a standard faster RCNN object detection network with pointwise weights where the loss function is replaced with the quantile regression loss function. In this implementation, program 3315 creates different models from those of technique A and B by changing the alpha parameter which refers to the desired quantile. A quantile of 0.5 is equivalent to the median which is value obtained in the standard approach, based on minimizing the Mean Absolute Error. For example, Figures 9A- 9C shows the ROC for three values of alpha. The result shows the uncertainty of the model in different parts of the ROC. [0519] Example 3 [0520] In the third example, program 3315 supports a triage system connected to a Picture Archiving Retrieval System. It is impractical for a radiologist or program 3315 to review each case in the training set, which can consist of tens or hundreds of thousands or even more images. To further complicate this issue, density classification by radiologists is done on standard 4 view screening images. DICOM tags usually, but not always, identify nonstandard views including magnification views where devices appear in the images. There can also be images with biopsy clips, pacemaker, and other artifacts that can confuse the model unless there are a sufficient number of examples in the training set. Program 3315 uses: (i) epistemic uncertainty to enable identification of all of the above cases; and (ii) retraining buckets (as described above). Thus program 3315 may identify which: (i) images are indicative of abnormal medical conditions requiring immediate medical care (i.e., low uncertainty); (ii) images that require further analysis (i.e., high uncertainty); and (iii) retraining. [0521] Example 4 [0522] In the fourth example, an anomaly in or on a tissue surface can be analyzed using program 3315, which provides a suspicion code. That code can, in real-time, populate a user interface to alert the radiologist or technician of a suspicious anomaly in or on the tissue surface, thereby leading to an auto-populating setting. In the auto-populating setting, a notification result file may be reported and thus generated to include the suspicion code. The notification result file can be in the form of a portable document format (PDF), DICOM® SR, JSON, HTTP, HTTPS, HTML, REST, or other communication format in which a “Suspicious” or “” (blank) code may be transmitted in a FFDM, CIS, RIS, HIS, or the like. [0523] Figure 42 is a flow chart of a method for determining a clinical score. The clinical score may be generated or determined by a clinical score engine (such as the cancer score engine) in a system. In general, the method 420 may involve processing and analyzing one or more pieces of medical information for generating, for one or more regions of the piece of tissue, an indication of a clinical indication (such as a type of cancer). For example, as described in more detail below, the method 4200 may gather variables that are pertinent to the clinical indication, programmatically analyze the piece(s) of medical information to determine values of the variables, transform these variables for use in generating a clinical indication or score, and then generate an indication of the clinical indication based on these variables or a transformation of these variables. A clinical score generator component may receive the indications of the clinical indication in the one or more regions of the tissue and generate a clinical score for at least one region of the tissue. For example, the clinical score may have a value that increases as a cancer tumor grows and decreases as a cancer tumor shrinks. In some embodiments, the clinical score may be normalized and have threshold levels so that, for example, a normalized clinical score of 1–3 indicates a benign tumor, a normalized clinical score of 4–6 indicates suspicious cells, and a normalized clinical score of 7–10 indicates that cancer is present in particular region(s) of tissue. [0524] As shown in step 4202, the method 4200 may include receiving one or more pieces of medical information for processing and analysis. The medical information may include information about a patient’s tissue, e.g., medical images of the tissue. The medical information may include preprocessed or raw data, which is then processed and analyzed by the systems or methods described herein. In an aspect, the clinical score engine may include a medical information analysis component that receives one or more pieces of medical information, where the clinical score engine then processes and analyzes this information. The medical information may be automatically streamed to the clinical score engine by an uneven length preprocessed time series input. For example, the header of a DICOM file may contain information on the image contained within it including, but not limited to, the pixel resolution in physical units, criteria for interpreting the pixel intensity, etc. [0525] As shown in step 4204, the method 4200 may include analyzing the one or more pieces of medical information about the tissue. This may include gathering variables values about the medical information (e.g., a mammogram), where generating the indication of the clinical indication may be based on the gathered variable values. The variables may include an intensity value for contours of instances of a type of tissue, a gradient of the instances of the type of tissue, one or more characteristics about each of the instances of the type of tissue, and a hierarchical structure of the instances of the type of tissue in a cluster. [0526] As shown in step 4206, the method 4200 may include generating an indication of the clinical indication (such as a biomarker). By way of example, the indication of the clinical indication may be generated for one or more regions of the tissue in the medical images. [0527] As shown in step 4208, the method 4200 may include generating a clinical quantification score. By way of example, generating a clinical quantification score may include generating a clinical score for each region of the tissue based on the indication of the clinical indication in each region of the tissue. The clinical quantification score may indicate an absence of the clinical indication in the region of the tissue. [0528] As shown in step 4210, the method 4200 may include generating guidance for a medical professional (such as a physician or a technician performing non-invasive medical imaging) based on one or more of the indications of the clinical indication and the clinical quantification score. The guidance may include, e.g., guidance for a radiologist based on the presence or absence of the clinical indication in the region of the tissue. The guidance may be generated by applying rules based on the analysis of the medical information, the indication of the clinical indication, or the clinical quantification score. [0529] Implementations may utilize one or more algorithms for detecting and quantifying the clinical from medical information supplied to the system. For example, for detecting and quantifying the clinical indication, the algorithm may detect and quantify micro-instances of the type of tissue (such as calcifications) in mammogram images. The algorithm may in general include (1) detecting and grouping the instances of the type of tissue into clusters, (2) classifying types of benign clusters, (3) quantifying clusters that are potentially instances of the clinical indication with a ‘Q factor’ as discussed herein, and (4) saving output quantities to evaluate performance. In an implementation, a first algorithm generates an indication of the clinical indication and a second algorithm generates a clinical score. [0530] In some embodiments, a system than implements method 4200 may receive a medical image associated with a non-invasive medical imaging technique. Then, the system may: analyze the medical image, generate a clinical score for the medical image; and transmit an instruction to a measurement device to a acquire a second medical image associated with a second non-invasive medical imaging technique based at least in part on the clinical score. [0531] Note that the analysis of the medical image may include: determining contours in the medical image satisfying one or more criterion; analyzing one or more geometric attributes and one or more contrast attributes of contours included in the first subset of contours to identify a second subset of contours based at least in part on the contours satisfying one or more predetermined geometric and contrast attributes; selecting a third subset of contours from the second subset of contours that corresponds to one or more potential instances of a type of tissue, the third subset selected based at least in part on contours within the third subset satisfying first criteria associated with the type of tissue; ranking contours included in the third subset of contours based at least in part on a selection metric, the selection metric accounting for a combination of contrast and intensity; grouping the contours included in the third subset of contours into nested structures; selecting one or more instances of the type of tissue from the nested structures satisfying second criteria associated with the type of tissue; grouping the selected one or more instances of the type of tissue into clusters based on neighboring instances of the type of tissue and a spatial cluster scale; classifying the clusters as benign or possibly associated with the clinical indication by performing one or more of: a regression analysis on the one or more instances of the type of tissue within the clusters, edge detection, a density analysis of the clusters, and/or a circularity analysis of the clusters; scoring the clusters using an analytic function to generate the clinical score; and combining physical features of each of the one or more instances of the type of tissue, as an individual structure and as part of a cluster of micro-instances of the type of tissue, together with clinical data, to construct a random-forest model and provide a scale of suspiciousness for the clinical indication. [0532] Moreover, subsequently the system may: receive a second medical image associated with a second non-invasive medical imaging technique; and revise the scale of suspiciousness for the clinical indication with the processor based at least in part on the second medical image. For example, the system may reduce a false positive rate for detecting the clinical indication by using information provided by different non-invasive imaging techniques. Notably, when a suspected cyst is detected in a mammogram medical image, additional information in an ultrasound medical image of the suspected cyst may be used to refine the assessment of the suspected cyst, [0533] Alternatively, in some embodiments, where the medical image is acquired using real-time ultrasound, the second medical image may provide additional information that corrects for artifacts associated with a quality of the medical image (such as operator motion or patient motion) and/or improves the clinical score. Thus, in some embodiments, the non- invasive medical imaging techniques may be different or may be the same non-invasive medical imaging technique. [0534] In some embodiments, the system does not provide the instruction to the measurement device. Instead, the medical image and the second medical image are acquired concurrently by the measurement device, and then are provided to the system. [0535] Note that instead of or in addition to detecting the clinical indication, in other embodiments the system diagnoses the clinical indication. For example, the system may compute a classification associated with the clinical indication based at least in part on the medical image and the second medical image. [0536] In some embodiments of any of the aforementioned methods, there may be additional or fewer steps or operations. Further, one or more different operations may be included. Moreover, the order of the operations may be changed, and/or two or more operations may be combined into a single operation or performed at least partially in parallel. [0537] The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. [0538] The systems and methods disclosed herein may be implemented via one or more components, systems, servers, appliances, other subcomponents, or distributed between such elements. When implemented as a system, such systems may include an/or involve, inter alia, components such as software modules, general-purpose CPU, GPUs, RAM, etc., found in general-purpose computers. In implementations where the innovations reside on a server, such a server may include or involve components such as CPU, GPUs, RAM, etc., such as those found in general-purpose computers. [0539] Additionally, the systems and methods herein may be achieved via implementations with disparate or entirely different software, hardware and/or firmware components, beyond that set forth above. With regard to such other components (e.g., software, processing components, etc.) and/or computer-readable media associated with or embodying the present implementations, for example, aspects of the innovations herein may be implemented consistent with numerous general purpose or special purpose computing systems or configurations. Various exemplary computing systems, environments, and/or configurations that may be suitable for use with the innovations herein may include, but are not limited to: software or other components within or embodied on personal computers, servers or server computing devices such as routing/connectivity components, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, consumer electronic devices, network PCs, other existing computer platforms, distributed computing environments that include one or more of the above systems or devices, etc. [0540] In some instances, aspects of the systems and methods may be achieved via or performed by logic and/or logic instructions including program modules, executed in association with such components or circuitry, for example. In general, program modules may include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular instructions herein. The embodiments may also be practiced in the context of distributed software, computer, or circuit settings where circuitry is connected via communication buses, circuitry or links. In distributed settings, control/instructions may occur from both local and remote computer storage media including memory storage devices. [0541] The software, circuitry and components herein may also include and/or utilize one or more type of computer readable media. Computer readable media can be any available media that is resident on, associable with, or can be accessed by such circuits and/or computing components. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and can accessed by computing component. Communication media may comprise computer readable instructions, data structures, program modules and/or other components. Further, communication media may include wired media such as a wired network or direct-wired connection, where media of any type herein does not include transitory media. Combinations of the any of the above are also included within the scope of computer readable media. [0542] In the present description, the terms component, module, device, etc. may refer to any type of logical or functional software elements, circuits, blocks and/or processes that may be implemented in a variety of ways. For example, the functions of various circuits and/or blocks can be combined with one another into any other number of modules. Each module may even be implemented as a software program stored on a tangible memory (e.g., random access memory, read only memory, CD-ROM memory, hard disk drive, etc.) to be read by a central processing unit to implement the functions of the innovations herein. Or, the modules can comprise programming instructions transmitted to a general-purpose computer or to processing/graphics hardware via a transmission carrier wave. Also, the modules can be implemented as hardware logic circuitry implementing the functions encompassed by the innovations herein. Finally, the modules can be implemented using special purpose instructions (SIMD instructions), field programmable logic arrays or any mix thereof which provides the desired level performance and cost. [0543] As disclosed herein, features consistent with the disclosure may be implemented via computer-hardware, software and/or firmware. For example, the systems and methods disclosed herein may be embodied in various forms including, for example, a data processor, such as a computer that also includes a database, digital electronic circuitry, firmware, software, or in combinations of them. Further, while some of the disclosed implementations describe specific hardware components, systems and methods consistent with the innovations herein may be implemented with any combination of hardware, software and/or firmware. Moreover, the above-noted features and other aspects and principles of the innovations herein may be implemented in various environments. Such environments and related applications may be specially constructed for performing the various routines, processes and/or operations according to the implementations described herein or they may include a general-purpose computer or computing platform selectively activated or reconfigured by code to provide the necessary functionality. The processes disclosed herein are not inherently related to any particular computer, network, architecture, environment, or other apparatus, and may be implemented by a suitable combination of hardware, software, and/or firmware. For example, various general-purpose machines may be used with programs written in accordance with teachings of the implementations herein, or it may be more convenient to construct a specialized apparatus or system to perform the required methods and techniques. [0544] Aspects of the method and system described herein, such as the logic, may also be implemented as functionality programmed into any of a variety of circuitry, including programmable logic devices ("PLDs"), such as field programmable gate arrays ("FPGAs"), programmable array logic ("PAL") devices, electrically programmable logic and memory devices and standard cell-based devices, as well as application specific integrated circuits. Some other possibilities for implementing aspects include: memory devices, microcontrollers with memory (such as EEPROM), embedded microprocessors, firmware, software, etc. Furthermore, aspects may be embodied in microprocessors having software-based circuit emulation, discrete logic (sequential and combinatorial), custom devices, fuzzy (neural) logic, quantum devices, and hybrids of any of the above device types. The underlying device technologies may be provided in a variety of component types, e.g., metal-oxide semiconductor field-effect transistor ("MOSFET") technologies like complementary metal- oxide semiconductor ("CMOS"), bipolar technologies like emitter-coupled logic ("ECL"), polymer technologies (e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures), mixed analog and digital, and so on. [0545] It should also be noted that the various logic and/or functions disclosed herein may be enabled using any number of combinations of hardware, firmware, and/or as data and/or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and/or other characteristics. Computer-readable media in which such formatted data and/or instructions may be embodied include, but are not limited to, non-volatile storage media in various forms (e.g., optical, magnetic or semiconductor storage media) though again does not include transitory media. Unless the context clearly requires otherwise, throughout the description, the words "comprise," "comprising," and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in a sense of "including, but not limited to." Additionally, the words "herein," "hereunder," "above," "below," and words of similar import refer to this application as a whole and not to any particular portions of this application. [0546] Moreover, the above systems, devices, methods, processes, and the like may be realized in hardware, software, or any combination of these suitable for a particular application. The hardware may include a general-purpose computer and/or dedicated computing device. This includes realization in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices or processing circuitry, along with internal and/or external memory. This may also, or instead, include one or more application specific integrated circuits, programmable gate arrays, programmable array logic components, or any other device or devices that may be configured to process electronic signals. It will further be appreciated that a realization of the processes or devices described above may include computer-executable code created using a structured programming language such as C, an object oriented programming language such as C++, or any other high-level or low-level programming language (including assembly languages, hardware description languages, and database programming languages and technologies) that may be stored, compiled or interpreted to run on one of the above devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software. In another aspect, the methods may be embodied in systems that perform the steps thereof, and may be distributed across devices in a number of ways. At the same time, processing may be distributed across devices such as the various systems described above, or all of the functionality may be integrated into a dedicated, standalone device or other hardware. In another aspect, means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure. [0547] Embodiments disclosed herein may include computer program products comprising computer-executable code or computer-usable code that, when executing on one or more computing devices, performs any and/or all of the steps thereof. The code may be stored in a non-transitory fashion in a computer memory, which may be a memory from which the program executes (such as random access memory associated with a processor), or a storage device such as a disk drive, flash memory or any other optical, electromagnetic, magnetic, infrared or other device or combination of devices. In another aspect, any of the systems and methods described above may be embodied in any suitable transmission or propagation medium carrying computer-executable code and/or any inputs or outputs from same. [0548] It will be appreciated that the devices, systems, and methods described above are set forth by way of example and not of limitation. Absent an explicit indication to the contrary, the disclosed steps may be modified, supplemented, omitted, and/or re-ordered without departing from the scope of this disclosure. Numerous variations, additions, omissions, and other modifications will be apparent to one of ordinary skill in the art. In addition, the order or presentation of method steps in the description and drawings above is not intended to require this order of performing the recited steps unless a particular order is expressly required or otherwise clear from the context. [0549] The method steps of the implementations described herein are intended to include any suitable method of causing such method steps to be performed, consistent with the patentability of the following claims, unless a different meaning is expressly provided or otherwise clear from the context. So for example performing the step of X includes any suitable method for causing another party such as a remote user, a remote processing resource (e.g., a server or cloud computer) or a machine to perform the step of X. Similarly, performing steps X, Y and Z may include any method of directing or controlling any combination of such other individuals or resources to perform steps X, Y and Z to obtain the benefit of such steps. Thus method steps of the implementations described herein are intended to include any suitable method of causing one or more other parties or entities to perform the steps, consistent with the patentability of the following claims, unless a different meaning is expressly provided or otherwise clear from the context. Such parties or entities need not be under the direction or control of any other party or entity, and need not be located within a particular jurisdiction. [0550] It should further be appreciated that the methods above are provided by way of example. Absent an explicit indication to the contrary, the disclosed steps may be modified, supplemented, omitted, and/or re-ordered without departing from the scope of this disclosure. [0551] It will be appreciated that the methods and systems described above are set forth by way of example and not of limitation. Numerous variations, additions, omissions, and other modifications will be apparent to one of ordinary skill in the art. In addition, the order or presentation of method steps in the description and drawings above is not intended to require this order of performing the recited steps unless a particular order is expressly required or otherwise clear from the context. Thus, while particular embodiments have been shown and described, it will be apparent to those skilled in the art that various changes and modifications in form and details may be made therein without departing from the spirit and scope of this disclosure and are intended to form a part of the invention as defined by the following claims, which are to be interpreted in the broadest sense allowable by law. [0552] References Cited [0553] All publications, patents, patent applications and other references cited in this application are incorporated herein by reference in their entirety for all purposes to the same extent as if each individual publication, patent, patent application or other reference was specifically and individually indicated to be incorporated by reference in its entirety for all purposes. Citation of a reference herein shall not be construed as an admission that such is prior art to the present invention. [0554] Allison, M. A., Criqui, M. H., & Wright, C. M. (2004). Patterns and risk factors for systemic calcified atherosclerosis. Arteriosclerosis, thrombosis, and vascular biology, 24(2), 331-336. [0555] Divakaran, S., Cheezum, M. K., Hulten, E. A., Bittencourt, M. S., Silverman, M. G., Nasir, K., & Blankstein, R. (2015). Use of cardiac CT and calcium scoring for detecting coronary plaque: implications on prognosis and patient management. The British journal of radiology, 88(1046), 20140594. [0556] Garcia, M., Mulvagh, S. L., Merz, C. N. B., Buring, J. E., & Manson, J. E. (2016). Cardiovascular disease in women: clinical perspectives. Circulation research, 118(8), 1273-1293. [0557] Go, A. S., Chertow, G. M., Fan, D., McCulloch, C. E., & Hsu, C. Y. (2004). Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization. New England Journal of Medicine, 351(13), 1296-1305. [0558] Hecht, H., Blaha, M. J., Berman, D. S., Nasir, K., Budoff, M., Leipsic, J., ... & Shaw, L. J. (2017). Clinical indications for coronary artery calcium scoring in asymptomatic patients: expert consensus statement from the Society of Cardiovascular Computed Tomography. Journal of cardiovascular computed tomography, 11(2), 157-168. [0559] Hong, Y. M. (2010). Atherosclerotic cardiovascular disease beginning in childhood. Korean Circulation .Journal, 40(1), 1-9. [0560] Insull, W. (2009). The pathology of atherosclerosis: plaque development and plaque responses to medical treatment. The American journal of medicine, 122(1), S3-S14. [0561] Kohli, P., Whelton, S. P., Hsu, S., Yancy, C. W., Stone, N. J., Chrispin, J., & Joshi, P. H. (2014). Clinician's guide to the updated ABCs of cardiovascular disease prevention. Journal of the American Heart Association, 3(5), e001098. [0562] Neves, P. O., Andrade, J., & Monção, H. (2017). Coronary artery calcium score: current status. Radiologia brasileira, 50(3), 182-189. [0563] Schiffrin, E. L., Lipman, M. L., & Mann, J. F. (2007). Chronic kidney disease: effects on the cardiovascular system. Circulation, 116(1), 85-97. [0564] World Health Organization (2017). Cardiovascular diseases (CVDs). Fact sheet. Updated May, 2017. [0565] Zaman, A. G., Helft, G., Worthley, S. G., & Badimon, J. J. (2000). The role of plaque rupture and thrombosis in coronary artery disease. Atherosclerosis, 149(2), 251- 266.

Claims

What is claimed is: 1. A computer-implemented method for determining a risk of a clinical indication, comprising: receiving medical information associated with an individual comprising a medical image associated with a non-invasive medical imaging technique through a network interface of a computing device over a data network; analyzing the medical image with a processor of the computing device; generating a clinical score for at least a portion of the medical image; and transmitting an instruction to a measurement device to a acquire a second medical image associated with a second non-invasive medical imaging technique based at least in part on the clinical score, wherein the analysis of the medical image comprises: determining contours in the medical image satisfying one or more criterion; analyzing one or more geometric attributes and one or more contrast attributes of contours included in the first subset of contours to identify a second subset of contours based at least in part on the contours satisfying one or more predetermined geometric and contrast attributes; selecting a third subset of contours from the second subset of contours that corresponds to one or more potential instances of a type of tissue, the third subset selected based at least in part on contours within the third subset satisfying first criteria associated with the type of tissue; ranking contours included in the third subset of contours based at least in part on a selection metric, the selection metric accounting for a combination of contrast and intensity; grouping the contours included in the third subset of contours into nested structures; selecting one or more instances of the type of tissue from the nested structures satisfying second criteria associated with the type of tissue; grouping the selected one or more instances of the type of tissue into clusters based on neighboring instances of the type of tissue and a spatial cluster scale; classifying the clusters as benign or possibly associated with the clinical indication by performing one or more of: a regression analysis on the one or more instances of the type of tissue within the clusters, edge detection, a density analysis of the clusters, or a circularity analysis of the clusters; scoring the clusters using an analytic function to generate the clinical score; and combining physical features of each of the one or more instances of the type of tissue, as an individual structure and as part of a cluster of micro-instances of the type of tissue, together with clinical data, to construct a predictive model and provide a scale of suspiciousness for the clinical indication; receiving a second medical image associated with the second non-invasive medical imaging technique through the network interface of the computing device over the data network; and revising the scale of suspiciousness for the clinical indication with the processor based at least in part on the second medical image.
2. The computer-implemented method of claim 1, wherein the clinical indication comprises a type of cancer
3. The computer-implemented method of claim 2, wherein the type of cancer comprises: neurological cancer, lung cancer, prostate cancer or breast cancer.
4. The computer-implemented method of claim 1, wherein the type of tissue comprises calcifications.
5. The computer-implemented method of claim 1, wherein the medical image comprises one or more of an x-ray image or a computerized tomography (CT) scan; and wherein the second medical image comprises one or more of a magnetic resonance (MRI) image or an ultrasound image.
6. The computer-implemented method of claim 1, wherein the instructions are provided while non-invasive imaging of patient is being performed or while the patient is available for the second medical image to be acquired following acquisition of the medical image.
7. The computer-implemented method of claim 1 further comprising extracting, with the processor, tagged data from the medical image, wherein the medical image is included in a computer file.
8. The computer-implemented method of claim 7, wherein the tagged data comprises one or more of: a side, a pixel spacing, an orientation, a protocol, or a date.
9. The computer-implemented method of claim 7, wherein the tagged data is included in a Digital Imaging and Communications in Medicine (DICOM) header.
10. The computer-implemented method of claim 1 further comprising converting, with the processor, the medical image to a real array of intensities for contouring.
11. The computer-implemented method of claim 1 further comprising selecting, with the processor, intensity levels for determining contours in the medical image.
12. The computer-implemented method of claim 1, wherein the one or more criterion comprises that each contour in the first subset of contours is (i) closed and (ii) includes a contour value larger than a surrounding area external to the contour.
13. The computer-implemented method of claim 1, wherein contours not satisfying the one or more criterion are discarded.
14. The computer-implemented method of claim 1, wherein the one or more geometric attributes of contours comprise at least one of: a centroid, an area, a perimeter, a circle ratio, or an interior flag.
15. The computer-implemented method of claim 1, wherein the one or more contrast attributes of contours comprise at least one of: an intensity, an inward contrast, an outward contrast, or a gradient scale.
16. The computer-implemented method of claim 1 further comprising detecting, with the processor, an object in the image for exclusion from further analysis.
17. The computer-implemented method of claim 16, wherein the object is an external object or a foreign object.
18. The computer-implemented method of claim 17, wherein the object is detected through the object having at least one of: an area greater than a predetermined area, an intensity greater than a predetermined intensity, or a circle ratio greater than a predetermined circle ratio.
19. The computer-implemented method of claim 1, wherein selecting the third subset of contours comprises excluding contours located within a predetermined distance from at least one of an edge of the medical image and an edge of tissue.
20. The computer-implemented method of claim 1, wherein the first criteria comprise contours having a predetermined area and a predetermined gradient scale.
21. The computer-implemented method of claim 20, wherein the predetermined area is between 0.003 mm2 and 800 mm2 and the predetermined gradient scale is less than 1.3 mm.
22. The computer-implemented method of claim 1, wherein the first criteria comprise contours having a predetermined intensity, a predetermined circle ratio, a predetermined inward contrast, or a predetermined outward contrast.
23. The computer-implemented method of claim 22, wherein the predetermined intensity is greater than 0.67 times a maximum intensity, the predetermined circle ratio is greater than 0.65, the predetermined inward contrast is greater than 1.06, or the predetermined outward contrast is greater than 1.22.
24. The computer-implemented method of claim 1, wherein the first criteria comprise contours having a predetermined area, a predetermined circle ratio, or at least one of a predetermined inward contrast and a predetermined gradient scale.
25. The computer-implemented method of claim 24, wherein the predetermined area is less than 0.30 mm2, the predetermined circle ratio is greater than 0.65, the predetermined inward contrast is greater than 1.04, or the predetermined gradient scale is greater than 0.3 mm.
26. The computer-implemented method of claim 1, wherein the first criteria comprise contours having a predetermined area, a predetermined circle ratio, or a predetermined intensity.
27. The computer-implemented method of claim 1 further comprising saving the third subset of contours in a memory of the computing device.
28. The computer-implemented method of claim 1 further comprising identifying, with the processor, instances of the type of tissue for each nested structure based on at least one of: a contour derivative or a grouping parameter computed for each nested structure.
29. The computer-implemented method of claim 28, wherein the contour derivative measures how rapidly intensity varies across a nested structure.
30. The computer-implemented method of claim 1 further comprising identifying, with the processor, outer contours in each nested structure representing a contour shape and inner contours in each nested structure providing data on internal gradients.
31. The computer-implemented method of claim 1, wherein the second criteria comprise a threshold on a contour derivate or a threshold on a grouping parameter.
32. The computer-implemented method of claim 1 further comprising computing cluster properties with the processor.
33. The computer-implemented method of claim 32, wherein the cluster properties comprise one or more of: a cluster centroid, a cluster half-length, a cluster half-width, an aspect ratio, a principal axis, or a packing fraction.
34. A computer program product comprising non-transitory computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, performs the operations of: receiving medical information associated with an individual comprising a medical image associated with a non-invasive medical imaging technique through a network interface of at least one of the one of more computing device over a data network; analyzing the medical image with a processor of the one or more computing devices; generating a clinical score for at least a portion of the medical image; and transmitting an instruction to a measurement device to a acquire a second medical image associated with a second non-invasive medical imaging technique based at least in part on the clinical score, wherein the analysis of the medical image comprises: determining contours in the medical image satisfying one or more criterion; analyzing one or more geometric attributes and one or more contrast attributes of contours included in the first subset of contours to identify a second subset of contours based at least in part on the contours satisfying one or more predetermined geometric and contrast attributes; selecting a third subset of contours from the second subset of contours that corresponds to one or more potential instances of a type of tissue, the third subset selected based at least in part on contours within the third subset satisfying first criteria associated with the type of tissue; ranking contours included in the third subset of contours based at least in part on a selection metric, the selection metric accounting for a combination of contrast and intensity; grouping the contours included in the third subset of contours into nested structures; selecting one or more instances of the type of tissue from the nested structures satisfying second criteria associated with the type of tissue; grouping the selected one or more instances of the type of tissue into clusters based on neighboring instances of the type of tissue and a spatial cluster scale; classifying the clusters as benign or possibly associated with the clinical indication by performing one or more of: a regression analysis on the one or more instances of the type of tissue within the clusters, edge detection, a density analysis of the clusters, or a circularity analysis of the clusters; scoring the clusters using an analytic function to generate the clinical score; and combining physical features of each of the one or more instances of the type of tissue, as an individual structure and as part of a cluster of micro-instances of the type of tissue, together with clinical data, to construct a predictive model and provide a scale of suspiciousness for the clinical indication; receiving a second medical image associated with the second non-invasive medical imaging technique through the network interface of at least the one of the one or more computing devices over the data network; and revising the scale of suspiciousness for the clinical indication with the processor based at least in part on the second medical image.
35. A system comprising: a computing device including a network interface for communications over a data network; a clinical score engine having a processor and a memory, the clinical score engine including a network interface for communications over the data network, the clinical score engine configured to receive medical information associated with an individual comprising a medical image from the computing device, the memory configured to store the medical image, and the processor configured to analyze the medical image, generate a clinical score for at least a portion of the medical image, and transmit an instruction to a measurement device to a acquire a second medical image associated with a second non-invasive medical imaging technique based at least in part on the clinical score, wherein the analysis of the medical image comprises: determining contours in the medical image satisfying one or more criterion; analyzing one or more geometric attributes and one or more contrast attributes of contours included in the first subset of contours to identify a second subset of contours based at least in part on the contours satisfying one or more predetermined geometric and contrast attributes; selecting a third subset of contours from the second subset of contours that corresponds to one or more potential instances of a type of tissue, the third subset selected based at least in part on contours within the third subset satisfying first criteria associated with the type of tissue; ranking contours included in the third subset of contours based at least in part on a selection metric, the selection metric accounting for a combination of contrast and intensity; grouping the contours included in the third subset of contours into nested structures; selecting one or more instances of the type of tissue from the nested structures satisfying second criteria associated with the type of tissue; grouping the selected one or more instances of the type of tissue into clusters based on neighboring instances of the type of tissue and a spatial cluster scale; classifying the clusters as benign or possibly associated with the clinical indication by performing one or more of: a regression analysis on the one or more instances of the type of tissue within the clusters, edge detection, a density analysis of the clusters, or a circularity analysis of the clusters; scoring the clusters using an analytic function to generate the clinical score; and combining physical features of each of the one or more instances of the type of tissue, as an individual structure and as part of a cluster of micro-instances of the type of tissue, together with clinical data, to construct a predictive model and provide a scale of suspiciousness for the clinical indication; and wherein the processor is configured to: receive a second medical image associated with the second non-invasive medical imaging technique through the network interface of the computing device over the data network; and revise the scale of suspiciousness for the clinical indication with the processor based at least in part on the second medical image.
PCT/US2022/025093 2021-04-15 2022-04-15 Detecting, scoring and predicting disease risk using multiple medical-imaging modalities WO2022221712A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202163175535P 2021-04-15 2021-04-15
US63/175,535 2021-04-15

Publications (1)

Publication Number Publication Date
WO2022221712A1 true WO2022221712A1 (en) 2022-10-20

Family

ID=83641036

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2022/025093 WO2022221712A1 (en) 2021-04-15 2022-04-15 Detecting, scoring and predicting disease risk using multiple medical-imaging modalities

Country Status (1)

Country Link
WO (1) WO2022221712A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115862850A (en) * 2023-02-23 2023-03-28 南方医科大学南方医院 Modeling method and device of hepatocellular carcinoma monitoring model based on longitudinal multidimensional data
CN116363155A (en) * 2023-05-25 2023-06-30 南方医科大学南方医院 Intelligent pectoral large muscle region segmentation method, device and storage medium
CN116485003A (en) * 2023-03-03 2023-07-25 大连海事大学 Multi-step channel water level prediction method and device based on echo algorithm and storage medium
CN116523704A (en) * 2023-04-03 2023-08-01 广州市德慷电子有限公司 Medical practice teaching decision method based on big data

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020164061A1 (en) * 2001-05-04 2002-11-07 Paik David S. Method for detecting shapes in medical images
WO2011151821A1 (en) * 2010-05-31 2011-12-08 Dvp Technologies Ltd. Inspection of region of interest
US20200219260A1 (en) * 2015-10-02 2020-07-09 Curemetrix, Inc. Cancer Detection Systems and Methods

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020164061A1 (en) * 2001-05-04 2002-11-07 Paik David S. Method for detecting shapes in medical images
WO2011151821A1 (en) * 2010-05-31 2011-12-08 Dvp Technologies Ltd. Inspection of region of interest
US20200219260A1 (en) * 2015-10-02 2020-07-09 Curemetrix, Inc. Cancer Detection Systems and Methods

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115862850A (en) * 2023-02-23 2023-03-28 南方医科大学南方医院 Modeling method and device of hepatocellular carcinoma monitoring model based on longitudinal multidimensional data
CN116485003A (en) * 2023-03-03 2023-07-25 大连海事大学 Multi-step channel water level prediction method and device based on echo algorithm and storage medium
CN116523704A (en) * 2023-04-03 2023-08-01 广州市德慷电子有限公司 Medical practice teaching decision method based on big data
CN116523704B (en) * 2023-04-03 2023-12-12 广州市德慷电子有限公司 Medical practice teaching decision method based on big data
CN116363155A (en) * 2023-05-25 2023-06-30 南方医科大学南方医院 Intelligent pectoral large muscle region segmentation method, device and storage medium
CN116363155B (en) * 2023-05-25 2023-08-15 南方医科大学南方医院 Intelligent pectoral large muscle region segmentation method, device and storage medium

Similar Documents

Publication Publication Date Title
Houssein et al. Deep and machine learning techniques for medical imaging-based breast cancer: A comprehensive review
Murtaza et al. Deep learning-based breast cancer classification through medical imaging modalities: state of the art and research challenges
Rezaei A review on image-based approaches for breast cancer detection, segmentation, and classification
US11430113B2 (en) Cancer detection systems and methods
US10111632B2 (en) System and method for breast cancer detection in X-ray images
Shah et al. Artificial intelligence for breast cancer analysis: Trends & directions
WO2022221712A1 (en) Detecting, scoring and predicting disease risk using multiple medical-imaging modalities
Mridha et al. A comprehensive survey on deep-learning-based breast cancer diagnosis
WO2016057960A1 (en) Apparatus, system and method for cloud based diagnostics and image archiving and retrieval
US20220254023A1 (en) System and Method for Interpretation of Multiple Medical Images Using Deep Learning
Dodia et al. Recent advancements in deep learning based lung cancer detection: A systematic review
CN108714034A (en) Abdominal pain source mark in medical imaging
Baccouche et al. Early detection and classification of abnormality in prior mammograms using image-to-image translation and YOLO techniques
WO2022110525A1 (en) Comprehensive detection apparatus and method for cancerous region
Songsaeng et al. Multi-scale convolutional neural networks for classification of digital mammograms with breast calcifications
US10957038B2 (en) Machine learning to determine clinical change from prior images
Banumathy et al. Breast Calcifications and Histopathological Analysis on Tumour Detection by CNN.
Rai et al. A comprehensive analysis of recent advancements in cancer detection using machine learning and deep learning models for improved diagnostics
Alsadoon et al. DFCV: a framework for evaluation deep learning in early detection and classification of lung cancer
Gupta et al. Texture and Radiomics inspired Data-Driven Cancerous Lung Nodules Severity Classification
Saroğlu et al. Machine learning, iot and 5g technologies for breast cancer studies: A review
US11901076B1 (en) Prediction of probability distribution function of classifiers
Duong et al. Edge detection and graph neural networks to classify mammograms: A case study with a dataset from Vietnamese patients
Kovalev et al. Automatic detection of pathological changes in chest X-ray screening images using deep learning methods
Munasinghe et al. Yuwathi: early detection of breast cancer and classification of mammography images using machine learning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22789045

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE