Classifications

• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
  • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
  • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
  • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/98—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
  • G06V10/987—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns with the intervention of an operator
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
  • G06V2201/03—Recognition of patterns in medical or anatomical images
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients

Definitions

• the present inventive concept relates generally to medical imaging and analysis; and, more particularly to a system and method to enhance clinical decision making capability within the context of an image reporting and data system (IRADS) for medical diagnosis.
• the present inventive concept provides a confidence level indication (CLI) for a computer-assisted diagnosis (CAD) system that is programmed to minimize the deviations in recommended clinical actions due to the biases of a group or individual in interpreting the rules of the system.
• CLI confidence level indication
• CAD computer-assisted diagnosis
• a trained medical professional such as a radiologist will generally attempt to identify and classify regions of suspicion within a medical image either manually or by using computer software.
• the radiologist may then manually characterize each region of suspicion in accordance with a relevant grading system.
• suspicious regions of interest within the breast may be characterized according to Breast Imaging Reporting and Data Systems (BI-RADS) guidelines.
• BI-RADS is a widely-accepted risk assessment and quality assurance tool used by radiologists for diagnosing breast cancer using mammography, ultrasound, or MRI.
• the classification assigned to each region of suspicion may dictate the future course of action. For example, if the region of suspicion is classified as likely malignant, then a biopsy may be ordered. If the region of suspicion is classified as normal, then no further action may be taken.
• the BI-RADS reporting methodology includes a standard lexicon and structure for reporting purposes which enables radiologists to provide a succinct review of image based findings and to communicate the results to the referring physician in a clear and consistent fashion with a final assessment and a specific course of action. Structured reporting also helps accelerate report turnaround time (TAT), simplify documentation for billing and regulatory compliance, and ease the process of data extraction for utilization review, quality assurance, and research.
• TAT report turnaround time
• the system provides important mechanisms for peer review and quality assurance data to improve the quality of patient care. Results compiled in a standardized manner permit the maintenance and collection analysis of demographic and outcome data.
• PI-RADS Prostate Imaging- Reporting and Data System
• TI- RADS Thyroid Imaging Reporting and Data System
• LI-RADS Liver Imaging Reporting and Data System
• Lung-RADS for prostate, thyroid, liver and lung cancer diagnosis, respectively.
• a BI-RADS 0 indicates an incomplete classification which warrants either an effort to ascertain prior imaging for comparison or to call the patient back for additional views, higher quality films or additional imaging modalities.
• a BI-RADS 6 indicates a proven malignancy previously proven by biopsy.
• the BI-RADS 4 classification is frequently divided into sub-categories of:
• the recommended patient management provided by the BI-RADS system is: if the region of interest is classified as suspicious or highly suspicious, i.e., BI-RADS 4 or BI-RADS 5, then a biopsy should be ordered. If the region of suspicion is classified as normal or benign, i.e., BI-RADS 1 or BI-RADS 2, then no further action may be taken. If, however, the region of suspicion is classified as probably benign, i.e., BI-RADS 3, then the recommendation is a 6- month follow-up to look for any changes.
• the BI-RADS score is a statistic that is correlated with malignancy and not a deterministic measure of malignancy.
• category 3 (less than 2 percent risk of malignancy) or category 4 (probability of cancer, ranging from 3 percent to 94 percent) lesions are considered different degrees of malignant breast lesions. This is especially true for hyperplastic nodules in category 3, which are considered to be uncertain ones. Such lesions do not have obvious characteristics of benign lesions, but they are still considered subjectively as category 3 lesions. There are 1-2 non-benign characteristics of category 4 lesions, but the American College of Radiology does not provide any detailed guidance. This leads to poor inter-observer consistency in classification with resulting discrepancies from the ideal use of the BI-RADS system.
• Category 4a typically consists of 90% to 98% benign lesions
• 4b consists of 50% to 90% benign lesions
• 4c consists of 5% to 50% benign lesions
• BI-RADS 5 has 0% to 5% benign lesions but all must go to biopsy. Therefore, as many as 80% of the biopsies performed on the patients with a category BI-RADS 4 or BI-RADS 5 are found to be benign.
• CAD Computer-aided diagnosis
• the present inventive concept provides a computerized system configured to utilize probabilistic classification incorporating intermediate values given by a human operator in order to better indicate a level of confidence in a CAD recommendation system.
• the aforementioned may be achieved in an aspect of the present inventive concept by providing a method of personalizing a diagnosis assistance system.
• the method may include the step of utilizing machine learning to train a device to provide a confidence level indication (CLI) for computer-assisted diagnosis (CAD) system recommendations.
• Such training may include the step of accessing a plurality of training image features.
• Each of the plurality of training image features may be associated with known classes of a plurality of classes.
• the known classes may correspond to known correct diagnostic decisions for each of the training image features.
• Such training may further include the step of accessing a plurality of initial clinician recommended diagnostic decisions from at least one operator corresponding to each of the plurality of training image features.
• Each of the plurality of initial clinician-recommended diagnostic decisions may include a clinician confidence factor.
• Such training may further include the step of accessing a plurality of CAD system recommended diagnostic decisions corresponding to each of the plurality of training image features.
• Each of the plurality of CAD system-recommended diagnostic decisions may include a classifier output score or a combination of the ensemble of scores.
• Such training may further include the step of accessing a subset of the plurality of initial clinician-recommended diagnostic decisions corresponding to a subset of the plurality of training image features.
• the subset of the plurality of initial clinician-recommended diagnostic decisions may differ with respect to certain CAD system-recommended diagnostic decisions corresponding to the subset of the plurality of training image features.
• Such training may further include the step of generating a function that defines a CLI score for each of the certain CAD system-recommended diagnostic decisions corresponding to the subset of the plurality of training image features.
• the method may further include the step of utilizing initial machine leaming to initially train the device, by providing an initial training data set associated with a series of training images. At least a portion of the initial training data set may include image features associated with initial known classes of the plurality of classes. The plurality of classes may be associated with initial predetermined possible clinical actions.
• the method may further include the step of determining a cost function of weighted error terms based on results of the providing of the initial training data set to the device.
• the method may further include the step of weighting and/or penalizing certain parameters of the cost function for certain image feature values associated with known examples of clinical significance that are predetermined as being important to diagnose.
• the method may further include the step of receiving a selected image by an interface.
• the selected image may include an image feature.
• the method may further include the step of utilizing the device to give a specific clinical action, by (i) extracting at least one image feature value from the selected image, and/or (ii) applying the at least one image feature value to the device trained using the weighted cost function to identify a class from the plurality of classes.
• the device may be trained using the initial machine learning before utilizing machine leaming to train the device to provide the CLI for the computer-assisted diagnosis (CAD) system recommendations.
• CAD computer-assisted diagnosis
• the CLI score for each of the certain CAD system-recommended diagnostic decisions corresponding to the subset of the plurality of training image features may be unique to image features of a particular type and/or unique to the at least one operator.
• One or more parameters of the function may include the clinician confidence factor, the classifier output score, and/or the known correct diagnostic decisions for each of the subset of the plurality of training image features.
• the step of generating the function that defines the CLI score may include providing a local area of an image where the CAD system weighed its confidence more heavily in its recommendation.
• the method may further include the step of periodically repeating, e.g., annually, one or more phases or training steps of the CLI, e.g., an initial training phase to adapt to any leamed behavior of a user, e.g., a radiologist, over a period of time.
• the leamed behavior may be a product of using the CDI/CAD system over the period of time, e.g., an improved ability to correctly detect and/or diagnose cancer and/or other learning experiences relative to diagnostic image analysis that the radiologist might have acquired over the period of time.
• a subset of the plurality of initial clinician-recommended diagnostic decisions corresponding to a subset of the plurality of training image features may further include an initial decision profile and/or a final decision profile by a specific operator, a plurality of specific operators, an institution, a locale, a workflow position, and/or an aggregation of final decisions made by a plurality of operators including to be utilized with the function at defines a CLI score for each of the certain CAD system-recommended diagnostic decisions corresponding to the subset of the plurality of training image features.
• the initial scores of an individual or group are recorded following their scores after seeing a CAD recommender score. These scores may be used to help train the CLI or function associated with the same to estimate the odds of a trained CAD device/system being correct over the individual or group's decisions for cases similar to each one seen during training.
• Each of the plurality of classes may be associated with different categories of a Breast Imaging Reporting and Data System (BI-RAD) lexicon.
• Each of the plurality of training image features image may include pixel values and/or a subset of pixel values associated with a region of interest for a legion.
• the function may be based on or factor one or more intermediate values given by an operator and/or CAD system recommendations to compute the CLI score.
• the function may utilize a probabilistic classification while incorporating intermediate values given by a human operator in order to better indicate a level of confidence of CAD systems recommendations as defined by the CLI score.
• the aforementioned may be achieved in another aspect of the present inventive concept by providing a method of training a diagnosis assistance system.
• the method may include the step of utilizing machine learning to train a device to provide a confidence level indication (CLI) for computer-assisted diagnosis (CAD) system recommendations.
• the training may include the step of accessing at least one training image feature associated with a known class corresponding to a known correct diagnostic decision for the training image feature.
• the training may further include the step of accessing a clinician-recommended diagnostic decision from at least one operator corresponding to the training image feature.
• the training may further include the step of accessing a CAD system-recommended diagnostic decision corresponding to the training image feature comprising a classifier output score or a combination of the ensemble of scores.
• the CAD-system recommended diagnostic decision may differ from the clinician- recommended diagnostic decision.
• the training may further include the step of generating a function that defines a CLI score for the CAD system-recommended diagnostic decision.
• One or more parameters of the function may include a clinician confidence factor, the classifier output score, and/or the known correct diagnostic decision for the training image feature.
• the clinician- recommended diagnostic decision may define an intermediate value given by the at least one operator.
• the apparatus may include at least one computing device.
• the computing device may be operable to be trained via machine learning to generate a recommended class based on one or more image features using a cost function of one or more weighted terms, with one or more parameters of the cost function being weighted and/or penalized for certain image features associated with predetermined known examples of clinical significance.
• Additional machine learning may be applied to the computing device.
• Such additional machine learning may include a plurality of clinician diagnostic decisions accessed by the computing device from an operator corresponding to each of a plurality of training image features.
• Each of the plurality of initial clinician diagnostic decisions may include a clinician confidence factor.
• Such additional machine learning may include a plurality of CAD system diagnostic decisions corresponding to each of the plurality of training image features accessed by the computing device.
• Each of the plurality of CAD system diagnostic decisions may include a classifier output score or a combination of the ensemble of scores.
• a subset of the plurality of clinician diagnostic decisions may correspond to a subset of the plurality of training image features that differ with respect to certain CAD system diagnostic decisions for the plurality of training image features accessed by the computing device.
• Such additional machine learning may include a function executed by the computing device that may define a CLI score for each of the certain CAD system diagnostic decisions corresponding to the subset of the plurality of training image features.
• the function may be defined as: P (c/X; and Z; and W; and Q), to compute a probability that being in class c, given a feature vector X, and the operator selects a label Z, and the CAD system recommendation is defined by label W, with a ground known truth defined as label Q.
• the CLI score may define a probability of the computing device correctly generating the recommended class by taking into account intermediate values given by the at least one operator using probabilistic classification.
• FIG. 1 is an exemplary process flow, according to aspects of the present inventive concept.
• FIG. 2 is an exemplary process flow, according to aspects of the present inventive concept.
• FIG. 3 is an exemplary process flow, according to aspects of the present inventive concept.
• FIG. 4 is an exemplary process flow, according to aspects of the present inventive concept.
• FIG. 5 is an exemplary process flow, according to aspects of the present inventive concept.
• FIG. 6 is an exemplary computing system that may implement various services, systems, and methods discussed herein.
• references to terms “one embodiment,” “an embodiment,” “the embodiment,” or “embodiments” mean that the feature or features being referred to are included in at least one aspect of the present inventive concept.
• references to terms “one embodiment,” “an embodiment,” “the embodiment,” or “embodiments” in this description do not necessarily refer to the same embodiment and are also not mutually exclusive unless so stated and/or except as will be readily apparent to those skilled in the art from the description.
• a feature, structure, process, step, action, or the like described in one embodiment may also be included in other embodiments, but is not necessarily included.
• the present inventive concept may include a variety of combinations and/or integrations of the embodiments described herein. Additionally, all aspects of the present disclosure as described herein are not essential for its practice.
• algorithm refers to logic, hardware, firmware, software, and/or a combination thereof that is configured to perform one or more functions including, but not limited to, those functions of the present inventive concept specifically described herein or are readily apparent to those skilled in the art in view of the description.
• Such logic may include circuitry having data processing and/or storage functionality. Examples of such circuitry may include, but are not limited to, a microprocessor, one or more processors, e.g., processor cores, a programmable gate array, a microcontroller, an application specific integrated circuit, a wireless receiver, transmitter and/or transceiver circuitry, semiconductor memory, or combinatorial logic.
• logic refers to computer code and/or instructions in the form of one or more software modules, such as executable code in the form of an executable application, an application programming interface (API), a subroutine, a function, a procedure, an applet, a servlet, a routine, source code, object code, a shared library/dynamic load library, or one or more instructions.
• software modules may be stored in any type of a suitable non-transitory storage medium, or transitory storage medium, e.g., electrical, optical, acoustical, or other form of propagated signals such as carrier waves, infrared signals, or digital signals.
• non- transitory storage medium may include, but are not limited or restricted to a programmable circuit; a semiconductor memory; non-persistent storage such as volatile memory (e.g., any type of random access memory "RAM”); persistent storage such as nonvolatile memory (e.g., readonly memory "ROM”, power-backed RAM, flash memory, phase-change memory, etc.), a solid- state drive, hard disk drive, an optical disc drive, or a portable memory device.
• volatile memory e.g., any type of random access memory "RAM”
• persistent storage such as nonvolatile memory (e.g., readonly memory "ROM”, power-backed RAM, flash memory, phase-change memory, etc.), a solid- state drive, hard disk drive, an optical disc drive, or a portable memory device.
• firmware the executable code is stored in persistent storage.
• the term "user” is generally used synonymously herein to represent a user of the system and/or method of the present inventive concept.
• the user may be a clinician, a diagnostician, a doctor, a technician, a student, and/or an administrator.
• CAD Computer- Assisted Diagnosis
• client means any program of software that connects to a CAD lesion application.
• server typically refers to a CAD lesion application that is listening for one or more clients unless otherwise specified.
• post-processing means an algorithm applied to an inputted ultrasound image.
• GSPS Grayscale Softcopy Presentation State.
• DIOM Digital Imaging and Communications in Medicine.
• UI User Interface
• PHI Private Health Information
• ensemble methods means multiple learning algorithms to obtain better performance that could be obtained from any of the constituent learning algorithms.
• stacking means training a learning algorithm to combine the predictions of several other learning algorithms. Stacking may also be referred to as stacked generalization.
• the term "combiner” means an algorithm trained to make a final prediction using all of the predictions of other algorithms as additional inputs.
• the term "diversity” means variation among ensemble methods/models. Many ensemble methods seek to promote diversity among the models they combine. More random algorithms (such as random decision trees) can be used to produce a stronger ensemble than very deliberate algorithms (such as entropy -reducing decision trees). Using a variety of strong learning algorithms (diversity), however, has been shown to be more effective than using techniques that attempt to dumb-down models in order to promote diversity.
• aspects of the present inventive concept provide a system and method that utilizes a preexisting Computer- Assisted Diagnosis (CAD) recommendation system implemented on a computing device as described in the related U.S. Patent Application Serial No. 15/200,719 incorporated herein by reference.
• the CAD recommendation system of U.S. Patent Application Serial No. 15/200,719 is specially programmed to minimize discrepancies between
• the CAD recommendation system of U.S. Patent Application Serial No. 15/200,719 reduces the number of erroneous clinical actions such as biopsies based on the operator's error profile.
• a training data set associated with a series of training images may be applied to a computing device.
• the series of training images may comprise medical images, showing a particular area of the human body, such as a human breast.
• the training images may comprise images of a portion of a breast with malignant or benign lesions.
• Each of the images, depending upon the lesion shown, may comprise different characteristics, such as color, shading, and the like.
• At least a portion of the training data set may comprise image features, from the training images, that are associated with known classes of a plurality of classes (of a medical lexicon).
• the image features have been proven to be linked to certain classes of a medical lexicon.
• a class 1 of a BI-RADS lexicon may be associated with or assigned to an image feature of a first training image
• a class 2 of a BI-RADS lexicon may be associated with or assigned to an image feature of a second training image.
• Image features may be vectors, or other values of a particular image, such as a medical image.
• at least a portion of the training data set may provide examples to the computing device about when images features should be assigned to one or more classes of a medical lexicon.
• Each of the classes of the medical lexicon may be associated with or correspond to predetermined possible clinical actions. In other words, it may be predetermined that when an image feature falls within a particular class, a certain clinical decision should be recommended that is specific to that class.
• Clinical actions may comprise, for example, certain tasks or procedures that should be taken based on image features.
• a clinical action may comprise performing a biopsy on a lesion to remove a tissue sample of the lesion and submit the tissue sample for testing and analysis.
• Another clinical action may comprise following up with a patient and lesion after a predetermined period of time, such as six months.
• a cost function of weighted terms may be determined based upon the training set data as applied to the computing device.
• certain parameters of the cost function may be weighted for certain image feature values associated with known examples of clinical significance that are predetermined as being important to diagnose.
• the certain parameters may be weighted to account for a difficulty of a radiologist or other clinician to accurately diagnose an image feature as belonging to one or more of the plurality of classes.
• the described machine learning may be utilized to train a computing device to give clinical decisions in the specific context of possible cancer diagnosis.
• a training data set in the form of a preselected group of data patterns, which identifies relationships between one or more image features and known correct BI-RAD classifications, may be presented to the computing device for classification.
• the training data set may identify, for example, that an image feature (of e.g. a lesion) that appears to the human eye as having a particular texture, shape, component/components, shade, color, or other visual characteristic that has been deemed to fall within class 4 according to a BI-RAD lexicon.
• the image feature may further comprise a particular value (of a pixel, group of pixels, or a function of pixels or group of pixels); e.g. the training data set identifies for the computing device a particular value (feature value) of one or more pixels (or a function of one or more pixels) associated with each image.
• the training data set further identifies for the computing device feature values or a set of feature values that are associated with images that have been deemed or predetermined (based e.g. on clinical evidence) to fall within one or more BI-RAD classes according to an individual radiologist, or a group of radiologist.
• the biopsy proven classification (as to whether the lesion is cancerous or benign) is known.
• An actual output produced by the computing device, in response to a training image, may be compared with the known biopsy proven result, with reference to a cost function.
• it may be ideal to minimize such a cost function, to minimize the errors outputted by the computing device regarding how image features should be classified with respect to the radiologist or group of radiologists and their selection of the class of a BI-RAD lexicon for the image.
• aspects of this comparison may be utilized to adjust certain parameters of the computing device and/or the cost function, such as weights, biases, or penalty functions added to the error terms.
• certain parameters of the cost function may be weighted for certain image feature values associated with known examples of clinical significance that are predetermined as being important to diagnose.
• the certain parameters may be weighted or penalized to account for a difficulty of a radiologist or other clinician to accurately diagnose an image feature as belonging to one or more of the plurality of classes which may be especially important where the clinician is faced with a decision as to whether or not to recommend a biopsy for a lesion.
• the above process may be repeated until the cost function, averaged over a suitable second preselected group of data patterns or validation set, is minimized.
• training of the computing device to give clinical actions for BI-RAD lesions may be deemed complete when subsequent test data is presented to the computing device, the computing device generates an output on that test data, and a comparison between the output and known correct results yields a difference or value that is within a predetermined acceptable margin.
• the trained computing device may be implemented for a variety of related applications such as training the computing device to generate decisions in the context of radiology, ultrasound, and the like.
• the trained computing device may be one element of the CAD recommendation system.
• Aspects of the CAD recommendation system utilize the machine learning/training of the training phase described above to recommend classifications for new images that have not already been diagnosed; the classifications corresponding to certain CAD-recommended clinical decisions for the image features.
• a new selected image may be received by the computing device of the CAD recommendation system using an interface.
• the computing device may scan the selected image, retrieve image features from the selected image, extract values from the selected image, and match the values to predetermined values that the computing device has been programmed to recognize as being associated with lesions or other abnormalities (as learned during the above training phase).
• a selected image comprises at least one image feature is made accessible to the computing device for analysis.
• the computing device may extract at least one image feature value from the selected image.
• the selected image feature value may be associated with a numeric value and may, in some embodiments, be a pixel value or set of pixel values. Extracting a value from a selected image feature in this fashion breaks down an image feature into data understandable by the computing device, and, data that can be utilized with one or more cost functions or other functions developed during machine learning.
• the computing device of the CAD recommendation system is utilized by applying the at least one feature value to the computing device which has been trained using the weighted cost function.
• the computing device then outputs a class from the plurality of classes as defined during the machine learning process.
• the weighted or penalty based training phase of the CAD system enhances the probability that the CAD system will give a correct score when the operator is more likely to be incorrect.
• the weighting or penalty function are used to enhance the CAD error profile be diverse from the operator by putting more weight on those errors typically made by the user or users. Alternatively the error terms that are most likely to be made by operator(s) are penalized to enhance their being corrected in the operation of the resulting CAD system.
• the computing device may output a score equivalent to a BI-RAD category of 2 which indicates the lesion is benign, however, this same image might be one that is most likely to be categorized as a BI-RADS 4 by the operator indicating that the image feature value is associated with a lesion that is suspicious and should be subject to further diagnosis.
• a score equivalent to a BI-RAD category of 2 which indicates the lesion is benign
• this same image might be one that is most likely to be categorized as a BI-RADS 4 by the operator indicating that the image feature value is associated with a lesion that is suspicious and should be subject to further diagnosis.
• CAD recommendation systems such as the CAD recommendation system of U.S. Patent Application No. 15/200,719 incorporated herein typically perform with error rates which are similar to expert radiologists.
• a CAD recommendation system outputs a class for an image classification, the class corresponding to a CAD-recommended diagnostic decision for the image feature, and that CAD-recommended clinical decision is consistent with an initial clinician recommended diagnostic decision, an operator (or other user) who generates the initial clinician-recommended diagnostic decision is provided with additional confidence in their diagnosis of the image feature.
• the operator is forced to make a decision to select between unaided diagnosis (the initial clinician-recommended diagnostic decision that does not take into account a recommendation from a CAD system); and the CAD-recommended diagnostic decision. [0077]
• the operator may then make a final decision based on confidence in the operator's initial diagnostic decision as compared to the operator's confidence in the CAD-recommended clinical decision. If the operator feels more confident in their own opinion than the CAD system's ability, the operator will be biased towards declining the CAD system's
• a CAD system will differ from the opinion of an operator such as a radiologist only when the radiologist is incorrect and the CAD system will be able to correct the radiologist.
• the CAD system would always agree when the radiologist is correct.
• the radiologist will sometimes be correct and the CAD system will be incorrect.
• An optimal decision making process could be obtained if the radiologist could have some additional measure of the confidence in the recommendation given by the CAD system.
• Standard output values of classifiers used in machine learning can be only be interpreted as probabilities under very limited conditions that are rarely met in practice.
• the present inventive concept is operable to solve this problem and help the radiologist decide when to accept the CAD recommendation over the initial opinion of the radiologist when the opinion of the radiologist differs with respect to the CAD system recommendation for a particular image/image feature.
• the present inventive concept addresses this problem by collecting training data while the radiologist employs the CAD system on a database of true data.
• the training session serves the purpose of familiarizing the radiologist with CAD systems strengths and weaknesses as well as collecting data as to when the CAD system can help correct the radiologist's initial decision.
• the training data is used to train a predictive model which provides a confidence score for particular CAD system recommendations by the CAD recommendation system.
• the predictive model gives a score which represents the likelihood of a CAD recommendation being correct given the image data and previous performance of a radiologist.
• the radiologist can use the personalized confidence score to help decide when it would best to defer to the CAD system.
• the present inventive concept allows radiologists to calibrate their trust in the CAD system more effectively.
• a CAD recommendation system may be trained to have errors which complement the errors made by the radiologist or radiologists who will use the system.
• the CAD system may be trained so as to more heavily penalize the errors made by the radiologists to provide a CAD system with errors that complement those of the radiologist.
• the present inventive concept utilizes machine learning/training to further enhance or tune a CAD recommendation system.
• the present inventive concept provides a method and means whereby the operator (for example a radiologist) is trained to learn to understand when they should trust the CAD system with more confidence than their initial opinion.
• the operator may be asked to diagnose a set of pre-stored cases based on observing a set of appropriate medical image data.
• the CAD system will provide a second opinion.
• the radiologist will be free to change their original opinion based on the CAD recommendation or decline the recommendation.
• the true diagnosis may be then be displayed along with images of similar cases to help the operator learn by example which cases the operator could benefit from by using the CAD systems recommendation and which cases the operator needs less help from the CAD system.
• This training phase may be used to form a statistical or predictive model of the confidence level indicator (CLI).
• the CLI's purpose is to determine, in cases of disagreement between the CAD system and an operator, which is more likely to be correct, given the case data, and the tendencies and potential biases of each.
• the CLI's parameters are determined by taking into consideration radiological data, the true (biopsy proven) diagnosis of that data, the CAD system's interpretation of that data, the operator's interpretation of that data, and the change in the operator's interpretation of the data when influenced by the CAD system.
• the true diagnosis of the data in this description, can refer to the biopsy proven diagnosis of the case in question.
• Multi-operator Models An extension of method 3, where many operators' behaviors are modelled, rather than a single one. This can be achieved by modelling the consensus of many operators, and/or considering the consensus of many operator-models.
• Fine-tuned Operator Models This method compromises between the personalized approach of method 3 and the generalized approach of method 4. In order to do so, a general multi-operator model is first developed. This model can be considered a baseline, which can then be fine-tuned to follow the predilections of an individual operator to whatever extent is deemed optimal.
• the previously mentioned factors may be used to resolve disagreements between the CAD and operator in the most optimal manner. This resolution can be considered a measure of confidence in the CAD's decision with respect to an operator of population of operators.
• Flow chart 100 illustrates pre-CAD training.
• a plurality of training images may be accessed from medical image database.
• the training images may depict areas of the human body that have characteristics that may indicate an ailment, injury, or affliction.
• the training images may include images of a portion of a breast that need to be diagnosed to determine whether the portion shown indicates a possible cancerous region.
• a feature extraction process may be implemented.
• a plurality of image features may be extracted from the training images.
• an operator such as a radiologist or other clinician, may undergo personalized radiologist model training.
• the operator may recommend a diagnostic decision, or a pre-CAD viewing diagnosis as shown in block 114, for each of at least a portion of the image features extracted from the training images.
• a personalized radiologist/operator pre-CAD model as shown in block 116, may be generated that is specific to the operator (or group of operators) comprising data about when the specific operator makes an erroneous diagnostic decision; i.e., a decision that differs from a known proven diagnostic decision (as shown in block 112).
• the personalized radiologist/operator pre-CAD model of block 116 may help to identify for what types of image features the specific operator tends to make an erroneous decision, i.e., a decision that differs from a known correct diagnostic decision.
• the flow chart 100 of FIG. 1 illustrates the personalized radiologist/operator pre-CAD model of block 116 that is derived without the operator having viewed CAD data.
• the operator in FIG. 1 undergoes pre-CAD training to develop the personalized radiologist/operator pre-CAD model of block 116.
• FIG. 1 further shows an optional radiologist pre-CAD viewing diagnosis database (pre- CAD database) 108.
• the pre-CAD database may be utilized to train an operator (e.g. radiologist) before actually measuring the performance of the operator with the personalized radiologist model training of block 110.
• the pre-CAD database 108 may assist to help the operator to better understand the functionality of a CAD system (that recommends diagnostic decisions during actual training).
• FIG. 2 illustrates a flow chart for a training process similar to FIG. 1.
• the flow chart indicates post-CAD viewing training, i.e., analyzing diagnostic decisions of the operator post- CAD viewing.
• a plurality of training images may be accessed from medical image database.
• the training images may depict areas of the human body that have characteristics that may indicate an ailment, injury, or affliction.
• the training images may include images of a portion of a breast that need to be diagnosed to determine whether the portion shown indicates a possible cancerous region.
• a feature extraction process may be implemented.
• a plurality of image features may be extracted from the training images.
• FIG. 2 further shows an optional radiologist pre-CAD viewing diagnosis database (pre- CAD database) 158.
• the pre-CAD database may be utilized to train an operator (e.g. radiologist) before actually measuring the performance of the operator with the personalized radiologist model training of block 130.
• the pre-CAD database 158 may assist to help the operator to better understand the functionality of a CAD system (that recommends diagnostic decisions during actual training).
• an operator such as a radiologist or other clinician, may undergo personalized radiologist model training. Specifically, the operator may recommend a diagnostic decision, or a post-CAD viewing diagnosis as shown in block 132, for each of at least a portion of the image features extracted from the training images.
• a personalized radiologist/operator post-CAD model as shown in block 152, may be generated that is specific to the operator (or group of operators) comprising data about when the specific operator makes an erroneous diagnostic decision; i.e., a decision that differs from a known proven diagnostic decision (as shown in block 112); and, when the operator makes the decision despite having access to a CAD decision for the same image feature.
• the personalized radiologist/operator post-CAD model of block 152 may help to identify for what types of image features the specific operator tends to make an erroneous decision, i.e., a decision that differs from a known correct diagnostic decision.
• the flow chart 150 of FIG. 2 illustrates the personalized radiologist/operator post-CAD model of block 152
• radiologist/operator post-CAD model of block 152 that is derived with the operator having viewed CAD data.
• the operator in FIG. 2 undergoes post-CAD training to develop the personalized radiologist/operator pre-CAD model of block 152.
• FIG. 3 illustrates another flow chart 200 for CLI model training; the output being a personalized operator/radiologist trained CLI model as shown in block 214.
• a plurality of training images may be accessed from medical image database.
• the training images may depict areas of the human body that have characteristics that may indicate an ailment, injury, or affliction.
• the training images may include images of a portion of a breast that need to be diagnosed to determine whether the portion shown indicates a possible cancerous region.
• a feature extraction process may be implemented.
• a plurality of image features may be extracted from the training images.
• confidence level indicator model training may take as inputs known correct diagnostic decision labels of block 112, a CAD system classifier output of block 210, a trained radiologist pre-CAD diagnosis of block 202, a personalized radiologist pre-CAD model of block 206, a trained radiologist post-CAD diagnosis of blocks 204, and a personalized radiologist post- CAD model of block 208.
• FIG. 4 is a flow chart 300 for classifying image features using the CLI model derived from FIG. 3. Similar to FIGS. 1 and 2, in block 302, a plurality of training images may be accessed from medical image database.
• the training images may depict areas of the human body that have characteristics that may indicate an ailment, injury, or affliction.
• the training images may include images of a portion of a breast that need to be diagnosed to determine whether the portion shown indicates a possible cancerous region.
• a feature extraction process may be implemented.
• a plurality of image features may be extracted from the training images.
• a personalized confidence level indicator model may be utilized which takes as inputs a CAD classifier of block 308 and an optional radiologist initial pre-CAD diagnosis of block 310.
• the CAD classifier 308 generates a CAD classifier output of block 314, and the personalized confidence level indicator model generates a CLI model output of block 316.
• a confidence level indicator can be generated for classifier outputs, or CAD system recommendations for diagnostic decisions.
• the confidence level indicator or CLI model output of 316 indicates how likely it is that the CAD classifier output of 314 is correct when the CAD system recommendation differs with respect to a clinician diagnostic decision.
• the confidence level indicator provides a confidence level similar to a probability of being correct relative to the CAD output so if the CAD system indicates that an image feature is associated with a cancerous lesion, the CLI might output, for example, .95 indicating that there is a 95% confidence level in the CAD diagnostic decision regarding the image feature being associated with a cancerous legion.
• a first phase of supervised machine learning may comprise training of the device to suggest classifications based on certain image features. Specifically, a fixed set of multi-dimensional feature vectors Xi may be used to train the device (machine learning) to output a score s(Xi) for each input vector Xi. The resulting trained device may then be used in operation on new data that the device was not trained on to generalize the pattern that the device learned during training (sometimes called learning).
• Most supervised learning/training methods produce classifiers that output scores s(x) where s(Xi) is a scalar value between 0 and 1 and Xi is vector of features (i.e. multiple value array) which can be used to rank the examples in the test set from the most probable member to the least probable member of a class c. That is, for two examples x and y, if S(x) ⁇ S(y) (the output score given by the classifier for x is less than the score given for vector y, then
• the present inventive concept is operable to produce a confidence estimate (CLI score) using training data collected on a set of images along with a user model and ground truth. The method used generalizes to provide useful confidence scores on data not used in the training data. Thus a confidence or CLI score is generated which can be used to help the operator select an optimum final decision based on one or more image features presented.
• the basic problem with classification is that all data values are labeled either as a "1" say for cancer or a "0" for not cancer during training.
• the values in between 0 and 1 are not trained as outputs of the classifier so there is much freedom as to how the intervening values are assigned to images when in actual operation.
• only the relative ranking is related to probability of being in the class C. That is ideally one would desire an output score of >.8 to indicate that 80% of images represented by a particular image example are cancerous but it just means that it is more likely to be cancer than image with a score of .7
• the present inventive concept utilizes training data with intermediate values given by the human operator as well as a trained CAD system to better indicate the level of confidence of the trained CAD systems recommendation.
• a more general value is given for the confidence (or probability of being correct) of cancer given an image with feature vector X as well as the CAD recommendation and anticipated or actual user initial decision (the operator's decision is obtained implicitly by using the CAD system which may be biased by his/her error profile) to give a confidence score of the CAD system recommendation.
• a function for generating a CLI score may be described as: P(c/X and Z and W and Q) which reads the probability of the class being in class c given the feature vector X, with the user/operator selected a label Z, and a CAD system recommending a label W, while a known ground truth is represented by the label Q.
• Labels may be cancer or not cancer, i.e., 1 or 0, respectively.
• Increasing specificity of an estimate by factoring information related to a user and/or a CAD system results in increased accuracy of the estimate and an improved or higher confidence score, which is specific to the user and the CAD system, thereby facilitating machine learning and processing of estimates.
• the function for computing the CLI score may take into account one or more specific operators and one or more specific CAD recommendation systems so that CLI scores reflect suggested confidence in CAD diagnostic decisions in the context of the one or more specific users and/or the one or more specific CAD systems.
• the operator may generally assign to the ith region of interest (ROI) of the N possible ROls in the images collected for a patient case study a score between 0 (lowest probability of cancer) to 1 (highest probability of cancer).
• the assigned scores are typically either 0 or 1 during operation but, for training purposes, intermediate values may be used.
• the CAD system is trained so that its errors E2(i) are statistically independent of the operator's errors. Thus, if the score assigned by the operator disagrees (or is inconsistent) with the independent score of the CAD system's output score (a first assigned score is high and a second assigned score is low relative to .5), then a "draw condition" is detected and the optimal decision is unclear.
• its output can be used as an independent opinion, e.g., to break a draw condition when such arises between the operator and the first CAD system by using a simple majority rule voting logic, i.e., if 2 out of the 3 scores are high, then select high, or if 2 out of three scores are low, then select low.
• Alternative logic such as trim means can be used on the three independent scores to get enhanced performance over the operator's unaided performance.
• the first CAD system may encompass CAD systems that have been trained to generate recommendations in some form, but have not been enhanced with CLI functionality as discussed herein.
• the system of the present inventive concept allows an operator to correct their error if they rely on the CAD system/device and follow its recommendation over their own, e.g., when it is correct and they are incorrect.
• One unique aspect is the initial training of the device, as disclosed by U.S. Patent Application Serial No. 15/200,719, is that the device is trained specifically to variably weigh errors made by an individual and/or group of operators, e.g., more heavily, to ensure that CAD system recommenders give correct recommendations when the operator is most likely to need help in correcting an unaided recommendation of the operator.
• the present inventive concept is operable to select and utilize one or more classifiers that are diverse from each to optimize their combined decision.
• U.S. Patent Application Serial No. 15/200,719 discloses a new method and means to obtain a personalized diverse classifier which provides a recommendation that can be combined with the operator's initial recommendation to improve the operator's accuracy by correcting one or more of their errors when the operator elects to accept the recommendation(s) at the appropriate time(s).
• the present inventive concept teaches a new way to combine the recommendations from the trained CAD system of U.S. Patent Application No. 15/200,719 with an operator's initial recommendation so as to improve the accuracy of the final decision or recommendation; and the operator also learns to select the CAD recommendations when the confidence level is high (much greater than .5) of the CD! score and this reassessment may correct the operator's initial incorrect assessment.
• the CLI system may provide a low confidence level (much lower than .5) when the operator is probably better off not to follow the CAD recommendation and proceed with their initial assessment.
• FIG. 5 is a process flow 400 for generating a CLI model to be utilized with CAD system recommendations for diagnostic decisions.
• machine learning may be utilized or otherwise implemented by a device to provide a confidence level indicator (CLI) for CAD recommended diagnostic decisions that are generated from the subject device or other devices.
• CLI confidence level indicator
• the device may have been trained using initial machine learning as set forth in the related U.S. Patent Application Serial No. 15/200,719 incorporated by reference.
• utilizing machine learning to train the device to provide a CLI may be an improvement upon a CAD system already trained to recommend diagnostic decisions; and, the same device/application may be modified to take into account the functionality (functions/algorithms) used to generate the CLI score.
• the device may access a plurality of training image features.
• Each of the training image features may be associated within known classes. In other words, certain image features may be associated with a lesion that has been proven as being cancer. In other cases, certain image features have been proven to be non-cancerous. In either case, all of the training image features are already assigned to one clinical class or another, based on evidence, proven test results, or the like.
• the classes may correspond to classes of an IRADS system as described in the related U.S. Patent Application Serial No. 15/200,719.
• a plurality of initial clinician-recommended diagnostic decisions from at least one operator are accessed. Each of the decisions may be made to diagnose one or more of the training image features (the true or known results of which are unknown to the operator).
• the operator's diagnostic decision or interpretation of the training image data may be obtained using a variety of methods. For example, direct labels consensus labels, operator models, multi- operator models, fine-tuned operator models, and/or archetypical operator models may be utilized as explained herein (although the disclosure is not limited to such models and additional models are contemplated).
• a plurality of CAD system-recommended diagnostic decisions may be accessed for each of the plurality of training image features.
• the device may simply apply the methodology and functions used during the initial training with the training images to compute such CAD system-recommended diagnostic decisions for each of the plurality of training image features.
• the device may access a subset of the plurality of initial clinician recommended diagnostic decisions corresponding to a subset of the plurality of training image features.
• the subset being where the operator disagrees or otherwise provides a diagnostic decision that is not consistent with or is different than the CAD system diagnostic decision for one or more certain training image features.
• a function may be generated that defines a CLI score for each of the certain CAD system-recommended diagnostic decisions corresponding to the subset of the plurality of training image features.
• one or more functions can be generated that compute CLI scores for certain CAD recommendation system diagnostic decisions where the operator has disagreed with such certain CAD recommendation system diagnostic decisions.
• FIG. 6 is an example schematic diagram of a computing system 700 that may implement various methodologies discussed herein.
• the computing system 700 may comprise a computing device used to implement a CLI application 70 for generating CLIs or CLI scores for certain CAD system recommendations.
• the computing system 700 includes a bus 701 (i.e., interconnect), at least one processor 702 or other computing element, at least one
• Processor(s) 702 can be any known processor, such as, but not limited to, an Intel ® Itanium ® or Itanium processor(s), AMD ® Opteron ® or Athlon MP ® processor(s), or Motorola ® lines of processors.
• Communication port 703 can be any of an RS-232 port for use with a modem based dial-up connection, a 10/100 Ethernet port, a Gigabit port using copper or fiber, or a USB port.
• Communication port(s) 703 may be chosen depending on a network such as a Local Area Network (LAN), a Wide Area Network (WAN), or any network to which the computer system 200 connects.
• Computing system may further include a transport and/or transit network 755, a display screen 760, an I/O port 740, and an input device 745 such as a mouse or keyboard.
• Main memory 704 can be Random Access Memory (RAM) or any other dynamic storage device(s) commonly known in the art.
• Read-only memory 706 can be any static storage device(s) such as Programmable Read-Only Memory (PROM) chips for storing static information such as instructions for processor 702.
• Mass storage device 707 can be used to store information and instructions.
• hard disks such as the Adaptec ® family of Small Computer Serial Interface (SCSI) drives, an optical disc, an array of disks such as Redundant Array of Independent Disks (RAID), such as the Adaptec ® family of RAID drives, or any other mass storage devices, may be used.
• SCSI Small Computer Serial Interface
• RAID Redundant Array of Independent Disks
• Bus 701 communicatively couples processor(s) 702 with the other memory, storage, and communications blocks.
• Bus 701 can be a PCI / PCI-X, SCSI, or Universal Serial Bus (USB) based system bus (or other) depending on the storage devices used.
• Removable storage media 705 can be any kind of external hard drives, thumb drives, Compact Disc - Read Only Memory (CD-ROM), Compact Disc - Re-Writable (CD-RW), Digital Video Disk - Read Only Memory (DVD-ROM), etc.
• Embodiments herein may be provided as a computer program product, which may include a machine-readable medium having stored thereon instructions which may be used to program a computer (or other electronic devices) to perform a process.
• the machine-readable medium may include, but is not limited to optical discs, CD-ROMs, magneto-optical disks, ROMs, RAMs, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or other type of media/machine-readable medium suitable for storing electronic instructions.
• embodiments herein may also be downloaded as a computer program product, wherein the program may be transferred from a remote computer to a requesting computer by way of data signals embodied in a carrier wave or other propagation medium via a
• communication link e.g., modem or network connection.
• main memory 704 is encoded with a CLI application 70 that supports functionality as discussed herein.
• the CLI application 70 (and/or other resources as described herein) can be embodied as software code such as data and/or logic instructions (e.g., code stored in the memory or on another computer readable medium such as a disk) that supports processing functionality according to different embodiments described herein.
• processor(s) 702 accesses main memory 704 via the use of bus 701 in order to launch, run, execute, interpret, or otherwise perform processes, such as through logic instructions, executing on the processor 702 and based on the CLI application 70 stored in main memory or otherwise tangibly stored.
• the described disclosure may be provided as a computer program product, or software, that may include a machine-readable medium having stored thereon instructions, which may be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure.
• a machine-readable medium includes any mechanism for storing information in a form (e.g., software, processing application) readable by a machine (e.g., a computer).
• the machine-readable medium may include, but is not limited to optical storage medium (e.g., CD-ROM); magneto-optical storage medium, read only memory (ROM); random access memory (RAM); erasable programmable memory (e.g., EPROM and EEPROM); flash memory; or other types of medium suitable for storing electronic instructions.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Health & Medical Sciences (AREA)
• General Physics & Mathematics (AREA)
• Multimedia (AREA)
• Artificial Intelligence (AREA)
• Public Health (AREA)
• Quality & Reliability (AREA)
• Medical Informatics (AREA)
• General Health & Medical Sciences (AREA)
• Epidemiology (AREA)
• Radiology & Medical Imaging (AREA)
• General Engineering & Computer Science (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Life Sciences & Earth Sciences (AREA)
• Primary Health Care (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Bioinformatics & Computational Biology (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Evolutionary Biology (AREA)
• Evolutionary Computation (AREA)
• Probability & Statistics with Applications (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Image Analysis (AREA)
• Medical Treatment And Welfare Office Work (AREA)

Priority Applications (6)

Applications Claiming Priority (2)

Publications (1)

Family

ID=61162555

Family Applications (1)

Country Status (6)

Cited By (8)

Families Citing this family (1)

Citations (6)

Family Cites Families (7)

• 2017
  • 2017-08-11 CA CA3033441A patent/CA3033441A1/en active Pending
  • 2017-08-11 WO PCT/US2017/046565 patent/WO2018031919A1/en unknown
  • 2017-08-11 CN CN201780059927.4A patent/CN109791804B/zh active Active
  • 2017-08-11 JP JP2019529462A patent/JP7021215B2/ja active Active
  • 2017-08-11 EP EP17840359.8A patent/EP3497603A4/en active Pending
  • 2017-08-11 AU AU2017308120A patent/AU2017308120B2/en active Active

Patent Citations (7)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events