JP2019526869A - CAD system personalization method and means for providing confidence level indicators for CAD system recommendations - Google Patents

Abstract

Methods and means for utilizing machine learning to train a device to generate a confidence level indicator (CLI). The device is a CAD system that was initially trained using initial machine learning to recommend classifications for image features presented to the device. In order to better indicate the confidence level of the CAD system's recommendation as to which class should be associated with a particular image feature, probabilistic classification is used to incorporate intermediate values given by human operators.

Description

Cross-reference of related applications
[0001] This patent application was filed on August 11, 2016 and is named "Method and Means of CAD System Personalization to Provide a Confidence Level Indicator for CAD System Recommendations" The entire content of which is a continuation of US patent application Ser. No. 15 / 235,050, incorporated herein by reference in its entirety.

background
[0002] Field
[0003] The inventive concept relates generally to medical imaging and analysis, and more specifically, within the framework of an image reporting and data system (IRADS) for medical diagnosis. The present invention relates to a system and method for improving decision making ability. The concept of the present invention is a confidence level display (CLI) for a computer-assisted diagnosis (CAD) system programmed to minimize deviations in recommended clinical practices due to group or individual bias in the interpretation of system rules. : Confidence level indication).

[0004] 2. Consideration of related fields
[0005] Each year, more than 1,300,000 breast biopsies are performed in the United States and more than 4,500,000 breast biopsies are performed worldwide. Of these biopsies, 80% have benign findings. Breast biopsy is a traumatic experience that puts patients at clinical risk, is uncomfortable and expensive.

  [0006] A trained medical professional, such as a radiologist, will typically attempt to identify and classify suspicious areas in a medical image, either manually or using computer software. The radiologist may then manually characterize each suspicious area according to an associated scoring scheme. For example, suspicious regions of interest within the breast may be characterized according to Breast Imaging Reporting and Data Systems (BI-RADS) guidelines. BI-RADS is a widely accepted risk assessment and quality assurance tool used by radiologists to diagnose breast cancer using mammography, ultrasound, or MRI. The classification assigned to each suspicious area can determine future policy. For example, if the suspicious area is classified as possibly malignant, a biopsy may be indicated. If the suspicious area is classified as normal, no further action may be taken. However, if the suspicious area is classified as possibly benign, the policy may be to repeat the test to see if there are any changes after 6 months. In the BI-RADS reporting approach, the radiologist provides a concise review of the image-based findings and communicates the results to the referring physician in a clear and consistent manner, with final assessment and specific policies Includes standard lexicons and structures for reporting purposes that allow Structured reporting accelerates report turnaround time (TAT), simplifies billing and regulatory compliance documentation, and simplifies the process of data review for usage review, quality assurance, and research purposes It also helps. Through medical audits and outcome monitoring, the system provides an important mechanism for mutual assessment and quality assurance data to enhance the quality of patient care. Results summarized in a standardized format allow for maintenance and collection analysis of demographic and outcome data.

  [0007] The success of BI-RADS since the start of mammography in 1993 spurred the introduction of many more similar checklist-based systems for various medical image reporting data systems. Some lists of similar reporting data systems include Prostate Imaging-Reporting and Data System (PI-RADS), Thyroid Imaging Reporting Data System (PI-RADS) for diagnosing prostate, thyroid, liver, and lung cancer, respectively ( TI-RADS (Thyroid Imaging Reporting and Data System), Liver Imaging Reporting and Data System (LI-RADS), and lung RADS.

[0008] The basic BI-RADS evaluation categories are as follows.
[0009] 1: negative,
[0010] 2: benign,
[0011] 3: Probably benign,
[0012] 4. Suspicious, and
[0013] 5: Strongly suggests malignancy.

  There are also category 0 (BI-RADS0) and category 6 (BI-RADS6). BI-RADS0 is an incomplete that requires previous images for comparison or an effort to get the patient back for additional views, higher quality films, or additional diagnostic imaging. Indicates the classification. BI-RADS6 shows a proven malignancy already proven by biopsy.

The classification of BI-RADS4 is often divided into the following subcategories.
[0016] 4A: Low suspicion of malignancy,
[0017] 4B: moderate suspicion of malignancy, and
[0018] 4C: High suspicion of malignancy.

  [0019] The recommended patient management provided by the BI-RADS system should be directed to biopsy if the area of interest is suspicious or very suspicious, ie, classified as BI-RADS4 or BI-RADS5. is there. If the suspicious area is classified as normal or benign, ie, BI-RADS1 or BI-RADS2, no further action may be taken. However, if the suspicious area is probably benign, ie, BI-RADS3, the recommendation is a 6 month follow-up to check for changes. The BI-RADS score is a statistical value correlated with a malignant tumor and is not a deterministic evaluation criterion of a malignant tumor. It has been shown that Category 3 (less than 2% risk of malignancy) or Category 4 (3% to 94% probability of cancer) lesions are considered to be of different degrees of malignant breast lesions. This is especially true for category 3 hyperplastic nodules that are considered uncertain. Such lesions do not have the obvious characteristics of benign lesions, but are still subjectively considered category 3 lesions. Although there are one or two non-benign features of Category 4 lesions, the American College of Radiology does not provide any detailed guidance. This leads to poor consistency between observers in the classification, resulting in differences from the ideal use of the BI-RADS system. Further, category 4a generally consists of 90% to 98% benign lesions, 4b consists of 50% to 90% benign lesions, and 4c consists of 5% to 50% benign lesions, As well as BI-RADS5 has 0% to 5% benign lesions, all have to go to biopsy. Thus, as many as 80% of biopsies performed on patients in the BI-RADS4 or BI-RADS5 category are found to be benign.

  [0020] Computer-aided diagnosis (CAD) systems have the potential to improve the diagnostic performance of a radiologist. In practice, however, it is difficult for the radiologist to know when to accept or decline a recommendation made by the CAD system.

  [0021] Particularly with these findings in mind, various aspects of the inventive concept have been conceived and developed.

Overview
[0022] The inventive concept is a computerized system configured to utilize a probabilistic classification incorporating intermediate values provided by a human operator to better indicate the confidence level for a CAD recommendation system. provide.

  [0023] The above can be achieved in one aspect of the inventive concept by providing a method for personalizing a diagnostic support system. The method may include utilizing machine learning to train the device to provide a confidence level display (CLI) for computer-aided diagnosis (CAD) system recommendations. Such training may include accessing a plurality of training image features. Each of the plurality of training image features may be associated with a known class of the plurality of classes. The known class may correspond to a known correct diagnostic decision for each of the training image features.

  [0024] The training may further include accessing a plurality of initial clinician recommended diagnostic decisions from at least one operator corresponding to each of the plurality of training image features. Each of the plurality of initial clinician recommended diagnosis decisions may include a clinician confidence.

  [0025] The training may further include accessing a plurality of CAD system recommended diagnosis decisions corresponding to each of the plurality of training image features. Each of the plurality of CAD system recommended diagnostic decisions may include a classifier output score or a combination of score ensembles.

  [0026] The training may further include accessing a plurality of initial clinician recommended diagnostic decision subsets corresponding to the plurality of training image feature subsets. The subset of initial clinician recommended diagnosis decisions may be different from the specific CAD system recommended diagnosis decision corresponding to the subset of training image features.

  [0027] The training may further include generating a function that defines a CLI score for each particular CAD system recommended diagnostic decision corresponding to a subset of the plurality of training image features.

  [0028] The method may further include utilizing initial machine learning to initially train the device by providing an initial training data set associated with the series of training images. At least a portion of the initial training data set may include image features associated with an initial known class of the plurality of classes. Multiple classes may be associated with an initial predetermined possible clinical practice. The method may further include determining a weighted error term cost function based on the results of providing the initial training data set for the device. The method weights specific parameters of the cost function for specific image feature values associated with known examples of clinical significance that are predetermined to be important for diagnosis, and / or Or it may further include the step of penalizing.

  [0029] The method may further include receiving an image selected by the interface. The selected image may include image features. The method is trained using a weighted cost function to (i) extract at least one image feature value from a selected image and / or (ii) identify a class from multiple classes. The method may further comprise utilizing the device to provide a specific clinical action by applying at least one image feature value to the device. The device may be trained using initial machine learning before utilizing machine learning to train the device to provide a CLI for computer-aided diagnosis (CAD) system recommendations. The CLI score for each particular CAD system recommended diagnostic decision corresponding to a subset of multiple training image features may be specific to a particular type of image feature and / or unique to at least one operator. The one or more parameters of the function may include a known correct diagnostic decision for each of the clinician confidence, classifier output score, and / or a subset of the plurality of training image features. The step of generating a function that defines a CLI score may include the CAD system providing local areas of the image that are weighted more heavily on the system's confidence in system recommendations.

  [0030] The method involves periodically (eg, annually) repeating one or more phases or training steps (eg, initial training phase) of the CLI over a period of time (eg, a radiologist). The method may further include a step of adapting to the learning behavior of Learning behavior is an improvement in the ability to correctly detect and / or diagnose a product using a CDI / CAD system over a period of time, for example, and / or diagnostic image analysis that a radiologist may have acquired over a period of time. Other learning experiences may be possible. A plurality of initial clinician-recommended diagnosis decisions corresponding to a plurality of training image feature subsets may be a specific operator, a plurality of specific operators, an institution, a locale, a workflow location, and / or a plurality of training image feature subsets. An initial decision profile and / or a final decision profile by aggregating final decisions made by a plurality of operators, including being used with a function that defines a CLI score for each of the specific CAD system recommended diagnostic decisions corresponding to May be included. That is, the initial score of an individual or group is recorded following the scores after viewing the CAD recommender score. These scores represent the CLI or a function associated with it so that the trained CAD device / system estimates a probability that is more correct than an individual or group decision for cases similar to each case seen during training. May be used to help train.

  [0031] Each of the multiple classes may be associated with a different category of Breast Image Reporting Data System (BI-RAD) lexicon. Each of the plurality of training image feature images may include a pixel value and / or a subset of pixel values associated with a corps region of interest. The function may be based on or factored in one or more intermediate values provided by the operator and / or CAD system recommendations to calculate a CLI score. The function may utilize probabilistic classification while incorporating intermediate values provided by human operators to better indicate the confidence level of the CAD system recommendation defined by the CLI score.

  [0032] The above can be achieved in another aspect of the inventive concept by providing a method for training a diagnosis support system. The method may include utilizing machine learning to train the device to provide a confidence level display (CLI) for computer-aided diagnosis (CAD) system recommendations. This training may include accessing a training image feature associated with a known class corresponding to a known correct diagnostic decision for at least one training image feature. This training may further include accessing a clinician recommended diagnostic decision from at least one operator corresponding to the training image feature. This training may further include accessing CAD system recommended diagnostic decisions corresponding to training image features having a classifier output score or ensemble combination of scores. The CAD system recommended diagnosis decision may be different from the clinician recommended diagnosis decision. This training may further include generating a function that defines a CLI score for the CAD system recommended diagnostic decision. The one or more parameters of the function may include known correct diagnostic decisions regarding clinician confidence, classifier output scores, and / or training image features. The clinician recommended diagnostic decision may define an intermediate value provided by at least one operator.

  [0033] In another aspect of the inventive concept, the above can be achieved by providing a diagnostic support training system or apparatus. The apparatus may include at least one computing device. The computing device is operable to be trained by machine learning to generate a recommendation class based on one or more image features using a cost function of one or more weighting terms. One or more parameters of the cost function may be weighted and / or penalized for specific image features associated with a known example of predetermined clinical significance. Good. Additional machine learning may be applied to the computing device. Such additional machine learning may include a plurality of clinician diagnostic decisions accessed by a computing device from an operator corresponding to each of the plurality of training image features. Each of the plurality of initial clinician diagnostic decisions may have a clinician confidence. Such additional machine learning may include a plurality of CAD system diagnostic decisions corresponding to each of a plurality of training image features accessed by the computing device. Each of the plurality of CAD system diagnostic decisions may include a combination of classifier output scores or ensembles of scores. The subset of clinician diagnostic decisions may correspond to a subset of training image features that are different from a particular CAD system diagnostic decision for the training image features accessed by the computing device. Such additional machine learning may include functions performed by the computing device that may define a CLI score for each particular CAD system diagnostic decision corresponding to a subset of the plurality of training image features. Good. This function is given in class c, given the feature vector X, and the operator selects label Z, and the CAD system recommendation is defined by label W and the known ground truth is defined as label Q. It may be defined as P (c / X; and Z; and W; and Q) for calculating a certain probability. The CLI score can use probabilistic classification to define the probability that a computing device will correctly generate a recommendation class by taking into account an intermediate value provided by at least one operator.

Brief Description of Drawings
[0034] The foregoing and other objects, features, and advantages of the inventive concepts described herein will become apparent from the following description of specific embodiments of the inventive concepts as illustrated in the accompanying drawings. It should be. The drawings show only typical embodiments of the inventive concept and are therefore not to be considered limiting in scope.

[0035] FIG. 4 is an exemplary process flow according to aspects of the inventive concept. [0036] FIG. 6 is an exemplary process flow according to aspects of the inventive concept. [0037] FIG. 4 is an exemplary process flow according to aspects of the inventive concept. [0038] Fig. 5 is an exemplary process flow according to aspects of the inventive concept. [0039] FIG. 6 is an exemplary process flow according to aspects of the inventive concept. [0040] An exemplary computing device capable of implementing the various services, systems, and methods described herein.

  [0041] The drawings do not limit the inventive concept to the specific embodiments disclosed and described herein. The drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of certain embodiments of the inventive concept.

Detailed description
[0042] The following detailed description refers to the accompanying drawings that illustrate various embodiments of the inventive concepts. The illustrations and descriptions are intended to describe aspects and embodiments of the inventive concepts in sufficient detail to enable those skilled in the art to practice the inventive concepts. Other components can be utilized and changes can be made without departing from the scope of the inventive concept. The following detailed description is, therefore, not to be construed in a limiting sense. The scope of the inventive concept is defined only by the full scope of equivalents to which such claims are entitled, along with the appended claims.

[0043] I. Terminology
[0044] The terminology is used herein to describe features of the inventive concept. For example, reference to the terms “one embodiment”, “an embodiment”, “the embodiment”, or “embodiments” is a reference. One or more of the features being included is meant to be included in at least one aspect of the inventive concept. Separate references to the terms “one embodiment”, “an embodiment”, “the above embodiment”, or “a plurality of embodiments” herein do not necessarily refer to the same embodiment, and And / or is not mutually exclusive, except where otherwise stated and / or readily apparent to one of ordinary skill in the art from this specification. For example, features, structures, processes, steps, or acts described in one embodiment may be included in other embodiments, but are not necessarily included. Accordingly, the inventive concepts may include various combinations and / or integrations of the embodiments described herein. Moreover, no aspect of the disclosure described herein is essential to its implementation.

  [0045] The term "algorithm" includes, but is not limited to, the functions of the inventive concept specifically described herein or readily apparent to one of ordinary skill in the art in view of this specification. Refers to logic, hardware, firmware, software, and / or combinations thereof configured to perform one or more functions. The logic may include a network having data processing and / or storage functions. Examples of such circuitry include a microprocessor, one or more processors, eg, a processor core, a programmable gate array, a microcontroller, an application specific integrated circuit, a wireless receiver, a transmitter, and / or a transceiver circuitry, Semiconductor memory, or combinational logic may be included, but is not limited to these.

  [0046] The term "logic" refers to an executable application, application programming interface (API), subroutine, function, procedure, applet, servlet, routine, source code, object code, shared library / dynamic load library, or one or Computer code and / or instructions in the form of one or more software modules, such as executable code in the form of instructions. These software modules can be any type of suitable non-transitory storage medium, or temporary storage medium (eg, electrical, optical, acoustic, or other form of propagating signal such as a carrier wave, infrared signal, or digital signal). May be stored. Examples of non-transitory storage media include non-persistent storage such as programmable circuits, semiconductor memory, volatile memory (eg, any type of random access memory “RAM”), non-volatile memory (eg, read-only memory “ROM”) ”, Power backup RAM, flash memory, phase change memory, etc.), solid state drives, hard disk drives, optical disk drives, or portable memory devices, but are not limited or limited thereto. As firmware, executable code is stored in persistent storage.

  [0047] The term "user" is generally used herein as a synonym for a user of the inventive system and / or method. As used herein, a user may be a clinician, a diagnostician, a doctor, a technician, a student, and / or an administrator.

  [0048] The terms "identified", "processed", and "selected" are used herein to refer to at least one processor, regardless of tense. Thus, it is commonly used as a synonym for a computerized process performed or performed automatically by the system in one or more processes.

  [0049] The acronym "CAD" means Computer-Assisted Diagnosis.

  [0050] The term "client" refers to any program of software that connects to a CAD lesion application.

  [0051] The term "server" generally refers to a CAD lesion application that listens to one or more clients, unless otherwise specified.

  [0052] The term "post processing" means an algorithm applied to the input ultrasound image.

  The acronym “PACS” means Picture Archival and Communication System.

  [0054] The acronym "GSPS" means Grayscale Softcopy Presentation State.

  [0055] The acronym "DICOM" means Digital Imaging and Communications in Medicine.

  The acronym “UI” means a user interface.

  [0057] The acronym "PHI" means personal health information.

  [0058] The term "computerized" generally refers to any corresponding operation being performed by hardware in combination with software and / or firmware.

  [0059] The term "ensemble method" means a plurality of learning algorithms for obtaining better performance that can be obtained from any of the constituent learning algorithms.

  [0060] The term "stacking" means training a learning algorithm to combine the predictions of several other learning algorithms. Stacking may also be referred to as stacked generalization.

  [0061] The term "combiner" refers to an algorithm that has been trained to make a final prediction using all other algorithm's predictions as additional inputs.

  [0062] The term "diversity" refers to variations between ensemble methods / models. Many of the ensemble methods seek to promote diversity between the models they combine. By using a more random algorithm (such as a random decision tree), a stronger ensemble can be generated than a very artificial algorithm (such as a reduced entropy decision tree). However, using various strong learning algorithms (diversity) has been shown to be more effective than using techniques that attempt to lower the model level to promote diversity.

  [0063] Finally, as used herein, the terms "or" and "and / or" are to be construed generically or to mean any one or any combination. Accordingly, “A, B, or C” or “A, B, and / or C” means “below: A; B; C; A and B; A and C; B and C; A, B, and C” Means "?" An exception to this definition occurs only when a combination of elements, functions, steps or actions is in some way inherently mutually exclusive.

  [0064] Since the inventive concept is capable of many different types of embodiments, it is intended that this disclosure be considered an example of the principles of the inventive concept, and that the inventive concept will be illustrated and described. It is not intended to be limited to the specific embodiments described.

[0065] II. Reference to US Patent Application No. 15 / 200,719
[0066] An aspect of the inventive concept is to provide an existing computer-aided diagnosis (CAD) recommendation system implemented on a computing device as described in related US patent application Ser. No. 15 / 200,719, incorporated herein by reference. Provide the system and method used. The CAD recommendation system of U.S. Patent Application No. 15 / 200,719 provides a correlation between recommended clinical practices for dealing with image features obtained from individuals or specific expert groups and proven or evidence-based data. Specially programmed to minimize differences between optimal clinical practice based. The CAD recommendation system of US patent application Ser. No. 15 / 200,719 reduces the number of false clinical actions such as biopsy based on the operator's error profile.

  [0067] A previous CAD recommendation system aspect utilizes machine learning to train a computing device to propose clinical decisions regarding image features. A training data set associated with the series of training images can be applied to the computing device. The series of training images may include medical images showing a specific area of the human body such as a human breast. Specifically, the training image may include an image of a part of a breast having a malignant or benign lesion. Each image may have different characteristics such as color and shading depending on the lesions shown. At least a portion of the training data set may have image features associated with a known class of multiple classes (of medical lexicons) from the training image. That is, image features have been proven to link to a specific class of medical lexicons. For example, a BI-RADS lexicon class 1 may be associated with or assigned an image feature of a first training image, and a BI-RADS lexicon class 2 may be an image of a second training image. It may be associated with or assigned to a feature. An image feature may be a vector of a specific image, such as a medical image, or other value. That is, at least a portion of the training data set can provide an example to the computing device regarding when image features should be assigned to one or more classes of medical lexicons.

  [0068] Each class of medical lexicon may be associated with or correspond to a predetermined possible clinical practice. That is, when an image feature enters a particular class, it can be predetermined that a particular clinical decision specific to that class should be recommended. A clinical action may have a specific task or procedure to be performed based on image features, for example. As an example, a clinical act may include performing a biopsy on the lesion to remove the tissue specimen of the lesion and submit the tissue specimen for examination and analysis. Another clinical practice may include following up patients and lesions after a predetermined period, such as 6 months.

  [0069] The cost function of the weighting term may be determined based on training set data applied to the computing device. Further, specific parameters of the cost function may be weighted against specific image feature values associated with known examples of clinical significance that are predetermined as important for diagnosis. For example, certain parameters may be weighted to take into account the difficulty of diagnosing certain image features accurately when a radiologist or other clinician belongs to one or more of several classes. Also good.

  [0070] In certain specific embodiments, the machine learning described above can be used to train a computing device to make clinical decisions in the specific context of a possible cancer diagnosis. A training data set in the form of a pre-selected data pattern group that identifies the relationship between one or more image features and a known correct BI-RAD classification is presented to the computing device for classification. May be. A training data set, for example, has a certain texture, shape, one component / multiple components, shading, color, or other visual features considered to fall into Class 4 with a BI-RAD lexicon. Visible (eg, lesion) image features can be identified. An image feature may further have a certain value (of a pixel, a group of pixels, or a function of a plurality of pixels or pixel groups), for example, a training data set for each computing device A particular value (feature value) of one or more pixels (or a function of one or more pixels) associated with the image is identified. As such, the training data set was considered or pre-determined by the individual radiologist, or group of radiologists, for a computing device, or entered into one or more BI-RAD classes (eg, Further identifying a feature value or feature value set associated with the image (based on clinical evidence). In addition, a biopsy-proven classification (related to whether the lesion is cancerous or benign) is known.

  [0071] In response to the training image, the actual output generated by the computing device may be compared to results proven by known biopsies with reference to a cost function. In one embodiment, output by a computing device regarding how image features should be classified with respect to a radiologist or group of radiologists and selection of a BI-RAD lexicon class for their images. In order to minimize errors, it may be ideal to minimize such cost functions. This aspect of comparison can be used to adjust certain parameters of the computing device and / or cost function, such as weighting, bias, or penalty function added to the error term. Further, specific parameters of the cost function may be weighted against specific image feature values associated with known examples of clinical significance that are predetermined as important for diagnosis. For example, a particular parameter may belong to one or more of several classes that may be particularly important when faced with a decision regarding whether a clinician recommends a biopsy of a lesion, a radiologist or others May be weighted to take into account the difficulty of accurately diagnosing certain image features, or may be penalized. The above process may be repeated until the cost function averaged over the appropriate second pre-selected data pattern group or validation set is minimized. In such an embodiment, training of the computing device to indicate clinical behavior for a BI-RAD lesion involves subsequent test data being presented to the computing device and the computing device outputting an output related to the test data. A generation and comparison between its output and the known correct result may be considered complete when it produces a difference or value within a predetermined tolerance range. The trained computing device may be implemented for a variety of related applications such as training the computing device to make decisions in situations such as radiology and ultrasonography. The trained computing device may be an element of a CAD recommendation system.

  [0072] Aspects of the CAD recommendation system utilize the above-described training stage machine learning / training to recommend new image classifications that have not yet been diagnosed, and these classifications are specific CAD recommendations for image features. Respond to clinical decisions. The newly selected image may be received by the CAD recommendation system computing device using the interface. The computing device scans the selected image, reads image features from the selected image, extracts values from the selected image, and extracts those values into lesions or other abnormalities (during the training stage described above). May be matched to a predetermined value that the computing device is programmed to recognize as being associated with it. In any case, the selected image has at least one image feature that is made accessible to the computing device for analysis.

  [0073] The computing device may extract at least one image feature value from the selected image. The selected image feature value may be associated with a numerical value, and in some embodiments may be a pixel value or a pixel value set. Extracting values in this manner from selected image features can be utilized with data understandable by the computing device and one or more cost functions or other functions constructed during machine learning. Decompose image features into data.

  [0074] The computing device of the CAD recommendation system is utilized by applying at least one feature value to a computing device trained using a weighted cost function. The computing device then outputs a class from among the multiple classes defined during the machine learning process. The weighting or penalty-based training phase of the CAD system increases the probability that the CAD system will show the correct score when the operator is likely to be incorrect. A weighting or penalty function is used to increase the likelihood that the CAD error profile is different from the operator, typically by placing more weight on errors by one or more users. Alternatively, penalizing error terms most likely to be created by one or more operators increases the likelihood that they will be corrected in the operation of the resulting CAD system. For example, for a given image of a benign image proven by biopsy, the computing device may output a score corresponding to two BI-RAD categories indicating that the lesion is benign, The same image may be most likely categorized as BI-RADS4 by the operator indicating that the image feature value is associated with a suspicious lesion and should be further diagnosed. Thus, if radiologists followed CAD system recommendations rather than their initial diagnosis, they may have been able to eliminate this unwanted biopsy.

[0075] III. Overall architecture of CLI system
[0076] CAD recommendation systems, such as the CAD recommendation system of US patent application Ser. No. 15 / 200,719, incorporated herein by reference, generally operate at error rates similar to those experienced by radiologists. If the CAD recommendation system outputs a class of image classification, this class corresponds to a CAD recommended diagnosis decision on image features, and if this CAD recommended clinical decision matches the initial clinician recommended diagnosis decision, the initial clinical The operator (or other user) who made the medically recommended diagnostic decision is given further confidence in his diagnosis of image features. However, if the CAD recommended diagnosis decision does not coincide with the initial clinician recommended diagnosis decision, the operator can determine whether the support is an unsupported diagnosis (initial clinician recommended diagnosis decision that does not take into account recommendations from the CAD system) or CAD recommended diagnosis decision You will be forced to make a decision to choose.

  [0077] The operator can then make a final decision based on the operator's confidence in the initial diagnostic decision compared to the operator's confidence in the CAD recommended clinical decision. If the operator is more convinced by his opinion than the capabilities of the CAD system, the operator will be biased towards refusing to recommend the CAD system. On the other hand, if the operators are more confident about the CAD system than their capabilities, they will be biased towards accepting the recommendation of the CAD system. This bias due to confidence perception or lack of confidence limits the performance of the CAD and radiologist / operator combination in the decision-making process within each step of the diagnostic workflow.

  [0078] Ideally, the CAD system differs from the opinion of an operator such as a radiologist only if the radiologist is not correct, and the CAD system can correct the radiologist. Furthermore, in the ideal case, the CAD system will always agree when the radiologist is right. However, there may be cases where the radiologist is correct and the CAD system is incorrect. An optimal decision-making process may be obtained if the radiologist can have some additional confidence in the recommendations given by the CAD system. Standard output values of classifiers used in machine learning can only be interpreted as probabilities under very limited conditions that are rarely met in practice.

  [0079] The inventive concept solves this problem and, for a particular image / image feature, if the radiologist's opinion differs from the recommendation of the CAD system, the radiologist It is operable to help determine when to accept CAD recommendations rather than initial opinions. The inventive concept addresses this problem by collecting training data while a radiologist uses a CAD system against a database of real data. The training session serves the purpose of acquainting the radiologist with the strengths and weaknesses of the CAD system and collecting data regarding when the CAD system can help correct the radiologist's initial decisions. The training data is used to train a predictive model that provides a confidence score for a particular CAD system recommendation by the CAD recommendation system. The predictive model provides a score that represents the likelihood that the CAD recommendation is correct given the image data and previous performance of the radiologist. The radiologist can use a personalized confidence score to help determine when it is best to follow the CAD system. The inventive concept allows radiologists to calibrate their confidence in the CAD system more effectively.

  [0080] The CAD recommendation system may be trained to have errors that complement the errors by one or more radiologists using the system. CAD systems may be trained to penalize errors by radiologists more heavily to provide errors that complement the radiologist's errors in the CAD system. The inventive concept utilizes machine learning / training to further improve or adjust the CAD recommendation system. The inventive concept is the way in which they are trained to learn when operators (eg, radiologist) should trust a CAD system with greater confidence than their initial opinions. And providing means. The operator can be asked to diagnose a pre-stored set of cases based on observing the appropriate medical image data set. The CAD system provides a second opinion. The radiologist is free to change his original opinion based on CAD recommendations, or can decline the recommendation. The examples then help the operator to learn which cases can benefit the operator by using the CAD system recommendations and which cases need less help from the CAD system. In addition, a true diagnosis may be displayed along with images of similar cases. This training phase can be used to form a confidence level indicator (CLI) statistical or predictive model.

  [0081] The purpose of the CLI is to determine which is more likely to be correct given the case data and the respective trends and potential biases in case of discrepancies between the CAD system and the operator It is to be. For this purpose, affected by radiological data, true (biopsy proven) diagnosis of the data, interpretation of the CAD system of the data, interpretation of the operator of the data, and the CAD system The CLI parameters are determined by taking into account changes in the interpretation of the operator's data. As used herein, a true diagnosis of data can refer to a diagnosis proven by biopsy of the case in question.

[0082] Interpretation of operator data can be obtained using a number of methods, and some of these methods can include certain combinations of:
1) Direct label-In this scenario, the operator is tasked with estimating the likelihood of malignancy of the lesion set. In each case situation, these labels are used to inform the CLI of operator trends. At that time, the CLI is personalized to the operator.
2) Consensus label-similar to the previous scenario, but many operators are considered. The CLI is then designed around a label determined by some degree of their consensus. In this case, CLI gains generality for multiple operators in exchange for personalization.
3) Operator model—Rather than directly using operator labels to influence the CLI, these are initially used to build a model of operator decision making. This model can then be used to generate a large number of simulated operator labels and eliminates the need for the operator to manually label many cases.
4) Multi-operator model—an extension of Method 3 in which many operator behaviors are modeled instead of one behavior. This can be achieved by modeling the consensus of many operators and / or considering the consensus of many operator models.
5) Fine-tuning operator model—This method strikes a trade-off between Method 3's personalization approach and Method 4's generalized approach. For this purpose, a general multi-operator model is first constructed. This model can be considered a baseline, which can then be fine-tuned to follow individual operator preferences, whatever the degree to be considered optimal.
6) Archetype operator model-This method extends both of the previous two. In this embodiment, several models are pretrained and represent different operator archetypes. Individual operator behavior can be matched to a model that reflects their behavior in most detail. The archetype model can then be used directly or can be further fine-tuned for the user.

  [0083] Using the aforementioned factors, the discrepancy between the CAD and the operator can be resolved in the most optimal manner. This solution can be viewed as some confidence in the CAD decision for one operator in the operator population.

  [0084] Aspects of the inventive concept can be described with reference to the flowchart 100 of FIG. Flowchart 100 shows pre-CAD training. As shown in block 102, multiple training images may be accessed from a medical image database. The training image may represent an area of the human body that has characteristics that may indicate illness, injury, or distress. For example, the training image may include an image of a portion of the breast that needs to be diagnosed to determine whether the indicated portion is a potential cancer area. At block 104, a feature extraction process may be performed. At block 106, a plurality of image features may be extracted from the training image. In block 110, an operator such as a radiologist or other clinician may receive personalized radiologist model training. Specifically, the operator may recommend a diagnostic decision or pre-CAD viewing diagnosis as shown in block 114 for each of at least some of the image features extracted from the training image. Specific to an operator (or operator group) having data on when a particular operator made a wrong diagnostic decision (ie, a decision different from a known proven diagnostic decision (as shown in block 112)), A personalized radiologist / operator pre-CAD model as shown at block 116 may be generated. The personalized radiologist / operator pre-CAD model at block 116 identifies which types of image features a particular operator tends to make wrong decisions (ie, decisions that are different from known correct diagnostic decisions). Can help you. It is important that the flow chart 100 of FIG. 1 shows the personalized radiologist / operator pre-CAD model of block 116 that is derived without the operator looking at the CAD data. That is, the operator of FIG. 1 undergoes pre-CAD training to build a personalized radiologist / operator pre-CAD model at block 116.

  FIG. 1 further illustrates an optional radiologist pre-CAD viewing diagnostic database (pre-CAD database) 108. The pre-CAD database may be utilized to train an operator (eg, a radiologist) prior to actually measuring the operator's performance using the personalized radiologist model training at block 110. As such, the pre-CAD database 108 can help an operator to better understand the functionality of the CAD system (which recommends diagnostic decisions during actual training).

  [0086] FIG. 2 shows a flowchart of a training process similar to FIG. This flowchart illustrates post-CAD viewing training (ie, analyzing diagnostic decisions for operator post-CAD viewing). Similar to FIG. 1, at block 102, multiple training images may be accessed from a medical image database. The training image may represent an area of the human body that has characteristics that may indicate illness, injury, or distress. For example, the training image may include an image of a portion of the breast that needs to be diagnosed to determine whether the indicated portion is a potential cancer area. At block 104, a feature extraction process may be performed. At block 106, a plurality of image features may be extracted from the training image.

  FIG. 2 further illustrates an optional radiologist pre-CAD viewing diagnostic database (pre-CAD database) 158. The pre-CAD database may be utilized to train an operator (eg, a radiologist) before actually measuring the operator's performance using the personalized radiologist model training of block 130. As such, the pre-CAD database 158 can help an operator to better understand the functionality of the CAD system (which recommends diagnostic decisions during actual training).

  [0088] At block 130, an operator, such as a radiologist or other clinician, may receive personalized radiologist model training. Specifically, the operator may recommend a diagnostic decision or a post-CAD viewing diagnosis as shown in block 132 for each of at least some of the image features extracted from the training image. When a particular operator made a wrong diagnostic decision (ie, a decision different from a known proven diagnostic decision (as shown in block 112)) and when the operator can access CAD decisions for the same image features Nevertheless, a personalized radiologist / operator post CAD model as shown in block 152 may be generated that is specific to the operator (or operator group) with data regarding what the decision was made. The personalized radiologist / operator post CAD model at block 152 identifies which types of image features a particular operator tends to make wrong decisions (ie, decisions that are different from known correct diagnostic decisions). Can help you. It is important that the flowchart 150 of FIG. 2 shows the personalized radiologist / operator post CAD model of block 152 derived with the operator looking at the CAD data. That is, the operator of FIG. 2 undergoes post-CAD training to build the personalized radiologist / operator pre-CAD model of block 152.

  FIG. 3 shows another flowchart 200 of CLI model training, the output of which is a personalized operator / radiologist trained CLI model as shown in block 214. Similar to FIGS. 1 and 2, at block 102, multiple training images may be accessed from a medical image database. The training image may represent an area of the human body that has characteristics that may indicate illness, injury, or distress. For example, the training image may include an image of a portion of the breast that needs to be diagnosed to determine whether the indicated portion is a potential cancer area. At block 104, a feature extraction process may be performed. At block 106, a plurality of image features may be extracted from the training image.

  [0090] As shown, confidence level indicator model training consists of known correct diagnostic decision labels in block 112, CAD system classifier output in block 210, trained radiologist pre-CAD diagnosis in block 202, block 206 The personalized radiologist pre-CAD model, the trained radiologist post-CAD diagnosis at block 204, and the personalized radiologist post-CAD model at block 208 may be captured as inputs.

  FIG. 4 is a flowchart 300 for classifying image features using the CLI model derived from FIG. Similar to FIGS. 1 and 2, at block 302, multiple training images may be accessed from a medical image database. The training image may represent an area of the human body that has characteristics that may indicate illness, injury, or distress. For example, the training image may include an image of a portion of the breast that needs to be diagnosed to determine whether the indicated portion is a potential cancer area. At block 304, a feature extraction process may be performed. At block 306, a plurality of image features may be extracted from the training image.

  [0092] At block 312, a personalized confidence level indicator model that takes as input the CAD classifier of block 308 and the optional radiologist initial pre-CAD diagnosis of block 310 may be utilized. As further illustrated, CAD classifier 308 generates the CAD classifier output of block 314, and the personalized confidence level indicator model generates the CLI model output of block 316. That is, a personalized confidence level index model can be used to generate a confidence level index for CAD system recommendations for classifier output or diagnostic decisions. The confidence level index or 316 CLI model output indicates that the 314 CAD classifier output may be correct if the CAD system recommendation is different from the clinician diagnostic decision. The confidence level indicator provides a confidence level that is similar to the correct probability for the CAD output, so if the CAD system indicates that an image feature is associated with the cancer lesion, the CLI may be, for example, this image A 0.95 may be output indicating a 95% confidence level for a CAD diagnostic decision regarding that the feature is associated with a cancer corps.

  [0093] In one embodiment, prior to training a device to calculate a CLI score for a CAD system recommendation, the first stage of supervised machine learning suggests a classification based on specific image features. You may have training the device. Specifically, the device can be trained (machine learning) to output a score s (Xi) for each input vector Xi using a fixed set of multidimensional feature vectors Xi. The resulting trained device is then used to generalize the patterns learned by the device during training (sometimes referred to as learning) in operations on new data for which the device is not trained. May be.

  [0094] For most supervised learning / training methods, s (Xi) is a scalar value between 0 and 1, and Xi is from the most probable member to the least probable member of class c. Generate a classifier that outputs a score s (x), which is a feature vector (ie, a multi-valued array) that can be used to rank the examples in the test set. That is, for the two examples x and y, if S (x) <S (y) (the output score given by the classifier of x is less than the score given for the vector y), then P The probability that (c / x) <P (c / y) (an image having a feature vector x is in class C (eg, cancer) is greater than the probability that an image having a feature vector Y is in class C. Low). However, in many applications, ranking the examples according to class membership probabilities may not be sufficient. Most pattern recognition / machine learning systems give a relative score for the probability of class membership. The inventive concept is operable to generate a confidence estimate (CLI score) using training data collected for an image set along with a user model and ground truth. The method used is generalized to provide a useful confidence score for data that is not used in the training data. Thus, a confidence or CLI score is generated that can be used to help the operator to select the optimal final decision based on the one or more presented image features.

  [0095] That is, the basic problem with classification is that all data values are labeled during training, for example, "1" for cancer or "0" for non-cancer. Values between 0 and 1 are not trained as the output of the classifier, so there is a high degree of freedom as to how intervening values are assigned to the image during actual operation. Thus, only relative ranking is tied to the probability of being in class C. That is, ideally one would want an output score of> 0.8 to indicate that 80% of the images represented by a particular image example are cancerous, but this would be It only means that it is more likely to be cancer than an image with a score of 7.

  [0096] The inventive concept utilizes training data with intermediate values provided by a human operator and a trained CAD system to better indicate the confidence level of the trained CAD system recommendation. Thus, to give a confidence score for the CAD system recommendation, the image with the feature vector X, as well as the CAD recommendation and the expected or actual user initial decision (operator decision may be biased by his / her error profile Given a implicit (using a certain CAD system), a more general value for cancer belief (or correct probability) is given.

[0097] In one embodiment, mathematically, the function for generating a CLI score is:
P (c / X and Z and W and Q)
This is the case where given a feature vector X, the user / operator selects label Z, and the CAD system recommends label W, where a known ground truth is represented by label Q. Indicates the probability that the class is in class c.

  [0098] The label may be cancer or not cancer, ie 1 or 0, respectively. Increasing the specificity of the estimation by factoring information related to the user and / or CAD system results in an increase in the accuracy of the estimation and an improvement or increase in the confidence score specific to the user and the CAD system. This facilitates the machine learning and estimation process. That is, the function for calculating the CLI score is such that, in the context of one or more specific users and / or one or more specific CAD systems, the CLI score reflects the proposed belief in CAD diagnosis decisions. One or more specific operators and one or more specific CAD recommendation systems may be taken into account.

  [0100] The operator is generally 0 (lowest likelihood of cancer) for the i th ROI of N possible regions of interest (ROI) in images collected for patient case studies. ) To 1 (most likely cancer) scores can be assigned. The assigned score is typically 0 or 1 during operation, but intermediate values may be used for training purposes. The operator's score can be expressed as S1 (i) when i = 1 to N. Since the operator's score is the error value (i is 1 to N) of the i-th score indicated by the true score S (i) + El (i), if i = 1 to N, 51 (i ) = S (i) + El (i). The CAD system may be trained to give a similar output score indicated by S2 (i) = S (i) + E2 (i), where i = 1 to N. The CAD system is trained so that its error E2 (i) is statistically independent of the operator's error. Thus, if the score assigned by the operator does not match (or does not match) the independent score of the output score of the CAD system (compared to 0.5, the first assigned score is higher and the second assigned Score is low), a “draw state” is detected, and the optimal decision is unknown. As used herein, the CLI may be equal to a third independent score generation device (similar to another CAD system) where S3 (i) = S (i) + E3 (i). In the CLI, the error E3 (i) (i = 1 to N) is the operator error El (i) (i = 1 to N) and the first CAD system error E2 (i) (i = 1 to N). May be designed to be statistically independent of both. Thus, for example, if a draw situation occurs between the operator and the first CAD system, by using a simple majority voting logic (ie if two of the three scores are high) , Select high, or select low if two of the three scores are low), the output can be used as an independent opinion to break the draw. Alternative logic, such as trimming means, can be used for the three independent scores to obtain a performance that is higher than the operator's unassisted performance. The first CAD system may include CAD systems that have been trained to generate some form of recommendation, but have not been enhanced using CLI functionality as described herein.

  [0101] The system of the present invention follows the recommendations of the CAD system / device rather than its own recommendations when the operator relies on the CAD system / device and, for example, the CAD system / device is correct and the operator is wrong. If so, it allows the operator to correct his error. One unique aspect is the initial training of the device as disclosed in US patent application Ser. No. 15 / 200,719, where the operator is most likely to need help in correcting the operator's unassisted recommendation. Sometimes, the CAD system recommender specifically trains the device to variably (eg, more heavily) weight errors by individuals and / or operator groups to ensure that the correct recommendation is given. It is to be done.

  [0102] The inventive concept is operable to select and utilize one or more different classifiers to optimize the combined decision. US patent application Ser. No. 15 / 200,719 corrects one or more of its errors if an operator chooses to accept one or more recommendations at one or more appropriate times. Accordingly, a new method and means for obtaining a personalized multi-classifier that provides recommendations that can be combined with the operator's initial recommendations to increase operator accuracy is disclosed.

  [0103] The inventive concept teaches a new way to combine the recommendations of the trained CAD system of US Patent Application No. 15 / 200,719 with the operator's initial recommendations to increase the accuracy of final decisions or recommendations, and The operator has a confidence level of CD! If the score is high (much greater than 0.5), you will also learn to choose CAD recommendations, and this reevaluation can correct the operator's incorrect initial evaluation. In addition, the CLI system can provide a low confidence level (much lower than 0.5) when an operator does not follow CAD recommendations and it is probably better to continue his initial evaluation.

  [0104] FIG. 5 is a process flow 400 for generating a CLI model for use with CAD system recommendations for diagnostic decisions. At block 402, machine learning may be utilized or otherwise performed by the device to provide a confidence level indicator (CLI) for CAD recommendation diagnostic decisions generated from the target device or other devices. May be. If the device itself is already initially trained to generate a recommendation for diagnostic determination of image features accessed by the device, the device is related US patent application Ser. No. 15 / 200,719, incorporated by reference. May have been trained using the initial machine learning described in. Thus, in one embodiment, utilizing machine learning to train a device to provide a CLI can be an improvement over a CAD system that has already been trained to recommend a diagnostic decision, And the same device / application may be modified to take into account the functionality (function / algorithm) used to generate the CLI score.

  [0105] In block 404, the device may access a plurality of training image features. Each training image feature may be associated within a known class. That is, a particular image feature may be associated with a lesion that has proven to be cancer. In other cases, certain image features have been proven to be non-cancerous. In either case, all of the training image features have already been assigned to one clinical class or another clinical class, such as based on evidence or proven test results. The class may correspond to the class of IRADS system described in the related US patent application Ser. No. 15 / 200,719.

  [0106] At block 406, a plurality of initial clinician recommended diagnostic decisions from at least one operator are accessed. Each decision may be made to diagnose one or more of the training image features (these true or known results are unknown to the operator). Operator diagnostic decisions or interpretation of the training image data can be obtained using various methods. For example, direct label consensus labels, operator models, multi-operator models, fine-tuning operator models, and / or archetype operator models may be utilized as described herein (provided that this disclosure Not limited to such models, and additional models are contemplated).

  [0107] At block 408, a plurality of CAD system recommended diagnostic decisions may be accessed for each of the plurality of training image features. If the target device is the same computing device / CAD system that was originally trained to make the CAD system diagnostic decision, this device will use the techniques and functions used during the initial training of the training image. By simply applying, the CAD system recommended diagnostic decision can be calculated for each of a plurality of training image features.

  [0108] At block 410, the device may access a plurality of initial clinician recommended diagnostic decision subsets corresponding to the plurality of training image feature subsets. A subset is when an operator disagrees or does not match a CAD system diagnostic decision or provides a different diagnostic decision for one or more specific training image features.

  [0109] In some embodiments, in this step, the operator / clinician may decide whether to change his / her diagnostic decision when taking the CAD system recommended diagnostic decision into account. it can. For example, in some cases, an operator can make a diagnostic decision for a training image feature and then change that decision given a CAD system recommended diagnostic decision for the same training image feature. Such data can be aggregated for this operator and can be used as part of a function to calculate a CLI score.

  [0110] At block 412, a function may be generated that defines a CLI score for each particular CAD system recommended diagnostic decision corresponding to a subset of the plurality of training image features. That is, if the operator disagrees with a particular CAD recommendation system diagnosis decision, one or more functions can be generated that calculate a CLI score for that particular CAD recommendation system diagnosis decision.

  [0111] FIG. 6 is an exemplary schematic diagram of a computing system 700 that may implement the various techniques described herein. For example, the computing system 700 may have a computing device that is used to implement a CLI application 70 for generating a CLI or CLI score for a particular CAD system recommendation. The computing system 700 includes a bus 701 (ie, an interconnect), at least one processor 702 or other computing element, at least one communication port 703, a main memory 704, a removable storage medium 705, a read only memory 706, and a mass. A storage device 707 is included. One or more processors 702 may be one of Intel (R), Itanium (R), or one or more Itanium processors, AMD (R), Opteron (R), Athlon MP (R). It may be any known processor, such as, but not limited to, one or more processors, or a Motorola® line processor. The communication port 703 may be any of an RS-232 port used with a modem-based dial-up connection, a 10/100 Ethernet port, a gigabit port using copper or fiber, or a USB port. The one or more communication ports 703 may be selected according to a network, such as a local area network (LAN), a wide area network (WAN), or any network to which the computer system 200 is connected. The computing system may further include a transport and / or transit network 755, a display screen 760, an input / output port 740, and an input device 745 such as a mouse or keyboard.

  [0112] Main memory 704 may be random access memory (RAM) or one or more other dynamic storage devices commonly known in the art. Read only memory 706 may be any one or more static storage devices such as programmable read only memory (PROM) for storing static information such as instructions of processor 702. Mass storage device 707 can be used to store information and instructions. For example, hard disks such as the Adaptec® family of small computer serial interface (SCSI) drives, optical disks, disk arrays such as RAID, RAIDtec RAID drives, or other mass storage devices. May be used.

  [0113] Bus 701 couples one or more processors 702 to be communicable with other memory, storage, and communication blocks. The bus 701 may be a PCI / PCI-X, SCSI, or Universal Serial Bus (USB) based system bus (or other) depending on the storage device used. The removable storage medium 705 is any type of external hard disk, thumb drive, compact disk read-only memory (CD-ROM), rewritable compact disk (CD-RW), digital video disk read-only memory (DVD-ROM), and the like. There may be.

  [0114] Embodiments herein are as a computer program product that can include a machine-readable medium having stored thereon instructions that can be used to program a computer (or other electronic device) to perform a process. May be provided. Machine-readable media include: optical disc, CD-ROM, magneto-optical disc, ROM, RAM, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), magnetic or optical card, flash memory, Or other types of media / machine-readable media suitable for storage of electronic instructions may be included, but are not limited to these. The embodiments herein can also be downloaded as a computer program product, in which case the program is transmitted on a carrier wave or other propagation medium via a communication link (eg, a modem or network connection). The embodied data signal can be transferred from the remote computer to the requesting computer.

  Thus, main memory 704 is encoded using CLI application 70 that supports functionality as described herein. The CLI application 70 (and / or other resources as described herein) may provide data and / or logical instructions (eg, memory or disk, etc.) that support processing functionality according to the different embodiments described herein. Code stored in another computer readable medium) may be embodied as software code. During operation of an embodiment, one or more processors 702 may be stored in main memory or otherwise tangibly stored, for example, by executing on processor 702, by logical instructions. Based on the CLI application 70, the bus 701 is used to access the main memory 704 for starting, running, executing, interpreting, or otherwise performing processes.

  [0116] The above description includes exemplary systems, methods, techniques, instruction sequences, and / or computer program products that embody the techniques of this disclosure. However, it is understood that the described disclosure may be practiced without these specific details. In the present disclosure, the disclosed methods may be implemented as an instruction set or software readable by a device. Further, it is understood that the specific order or hierarchy of steps in the disclosed methods is illustrative of an exemplary approach. Based on design preferences, it is understood that the specific order or hierarchy of steps in the method can be rearranged while remaining within the scope of the disclosure. The accompanying method claims present elements of the various steps in a sample order, and are not necessarily limited to the specific order or hierarchy presented.

  [0117] The described disclosure may include a machine-readable medium having stored thereon instructions that can be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. Alternatively, it may be provided as software. A machine-readable medium includes any mechanism for storing information in a form (eg, software, processing application) readable by a machine (eg, a computer). Machine-readable media include optical storage media (eg, CD-ROM), magneto-optical storage media, read only memory (ROM), random access memory (RAM), erasable programmable memory (eg, EPROM and EEPROM), flash memory Or other types of media suitable for storage of electronic instructions may be included, but are not limited to these.

  [0118] Additional aspects, advantages, and utilities of the inventive concept are described in part in the specification and drawings, and in part are apparent from the specification and drawings, or of the inventive concept. Can be learned through implementation.

  [0119] The specification and drawings are intended to be illustrative and not restrictive. Many features and sub-combinations of the inventive concept can be created and will be readily apparent from a study of the specification and drawings. These features and sub-combinations can be used independently of other features and sub-combinations.

Claims (20)

A method for personalizing a diagnostic support system,
Accessing a plurality of training image features, wherein each of the plurality of training image features is associated with a known class of a plurality of classes, and the known class is a known for each of the training image features. Responding to correct diagnosis decisions
Accessing a plurality of initial clinician recommended diagnosis decisions from at least one operator corresponding to each of the plurality of training image features, wherein each of the plurality of initial clinician recommended diagnosis decisions is a clinician belief. Having a degree,
Accessing a plurality of computer-aided diagnosis (CAD) system recommended diagnosis decisions corresponding to each of the plurality of training image features, each of the plurality of CAD system recommended diagnosis decisions being a classifier output score or a score Having a combination of ensembles;
Accessing a subset of the plurality of initial clinician recommended diagnosis decisions corresponding to the subset of the plurality of training image features, wherein the subset of the plurality of initial clinician recommended diagnosis decisions is the plurality of training image features. Different from a specific CAD system recommended diagnosis decision corresponding to the subset of
Generating a function defining a confidence level display (CLI) score for each of the particular CAD system recommended diagnostic decisions corresponding to the subset of the plurality of training image features;
Using machine learning to train a device to provide a CLI for CAD system recommendations. Providing an initial training data set associated with a series of training images, wherein at least a portion of the initial training data set includes image features associated with an initial known class of the plurality of classes. The plurality of classes are associated with an initial predetermined possible clinical practice;
Determining a weighted error term cost function based on the results of the provision of the initial training data set for the device;
Weight or penalize certain parameters of the cost function for specific image feature values associated with known examples of clinical significance that are predetermined as important for diagnosis When,
The method of claim 1, further comprising utilizing initial machine learning to initially train the device. Receiving an image selected by an interface, wherein the selected image has image features;
Extracting at least one image feature value from the selected image and the at least one image on the device trained using the weighted cost function to identify a class from the plurality of classes Utilizing the device to provide a specific clinical action by applying feature values;
The method of claim 2 further comprising:   The device is trained using the initial machine learning prior to utilizing machine learning to train the device to provide the CLI for the computer-aided diagnosis (CAD) system recommendation. 2. The method according to 2.   The CLI score for each of the particular CAD system recommended diagnostic decisions corresponding to the subset of the plurality of training image features is unique to a particular type of image feature and unique to the at least one operator; The method of claim 1.   The method of claim 1, wherein the function parameters include the clinician confidence, the classifier output score, and the known correct diagnostic decision for each of the subsets of the plurality of training image features.   The method of claim 1, wherein the step of generating the function defining the CLI score comprises providing a local area of an image that the CAD system weights more heavily on system belief in system recommendations. the method of.   Further adapting to the radiologist learning behavior that the radiologist may have acquired over the period as a product of using the CDI / CAD system over a period of time by repeating the initial training phase of the CLI. The method of claim 1, comprising:   The subset of the plurality of initial clinician recommended diagnosis decisions corresponding to the subset of the plurality of training image features is a particular operator, a plurality of specific operators, an institution, a locale, a workflow position, or the plurality of training image features. Further comprising an initial and final decision profile by aggregation of final decisions made by a plurality of operators utilized with the function defining a CLI score for each of the particular CAD system recommended diagnostic decisions corresponding to the subset of The method of claim 1.   The method of claim 1, wherein each of the plurality of classes is associated with a different category of breast image reporting data system (BI-RAD) lexicon.   The method of claim 1, wherein each of the plurality of training image feature images has a pixel value or a subset of pixel values associated with a corps region of interest.   The method of claim 1, wherein the function is based on one or more intermediate values provided by an operator and / or one or more CAD system recommendations to calculate the CLI score. The function is
Given the feature vector X, and the operator selects label Z, and the CAD system recommendation is defined by label W, the probability that the known ground truth is in class c, defined as label Q 2. The method of claim 1 having P (c / X; and Z; and W; and Q) for calculating.   The method of claim 1, wherein the function utilizes probabilistic classification while incorporating an intermediate value provided by a human operator to indicate a confidence level of CAD system recommendation defined by the CLI score. Accessing the training image feature associated with a known class corresponding to a known correct diagnostic decision for at least one training image feature;
Accessing a clinician recommended diagnosis decision from at least one operator corresponding to the training image feature;
Accessing a computer-aided diagnosis (CAD) system recommended diagnostic decision corresponding to the training image feature having a classifier output score or ensemble combination of scores, wherein the CAD system recommended diagnostic decision is the clinician recommended diagnostic Different from the decision,
Generating a function that defines a confidence level display (CLI) score for the CAD system recommended diagnosis decision;
Through utilizing machine learning to train a device to provide a CLI for CAD system recommendations.   The method of claim 15, wherein the parameters of the function include clinician confidence, the classifier output score, and the known correct diagnostic decision regarding the training image feature.   The method of claim 16, wherein the clinician recommended diagnostic decision defines an intermediate value provided by the at least one operator. An apparatus having a computing device, wherein the computing device is trained by machine learning to generate a recommendation class based on image features using a cost function of a weighting term, and a specific parameter of the cost function Are weighted or penalized for specific image features associated with known examples of predetermined clinical significance,
A plurality of clinician diagnostic decisions accessed by the computing device from an operator corresponding to each of a plurality of training image features, each of the plurality of initial clinician diagnostic decisions having a clinician confidence. Multiple clinician diagnostic decisions,
A plurality of CAD system diagnostic decisions corresponding to each of the plurality of training image features accessed by the computing device, each of the plurality of CAD system diagnostic decisions being a combination of a classifier output score or an ensemble of scores A plurality of CAD system diagnostic decisions,
A subset of the plurality of clinician diagnostic decisions corresponding to the subset of the plurality of training image features different from a particular CAD system diagnostic decision for the plurality of training image features accessed by the computing device;
A function executed by the computing device that defines a CLI score for each of the particular CAD system diagnostic decisions corresponding to the subset of the plurality of training image features;
An apparatus wherein additional machine learning is applied to the computing device. The function is
Given the feature vector X and the operator selects label Z, and the probability that the CAD system recommendation is in class c, defined by label W and known ground truth is defined as label Q 19. An apparatus according to claim 18 having P (c / X; and Z; and W; and Q) for calculation.   The CLI score defines a probability that the computing device correctly generates the recommendation class by taking into account an intermediate value provided by the at least one operator using probabilistic classification. The device according to claim 18.

Images