JP2020507149A - Methods and systems for automated detection of inclusion or exclusion criteria - Google Patents

Abstract

A method (100) for training a scoring system (600), the method comprising: (i) providing a scoring system including a scoring module (110); Receiving (120) a training data set comprising patient data and treatment results for the training; and (iii) using the clinical decision support algorithm to generate a plurality of clinical decision support recommendations. Analyzing the data set (130); (iv) clustering the plurality of clinical decision support recommendations into a plurality of clusters using the scoring module (140); (v) Using the scoring module, one or more of at least one of the plurality of clusters Identifying a feature and generating 160 the at least one inclusion criterion of the plurality of clusters based on the identified one or more features.

Description

  The present disclosure relates generally to methods and systems for improving clinical decision support algorithms by adding a confidence score based on clustering.

  Clinical decision support (CDS) systems analyze data about patients from a health knowledge perspective to provide information to health professionals for the purpose of improving health care (health care). It is a health information algorithm. This information can be, for example, a diagnosis, a course of action to be recommended, or both. Most health professionals add CDS results to their analysis and decision-making processes based on past experience with CDS, rather than simply agreeing on CDS results and applying CDS results . In fact, experimental experience indicates that there are some characteristics of patients that can cause higher misclassification, including both false positive and false negative recommendations. It has been suggested to. For example, certain medications, interventions, and / or chronic conditions can lead to erroneous recommendations of the CDS algorithm.

  The output of a simple CDS algorithm, including false positives and false negatives, is easily interpreted by a healthcare professional, who will be told when the output from the CDS algorithm should be ignored or when the response to the algorithm is modified. Learn more about what to do. However, as more complex and sophisticated CDS services are provided, it becomes increasingly difficult for medical professionals to interpret CDS output with simple algorithms as confidently as possible. As a result, medical professionals may not fully understand when the output of the CDS algorithm should be used, trusted, or ignored. Moreover, relying on healthcare professionals to interpret the output of complex CDS services not only increases the heavy workload in most healthcare settings, but also reduces the reliability of the output of the CDS algorithm. .

  In view of the above, it would be beneficial to provide methods and systems for improving clinical decision support algorithms and to provide health professionals with reliable and actionable information.

  Accordingly, the present disclosure relates to methods and systems for adding a confidence score to the output of a clinical decision support algorithm. Various embodiments and implementations herein are directed to a system that clusters the output of a CDS algorithm that analyzes training data into a multidimensional parameter space based on accuracy and calculates a confidence score for each cluster. The confidence score for a given cluster indicates the reliability of the prediction from the CDS algorithm that falls within the region of the multidimensional parameter space corresponding to the given cluster. When the system is trained, it receives the output from the CDS algorithm, compares the output with the generated cluster, correlates the output with a confidence score corresponding to the cluster into which the output enters, and provides a confidence score to the medical professional.

  In general, in one aspect, a method for training a scoring system is provided. The method includes: (i) providing a scoring system that includes a scoring module; (ii) receiving a training data set that includes a plurality of patient data and treatment results; and (iii) a clinical will. Analyzing the training data set to generate a plurality of clinical decision support recommendations using a decision support algorithm; and (iv) using the scoring module to generate the plurality of clinical decision support recommendations. Clustering the decision support recommendations into a plurality of clusters; (v) using the scoring module to identify one or more features of at least one of the plurality of clusters; One or more of the at least one of the plurality of clusters based on one or more characteristics Generating an inclusion criterion.

  According to one embodiment, clustering is based, at least in part, on comparing each of the clinical decision support recommendations with treatment outcomes in the training data.

  According to one embodiment, the step of identifying further includes generating one or more exclusion criteria for at least one of the plurality of clusters based on the identified one or more features.

  According to one embodiment, the method includes obtaining health data for the patient, and using the clinical decision support algorithm to generate the clinical decision support recommendation for the patient using the health data. Analyzing; assigning the clinical decision support recommendation to one of the plurality of clusters based on the generated inclusion criteria using the scoring module; Assigning a confidence score to said clinical decision support recommendation based at least in part on assigning said recommendation to said one of said plurality of clusters.

  According to one embodiment, the method further comprises communicating the confidence score to the user.

  According to one embodiment, the confidence score is a quantitative score.

  According to a second aspect, there is provided a method of training a scoring system. The method includes providing a scoring system that includes a scoring module, receiving a training data set that includes a plurality of patient data and treatment results, and using the scoring module to: Clustering the training data set into a plurality of clusters, and analyzing the training data set to generate a plurality of clinical decision support recommendations using a clinical decision support algorithm. Analyzing and clustering using the scoring module to identify one or more features of at least one of the plurality of clusters. The clustering and analyzing are performed separately. And, based on the one or more identified features, And generating the at least one of the one or more inclusion criteria number of clusters.

  According to one embodiment, the step of identifying includes identifying at least one of the plurality of clusters as a cluster having a plurality of incorrect recommendations based on the plurality of clusters compared to the plurality of clinical decision support recommendations. The method further includes a step.

  According to one embodiment, the method further comprises using a scoring module to identify at least one of the plurality of clusters as a noisy cluster, wherein the identified noisy cluster is A cluster for which one or more exclusion criteria are generated.

  According to one embodiment, the step of analyzing comprises analyzing each of the plurality of clusters to generate a plurality of clinical decision support recommendations for each of the plurality of clusters using a clinical decision support algorithm. The method further comprises using the scoring module to determine a low area under the score curve based on at least a portion of the plurality of clusters and the plurality of clinical decision support recommendations. Identifying at least one of said plurality of clusters.

According to one embodiment, the method comprises: (i) obtaining health data for a patient;
(Ii) analyzing the health data to generate clinical decision support recommendations for the patient using a clinical decision support algorithm; and (iii) extracting using the scoring module. Assigning the clinical decision support recommendation to one of a plurality of clusters based on the included inclusion criteria; and (iv) using the scoring algorithm to assign the clinical decision support recommendation to the one of the plurality of clusters. Assigning a confidence score to the clinical decision support recommendation based at least in part on the assignment of the recommendation.

  According to another aspect, a method for providing a confidence score is provided. The method includes: (i) obtaining health data for a patient; and (ii) using the clinical decision support algorithm to analyze the health data to generate a clinical decision support recommendation for the patient. (Iii) assigning the clinical decision support recommendation to one of a plurality of clusters based on the inclusion criteria using a scoring module, the inclusion criteria comprising: (1) Clustering a training data set or clinical decision support recommendation into a plurality of clusters using the scoring module; and (2) using the scoring module to generate at least one of the plurality of clusters. Identifying a feature and, based on the identified one or more features, Assigning, generated by generating one or more inclusion criteria for said at least one of said plurality of clusters; and (iv) using said scoring algorithm to generate said one of said plurality of clusters. Assigning a confidence score to the clinical decision support recommendation based, at least in part, on the assignment of the recommendation to a user; and (v) communicating the confidence score to a user.

  According to one embodiment, the confidence score is a quantitative score that includes an indication of whether to use clinical decision support recommendations.

  As used herein for the purposes of this disclosure, the term "processor" generally describes various device components that are associated with the operation of a recommendation device, system, or method. Used to A processor may be implemented in numerous ways (e.g., with dedicated hardware) to perform the various functions described herein. A “processor” can utilize one or more microprocessors that are programmed using software (eg, microcode) and that can perform the various functions described herein. A processor may also be implemented as a combination of dedicated hardware to perform some functions. Examples of processor components that may be utilized in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). Not done.

  In various implementations, a processor includes one or more storage media (commonly referred to herein as "memory", for example, volatile and non-volatile computer memory, such as RAM, PROM, EPROM, and EEPROM, Floppy disk, compact disk, optical disk, magnetic tape, etc.). In some implementations, the storage medium is encoded with one or more programs that, when executed on one or more processors and / or controllers, perform at least some of the functions discussed herein. You may. The various storage media may be fixed in the processor or the controller, or may be transportable, such that one or more programs stored thereon may execute the various aspects described herein. May be loaded into a processor or controller to implement

  The terms "program" or "computer program" are used herein to refer to any type of computer code (e.g., , Software or microcode). As used herein, the term "non-transitory machine-readable medium" includes both volatile and non-volatile memory, but excludes transient signals. Then you will understand. As used herein, the term "user interface" refers to an interface between a human user or operator and one or more devices that enable communication between the user and the device. Point. Examples of user interfaces that can be used in various implementations of the present disclosure include switches, potentiometers, buttons, dials, sliders, trackballs, display screens, various types of graphical user interfaces, touch screens, Includes but is not limited to microphones and other types of sensors that can receive and generate signals in response to some form of artificial stimulus.

  Various embodiments may further include a non-transitory computer readable storage medium having embodied thereon a firewall program executable by a processor to perform the methods described herein.

  All combinations of the above concepts and additional concepts discussed in more detail below (provided such concepts are not mutually exclusive) are part of the subject matter disclosed herein. Please understand that it is considered. In particular, all combinations of claimed subject matter appearing at the end of the disclosure are considered to be part of the subject matter disclosed herein.

  These and other aspects are apparent from and will be elucidated with reference to the embodiments described hereinafter.

  In the drawings, like reference characters generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles set forth herein.

4 is a flowchart of a method for training a clustering and scoring algorithm according to one embodiment.

4 is a flowchart of a method for training a clustering and scoring algorithm according to one embodiment.

4 is a flowchart of a method for training a clustering and scoring algorithm according to one embodiment.

5 is a flowchart of a method for implementing a clustering and scoring algorithm according to one embodiment.

5 is a flowchart of a method for implementing a clustering and scoring algorithm according to one embodiment.

1 is a schematic diagram of a system for training and implementing a clustering and scoring algorithm according to one embodiment.

  This disclosure describes various embodiments of methods and systems for providing reliable and actionable information to healthcare professionals. More generally, Applicants have recognized and appreciated that it would be beneficial to provide a method or system for increasing the output of a clinical decision support algorithm with a confidence score. A particular goal of utilization of certain embodiments of the present disclosure is to provide a reporting system that provides the output and confidence score of the CDS algorithm to health professionals.

  In view of the above, various embodiments and implementations cluster the output from a CDS algorithm that analyzes training data in a multi-dimensional parameter space based on accuracy, and calculate a confidence score for each cluster. Pointed to the system. The confidence score for a given cluster indicates the reliability of the prediction from the CDS algorithm that falls within the region of the multidimensional parameter space corresponding to the given cluster. The trained algorithm can then receive the output from the CDS algorithm, compare the output with the generated cluster, and associate the output with a confidence score corresponding to the cluster into which the output falls. The confidence score is then reported to the healthcare professional and the healthcare professional can interpret the CDS taking into account that score.

  Referring to FIG. 1, in one embodiment, a method 100 for training a clustering algorithm. This method clusters the output of a CDS algorithm that analyzes training data into a multidimensional parameter space based on accuracy, and calculates a confidence score for each cluster. The confidence score for a given cluster indicates the reliability of the prediction from the CDS algorithm that falls within the region of the multidimensional parameter space corresponding to the given cluster. Once the system has been trained, it can correlate the output of the CDS algorithm with a confidence score, leading to the output of patient-centered personalized CDS medicine over the course of care.

  In step 110 of the method, a scoring service, system, or device is provided. The scoring service, system, or device may be any system described or otherwise envisioned herein. For example, the scoring service may be a cloud-based service provided to healthcare professionals, such as software as a service option, and thus may be hosted on one or more specialized remote servers. May be done. Alternatively, the scoring service may be hosted locally to the medical professional, such as on a special computer or server at a hospital or other medical facility.

  At step 120 of the method, training data is obtained. The training data is used to train a clustering algorithm based on a comparison of the output of a CDS algorithm that provides a diagnosis and / or a recommendation to actual diagnosis and / or treatment decisions. Thus, training data may include clinical and / or non-clinical measurements, or other data from the patient, as well as treatment decisions and results. Training data can be obtained from one or more sources and can be generalized for all locations, settings, or one or more other variables. Alternatively, the training data can be obtained from a specific location that is most applicable to where the clustering algorithm is implemented. For example, training data may be specific to a hospital or other healthcare facility, and / or a physician or other healthcare professional. Training data can include, for example, health data, background information, demographic data, and clinical measurements for a plurality of patients. Each of these patients in the training data may be associated with one or more treatment decisions made and one or more results of these treatment decisions.

  At step 130 of the method, the CDS algorithm analyzes the training data to generate one or more diagnoses and / or one or more treatment recommendations. According to one embodiment, the CDS algorithm for analyzing the training data is the same CDS algorithm utilized by medical professionals in cooperation with the scoring algorithm. Alternatively, the CDS algorithm is different from the CDS algorithm utilized by medical professionals in cooperation with the scoring algorithm. The CDS analyzes information in the training data normally provided to the CDS, such as health data, background information, demographic data, and clinical and / or non-clinical measurements, or other data regarding the patient. .

  In step 140 of the method, the clustering algorithm clusters the output of the CDS algorithm. According to one embodiment, the clustering algorithm includes known clustering, including, but not limited to, K-means, spectral clustering, and other clustering approaches. A system, approach, or algorithm. The clustering algorithm may cluster the output of the CDS algorithm based on a comparison of the CDS-based diagnosis and / or recommended treatment with known diagnoses and treatment decisions. Therefore, the clustering algorithm outputs the output of the CDS algorithm as True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). It can be clustered into groups related to accuracy as compared to such known diagnostic and treatment decisions. According to one embodiment, clusters can then optionally be fixed, so that clustering can be verified on an independent set of patients before deployment.

  At step 160, the clustering algorithm identifies or extracts one or more features of one or more clusters whose features are strongly correlated with the cluster. The identified or extracted features provide inclusion and exclusion criteria that can be utilized by scoring and / or CDS algorithms for analysis of future data. For example, the identified or extracted features may be associated with false positive (FP) and / or false negative (FN) clusters. According to one embodiment, the output of the clustering algorithm is a cluster and one or more identified or extracted features associated with one or more of these clusters. Each cluster may be associated with a generated feature, an identified feature, or an extracted feature, or only one or a few of the clusters may be associated with the generated feature, the identified feature, or the extracted feature. May be associated with a given feature.

  There are several ways to cluster, identify features, and calculate confidence scores. In addition to the methods described above, a clustering algorithm can be first applied to the training data. Accordingly, in step 132 of the method, a clustering algorithm is applied to the predetermined data to generate one or more clusters.

  At step 142, the data in each cluster is analyzed using the CDS algorithm. Each cluster is analyzed separately. As a result, an area under the curve (AUC) score of each cluster is obtained.

  At step 150, clusters with low AUC scores are identified. This may be based on a learned threshold or a predetermined threshold. For example, a health professional or medical facility may determine a particular threshold based on one or more factors, internal factors or external factors. As an example, a hospital or health care professional can intentionally set a very high threshold to eliminate as many false positives and / or false negatives as possible. Instead, the hospital or healthcare professional may deliberately set the threshold low, with the understanding that there may be more false positives and / or false negatives in the output to the healthcare professional. Is also good. The threshold may be a learned threshold and aims to identify and / or trust the cluster as much as possible. Thus, the threshold may change over time or in response to one or more internal and / or external factors.

  In step 160, the scoring algorithm identifies or extracts one or more features of one or more clusters whose features are strongly correlated with the cluster. The identified or extracted features provide inclusion and exclusion criteria that can be utilized by scoring and / or CDS algorithms for analysis of future data.

  Referring to FIG. 2, in one embodiment, a schematic representation of the method described by steps 132-160. The training data that can be stored in the database 10 is clustered in step 132 by a clustering algorithm, which may be a known clustering algorithm, such as K-means, spectral clustering, and other clustering approaches. . This results in a plurality of clusters 20. In step 142, the data in each cluster is analyzed using the CDS algorithm, and as a result, the AUC score of each cluster is obtained. In FIG. 2, for example, each of the three clusters receives an AUC score, and these particular clusters receive scores of 0.5, 0.8, and 0.9. Next, clusters with low AUC scores are identified in step 150, and in step 160, one or more of the clusters are generated to generate a set of exclusion and inclusion criteria that can be used to analyze and classify subsequent data. Are identified or extracted.

  As mentioned above, there are several ways to cluster, identify features, and calculate confidence scores. In addition to the methods described above, the CDS and clustering algorithms can be simultaneously applied separately to the training data. Thus, steps 130 and 132 are performed separately on the same training data. At step 130 of the method, the training data is analyzed using a CDS algorithm, where each data point is one of four sets: true positive (TP), true negative (TN), false positive (FP), or Evaluate predictions for known labels belonging to false negatives (FN). In step 132 of the method, the training data is clustered using a clustering algorithm to divide the data into meaningful clusters. Steps 130 and 132 may be performed simultaneously or separately on the same training data or at different times.

  According to this embodiment, at step 152, the system utilizes the output from the CDS algorithm at step 130 and the clustering algorithm at step 132 to generate one or more clusters with high false prediction densities. , And these clusters are candidates for exclusion, while the rest are candidates for inclusion. The method then proceeds to step 160 to generate one or more features from one or more clusters to generate a set of exclusion and inclusion criteria that can be used to analyze and classify subsequent data. Is identified or extracted.

  Referring to FIG. 3, in one embodiment, a schematic representation of the method described by steps 130, 132, 152, and 160 where the CDS algorithm and the clustering algorithm are applied separately to the training data. The training data, which may be stored in the database 10, is analyzed with the CDS algorithm at step 130 to generate an output 30, and separately, the training data is analyzed with a clustering algorithm to generate a plurality of clusters 20. At step 152, the system utilizes the output from the CDS algorithm at step 130 and the clustering algorithm at step 132 to identify one or more clusters with a high false prediction density, and these clusters are excluded. Candidates, while one or more clusters of other clusters are candidates for inclusion. At step 160, one or more features are identified or extracted from one or more clusters to generate a set of exclusion and inclusion criteria that can be used to analyze and classify subsequent data.

  Once the clustering algorithm creates clusters and generates, identifies, or extracts one or more features of these clusters to generate a set of exclusion and inclusion criteria, the resulting output is used to generate subsequent data Can be used to analyze and classify In addition, how well subsequent data fits within the cluster, and thus how accurate the CDS decision is based on the training data, can be determined based on confidence scores provided to health care professionals. it can.

  Referring to FIG. 4, in one embodiment, is a method 400 for providing a confidence score at the output of a CDS algorithm. In step 410 of the method, a scoring service, system, or device is provided. The scoring service, system, or device may be any system described or otherwise envisioned herein. For example, the scoring service may be a cloud-based service provided to healthcare professionals, such as software as a service option, and thus may be hosted on one or more specialized remote servers. May be done. Alternatively, the scoring service may be hosted locally by a medical professional, such as a special computer or server at a hospital or other healthcare facility.

  In step 420 of the method, input to the CDS algorithm is obtained or provided. For example, the inputs may include health data, background information, demographic data, clinical and / or non-clinical measurements, and / or other data from or about the patient or subject. In step 430 of the method, a CDS algorithm is applied to the input to generate an output. The output can be, for example, a diagnostic and / or therapeutic recommendation, among other outputs of the CDS.

  In step 440 of the method, the scoring algorithm determines which of the clusters generated by the clustering algorithm the output of the CDS and / or the patient data uses the training data. Some outputs of the CDS algorithm do not fit into any of the clusters, and this information may be reported to health care professionals.

  In step 450 of the method, a confidence score is assigned to the output of the CDS algorithm based at least in part on the cluster membership of the predetermined patient data and / or the output of the CDS algorithm. According to one embodiment, the confidence score is quantitative and / or qualitative. For example, the confidence score can be a positive number between 0 and 1. Alternatively, the confidence score may be “yes” or “no”. A "1" or "Yes" can indicate that the output of the CDS algorithm is reliably available, while a "0" or "No" indicates that the output of the CDS algorithm is not reliably available. Can be shown. If "0" or "No", it is excluded from further CDS evaluation and a flag can be set to alert the physician.

  Thus, the output of the scoring algorithm and method 400 may be, for example, a decision whether to utilize the CDS algorithm for a particular patient or to use on particular patient data. Alternatively, the output of the scoring algorithm and method 400 may be, for example, a confidence score associated with deciding whether to utilize the CDS algorithm. Alternatively, the output of the scoring algorithm and method 400 may be, for example, a confidence score associated with the output of the CDS algorithm, whereby the medical professional determines whether the output of the CDS algorithm is reliable. be able to.

  Referring to FIG. 5, in one embodiment, a simplified flowchart of a method 400 is shown. A new patient record 50 is received and its assignment in a cluster is made at step 440 of the method. The confidence score is applied by the scoring algorithm at step 450, and at step 460 of the method, the confidence score (here, "yes" or "no") is communicated to the healthcare professional. Thus, according to one embodiment, the output of the system is a decision whether to analyze the patient record with the CDS algorithm in connection with the confidence score in the determination. According to another embodiment, the output of the system is the output of the CDS algorithm and a confidence score for that output.

  According to one embodiment, the systems and methods described or otherwise envisioned herein are general platforms that can detect exclusion and / or inclusion criteria for any CDS algorithm. Generate Thus, the systems and methods are not limited to existing CDS algorithms.

  Referring to FIG. 6, in one embodiment, a scoring system 600. The scoring service, system, or device may be any system described or otherwise envisioned herein. For example, the scoring service may be a cloud-based service provided to healthcare professionals, such as software as a service option, and thus may be hosted on one or more specialized remote servers. May be done. Alternatively, the scoring service may be hosted locally by a medical professional, such as a special computer or server at a hospital or other healthcare facility.

  According to one embodiment, scoring system 600 includes a training database 602 that includes training data. Training data can include, for example, health data, background information, demographic data, and clinical measurements for a plurality of patients. Each of these patients in the training data can be associated with one or more treatment decisions made, as well as one or more results of these treatment decisions. Training database 602 may be local or remote, and thus the system may include a network 604 configured to communicate with the scoring system by wired and / or wireless communication.

  According to one embodiment, scoring system 600 is configured to train a clustering algorithm using training data from training database 602 and further configured to analyze new patient data from patient 608. A scoring module 606 is included.

  According to one embodiment, the scoring module 606 may include a processor 610 configured or programmed to perform one or more processes on or for the scoring module. it can. For example, the processor may be configured or programmed to receive and analyze training data from the training database 602 and receive and analyze patient data from the patient 608. Processor 610 may be programmed using software to perform various functions described herein, and may be utilized in combination with memory 612 and / or database 614. Memory 612 and / or database 614 may store data, including one or more commands or software programs for execution by processor 610, as well as various types of data. For example, memory 612 includes a set of instructions executable by processor 610 and may include non-transitory computer-readable storage media that causes a system to perform one or more steps of the methods described herein. Good. According to one embodiment, the scoring module 606 includes a communication module 616 to facilitate wired and / or wireless communication between the report generator and other devices and / or networks, such as the network 604. be able to. Communication module 616 can be facilitated, for example, by the use of one or more antennas. The communication module 616 can be implemented using known wireless methods, including, but not limited to, Wi-Fi, Bluetooth®, 3G, 4G, LTE, and / or ZigBee®, among others. Communication with one or more networks or other devices can be facilitated.

  Scoring module 606 may include a user interface and display 618 to receive input from healthcare provider 620 and / or to provide output or other information to healthcare provider 620. The user interface may be a button or buttons, a microphone, a keystroke input, a slider, a touch screen, or any of a variety of other inputs. The display may be, for example, an LED-based, LCD-based, or electronic paper-type display. In other embodiments, the display may be a touch screen display that allows a user to interact directly with the wearable device via physical contact and / or gestures.

  According to one embodiment, system 600 can optionally include a display 622 that can also function as a user input device. Device 622 may be, for example, a smartphone or other portable device. According to yet another embodiment, device 622 may be a computer such as a desktop, laptop, tablet, or other permanent or semi-permanent computing device. Device 622 may receive input from health care provider 620. Device 622 may also display information to health care provider 620 or other intended or authorized entities.

  All definitions, as defined and used herein, should be understood to govern the lexical definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms. It is.

  The indefinite articles "a" and "an", as used herein in the description and in the claims, unless used to the contrary clearly indicate "at least one". It should be understood to mean "at least one."

  The expression "and / or" is used in the description and in the claims, as used herein, to indicate that an element is so connected, i.e., in some cases, connected, It is to be understood as meaning "either or both" of elements that are present, and not otherwise connected. Multiple elements listed with "and / or" should be construed in the same fashion, ie, "one or more" of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” section, whether or not related to those elements specifically identified.

  In the description and the claims, as used herein, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" is inclusive, i.e., the number or list of elements and, optionally, further lists. It should be construed to include at least one, or more than one, of the unlisted items. Conversely, only terms expressly stated, for example, "only one of" or "exactly one of" or, when used in a claim, "consisting of" of) "refers to including the number of elements or exactly one element of a list. In general, the term "or", as used herein, refers to "either", "one of", "only one of" or When preceded by an exclusivity term such as "exactly one of", it is merely construed to indicate an exclusive alternative (i.e., "one or the other, but not both"). Should be.

  As used herein in the specification and claims, the expression "at least one" refers to any one of the elements in the list of elements, with reference to the list of one or more elements. Means at least one selected from the above elements, does not necessarily include at least one of each and every element specifically listed in the list of elements, and any combination of elements in the list of elements Should not be excluded. This definition implies that any element other than those specifically identified in the list of elements referred to by the expression "at least one", whether or not related to the specifically identified element, is optional. Allows to be present in the selection.

  Also, unless explicitly stated to the contrary, in any method claimed herein that includes more than one step or operation, the order of the steps or operations in the method may be It should be understood that the steps or operations in the description are not necessarily limited to the order in which they are cited.

  In the claims and in the above specification, "comprising", "including", "carrying", "having", "containing", "relationship" All transition phrases such as "involving," "holding," "composed of," etc., are open-ended, that is, include, but are not limited to It should be understood to mean that. Only the transitional phrases “consisting of” and “consisting essentially of” are closed or semi-closed transitional phrases, respectively.

  Although several embodiments are described and illustrated herein, those skilled in the art will appreciate that they can perform the functions described herein and / or obtain the results and / or one or more advantages. A variety of other means and / or structures are readily imaginable, and each such variation and / or modification is considered to be within the scope of the embodiments described herein. More generally, one of ordinary skill in the art would understand that all parameters, dimensions, materials, and configurations described herein are exemplary, and that the actual parameters, dimensions, materials, and / or configurations may be different. It will be readily understood that the teachings will depend on the particular application for which it is used. One skilled in the art will recognize many equivalents to the specific embodiments described herein, or will be able to ascertain using only routine experimentation. Therefore, it is to be understood that the above-described embodiments are provided by way of example only, and that within the scope of the appended claims and their equivalents, the embodiments may be practiced other than as specifically described and claimed. Various embodiments of the present disclosure are directed to individual features, systems, articles, materials, kits, and / or methods described herein. Further, any combination of two or more such features, systems, articles, materials, kits, and / or methods may be used if such features, systems, articles, materials, kits, and / or methods are mutually exclusive. If not contradictory, it falls within the scope of the present disclosure.

Claims (17)

A method for training a scoring system, comprising:
Providing a scoring system including a scoring module;
Receiving a training data set including a plurality of patient data and treatment results;
Analyzing the training data set to generate a plurality of clinical decision support recommendations using a clinical decision support algorithm;
Clustering the plurality of clinical decision support recommendations into a plurality of clusters using the scoring module;
Using the scoring module to identify one or more features of at least one of the plurality of clusters;
Generating one or more inclusion criteria of the at least one of the plurality of clusters based on the identified one or more features;
Including, methods.   The method of claim 1, wherein the step of clustering is based at least in part on comparing each of the clinical decision support recommendations with the treatment results in the training data.   The method of claim 1, wherein the identifying step further comprises generating one or more exclusion criteria for the at least one of the plurality of clusters based on the identified one or more features. . Obtaining health data about the patient;
Analyzing the health data to create a clinical decision support recommendation for the patient using a clinical decision support algorithm;
Assigning the clinical decision support recommendation to one of the plurality of clusters based on the generated inclusion criteria using the scoring module;
Using the scoring module to assign a confidence score to the clinical decision support recommendation based at least in part on the assignment of the recommendation to the one of the plurality of clusters;
The method of claim 1, further comprising:   5. The method of claim 4, further comprising communicating the confidence score to a user.   The method of claim 5, wherein the confidence score is a quantitative score. A method for training a scoring system, comprising:
Providing a scoring system including a scoring module;
Receiving a training data set including a plurality of patient data and treatment results;
Clustering the training data set into a plurality of clusters using the scoring module;
Analyzing the training data set to generate a plurality of clinical decision support recommendations using a clinical decision support algorithm, wherein the clustering and the analyzing are separately performed. Performing an analyzing step;
Using the scoring module to identify one or more features of at least one of the plurality of clusters;
Generating one or more inclusion criteria of the at least one of the plurality of clusters based on the identified one or more features;
Including, methods.   Identifying the at least one of the plurality of clusters as a cluster having a plurality of erroneous recommendations based on the plurality of clusters compared to the plurality of clinical decision support recommendations; The method of claim 7, further comprising:   Using the scoring module to identify at least one of the plurality of clusters as a noisy cluster, wherein the identified noisy clusters are generated by the one or more exclusion criteria. 8. The method of claim 7, further comprising the step of identifying which cluster is to be performed.   The step of analyzing includes analyzing each of the plurality of clusters to generate a plurality of clinical decision support recommendations for each of the plurality of clusters using a clinical decision support algorithm, further comprising: Using the scoring module to determine, based on the plurality of clusters and at least a portion of the plurality of clinical decision support recommendations, the plurality of clusters having a low area under a score curve. The method of claim 7, comprising identifying at least one. Obtaining health data about the patient;
Analyzing the health data to generate a clinical decision support recommendation for the patient using a clinical decision support algorithm;
Using the scoring module to assign the clinical decision support recommendation to one of the plurality of clusters based on the generated inclusion criteria;
Using the scoring module to assign a confidence score to a decision support recommendation based at least in part on the assignment of the recommendation to the one of the plurality of clusters;
The method of claim 7, further comprising:   The method of claim 11, further comprising communicating the confidence score to a user. A method for providing a confidence score for clinical decision support recommendations, comprising:
Obtaining health data about the patient;
Analyzing the health data to generate a clinical decision support recommendation for the patient using a clinical decision support algorithm;
Assigning the clinical decision support recommendation to one of a plurality of clusters based on inclusion criteria using a scoring module, the inclusion criteria comprising: (i) using the scoring module. Clustering the training data set or clinical decision support recommendations into a plurality of clusters; and (ii) using the scoring module to identify at least one feature of the plurality of clusters; Assigning, generated by generating one or more inclusion criteria for the at least one of the plurality of clusters based on the identified one or more features;
Using the scoring algorithm to assign a confidence score to the clinical decision support recommendation based at least in part on the assignment of the recommendation to the one of the plurality of clusters;
Communicating the confidence score to a user;
Including, methods.   14. The method of claim 13, wherein said confidence score is a quantitative score.   14. The method of claim 13, wherein the confidence score is a quantitative score that includes an indication of whether to use the clinical decision support recommendation.   16. Apparatus for training a scoring system, said apparatus comprising a memory and a processor configured to perform the steps of the method according to any one of claims 1 to 15. .   16. A non-transitory machine-readable storage medium encoded with instructions for execution by a processor to train a scoring system, performing the steps of the method of any one of claims 1-15. A non-transitory machine-readable storage medium containing instructions for performing.

Images