EP4712858A1 - Operation of a computing system to perform discordant pair analysis for cardiac event detection models - Google Patents

Abstract

A medical system can be configured to apply an ECG data set to first and second heart event models to generate a first labeled data set of arrhythmia episodes and a second labeled data set of arrhythmia episodes; compare the first labeled data set to the second labeled set to generate a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set; output, via the communication circuitry, the discordant pair data set to at least one reviewer computing device to be adjudicated by at least one reviewer; receive, from at least one reviewer computing device, an adjudicated discordant pair data set; and determine a performance of the second heart event model based on the adjudicated discordant pair data set.

Description

Atty Ref. No.: A0010217WO01 OPERATION OF A COMPUTING SYSTEM TO PERFORM DISCORDANT PAIR ANALYSIS FOR CARDIAC EVENT DETECTION MODELS FIELD [0001] This application claims the benefit of U.S. Provisional Patent Application Serial No.63/502,357, filed May 15, 2023, the entire content of which is incorporated herein by reference. [0002] This disclosure generally relates to medical devices and, more particularly, analysis of signals sensed by medical devices. BACKGROUND [0003] Medical devices may be used to monitor physiological signals of a patient. For example, some medical devices are configured to sense cardiac electrogram (EGM) signals, e.g., electrocardiogram (ECG) signals, indicative of the electrical activity of the heart via electrodes. Some medical devices are configured to detect occurrences of cardiac arrhythmia, often referred to as episodes, based on the cardiac EGM and, in some cases, data from additional sensors. Example arrhythmia types include asystole, bradycardia, ventricular tachycardia, supraventricular tachycardia, wide complex tachycardia, atrial fibrillation, atrial flutter, ventricular fibrillation, atrioventricular block, premature ventricular contractions, and premature atrial contractions. The medical devices may store the cardiac EGM and other data collected during a time period including an episode as episode data. The medical device may also store episode data for a time period in response to user input, e.g., from the patient. [0004] A computing system may obtain episode data from medical devices to allow a clinician or other user to review the episode. A clinician may diagnose a medical condition of the patient based on identified occurrences of cardiac arrhythmias within the episode. In some examples, a clinician or other reviewer may review episode data to annotate the episodes, including determining whether arrhythmias detected by the medical device actually occurred, to prioritize the episodes and generate reports for further review by the clinician that prescribed the medical device for a patient or is otherwise responsible for the care of the particular patient. Atty Ref. No.: A0010217WO01 SUMMARY [0005] This disclosure describes systems that may determine a performance of a machine learning (ML) model and based on the determined performance, determine whether a particular ML model is suitable for detecting certain types of arrhythmias or whether a particular ML model should replace or supplement a different ML model. The techniques of this disclosure may be used for updating a neural network model used for the classification of arrhythmias in electrocardiogram (ECG) data collected from implantable cardiac monitors (ICMs) or other implantable or external medical devices. For example, using the techniques of this disclosure, a system may determine if a new model is suitable to replace an existing model running in a production environment. The techniques of this disclosure may, for example, enable a system to determine if a new model is a better performing model that has statistically significantly superior specificity for detecting atrial fibrillation or other arrhythmias. As will be explained in more detail below, generating a discordant pair data set that includes data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set may enable a computing system to assess the performance of a new model relative to an existing model in a more efficient manner and while potentially requiring less user input. [0006] According to one example, a medical system includes communication circuitry configured to receive an electrocardiogram (ECG) data set; a memory configured to store a first heart event model, a second heart event model, and an electrocardiogram (ECG) data set; and processing circuitry configured to: apply the ECG data set to a first heart event model to generate a first labeled data set of arrhythmia episodes in the ECG data set; apply the ECG data set to a second heart event model to generate a second labeled data set of arrhythmia episodes in the ECG data set, the second heart event model being trained on different data than the first heart event model; compare the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set; generate a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set; output, via the communication circuitry, the discordant pair data set to at least one reviewer computing device to be adjudicated by at Atty Ref. No.: A0010217WO01 least one reviewer; receive, from the at least one reviewer computing device via the communication circuitry, an adjudicated discordant pair data set; and determine a performance of the second heart event model based on the adjudicated discordant pair data set; and output an indication of the performance. [0007] According to one example, a computer-implemented method for operating a computing system to evaluate performance of heart event models, the method comprising: applying, by processing circuitry, an electrocardiogram (ECG) data set to a first heart event model to generate a first labeled data set of arrhythmia episodes in the ECG data set; applying, by the processing circuitry, the ECG data set to a second heart event model to generate a second labeled data set of arrhythmia episodes in the ECG data set, the second heart event model being trained on different data than the first heart event model; comparing, by the processing circuitry, the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set; generating, by the processing circuitry, a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set; outputting, by the processing circuitry, the discordant pair data set to be adjudicated by a reviewer; receiving, by the processing circuitry, an adjudicated discordant pair data set; determining, by the processing circuitry, a performance of the second heart event model based on the adjudicated discordant pair data set; and outputting, by the processing circuitry, an indication of the performance. [0008] According to one example, a non-transitory computer-readable storage medium stores instructions that when executed by processing circuitry of a computing system, cause the processing circuitry to: apply an electrocardiogram (ECG) data set to a first heart event model to generate a first labeled data set of arrhythmia episodes in the ECG data set; apply the ECG data set to a second heart event model to generate a second labeled data set of arrhythmia episodes in the ECG data set, the second heart event model being trained on different data than the first heart event model; compare the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set; generate a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set; output the discordant pair data set to be Atty Ref. No.: A0010217WO01 adjudicated by a reviewer; receive an adjudicated discordant pair data set; and output an indication of a performance of the second heart event model. [0009] This summary is intended to provide an overview of the subject matter described in this disclosure. It is not intended to provide an exclusive or exhaustive explanation of the apparatus and methods described in detail within the accompanying drawings and description below. Further details of one or more examples are set forth in the accompanying drawings and the description below. BRIEF DESCRIPTION OF DRAWINGS [0010] FIG.1 is a conceptual drawing illustrating an example of a medical device system configured to utilize machine learning models to detect cardiac arrhythmias in accordance with the techniques of the disclosure. [0011] FIG.2 shows an overview of a discordant pair analysis process that may be performed in accordance with the techniques of this disclosure. [0012] FIG.3 is a block diagram illustrating an example configuration of the implantable medical device (IMD) of FIG.1. [0013] FIG.4 is a conceptual side-view diagram illustrating an example configuration of the IMD of FIGS.1 and 2. [0014] FIG.5 is a functional block diagram illustrating an example configuration of the computing system of FIG.1. [0015] FIG.6 is a flow diagram illustrating an example operation for utilizing discordant pair analysis to analyze the performance of cardiac event detection models in accordance with the techniques of the disclosure. [0016] FIG.7 is a conceptual diagram illustrating an example machine learning model configured to determine a likelihood of recurrence of AF. [0017] FIG.8 is a conceptual diagram illustrating an example training process for an artificial intelligence model, in accordance with examples of the current disclosure. [0018] Like reference characters refer to like elements throughout the figures and description. Atty Ref. No.: A0010217WO01 DETAILED DESCRIPTION [0019] A variety of types of implantable and external medical devices detect arrhythmia episodes based on sensed cardiac EGMs and, in some cases, other physiological parameters. External devices that may be used to non-invasively sense and monitor cardiac EGMs include wearable devices with electrodes configured to contact the skin of the patient, such as patches, watches, or necklaces. One example of a wearable physiological monitor configured to sense a cardiac EGM is the SEEQ™ Mobile Cardiac Telemetry System, available from Medtronic, Inc., of Minneapolis, Minnesota. Such external devices may facilitate relatively longer-term monitoring of patients during normal daily activities, and may periodically transmit collected data, e.g., episode data for detected arrhythmia episodes, to a remote patient monitoring system, such as the Medtronic Carelink™ Network. [0020] Implantable medical devices (IMDs) also sense and monitor cardiac EGMs, and detect arrhythmia episodes. Example IMDs that monitor cardiac EGMs include pacemakers and implantable cardioverter-defibrillators, which may be coupled to intravascular or extravascular leads, as well as pacemakers with housings configured for implantation within the heart, which may be leadless. Some IMDs that do not provide therapy, e.g., implantable patient monitors, sense cardiac EGMs. One example of such an IMD is the Reveal LINQ™ or LINQ II™ Insertable Cardiac Monitor (ICM), available from Medtronic, Inc., which may be inserted subcutaneously. Such IMDs may facilitate relatively longer-term monitoring of patients during normal daily activities, and may periodically transmit collected data, e.g., episode data for detected arrhythmia episodes, to a remote patient monitoring system, such as the Medtronic Carelink™ Network. [0021] By uploading episode data from medical devices, and distributing the episode data to various users, various network services may support centralized or clinic-based arrhythmia episode review. The episode data may include an indication of the one or more arrhythmias that the medical device detected during the episode. The episode data may also include data collected by the medical device during a time period including time before and after the instant the medical device determined the one or more arrhythmias to have occurred. The episode data may include the digitized cardiac EGM during that time period, heart rates or other parameters derived from the EGM during that time period, and Atty Ref. No.: A0010217WO01 any other physiological parameter data collected by the medical device during the time period. [0022] This disclosure describes a system that uses machine learning (ML) models to improve the ability of a medical device to detect and classify cardiac episodes. The medical device may, for example, be one of the devices described above or any other type of implantable device, such as a subcutaneous cardiac monitoring device, a single chamber ICD, an extravascular ICD, a subcutaneous ICD, or any other type of device configured to classify detected AF episodes. The system may also include an external device, such as a cloud-based system that is external to the cardiac monitoring device, like Medtronic Carelink™ Network introduced above. [0023] Cardiac monitoring devices like those introduced above have the capability of detecting atrial fibrillation (AF) and other types of arrythmias. Despite various enhancements over the years to the arrhythmia detection algorithms, many cardiac monitoring devices still generate a significant number of inappropriate detections of arrhythmia episodes, which can cause a significant data review burden for physicians leading to a significant amount of time being spent on reviewing episodes rather than treating patients. The techniques of this disclosure may improve the accuracy of the arrhythmia detection and classification performed by the system, and/or the cardiac monitoring device specifically, and thus reduce the arrhythmia episode review time for physicians. [0024] As cardiac monitoring devices are typically battery powered and, in the case of IMDs, need to have a sufficient enough battery life to justify implantation, the devices usually have limited processing capabilities in order to limit battery drain, which limits the complexity of the algorithms that can be implemented inside the cardiac monitoring device. Thus, cardiac monitoring devices can be configured to transmit data collected for suspected cardiac episodes to the external system so the external system can use advanced signal processing techniques to post process stored and transmitted data for episodes prior to review by physicians. This disclosure describes advanced signal processing techniques that may be used by the external system to post process the arrhythmia episodes detected by the cardiac monitoring device. Although the techniques of this disclosure will be described as being performed by an external system, it should be understood that in other implementations the described techniques may be performed by the IMD itself. Atty Ref. No.: A0010217WO01 [0025] This disclosure describes systems that may determine a performance of an ML model and based on the determined performance, determine whether a particular ML model is suitable for detecting certain types of arrhythmias or whether a particular ML model should replace or supplement a different ML model. Supervised machine learning models require large quantities of labeled data, e.g., labeled true or false, to optimize and validate performance. Supervised learning generally refers to learning where the input data has been labeled or categorized to assist the AI/ML system in processing it. This pre- processing may be manual, automated, or some combination. The quality of these data labels is especially important in applications such as medical diagnostics, where the model decisions can have large impacts on patient outcomes. Ensuring proper quality relies on sourcing data labels from certified professionals, which is both expensive and time consuming. Thus, in resource constrained environments, it becomes necessary to seek to minimize the number of data labels required to characterize a machine learning model’s performance. A separate but related issue arises when updating machine learning models, by training new data either on an existing model or on a completely new modeling architecture. In both cases, characterizing the performance of the new model may require a completely new set of data labels for the validation dataset. That is, in many regulatory settings, reusing validation datasets across model updates is not feasible due to the potential for data leakage and concerns around generalizability. Another issue is that with high performing models, challenging cases, or “edge” cases, become critical for assessing model quality. These challenging cases can be difficult to find and may require large amounts of manual adjudications before enough edge cases are obtained and labeled to have an accurate quantification of performance of any model. A final potential issue is related to the regulatory burden associated with medical diagnostics. In many cases a full description and pre-specification of the validation data label acquisition approach is necessary, including potential sample sizes and desired effect sizes. [0026] Existing techniques for improving or reducing the number of data labels have involved ranking schemes or active learning approaches that use the model under consideration to subsample data requiring labels. In Bernhardt, M., Castro, D. C., Tanno, R., Schwaighofer, A., Tezcan, K. C., Monteiro, M., Bannur, S., Lungren, M.P., Nori, A., Glocker, B., Alvarez-Valle, J., Oktay, O. (2022). Active label cleaning for improved dataset quality under resource constraints. Nature communications, 13(1), 1161, a label Atty Ref. No.: A0010217WO01 cleaning process was proposed that iteratively ranks data instances based on the estimated label correctness and labelling difficulty associated with each sample. Based on the ranking scheme, annotators relabel data until a budget is exhausted, where the budget can be time and/or monetary based. This active learning model can be applied to the use case of data labeling in the context of model validation, but the methodology was developed with the existence of noisy labels in mind, rather than no labels at all. Kossen, J., Farquhar, S., Gal, Y., & Rainforth, T. (2021, July). Active testing: Sample-efficient model evaluation. In International Conference on Machine Learning (pp.5753-5763). PMLR proposed an active learning framework that is a sample efficient technique that the authors name active testing. With the active testing approach, sample points are selected for labeling based on maximizing the accuracy of an empirical risk estimate. [0027] In both of these described approaches, the underlying methodology relies on labeling data based on the quality of existing labels or maximizing some performance metric. In a high regulatory burden setting, where pre-specification of all validation details is required, the label cleaning approach of Bernhardt, et al. would not reduce the number of labels requiring adjudication. Similarly, the active testing approach of Kossen, J. et al., would not be applicable due to all sample points needing to be selected for labeling in order to maximize accuracy. Random subsampling is not a preferred approach in a high regulatory burden environment. [0028] To address the issues introduced above, this disclosure proposes a process for achieving a reduced size validation dataset using a so-called discordant pair analysis based on the model predictions from a baseline model and a new model. The discordant pair analysis of this disclosure uses the diagonals of a 2x2 confusion matrix, where the predictions of a baseline and updated model disagree. Table 1 below shows an example of a 2x2 confusion matrix for a baseline model (Model 0) and an updated model (Model 1). Table 1 Model 1 Model 0 True False True C₁ D₁ False D₂ C₂ [0029] For purposes of explanation, Model 0 and Model 1 can be assumed to be models for classifying ECG data for a heart event as either corresponding to AF (True) or not corresponding to AF (False). In Table 1, result C1 corresponds to the concordant pair Atty Ref. No.: A0010217WO01 where both Model 0 and Model 1 classify the heart event as being AF, and result C2 corresponds to the concordant pair where both Model 0 and Model 1 classify that heart event as not being AF. Result D1 corresponds to the discordant pair wherein Model 0 classifies the heart event as being AF, but Model 1 classifies the heart event as not being AF. Result D2 corresponds to the discordant pair wherein Model 0 classifies the heart event as not being AF, but Model 1 classifies the heart event as being AF. As will be explained in more detail below, generating a discordant pair data set that includes data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set may enable a computing system to assess the performance of Model 1 relative to Model 0 in a more efficient manner and while potentially requiring less user input. [0030] The techniques of this disclosure may be used for updating a neural network model used for the classification of arrhythmias, such as AF, and other arrhythmias, in ECG data collected from ICMs or other implantable or external medical devices. For example, using the techniques of this disclosure, a system may determine if a new model is suitable to replace an existing model running in a production environment. The techniques of this disclosure may, for example, enable a system to determine if new model is a better performing model that has statistically significantly superior specificity for detecting AF or other arrhythmias. [0031] FIG.1 is a conceptual drawing illustrating an example of a medical device system 2 configured to utilize machine learning models to detect cardiac arrhythmias in accordance with the techniques of the disclosure. The example techniques may be used with an IMD 10, which may be in wireless communication with an external device 12. In some examples, IMD 10 is implanted outside of a thoracic cavity of patient 4 (e.g., subcutaneously in the pectoral location illustrated in FIG.1). IMD 10 may be positioned near the sternum near or just below the level of the heart of patient 4, e.g., at least partially within the cardiac silhouette. IMD 10 includes a plurality of electrodes (not shown in FIG. 1) and is configured to sense a cardiac EGM via the plurality of electrodes. In some examples, IMD 10 takes the form of a LINQ™ ICM. Although described primarily in the context of examples in which the medical device that collects episode data takes the form of an ICM, the techniques of this disclosure may be implemented in systems including any Atty Ref. No.: A0010217WO01 one or more implantable or external medical devices, including monitors, pacemakers, or defibrillators. [0032] External device 12 is a computing device configured for wireless communication with IMD 10. External device 12 may be configured to communicate with computing system 24 via network 25. In some examples, external device 12 may provide a user interface and allow a user to interact with IMD 10. Computing system 24 may comprise computing devices configured to allow a user to interact with IMD 10, or data collected from IMD, via network 25. [0033] External device 12 may be used to retrieve data from IMD 10 and may transmit the data to computing system 24 via network 25. The retrieved data may include values of physiological parameters measured by IMD 10, indications of episodes of arrhythmia or other maladies detected by IMD 10, episode data collected for episodes, and other physiological signals recorded by IMD 10. The episode data may include EGM segments recorded by IMD 10, e.g., due to IMD 10 determining that an episode of arrhythmia or another malady occurred during the segment, or in response to a request to record the segment from patient 4 or another user. [0034] In some examples, computing system 24 includes one or more handheld computing devices, computer workstations, servers or other networked computing devices. In some examples, computing system 24 may include one or more devices, including processing circuitry and storage devices, that implement a monitoring system 550. Computing system 24, network 25, and monitoring system 550 may be implemented by the Medtronic Carelink™ Network or other patient monitoring system, in some examples. [0035] Network 25 may include one or more computing devices (not shown), such as one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection, and/or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. Network 25 may include one or more networks administered by service providers, and may thus form part of a large-scale public network infrastructure, e.g., the Internet. Network 25 may provide computing devices, such as computing system 24 and IMD 10, access to the Internet, and may provide a communication framework that allows the computing devices to communicate with one Atty Ref. No.: A0010217WO01 another. In some examples, network 25 may be a private network that provides a communication framework that allows computing system 24, IMD 10, and/or external device 12 to communicate with one another but isolates one or more of computing system 24, IMD 10, or external device 12 from devices external to network 25 for security purposes. In some examples, the communications between computing system 24, IMD 10, and external device 12 are encrypted. [0036] Monitoring system 550, e.g., implemented by processing circuitry of computing system 24, may implement some of techniques of this disclosure including applying machine learning models to episode data to detect cardiac arrhythmias. Monitoring system 550 may receive episode data for episodes from medical devices, including IMD 10, which may store the episode data in response to their detection of an arrhythmia and/or user input. Based on the application of one or more arrhythmia classification machine learning models, monitoring system 550 may determine the likelihood that one or more arrhythmias of one or more types occurred during the episode including, in some examples, the arrhythmia identified by the medical device that stored the episode data. [0037] This disclosure describes techniques for assessing the effectiveness of a classification algorithm using discordant pair analysis. The techniques of this disclosure may, for example, be performed by computing system 24 to assess arrhythmia classification models, e.g., machine learning models, for use by monitoring system 550. In some examples, the described techniques may be performed, in full or in part, by monitoring system 550. In other examples, the described techniques may be performed by different units or modules of computing system 24. For ease of explanation, the techniques of this disclosure will generally be described as being performed by computing system 24, but it should be understood that the described techniques may also be performed in full or in part by other computing devices as well. [0038] The techniques described herein utilize a performance baseline algorithm and a large unlabeled dataset with an assumed class distribution to obtain overall performance estimates by only assessing the subset of examples that the algorithms classify discordantly. The techniques of this disclosure may enable computing system 24 to efficiently evaluate the performance of an algorithm that minimizes the amount of human adjudications needed while also maintaining precision in the evaluation. In some cases, Atty Ref. No.: A0010217WO01 the techniques of this disclosure may improve the evaluation quality of computing system 24 by reducing human adjudication errors. The techniques of this disclosure are a computationally efficient alternative to the traditional exhaustive processes of performance evaluation and has the potential to improve the accuracy of performance estimates. Simulation studies show that, in some contexts, the discordant pair process described herein may reduce the number of adjudications by over 90%, while maintaining the same level of sensitivity and specificity. In this context, sensitivity generally refers to the model's ability to correctly identify positive instances or true positives, and specificity generally refers to the model's ability to correctly identify negative instances or true negatives. [0039] The techniques of this disclosure may reduce the number of model validation samples required for labeling based on disagreement between a baseline model and a new, updated model, where the baseline model has known performance that is expected to generalize to the validation data. The techniques of this disclosure may be used in conjunction with binary labels, such as 0 and 1, true and false, or positive and negative. [0040] FIG.2 shows an overview of the discordant pair analysis process 200 that may be performed by computing system 24 in accordance with the techniques of this disclosure. In the example of FIG.2, computing system 24 generates or receives ECG data set 202. ECG data set 202 may include ECG data for a plurality of heart events for a plurality of patients. Computing system 24 then applies heart event models 204 to ECG data set 202. Heart event models 204 may, for example, include a first heart event model that corresponds to a baseline model (M₀) and a second heart event model that corresponds to a new or updated model (M₁). In some use cases, model M₁ may have been trained on different data than model M₀. In some examples, the different data my include less data. [0041] By applying heart models 204, computing system 24 generates a first labeled data set of arrhythmia episodes in the ECG data set determined by M0 and a second labeled data set of arrhythmia episodes in the ECG data set determined by M1. Computing system 24 compares the first labeled data set to the second labeled set to generate results matrix 206. Using results matrix 206, computing system 24 determines which labeled data of the first labeled data set differs than labeled data of the second labeled data set. The labeled data that differs corresponds to discordant pairs D_NP and D_PN, or more generally, to episodes where the two models produce different Atty Ref. No.: A0010217WO01 classifications. Discordant pairs D_NP correspond to heart events where model M₀ did not identify an arrhythmia, but model M₁ did. Discordant pairs D_PN correspond to heart events where model M₀ identified an arrhythmia, but model M₁ did not. [0042] Computing system 24 generates discordant pair data set (D = D_NP + D_PN) 208, which corresponds to the instances in which first labeled data set generated by M0 differs from the second labeled data set generated by M₁. Computing system 24 then outputs discordant pair data set 208, to reviewer computing devices 30, to be adjudicated by one or more reviewers. Computing system 24 generates based on the adjudications received from the one or more users of reviewer computing devices 30 an adjudicated discordant pair data set 210. Computing system 24 may then output an indication of performance 212, which represents an indication of the performance model M₁ relative to model M₀. [0043] The following description provides a more detailed analysis of process 200. In the following analysis, PREV represents the prevalence of a binary outcome, and n represents a sample size. The estimated number of positive outcomes(P) is P=n×PREV. The performance of a model is based on a requirement definition, along with statistical power considerations, of the desired sensitivity of the updated model. Bujang, M. A., & Adnan, T. H, (2016), Requirements for minimum sample size for sensitivity and specificity analysis, Journal of clinical and diagnostic research: JCDR, 10, YE01 describes further sample size considerations around diagnostic metrics. The n samples, total sample size of ^^^^ samples, are evaluated similarly to an A/B testing framework, where each of the samples is evaluated on both the baseline (M₀) and updated (M₁) models. Table 2 shows an example of a 2x2 matrix that may correspond to results matrix 206. Table 2. Baseline model ( ^^^^₀) and updated model ( ^^^^₁) 2x2 results table. U_{pdated Model - ^^^^ ^^^^ Baseline Model - ^^^^ ^^^^} ^{Outcome = Positive Outcome = Negative Outcome = Positive} _{^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^} O^{utcome = Negative} _{^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^} [0044] Computing system 24 may identify binary model outcomes from two models as concordant, ^^^^ = ^^^^ _{^^^^ ^^^^} + ^^^^ _{^^^^ ^^^^}, or discordant, ^^^^ = ^^^^ _{^^^^ ^^^^} + ^^^^ _{^^^^ ^^^^}, with the total sample size n=C+D. Computing system 24 may further decompose each of the four paired concordance and discordant outcomes as ^^^^ _{^^^^ ^^^^} = ^^^^ ^^^^_{0 ^^^^} + ^^^^ ^^^^_{0 ^^^^} + ^^^^ ^^^^_{1 ^^^^} + ^^^^ ^^^^_{1 ^^^^} , Atty Ref. No.: A0010217WO01 ^^^^ _{^^^^ ^^^^} = ^^^^ ^^^^_{0 ^^^^} + ^^^^ ^^^^_{0 ^^^^} + ^^^^ ^^^^_{1 ^^^^} + ^^^^ ^^^^_{1 ^^^^} , ^^^^ _{^^^^ ^^^^} = ^^^^ ^^^^_{0 ^^^^} + ^^^^ ^^^^_{0 ^^^^} + ^^^^ ^^^^_{1 ^^^^} + ^^^^ ^^^^_{1 ^^^^} , ^^^^ _{^^^^ ^^^^} = ^^^^ ^^^^_{0 ^^^^} + ^^^^ ^^^^_{0 ^^^^} + ^^^^ ^^^^_{1 ^^^^} + ^^^^ ^^^^_{1 ^^^^} , where ^^^^ ^^^^ _{^^^^ ^^^^}, ^^^^ ^^^^ _{^^^^ ^^^^}, ^^^^ ^^^^ _{^^^^ ^^^^}, and ^^^^ ^^^^ _{^^^^ ^^^^} are the true positive, true negative, false positive, and false negative counts, respectively, ^^^^ ∈ (0, 1) indexes the baseline and updated models, respectively, and ^^^^ ∈ ⁽ ^^^^, ^^^^⁾ indexes the concordant and discordant sets, respectively. As an illustrative example to help understanding, ^^^^ ^^^^_{0 ^^^^} is the number of true positive outcomes from the baseline model in the concordant set. [0045] Computing system 24 may then label the discordant outcomes where the models disagree. The discordant outcomes are the samples that contribute to ^^^^ _{^^^^ ^^^^} and ^^^^ _{^^^^ ^^^^}, to facilitate a final estimate of the updated model performance. The sensitivity of the updated model, SENS₁, may be estimated using the following approach. A sensitivity for the baseline model (SENS₀) may be estimated from a previous validation or performance surveillance effort. Thus, it can be assumed that: and solving for ^^^^ ^^^^_{0 ^^^^} gives ^^^^ ^^^^_{0 ^^^^} = ^^^^ ^^^^ ^^^^ ^^^^₀ × ^^^^ − ^^^^ ^^^^_{0 ^^^^} . [0046] By definition of the concordance and discordant sets, TP_0C = TP_1C. That is, the number of true positives between the two models in the concordant set are equivalent. The sensitivity of the updated model can be estimated as: where ^^^^ is the estimated number of binary outcomes and ^^^^ ^^^^ _^^^^, ^^^^ ∈ (0,1), is the number of true positives from model ^^^^. This derivation shows that the sensitivity of the updated model can be calculated using only the baseline model sensitivity, positive outcome prevalence, and values from the discordant set. Atty Ref. No.: A0010217WO01 [0047] A similar application can be used to estimate the specificity of the updated model, ^^^^ ^^^^ ^^^^ ^^^^₁, as well. Assuming a specificity for the baseline model, ^^^^ ^^^^ ^^^^ ^^^^₀: where ^^^^ = ^^^^ − ^^^^ is the assumed number of negative binary outcomes. Solving for ^^^^ ^^^^_{0 ^^^^} gives ^^^^ ^^^^_{0 ^^^^} = ^^^^ ^^^^ ^^^^ ^^^^₀ × ^^^^ − ^^^^ ^^^^_{0 ^^^^} , where, ^^^^ ^^^^_{0 ^^^^} is the number of true negative outcomes from the baseline model in the concordant set. Similar to the sensitivity calculation above, ^^^^ ^^^^_{0 ^^^^} = ^^^^ ^^^^_{1 ^^^^}. That is, the number of true negatives between the two models in the concordant set are equivalent. Then, we can estimate the specificity of the updated model, ^^^^ ^^^^ ^^^^ ^^^^₁, as ^^{^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ 1 1 =} ^_^^^ ^, ^_{^^^ ^^^^ + ^ = 1 ^^^^ ^^^ ^^^^1 ^^^^} ^^^^ ^{, (} _{^^^^ ^^^^ ^^^^ ^^^^0 × ^^^^ − ^^^^ ^^^^0 ^^^^} ⁾ _{+ ^^^^ ^^^^ = 1 ^^^^} ^^^^ _. [0048] Thus, computing system 24 may calculate the specificity of the updated model using only the baseline model specificity, negative outcome prevalence, and values from the discordant set. [0049] In addition to point estimates of the sensitivity and specificity of an updated model, computing system 24 may be configured to estimate confidence intervals, for the sensitivity and/or specificity, for demonstrating the non-inferiority or superiority of the updated model. The sensitivity and specificity metrics may depend in part on assumptions based on data sets collected in the past. A bootstrapping approach, such as that describe in DiCiccio, T. J., & Efron, B, (1996), Bootstrap confidence intervals, Statistical science, 11, 189–228 may be used to propagate uncertainties and obtain accurate confidence intervals for the sensitivity and specificity of the updated model. [0050] Computing system 24 may be configured to implement a multi-stage Monte Carlo sampling. Beginning with baseline model sensitivity and specificity values, ^^^^ ^^^^ ^^^^ ^^^^₀ and ^^^^ ^^^^ ^^^^ ^^^^₀, respectively, and positive outcome prevalence ^^^^ ^^^^ ^^^^ ^^^^, for the ^^^^th Monte Carlo sample, ^^^^ = 1, … , the following can be determined: Atty Ref. No.: A0010217WO01 ^_{^^^ ^^^^ ^^^^ ^^^^ ^^^^ ~ ^^^^ ^^^^ ^^^^ ^^^^} ⁽ _{100, 100/ ^^^^ ^^^^ ^^^^ ^^^^ − 100} ⁾ _, ^^^^ _^^^^ ~ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^( ^^^^, ^^^^ ^^^^ ^^^^ ^^^^ _^^^^ ), ^^^^ ^^^^_{0 ^^^^} ~ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^( ^^^^ _^^^^, ^^^^ ^^^^ ^^^^ ^^^^₀), ^_{^^^ ^^^^ ^^^^ ^^^^1 ^^^^ ~ ^^^^ ^^^^ ^^^^ ^^^^} ⁽ _{^^^^ ^^^^0 ^^^^ − ^^^^ ^^^^0 ^^^^ + ^^^^ ^^^^1 ^^^^ + 1, ^^^^ ^^^^ −} ⁽ _{^^^^ ^^^^0 ^^^^ − ^^^^ ^^^^0 ^^^^ + ^^^^ ^^^^1 ^^^^} ⁾ _{+ 1} ⁾ _, resulting in ^^^^ Monte Carlo samples of the updated model sensitivity. We then sample the updated model specificity: where, in general, ^^^^ ^^^^ ^^^^ ^^^^( ^^^^, ^^^^) is a beta distribution with rates ^^^^ and ^^^^, and ^^^^ ^^^^ ^^^^ ^^^^ ^^^^( ^^^^, ^^^^) is a Binomial distribution with ^^^^ trialsprobability ^^^^. Upon sampling each ^^^^ ^^^^ ^^^^ ^^^^_{1 ^^^^} and ^^^^ ^^^^ ^^^^ ^^^^_{1 ^^^^}, computing system 24 may be configured to compute upper and lower quantiles of the samples to estimate the confidence bounds, e.g., the 2.5^th and 97.5^th quantiles correspond to two-sided 95% confidence bounds. [0051] FIG.3 is a block diagram illustrating an example configuration of IMD 10 of FIG.1. As shown in FIG.3, IMD 10 includes processing circuitry 50 sensing circuitry 52, communication circuitry 54, memory 56, sensors 58, switching circuitry 60, and electrodes 16A, 16B (hereinafter “electrodes 16”), one or more of which may be disposed on a housing of IMD 10. In some examples, memory 56 includes computer-readable instructions that, when executed by processing circuitry 50, cause IMD 10 and processing circuitry 50 to perform various functions attributed herein to IMD 10 and processing circuitry 50. Memory 56 may include any volatile, non-volatile, magnetic, optical, or electrical media, such as a random-access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), electrically-erasable programmable ROM (EEPROM), flash memory, or any other digital media. [0052] Processing circuitry 50 may include fixed function circuitry and/or programmable processing circuitry. Processing circuitry 50 may include any one or more of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent discrete or analog logic circuitry. In some examples, processing circuitry 50 may include multiple components, such as any combination of one or more microprocessors, one or more controllers, one or more DSPs, one or more ASICs, or one or more FPGAs, as well as Atty Ref. No.: A0010217WO01 other discrete or integrated logic circuitry. The functions attributed to processing circuitry 50 herein may be embodied as software, firmware, hardware or any combination thereof. [0053] Sensing circuitry 52 may be selectively coupled to electrodes 16A, 16B via switching circuitry 60 as controlled by processing circuitry 50. Sensing circuitry 52 may monitor signals from electrodes 16A, 16B in order to monitor electrical activity of a heart of patient 4 of FIG.1 and produce cardiac EGM data, e.g., ECG data including digitized ECG signals, for patient 4. In some examples, processing circuitry 50 may identify features of the sensed cardiac EGM to detect an episode of cardiac arrhythmia of patient 4. Processing circuitry 50 may store the digitized cardiac EGM and features of the EGM used to detect the arrhythmia episode in memory 56 as episode data for the detected arrhythmia episode. In some examples, processing circuitry 50 stores one or more segments of the cardiac EGM data, features derived from the cardiac EGM data, and other episode data in response to instructions from external device 12 (e.g., when patient 4 experiences one or more symptoms of arrhythmia and inputs a command to external device 12 instructing IMD 10 to upload the data for analysis by a monitoring center or clinician). [0054] In some examples, processing circuitry 50 transmits, via communication circuitry 54, the episode data for patient 4 to an external device, such as external device 12 of FIG.1. For example, IMD 10 sends digitized cardiac EGM and other episode data to network 25 for processing by monitoring system 550 of FIG.1. [0055] Sensing circuitry 52 and/or processing circuitry 50 may be configured to detect cardiac depolarizations (e.g., P-waves of atrial depolarizations or R-waves of ventricular depolarizations) when the cardiac EGM amplitude crosses a sensing threshold. For cardiac depolarization detection, sensing circuitry 52 may include a rectifier, filter, amplifier, comparator, and/or analog-to-digital converter, in some examples. In some examples, sensing circuitry 52 may output an indication to processing circuitry 50 in response to sensing of a cardiac depolarization. In this manner, processing circuitry 50 may receive detected cardiac depolarization indicators corresponding to the occurrence of detected R-waves and P-waves in the respective chambers of heart. Processing circuitry 50 may use the indications of detected R-waves and P-waves for determining features of the cardiac EGM including inter-depolarization intervals, heart rate, and detecting arrhythmias, such as tachyarrhythmias and asystole. Sensing circuitry 52 may also provide Atty Ref. No.: A0010217WO01 one or more digitized cardiac EGM signals to processing circuitry 50 for analysis, e.g., for use in cardiac rhythm discrimination and/or to identify and delineate features of the cardiac EGM, such as QRS amplitudes and/or width, or other morphological features. [0056] In some examples, IMD 10 includes one or more sensors 58, such as one or more accelerometers, microphones, optical sensors, and/or pressure sensors. In some examples, sensing circuitry 52 may include one or more filters and amplifiers for filtering and amplifying signals received from one or more of electrodes 16A, 16B and/or other sensors 58. In some examples, sensing circuitry 52 and/or processing circuitry 50 may include a rectifier, filter and/or amplifier, a sense amplifier, comparator, and/or analog-to- digital converter. Processing circuitry 50 may determine values of physiological parameters of patient 4 based on signals from sensors 58, which may be used to identify arrhythmia episodes and stored as episode data in memory 56. [0057] Communication circuitry 54 may include any suitable hardware, firmware, software or any combination thereof for communicating with another device, such as external device 12. Under the control of processing circuitry 50, communication circuitry 54 may receive downlink telemetry from, as well as send uplink telemetry to, external device 12 or another device with the aid of an internal or external antenna, e.g., antenna 26. In some examples, processing circuitry 50 may communicate with a networked computing device via an external device (e.g., external device 12) and a computer network, such as the Medtronic CareLink® Network developed by Medtronic, Inc. [0058] Although described herein in the context of example IMD 10, the techniques for cardiac arrhythmia detection disclosed herein may be used with other types of devices. For example, the techniques may be implemented with an extra-cardiac defibrillator coupled to electrodes outside of the cardiovascular system, a transcatheter pacemaker configured for implantation within the heart, such as the Micra^TM transcatheter pacing system commercially available from Medtronic, Inc., an ICM, such as a LINQ ^TM ICM, also commercially available from Medtronic, Inc., a neurostimulator, a drug delivery device, a medical device external to patient 4, a wearable device such as a wearable cardioverter defibrillator, a fitness tracker, or other wearable device, a mobile device, such as a mobile phone, a “smart” phone, a laptop, a tablet computer, a personal digital assistant (PDA), or “smart” apparel such as “smart” glasses, a “smart” patch, or a “smart” watch. Atty Ref. No.: A0010217WO01 [0059] FIG.4 is a conceptual side-view diagram illustrating an example configuration of IMD 10. In the example shown in FIG.4, IMD 10 may include a leadless, subcutaneously-implantable monitoring device having a housing 14 and an insulative cover 74. Electrode 16A and electrode 16B may be formed or placed on an outer surface of cover 74. Circuitries 50–56 and 60, described above with respect to FIG.3, may be formed or placed on an inner surface of cover 74, or within housing 14. In the illustrated example, antenna 26 is formed or placed on the inner surface of cover 74, but may be formed or placed on the outer surface in some examples. Sensors 58 may also be formed or placed on the inner or outer surface of cover 74 in some examples. In some examples, insulative cover 74 may be positioned over an open housing 14 such that housing 14 and cover 74 enclose antenna 26, sensors 58, and circuitries 50–56 and 60, and protect the antenna and circuitries from fluids such as body fluids. [0060] One or more of antenna 26, sensors 58, or circuitries 50–56 may be formed on insulative cover 74, such as by using flip-chip technology. Insulative cover 74 may be flipped onto a housing 14. When flipped and placed onto housing 14, the components of IMD 10 formed on the inner side of insulative cover 74 may be positioned in a gap 76 defined by housing 14. Electrodes 16 may be electrically connected to switching circuitry 60 through one or more vias (not shown) formed through insulative cover 74. Insulative cover 74 may be formed of sapphire (i.e., corundum), glass, parylene, and/or any other suitable insulating material. Housing 14 may be formed from titanium or any other suitable material (e.g., a biocompatible material). Electrodes 16 may be formed from any of stainless steel, titanium, platinum, iridium, or alloys thereof. In addition, electrodes 16 may be coated with a material such as titanium nitride or fractal titanium nitride, although other suitable materials and coatings for such electrodes may be used. [0061] FIG.5 is a block diagram illustrating an example configuration of computing system 24. In the illustrated example, computing system 24 includes processing circuitry 502 for executing applications 524 that include monitoring system 550 or any other applications described herein. Computing system 24 may be any component or system that includes processing circuitry or other suitable computing environment for executing software instructions and, for example, need not necessarily include one or more elements shown in FIG.5 (e.g., input devices 504, communication circuitry 506, user interface devices 510, or output devices 512; and in some examples components such as storage Atty Ref. No.: A0010217WO01 device(s) 508 may not be co-located or in the same chassis as other components). In some examples, computing system 24 may be a cloud computing system distributed across a plurality of devices. [0062] In the example of FIG.5, computing system 24 includes processing circuitry 502, one or more input devices 504, communication circuitry 506, one or more storage devices 508, user interface (UI) device(s) 510, and one or more output devices 512. Computing system 24, in some examples, further includes one or more application(s) 524 such as monitoring system 550, and operating system 516 that are executable by computing system 24. Each of components 502, 504, 506, 508, 510, and 512 may be coupled (physically, communicatively, and/or operatively) for inter-component communications. In some examples, communication channels 514 may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data. As one example, components 502, 504, 506, 508, 510, and 512 may be coupled by one or more communication channels 514. [0063] Processing circuitry 502, in one example, is configured to implement functionality and/or process instructions for execution within computing system 24. For example, processing circuitry 502 may be capable of processing instructions stored in storage device 508. Examples of processing circuitry 502 may include any one or more of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry. [0064] One or more storage devices 508 may be configured to store information within computing device 500 during operation. Storage device 508, in some examples, is described as a computer-readable storage medium. In some examples, storage device 508 is a temporary memory, meaning that a primary purpose of storage device 508 is not long- term storage. Storage device 508, in some examples, is described as a volatile memory, meaning that storage device 508 does not maintain stored contents when the computer is turned off. Examples of volatile memories include RAM, dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art. In some examples, storage device 508 is used to store program instructions for execution by processing circuitry 502. Storage device 508, in one Atty Ref. No.: A0010217WO01 example, is used by software or applications 524 running on computing system 24 to temporarily store information during program execution. [0065] Storage devices 508, in some examples, also include one or more computer- readable storage media. Storage devices 508 may be configured to store larger amounts of information than volatile memory. Storage devices 508 may further be configured for long-term storage of information. In some examples, storage devices 508 include non- volatile storage elements. Examples of such non-volatile storage elements include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable memories (EEPROM). [0066] Computing system 24, in some examples, also includes communication circuitry 506 to communicate with other devices and systems, such as IMD 10 and external device 12 of FIG.1. Communication circuitry 506 may include a network interface card, such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device that can send and receive information. Other examples of such network interfaces may include 4G, 5G, and WiFi radios. [0067] Computing system 24, in one example, also includes one or more user interface devices 510. User interface devices 510, in some examples, are configured to receive input from a user through tactile, audio, or video feedback. Examples of user interface devices(s) 510 include a presence-sensitive display, a mouse, a keyboard, a voice responsive system, video camera, microphone or any other type of device for detecting a command from a user. In some examples, a presence-sensitive display includes a touch- sensitive screen. [0068] One or more output devices 512 may also be included in computing system 24. Output devices 512, in some examples, are configured to provide output to a user using tactile, audio, or video stimuli. Output devices 512, in one example, include a presence- sensitive display, a sound card, a video graphics adapter card, or any other type of device for converting a signal into an appropriate form understandable to humans or machines. Additional examples of output devices 512 include a speaker, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD), or any other type of device that can generate intelligible output to a user. Atty Ref. No.: A0010217WO01 [0069] Computing system 24 may include operating system 516. Operating system 516, in some examples, controls the operation of components of computing system 24. For example, operating system 516, in one example, facilitates the communication of one or more applications 524 and monitoring system 550 with processing circuitry 502, communication circuitry 506, storage device 508, input device 504, user interface devices 510, and output device 512. [0070] Applications 524 may also include program instructions and/or data that are executable by computing device 500. Example application(s) 524 executable by computing device 500 may include monitoring system 550. Other additional applications not shown may alternatively or additionally be included to provide other functionality described herein and are not depicted for the sake of simplicity. [0071] In some examples, applications 524 may include an arrhythmia classification model analysis application that determines a performance of various arrhythmia classification models in comparison to arrhythmia classification models 552. The analysis application may, for example, apply an ECG data set to arrhythmia classification models 552 to generate a first labeled data set of arrhythmia episodes in the ECG data set. The analysis application may then apply the ECG data set to a new arrhythmia classification model and compare the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set. The analysis application may then generate a discordant pair data set that includes data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set and output the discordant pair data set to be adjudicated by a reviewer. [0072] The analysis application may receive, via input devices 504, an adjudicated discordant pair data set and determine a performance of the new arrhythmia classification model based on the adjudicated discordant pair data set and output, via output devices 512, an indication of the performance. The indication of the performance of the new arrhythmia classification model may, for example, be an indication of the performance of the new arrhythmia classification model relative to arrhythmia classification models 552. Based on the performance of the new arrhythmia classification model relative to arrhythmia classification models 552, computing system 24 may determine whether to update arrhythmia classifications models 552 with the new arrhythmia classification Atty Ref. No.: A0010217WO01 model or whether to recommend to a user that the user update arrhythmia classifications models 552 with the new arrhythmia classification model. [0073] Reviewer computing device 30 may generally include the same components, such as processing circuitry, one or more input devices, communication circuitry, one or more storage devices, UI devices, and one or more output devices, that are similar to processing circuitry 502, one or more input devices 504, communication circuitry 506, one or more storage devices 508, user interface (UI) device(s) 510, and one or more output devices 512 as described with respect to FIG.5. [0074] FIG.6 is a flow diagram illustrating an example operation for utilizing discordant pair analysis to analyze the performance of cardiac event detection models in accordance with the techniques of the disclosure. The techniques of FIG.6. will be described with respect to computing system 24, but it should be understood that the techniques of FIG.6 are not limited to any particular computing system. According to the example illustrated by FIG.6, computing system 24 receives an ECG data set (602). The ECG data set may include a plurality of ECG signals correspond to a plurality of episodes. [0075] Computing system 24 applies the ECG data set to a first heart event model to generate a first labeled data set of arrhythmia episodes in the ECG data set (604) and applies the ECG data set to a second heart event model to generate a second labeled data set of arrhythmia episodes in the ECG data set (606). The first and second heart event models may, for example, be different machine learning models. The first labeled data set of arrhythmia episodes includes determinations, using the first heart event model, of whether or not the episodes correspond to a certain arrhythmia type. The second labeled data set of arrhythmia episodes includes determinations, using the second heart event model, of whether or not the episodes correspond to the certain arrhythmia type. The second heart event model may, for example, have been trained on different data than the first heart event model. The arrhythmia type may be AF or any other arrhythmia. [0076] Computing system 24 compares the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set (608). In this context, the first labeled data sets differs from the second labeled data set when the first labeled data set has an event corresponding to the arrhythmia type and the second labeled data set does not have the event correspond to the arrhythmia, or vice versa. Atty Ref. No.: A0010217WO01 [0077] Computing system 24 generates a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set (610). Computing system 24 outputs the discordant pair data set to be adjudicated by a reviewer (612). In some example, computing system 24 outputs only the discordant pair data set for adjudication. That is, computing system 24 may be configured to separate out, and not output for adjudication, the concordant pairs. [0078] Computing system 24 receives an adjudicated discordant pair data set (614). One or more reviewers may review the discordant pair data set, and based, for example, on a voting of the adjudications of the reviewers, computing system 24 may determine the adjudicated discordant pair data set. Computing system 24 determines a performance of the second heart event model based on the adjudicated discordant pair data set (616). The performance may, for example, correspond to one or both of a specificity or a sensitivity of the second heart event model. [0079] Computing system 24 may then output an indication of the performance (618). The indication of the performance of the second heart event model may, for example, be an indication of the performance of the second heart event model relative to the first heart event model or an absolute performance of the second heart event model based on a known performance of the first heart event model. The indication of the performance of the second heart event model may additionally or alternatively include an indication of whether the performance of the second heart event model exceeds a performance of the first heart event model by a threshold amount. The threshold amount may for example be zero or greater than zero. Computing system 24 may additionally output a confidence value indicative of a confidence in whether the performance of the second heart event model exceeds a performance of the first heart event model by a threshold amount. [0080] In some implementations, computing system 24 may be configured to save the adjudicated discordant pair data set for use as training data and use the adjudicated discordant pair data set to train a third heart event model. [0081] FIG.7 is a conceptual diagram illustrating an example machine learning model 700 configured to determine a likelihood of recurrence of AF. Machine learning model may be one of arrhythmia classifications models 752. Machine learning model 700 is an example of a deep learning model, or deep learning algorithm. One or more of IMD 10, external device 12, or computing system 24 may train, store, and/or utilize machine Atty Ref. No.: A0010217WO01 learning model 700, but other devices may apply inputs associated with a particular patient to machine learning model 700 in other examples. Some non-limiting examples of machine learning techniques include Bayesian probability models, Hawkes processes, Support Vector Machines, K-Nearest Neighbor algorithms, and Multi-layer Perceptron. [0082] As shown in the example of FIG.7, machine learning model 700 may include three layers. These three layers include input layer 702, hidden layer 704, and output layer 706. Output layer 706 comprises the output from the transfer function 705 of output layer 706. Input layer 702 represents each of the input values X1 through X4 provided to machine learning model 700. The number of inputs may be less than or greater than 4, including much greater than 4, e.g., hundreds or thousands. In some examples, the input values may be parameters determined based on AF episode data, including those described herein, and in some cases other parameter data. In some examples, the input values may include those described in U.S. Provisional Patent Application Serial No.63/365,188, filed May 23, 2022 and titled “SYSTEM USING HEART RATE VARIABILITY FEATURES FOR PREDICTION OF MEDICAL PROCEDURE EFFICACY”, the entire content of which is incorporated herein by reference. [0083] Each of the input values for each node in the input layer 702 is provided to each node of hidden layer 704. In the example of FIG.7, hidden layers 704 include two layers, one layer having four nodes and the other layer having three nodes, but fewer or greater number of nodes may be used in other examples. Each input from input layer 702 is multiplied by a weight and then summed at each node of hidden layers 704. During training of machine learning model 700, the weights for each input are adjusted to establish the relationship between input physiological parameter values and one or more output values indicative of a health state of the patient. In some examples, one hidden layer may be incorporated into machine learning model 700, or three or more hidden layers may be incorporated into machine learning model 700, where each layer includes the same or different number of nodes. [0084] The result of each node within hidden layers 704 is applied to the transfer function of output layer 706. The transfer function may be liner or non-linear, depending on the number of layers within machine learning model 700. Example non-linear transfer functions may be a sigmoid function or a rectifier function. The output 707 of the transfer function may be a value or values indicative of a likelihood (e.g., a probability) of Atty Ref. No.: A0010217WO01 recurrence of AF after PVI or another procedure to treat AF. By applying the patient parameter data to a machine learning model, such as machine learning model 700, processing circuitry of system computing system 24 or other such systems and devices is able to determine the likelihood of AF recurrence with great accuracy, specificity, and sensitivity. [0085] FIG.8 is an example of a machine learning model 700 being trained using supervised and/or reinforcement learning techniques. Machine learning model 700 may be implemented using any number of models for supervised and/or reinforcement learning, such as but not limited to, an artificial neural network, a decision tree, naïve Bayes network, support vector machine, or k-nearest neighbor model, to name only a few examples. In some examples, processing circuitry one or more of IMD 10, external device 12, and/or computing system 24 initially trains the machine learning model 700 based on training set data 800 including numerous instances of input data, e.g., parameters determined from AF episode data, corresponding to and labeled as either recurrence or non-recurrence of AF. An output of the machine learning model 700 may be compared 804 to the target output 803, e.g., as determined based on the label. Based on an error signal representing the comparison, the processing circuitry implementing a learning/training function 805 may send or apply a modification to weights of machine learning model 700 or otherwise modify/update the machine learning model 700. For example, one or more of IMD 10, external device 12, and/or computing system 24 may, for each training instance in the training set 800, modify machine learning model 700 to change a probability or other likelihood output generated by the machine learning model 700 in response to data applied to the machine learning model 700. [0086] In various aspects, processing circuitry determines a likelihood of AF recurrence based on modeling AF episode timing as a bivariate Hawkes process. The bivariate Hawkes process is a statistical approach. The bivariate Hawkes process characterizes AF episode patterns where the AF episodes are assumed to be history dependent. The bivariate Hawkes process assumes that the heart switches between an AF state and an SR state, although the heart may take other non-AF rhythms. [0087] The AF episode pattern is modeled by two alternating point processes { ^^^^₁( ^^^^), ^^^^₂( ^^^^), ^^^^ > 0}, which describe the number of transitions that have occurred up to time t: one accounting for transitions from SR to AF occurring at times (points) Atty Ref. No.: A0010217WO01 ^^^^_1,1, ^^^^_1,2, …, and another for transitions from AF to SR occurring at times ^^^^_2,1, ^^^^_2,2, …. Here, the first subscript denotes the type of transition (SR-to-AF is denoted 1, while AF-to-SR is denoted 2), and the second subscript denotes the transition number. [0088] The following examples are illustrative of the techniques and systems described herein. [0089] Example 1. A medical system comprising: communication circuitry configured to receive an electrocardiogram (ECG) data set; a memory configured to store a first heart event model, a second heart event model, and an electrocardiogram (ECG) data set; and processing circuitry configured to: apply the ECG data set to a first heart event model to generate a first labeled data set of arrhythmia episodes in the ECG data set; apply the ECG data set to a second heart event model to generate a second labeled data set of arrhythmia episodes in the ECG data set, the second heart event model being trained on different data than the first heart event model; compare the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set; generate a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set; output, via the communication circuitry, the discordant pair data set to at least one reviewer computing device to be adjudicated by at least one reviewer; receive, from the at least one reviewer computing device via the communication circuitry, an adjudicated discordant pair data set; and determine a performance of the second heart event model based on the adjudicated discordant pair data set; and output an indication of the performance. [0090] Example 2. The medical system of example 1, wherein the indication of the performance of the second heart event model comprises an indication of the performance of the second heart event model relative to the first heart event model. [0091] Example 3. The medical system of example 1 or 2, wherein the indication of the performance of the second heart event model comprises an indication of whether the performance of the second heart event model exceeds a performance of the first heart event model by a threshold amount. [0092] Example 4. The medical system of example 3, wherein the processing circuitry is further configured to: output a confidence value indicative of a Atty Ref. No.: A0010217WO01 confidence in whether the performance of the second heart event model exceeds a performance of the first heart event model by a threshold amount. [0093] Example 5. The medical system of any of examples 1-4, wherein the indication of the performance of the second heart event model comprises an indication of a specificity of the second heart event model. [0094] Example 6. The medical system of nay of examples 1-5, wherein the indication of the performance of the second heart event model comprises an indication of a sensitivity of the second heart event model. [0095] Example 7. The medical system of any of examples 1-6, wherein the processing circuitry is further configured to: determine the performance of the second heart event model based on a known performance of the first heart event model. [0096] Example 8. The medical system of any of examples 1-7, wherein the first heart event model comprises a first machine learning model and the second heart event model comprises a second machine learning model. [0097] Example 9. The medical system of any of examples 1-8, wherein the processing circuitry is further configured to: train a third heart event model based on the adjudicated discordant pair data set. [0098] Example 10. The medical system of any of examples 1-9, wherein the processing circuitry comprises a cloud-based computing system. [0099] Example 11. A computer-implemented method for operating a computing system to evaluate performance of heart event models, the method comprising: applying, by processing circuitry, an electrocardiogram (ECG) data set to a first heart event model to generate a first labeled data set of arrhythmia episodes in the ECG data set; applying, by the processing circuitry, the ECG data set to a second heart event model to generate a second labeled data set of arrhythmia episodes in the ECG data set, the second heart event model being trained on different data than the first heart event model; comparing, by the processing circuitry, the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set; generating, by the processing circuitry, a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set; outputting, by the processing circuitry, the discordant pair data set to be adjudicated by a reviewer; receiving, by the Atty Ref. No.: A0010217WO01 processing circuitry, an adjudicated discordant pair data set; determining, by the processing circuitry, a performance of the second heart event model based on the adjudicated discordant pair data set; and outputting, by the processing circuitry, an indication of the performance. [0100] Example 12. The method of example 11, wherein the indication of the performance of the second heart event model comprises an indication of the performance of the second heart event model relative to the first heart event model. [0101] Example 13. The method of example 11 or 12, wherein the indication of the performance of the second heart event model comprises an indication of whether the performance of the second heart event model exceeds a performance of the first heart event model by a threshold amount. [0102] Example 14. The method of example 13, further comprising: outputting a confidence value indicative of a confidence in whether the performance of the second heart event model exceeds a performance of the first heart event model by a threshold amount. [0103] Example 15. The method of any of examples 11-14, wherein the indication of the performance of the second heart event model comprises an indication of a specificity of the second heart event model. [0104] Example 16. The method of any of examples 11-15, wherein the indication of the performance of the second heart event model comprises an indication of a sensitivity of the second heart event model. [0105] Example 17. The method of any of examples 11-16, further comprising: determining the performance of the second heart event model based on a known performance of the first heart event model. [0106] Example 18. The method of any of examples 11-17, wherein the first heart event model comprises a first machine learning model and the second heart event model comprises a second machine learning model. [0107] Example 19. The method of any of examples 11-18, further comprising: training a third heart event model based on the adjudicated discordant pair data set. [0108] Example 20. A non-transitory computer-readable storage medium storing instructions that when executed by processing circuitry of a computing system, Atty Ref. No.: A0010217WO01 cause the processing circuitry to: apply an electrocardiogram (ECG) data set to a first heart event model to generate a first labeled data set of arrhythmia episodes in the ECG data set; apply the ECG data set to a second heart event model to generate a second labeled data set of arrhythmia episodes in the ECG data set, the second heart event model being trained on different data than the first heart event model; compare the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set; generate a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set; output the discordant pair data set to be adjudicated by a reviewer; receive an adjudicated discordant pair data set; and output an indication of a performance of the second heart event model. [0109] In some examples, the techniques of the disclosure include a system that comprises means to perform any method described herein. In some examples, the techniques of the disclosure include a computer-readable medium comprising instructions that cause processing circuitry to perform any method described herein. [0110] It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the techniques). In addition, while certain aspects of this disclosure are described as being performed by a single module, unit, or circuit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units, modules, or circuitry associated with, for example, a medical device. [0111] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any Atty Ref. No.: A0010217WO01 other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer). [0112] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” or “processing circuitry” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements. [0113] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

Atty Ref. No.: A0010217WO01 WHAT IS CLAIMED IS: 1. A medical system comprising: communication circuitry configured to receive an electrocardiogram (ECG) data set; a memory configured to store a first heart event model, a second heart event model, and an electrocardiogram (ECG) data set; and processing circuitry configured to: apply the ECG data set to a first heart event model to generate a first labeled data set of arrhythmia episodes in the ECG data set; apply the ECG data set to a second heart event model to generate a second labeled data set of arrhythmia episodes in the ECG data set, the second heart event model being trained on different data than the first heart event model; compare the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set; generate a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set; output, via the communication circuitry, the discordant pair data set to at least one reviewer computing device to be adjudicated by at least one reviewer; receive, from the at least one reviewer computing device via the communication circuitry, an adjudicated discordant pair data set; and determine a performance of the second heart event model based on the adjudicated discordant pair data set; and output an indication of the performance. 2. The medical system of claim 1, wherein the indication of the performance of the second heart event model comprises an indication of the performance of the second heart event model relative to the first heart event model. Atty Ref. No.: A0010217WO01 3. The medical system of claim 1 or 2, wherein the indication of the performance of the second heart event model comprises an indication of whether the performance of the second heart event model exceeds a performance of the first heart event model by a threshold amount. 4. The medical system of claim 3, wherein the processing circuitry is further configured to: output a confidence value indicative of a confidence in whether the performance of the second heart event model exceeds a performance of the first heart event model by a threshold amount. 5. The medical system of any of claims 1-4, wherein the indication of the performance of the second heart event model comprises an indication of a specificity of the second heart event model. 6. The medical system of any of claims 1-5, wherein the indication of the performance of the second heart event model comprises an indication of a sensitivity of the second heart event model. 7. The medical system of any of claims 1-6, wherein the processing circuitry is further configured to: determine the performance of the second heart event model based on a known performance of the first heart event model. 8. The medical system of any of claims 1-7, wherein the first heart event model comprises a first machine learning model and the second heart event model comprises a second machine learning model. 9. The medical system of any of claims 1-8, wherein the processing circuitry is further configured to: train a third heart event model based on the adjudicated discordant pair data set. Atty Ref. No.: A0010217WO01 10. The medical system of any of claims 1-9, wherein the processing circuitry comprises a cloud-based computing system. 11. A computer-implemented method for operating a computing system to evaluate performance of heart event models, the method comprising: applying, by processing circuitry, an electrocardiogram (ECG) data set to a first heart event model to generate a first labeled data set of arrhythmia episodes in the ECG data set; applying, by the processing circuitry, the ECG data set to a second heart event model to generate a second labeled data set of arrhythmia episodes in the ECG data set, the second heart event model being trained on different data than the first heart event model; comparing, by the processing circuitry, the first labeled data set to the second labeled set to determine which labeled data of the first labeled data set differs than labeled data of the second labeled data set; generating, by the processing circuitry, a discordant pair data set comprising data of the ECG data set in which labeled data of the first labeled data set differs than labeled data of the second labeled data set; outputting, by the processing circuitry, the discordant pair data set to be adjudicated by a reviewer; receiving, by the processing circuitry, an adjudicated discordant pair data set; determining, by the processing circuitry, a performance of the second heart event model based on the adjudicated discordant pair data set; and outputting, by the processing circuitry, an indication of the performance. 12. The method of claim 11, wherein the indication of the performance of the second heart event model comprises an indication of the performance of the second heart event model relative to the first heart event model. 13. The method of claim 11 or 12, wherein the indication of the performance of the second heart event model comprises an indication of whether the performance of the Atty Ref. No.: A0010217WO01 second heart event model exceeds a performance of the first heart event model by a threshold amount. 14. The method of claim 13, further comprising: outputting a confidence value indicative of a confidence in whether the performance of the second heart event model exceeds a performance of the first heart event model by a threshold amount. 15. A non-transitory computer-readable storage medium storing instructions that when executed by processing circuitry of a computing system, cause the processing circuitry to perform the method of any of claims 11-14.