WO2006079530A1 - Selection and evaluation of diagnostic tests by means of discordance analysis characteristics (dac) - Google Patents
Selection and evaluation of diagnostic tests by means of discordance analysis characteristics (dac) Download PDFInfo
- Publication number
- WO2006079530A1 WO2006079530A1 PCT/EP2006/000670 EP2006000670W WO2006079530A1 WO 2006079530 A1 WO2006079530 A1 WO 2006079530A1 EP 2006000670 W EP2006000670 W EP 2006000670W WO 2006079530 A1 WO2006079530 A1 WO 2006079530A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- cut
- test
- dac
- coi
- pairs
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
- G01N33/48—Biological material, e.g. blood, urine; Haemocytometers
- G01N33/50—Chemical analysis of biological material, e.g. blood, urine; Testing involving biospecific ligand binding methods; Immunological testing
- G01N33/68—Chemical analysis of biological material, e.g. blood, urine; Testing involving biospecific ligand binding methods; Immunological testing involving proteins, peptides or amino acids
- G01N33/6803—General methods of protein analysis not limited to specific proteins or families of proteins
Definitions
- the invention relates to the use of a mathematical method with the aid of which a method for diagnosing an illness can be selected and evaluated.
- the selection and evaluation of a diagnostic method is performed on the basis of various parameters. Those in the foreground here are
- the diagnostic quality (sensitivity, specificity of the positive/negative decision, calculated with the aid of the numbers for correctly positive, correctly negative, falsely positive and falsely negative decisions),
- cost factor monetary costs, "costs”, that is to say strain (for example invasive subsequent examinations and additional examinations for the patient)
- the invention relates to selection with reference to diagnostic quality. Also possible, in addition, are statements on the "cost", a direct link being made between diagnostic evaluation and cost evaluation. Two methods are applied according to the prior art on the basis of the values of sensitivity and specificity for various cut-offs (decision thresholds) of the test or tests to be investigated:
- Sensitivity and specificity are determined and specified for prescribed cut-off values.
- the associated values for sensitivity (or specificity) can be determined for prescribed values of specificity (or sensitivity), for example 95%.
- the sensitivity and/or specificity are determined in the latter way, their values can be compared for two diagnostic tests by entering the respective positive/negative test results into a four-field table.
- the fields contain the combinations of "both tests positive", “both tests negative” as well as the two discordant cases.
- the statistical appraisal with regard to distinguishing sensitivity and specificity is performed with the aid of the McNemar test (Altaian DG. Practical statistics for medical research, London: Chapman & Hall, 1991: page 416).
- the integral under the function, the AUC (Area Under Curve) is used as parameter for the purpose of comparing methods. A higher AUC is attended by an improved diagnostic quality.
- the AUC can assume values of between 0.5 (missing sharpness of separation of the tests) and 1 (complete sharpness of separation of the test).
- Pointwise analyses depends strongly on the specific fundamental data and distributions. This means it is difficult to compare different data compilations related to one and the same diagnostic test, and to compare different diagnostic methods. Pointwise analyses are problematic, in particular, when only subranges of the value range described by the universe are 20 investigated within the data compilations. In the case of expensive measurement methods, it is usual, for example, to include in a data compilation only patients who have measured values about the decision limit, the cut-off value. The values ascertained here for sensitivity and specificity differ completely and fundamentally from the corresponding values for the universe.
- the McNemar test permits statements for specific value pairs (sensitivity/specificity), that is to say for special points on the ROC curve. The selection undertaken in this case of only one cut-off value leads to correct but restricted statements.
- the disadvantage of the ROC analysis which is, in particular, to be improved with the method according to the invention consists in that the diagnostic quality about the decision point (cut off) - which amounts to the actual value of a diagnostic method - is covered by the equivalent results of two methods far from the decision point.
- a conventional strategy for overcoming this disadvantage is to carry out the ROC analysis only with the aid of subgroups, the subgroups being defined by an excised value range of a marker.
- this method leads to distortions which do not permit the correct and objective evaluation of the differences of two methods.
- the distortions consist, in particular, in that patients with an illness are excluded from the analysis for low values of the marker used for selection, although they would be falsely detected as negative by this marker and possibly correctly detected as positive by the second marker.
- Such a subgroup analysis by means of ROC curves will thus favour the marker used for the selection. Similar selection artefacts exist at the uppe ⁇ .bDundary of the excised value range.
- the ROC analysis leads to a curve in which the correctly positive rate is plotted against the falsely positive rate.
- the curve produced can be interpreted only by a specialist with prior statistical training. Since the ROC curve is frequently used in medical articles or specialist information, no critical assessment of the potential and the limits of the diagnostic test is possible by the customary users of diagnostic tests (doctor). ⁇
- patient characteristics such as age, disease stage and disease subclass
- the methods according to the prior art cannot be used to ascertain directly or to represent the relationships between the diagnostic quality, which is represented, for example, with the aid of sensitivity/specificity value pairs or of an ROC curve, and the patient characteristics.
- the method according to the invention provides that only the discordant test results feature in the analysis.
- the discordant test results constitute the advantage of a diagnostic test or the difference between the two diagnostic tests. This is explained below by comparing two different diagnostic tests. To evaluate an individual diagnostic test, the second test required for this purpose is replaced by a mathematical description of an assumed, random distribution of test results.
- ⁇ a regression method such as Passing Bablok Regression, for example, in which "the method values of only one of the two groups to be discriminated feature.
- the measurement results are subdivided into 4 quadrants by the cut-off pair, which is manifested in the scatter plot by one vertical and one horizontal line each. This is illustrated in Figure 1.
- the quadrants subdivide the patients into four groups: either the two test results are uniformly negative (Ql) or positive (Q3) or they differ from one another (Q2 and Q4).
- Ql uniformly negative
- Q3 positive
- Q2 and Q4 the patients with a discordant test result
- Patients with discordant test results are ascertained for all possible cut-off pairs which can be calculated using the above-named methods. When k different cut-off pairs are yielded thereby, it is possible in each case to specify k different quadrants Ql, Q2, Q3 and Q4.
- the patients for in each case different cut-off pairs are mostly assigned to the quadrants Ql or Q3.
- the patients are assigned at least once either to quadrant Q2 or quadrant Q4.
- a patient is assigned to a quadrant Q2 for one cut-off pair and to a quadrant Q4 for another cut-off pair.
- all the patients are assigned uniquely to one of two subpopulations.
- the patients of one subpopulation are distinguished by the fact that it is possible to specify at least one cut-off pair for which it belongs to the quadrant Q2.
- Q4 for the other subpopulation.
- characteristics therefrom that is to say selected parameters are determined for the two subpopulations of each cut-off pair and plotted over all or one range of cut-off pairs. These characteristics are denoted in the further text as discordance analysis characteristics (DAC).
- DAC discordance analysis characteristics
- These parameters can be both test results (referred in each case to Q2 and Q4 correctly and falsely test positive and/or test negative), as well as parameters derived therefrom such as sensitivity, specificity and also patient characteristics such as, for example, age or disease stage.
- the numbers rp (correctly test positive-), fp (falsely test positive), rn (correctly test negative) and fn (falsely test positive) are analyzed by quadrant into Q2 and Q4 within the discordantly classified patient.
- sensitivities rp/(rp+fh) and specificities rn/(rn+fp) can be specified - referred in each case to the two diagnostic tests - within these discordantly classified patients.
- These parameters may be denoted as DAC sensitivity (DAC-SENS) or DAC specificity (DAC-SPEC) in order to avoid confusion. Since the patients in quadrants Q2 and Q4 have the characteristic that a test result for one test Tl is linked in each case with the opposite test result for the other test T2, it holds that:
- the positive predictive value DAC-pVw of one test is equal to the negative predictive value DAC- nVw of the other test, for example:
- such parameters, or new parameters yielded by arithmetic operations are plotted in a graph against the cut-off pairs or against the variables used in selecting the cut-off.
- the difference between the DAC specificity for Test 2 and the DAC specificity for Test 1 can be plotted against the associated cut-off pairs. If the criterion of equal sensitivity has been used for selecting the cut-off values, this difference of the DAC specificities can also be plotted against sensitivity.
- a curve obtained in this way has the property that the respective function value belonging to a-cut ⁇ off pair depends only on patients who have the measure ⁇ values in the local surroundings" of the cut-off values.
- the method according to the invention can be used to submit patient characteristics (PE) to analysis in the same way as the test results.
- the expectations of the patient characteristic PE for example mean value of age
- the expectations of the patient characteristic PE are ascertained for this purpose within the quadrants Q2 and Q4, respectively. This procedure is repeated again for all cut-off pairs, and a characteristic is produced as described above.
- statistical methods such as, for example, contingency table analyses, correlation analyses or variance analyses, it is possible to uncover relationships between these elected patient characteristics PE and diagnostic quality.
- the method according to the invention can also be applied when the diagnostic quality of only one diagnostic test is to be ascertained and evaluated.
- the second test is replaced by a mathematical description of a random test result in the same measured value distribution as that of the first test.
- Figure 1 shows a scatter plot for two different tests relating to the same illness
- Figure 2 shows an evaluation of the data of Lein et al. (2003) relating to the diagnostics of a prostate carcinoma by means of the DAC method
- Figure 3 shows an evaluation of age for patients with prostate carcinoma from the study by
- Figure 4 shows an evaluation of the data from the study by Keller et al. (1998) relating to the. significance of the tumour markers CYFRA 21-1 and CEA for distinguishing "" between squamous epithelium carcinoma and glandular carciRoma,
- Figure 5 shows an evaluation of the data from the study by Keller et al. (1998) relating to the significance of the tumour markers CYFRA 21-1 and NSE for the diagnostics of a bronchial carcinoma by comparison with benign illnesses,
- Figure 6 shows an evaluation of the tumour type for patients having bronchial carcinoma from the study by Keller et al. (1998) with the aid of the DAC method
- Figure 7a shows an application of the DAC method to a single marker: NSE for the diagnostics of bronchial carcinoma for small-cell tumours, and
- Figure 7b shows an application of the DAC method to a single marker: NSE for the diagnostics of bronchial carcinoma for non-small-cell tumours.
- Figure 2 shows a scatter plot of the measurement results of two different diagnostic tests, that is to say with the aid of two different diagnostic marks for patients with ( A. ) and without (X) illness.
- Four quadrants are yielded by drawing in the respective threshold values (cut-offs) COj and CO 2 for each marker.
- cut-offs threshold values
- the method according to the invention was applied to the data from Lein et al. (2003), the difference of the DAC specificities being used as parameter.
- the two abscissa axes specify cut-off pairs which were ascertained in accordance with the criterion of equal sensitivity.
- cPSA is superior if the DAC-SPEC difference > 0, otherwise it is tPSA.
- the curve (thick line) is specified together with its pointwise 95% confidence limits. The curves were smoothed for the representation.
- the confidence interval additionally indicated in Figure 2 indicates approximately the range in which it is possible to assume a significantly better diagnostic quality of the cPSA.
- the results of the DAC method can be used, inter alia, for the purpose of undertaking the definition of cut-off values and of then assessing these.
- the finding is that it is precisely the frequently recommended tPSA cut-off value of 4 ⁇ g/L cPSA which is particularly suitable for diagnostic statements.
- Example 2 Age was known for the patients of the study Lein et al. (2003) named in Example 1. For Example 2, age was incorporated into an evaluation in accordance with the DAC method.
- Figure 3 shows the result.
- the mean value of the cPSA Test Positive (continuous line) calculated in relation to the respective cut-off pairs (abscissa axes) is plotted against tPSA Test Positives (dashed line). It is to be observed that the PCA patients indicated as correctly positive by means of cPSA are younger on average than the patients indicated as correctly positive by means of tPSA.
- tumour markers in the diagnosis of bronchial carcinoma new options using fuzzy logic based tumour marker profiles.
- the analysis was based on the fact that high CYFRA 21-1 values are more readily associated with a squamous epithelium carcinoma and high CEA values are more readily associated with a glandular carcinoma.
- the detection of the glandular carcinoma was evaluated as "positive” (squamous epithelium: ' ⁇ negative").
- The-DAC specificities for the two markers are given in Figure -4 as a function of the sensitivity which was used in selecting the cut-off pairs.
- CEA is more suitable than CYFRA 21-1 for differentiating between a squamous epithelium carcinoma and a glandular carcinoma. This ability is largely uniform with the entire possible range of possible cut-off pairs.
- the diagnostic value of CYFRA 21-1 and NSE in the diagnostics of bronchial carcinoma was investigated.
- the difference of the DAC sensitivities for the two markers is plotted in Figure 5 as a function of the specificity which was used in selecting the cut-off pairs.
- the result was a better diagnostic value for CYFRA 21-1 over the entire range of possible cut-off pairs.
- NSCLC non-small-cell
- SCLC small-cell
- NSE and CYFRA 21-1 respectively specifically detect one tumour type.
- the DAC method was applied to evaluate the diagnostic quality of a single test.
- tumour marker NSE in the detection of small-cell bronchial carcinomas is to be demonstrated by comparison with the value of NSE in the detection of non-small-cell bronchial carcinoma, use being made of data from the study Keller et al. (1998).
- Figure 7a shows results for NSE in the diagnostics of bronchial carcinoma for small-cell tumours
- Figure 7b shows results for NSE in diagnostics of bronchial carcinoma for non-small-cell tumours.
- the difference of the positive predictive value pVw of the NSE in relation to the positive predictive value of a test, suitably described in mathematical terms, with random results is plotted.
- the abscissa describes the sensitivity which was used in ascertaining the cut-off pairs.
Landscapes
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Molecular Biology (AREA)
- Hematology (AREA)
- Chemical & Material Sciences (AREA)
- Urology & Nephrology (AREA)
- Physics & Mathematics (AREA)
- Immunology (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Food Science & Technology (AREA)
- Biotechnology (AREA)
- Proteomics, Peptides & Aminoacids (AREA)
- Microbiology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Cell Biology (AREA)
- Medicinal Chemistry (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Pathology (AREA)
- Investigating Or Analysing Biological Materials (AREA)
Abstract
The invention relates to a method for determining the diagnostic quality of diagnostic tests with the aid of measured values, of two diagnostic tests measured on patients with (positive) and without (negative) the illness to be diagnosed. Cut-off pairs (CO1, CO2)k are defined in accordance with the criterion of equal sensitivity of the measured values of two tests, and for each cut-off pair (CO1, CO2)k those measured values are selected which either lie above the associated cut-off (CO1) for the first test^v- below the associated cut-off (CO2) for the second test, or which he below the associated cut-off (CO1) for the first test and above the associated cut-off (CO2) for the second test For each cut-off pair (CO1, CO2)kand for the selected measured values, the number of those measured values is determined which belong to patients who correctly tested positive, falsely tested positive, correctly tested negative and falsely tested negative. The specificities for two diagnostic tests are calculated from these numbers.
Description
Selection and evaluation of diagnostic tests by means of discordance analysis characteristics (DAC)
The invention relates to the use of a mathematical method with the aid of which a method for diagnosing an illness can be selected and evaluated. The selection and evaluation of a diagnostic method is performed on the basis of various parameters. Those in the foreground here are
the laboratory properties (correctness, precision, selectivity etc. in production of the measured value),
the diagnostic quality (sensitivity, specificity of the positive/negative decision, calculated with the aid of the numbers for correctly positive, correctly negative, falsely positive and falsely negative decisions),
the cost factor (monetary costs, "costs", that is to say strain (for example invasive subsequent examinations and additional examinations for the patient)), and
the effect (improved therapy options, improved survival on the basis of the diagnostics).
The invention relates to selection with reference to diagnostic quality. Also possible, in addition, are statements on the "cost", a direct link being made between diagnostic evaluation and cost evaluation. Two methods are applied according to the prior art on the basis of the values of sensitivity and specificity for various cut-offs (decision thresholds) of the test or tests to be investigated:
a. Pointwise analysis of sensitivity and specificity, comparison by means of the McNemar test:
Sensitivity and specificity are determined and specified for prescribed cut-off values. Alternatively, the associated values for sensitivity (or specificity) can be determined for prescribed values of specificity (or sensitivity), for example 95%. The sensitivity and/or specificity are determined in the latter way, their values can be compared for two diagnostic tests by entering the respective positive/negative test results into a four-field table. The fields contain the combinations of "both tests positive", "both tests negative" as well as the two discordant cases. The statistical appraisal with regard to distinguishing sensitivity and specificity is performed with the aid of the McNemar test (Altaian DG. Practical statistics for medical research, London: Chapman & Hall, 1991: page 416).
b. ROC CReceiver Operating Characteristics') analysis:
The sensitivity/specificity value pairs for all conceivable cut-offs are plotted in the form of correctly positive rate = f(falsely positive rate), or otherwise formulated as sensitivity = f(l specificity) for the purpose of the ROC analysis. That is to say, all values of the value range 5 (measuring range) are used as cut-off value, and the associated sensitivities and specificities are ascertained. The integral under the function, the AUC (Area Under Curve) is used as parameter for the purpose of comparing methods. A higher AUC is attended by an improved diagnostic quality. The AUC can assume values of between 0.5 (missing sharpness of separation of the tests) and 1 (complete sharpness of separation of the test). Statistical evaluation of an AUC by means of, for
10 example, the Wilcoxon test or the comparison of a number of AUCs and the calculation of a statistical significance for the difference by means of the methods specified, for example, in Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29-36 or in DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric
15 approach. Biometrics 1988;44:837-45 are established as mathematical methods.
The results of the pointwise analyses depends strongly on the specific fundamental data and distributions. This means it is difficult to compare different data compilations related to one and the same diagnostic test, and to compare different diagnostic methods. Pointwise analyses are problematic, in particular, when only subranges of the value range described by the universe are 20 investigated within the data compilations. In the case of expensive measurement methods, it is usual, for example, to include in a data compilation only patients who have measured values about the decision limit, the cut-off value. The values ascertained here for sensitivity and specificity differ completely and fundamentally from the corresponding values for the universe.
Pointwise comparisons between two different tests or between a number of data compilations 25 relating to one and the same test are difficult, moreover, when they are not performed on the basis of equal sensitivity (or specificity). This is the case, for example, when value pairs (sensitivity, specificity)=(70%, 80%) and (50%, 90%) are to be compared. Aggregate measures which feature in the two values, such as the Youden index (sensitivity + specificity 1) or the efficiency (proportion of all correctly evaluated test results in the total number of cases) have not become established, 30 since their informativeness is restricted and erroneous deductions are possible. Thus, the McNemar test permits statements for specific value pairs (sensitivity/specificity), that is to say for special points on the ROC curve. The selection undertaken in this case of only one cut-off value leads to correct but restricted statements.
Because of these disadvantages, it is customary to undertake ROC analyses in order to evaluate the
T C rHαonnctTr, πuaiitv. Because of the fact that these analyses use the entire value range, it is possible
to avoid erroneous statements in the pointwise analyses. However, by virtue of its nature the ROC analysis leads to an overview consideration of the diagnostic quality. This state of affairs is formulated in the technical language by saying that each false/positive rate has the same weight in the ROC analysis (Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine [Review]. Clin Chem 1993;39:561-77).
When using a diagnostic test, however, the user is mostly interested only in a selected sensitivity range or specificity range. Existing or missing differences in this range are covered by the overall consideration such as is undertaken by the AUC in particular. Thus, existing differences can be ascertained as insignificant. On the other hand, missing differences can be falsely detected as differences through the occurrence of differences in other sensitivity/specificity ranges.
The disadvantage of the ROC analysis which is, in particular, to be improved with the method according to the invention consists in that the diagnostic quality about the decision point (cut off) - which amounts to the actual value of a diagnostic method - is covered by the equivalent results of two methods far from the decision point.
A conventional strategy for overcoming this disadvantage is to carry out the ROC analysis only with the aid of subgroups, the subgroups being defined by an excised value range of a marker. However, this method leads to distortions which do not permit the correct and objective evaluation of the differences of two methods. At the lower boundary of the excised value range, the distortions consist, in particular, in that patients with an illness are excluded from the analysis for low values of the marker used for selection, although they would be falsely detected as negative by this marker and possibly correctly detected as positive by the second marker. Such a subgroup analysis by means of ROC curves will thus favour the marker used for the selection. Similar selection artefacts exist at the uppeχ.bDundary of the excised value range.
Another strategy is to use selected portions of the ROC curve, that is to say the analysis of pAUC (partial Area Under Curve) (McClish DK. Analyzing a portion of the ROC curve. Med Decis Making 1989;9: 190-5.). This strategy leads to correct results. However, it has not been able to establish itself in practice because it is excessively complicated and, in particular, because the statistical verification of differences (calculation of the significance and the confidence interval) is difficult.
The ROC analysis leads to a curve in which the correctly positive rate is plotted against the falsely positive rate. The curve produced can be interpreted only by a specialist with prior statistical training. Since the ROC curve is frequently used in medical articles or specialist information, no critical assessment of the potential and the limits of the diagnostic test is possible by the customary users of diagnostic tests (doctor).
λ
_ 4 _
Furthermore, patient characteristics such as age, disease stage and disease subclass, can influence the diagnostic quality. The methods according to the prior art cannot be used to ascertain directly or to represent the relationships between the diagnostic quality, which is represented, for example, with the aid of sensitivity/specificity value pairs or of an ROC curve, and the patient characteristics.
The inventive solution to the problems of the methods known from the prior art consists in the method according to Claim 1 and the dependent claims.
Defining cut-off pairs and ascertaining discordant test results by means of quadrants
The method according to the invention provides that only the discordant test results feature in the analysis. The discordant test results constitute the advantage of a diagnostic test or the difference between the two diagnostic tests. This is explained below by comparing two different diagnostic tests. To evaluate an individual diagnostic test, the second test required for this purpose is replaced by a mathematical description of an assumed, random distribution of test results.
The diagnostic measured values of two diagnostic tests, Test 1 and Test 2, are plotted in a scatter plot. Two cut-off values (COi, CO2, called cut-off pair below) can be determined using the following methods from the measured values thus plotted:
- applying the criteria of equal sensitivity or specificity.
- a regression method such as, for example, Passing-Bablok Regression (Passing H., Bablok W. A new biometrical procedure for testing the equality of measurements from two different analytical methods. Application of linear regression procedures for method comparison studies in clinical chemistry, Part I. J Clin Chem Clin Fiochem 1983:21 :709- 20), in which all measured values feature.
■ a regression method such as Passing Bablok Regression, for example, in which" the method values of only one of the two groups to be discriminated feature.
- equal occupancy numbers for quadrants Q2 and Q4.
The measurement results are subdivided into 4 quadrants by the cut-off pair, which is manifested in the scatter plot by one vertical and one horizontal line each. This is illustrated in Figure 1. The quadrants subdivide the patients into four groups: either the two test results are uniformly negative (Ql) or positive (Q3) or they differ from one another (Q2 and Q4). Following the method according to the invention, only the patients with a discordant test result are included, that is to say patients from quadrants 2 and 4.
The procedure is the same over the entire value range of the test or tests investigated. Patients with discordant test results are ascertained for all possible cut-off pairs which can be calculated using the above-named methods. When k different cut-off pairs are yielded thereby, it is possible in each case to specify k different quadrants Ql, Q2, Q3 and Q4.
Within this overall consideration, the patients for in each case different cut-off pairs are mostly assigned to the quadrants Ql or Q3. However, for specific cut-off pairs the patients are assigned at least once either to quadrant Q2 or quadrant Q4. In this case, it is not possible within an overall consideration that a patient is assigned to a quadrant Q2 for one cut-off pair and to a quadrant Q4 for another cut-off pair. In other words, all the patients are assigned uniquely to one of two subpopulations.
The patients of one subpopulation are distinguished by the fact that it is possible to specify at least one cut-off pair for which it belongs to the quadrant Q2. A similar statement holds correspondingly for Q4 for the other subpopulation. It is possible to derive characteristics therefrom, that is to say selected parameters are determined for the two subpopulations of each cut-off pair and plotted over all or one range of cut-off pairs. These characteristics are denoted in the further text as discordance analysis characteristics (DAC). These parameters can be both test results (referred in each case to Q2 and Q4 correctly and falsely test positive and/or test negative), as well as parameters derived therefrom such as sensitivity, specificity and also patient characteristics such as, for example, age or disease stage.
It is advantageously possible in this way to ascertain and evaluate the diagnostic quality itself, on the one hand, and also relationships between the diagnostic quality and patient characteristics, on the other hand.
Ascertaining: and evaluating diagnostic quality
In order to ascertain and evaluate the diagnostic quality, the numbers rp (correctly test positive-), fp (falsely test positive), rn (correctly test negative) and fn (falsely test positive) are analyzed by quadrant into Q2 and Q4 within the discordantly classified patient.
Thus, sensitivities rp/(rp+fh) and specificities rn/(rn+fp) can be specified - referred in each case to the two diagnostic tests - within these discordantly classified patients. These parameters may be denoted as DAC sensitivity (DAC-SENS) or DAC specificity (DAC-SPEC) in order to avoid confusion. Since the patients in quadrants Q2 and Q4 have the characteristic that a test result for one test Tl is linked in each case with the opposite test result for the other test T2, it holds that:
DAC-SENSτi=rpτi/(rpτi+foτi)=fnτ2/(fiiT2+rpτ2)::=l-rpτ2/(φτ2+fiiΩ)=l-DAC-SENST2
Consequently, the DAC sensitivities of Test 1 and Test 2 complement one another to yield 1. A corresponding result holds for the DAC specificities.
Similarly, the following may be demonstrated to hold: within the discordantly classified patients, the positive predictive value DAC-pVw of one test is equal to the negative predictive value DAC- nVw of the other test, for example:
PVwτi=rpτi/(rpτl+fpτ,)=fnT2/(fnT2+rnT2)=nVwT2
Because of these dependencies, it suffices for the purpose of describing the diagnostic quality to specify DAC sensitivity (when using cut-off pairs according to the criterion of equal specificity) or DAC specificity (when using cut-off pairs according to the criterion of equal sensitivity) of only one of the tests. It is sufficient, furthermore, to specify either the positive or the negative predictive value.
In accordance with the method according to the invention, such parameters, or new parameters yielded by arithmetic operations, such as the difference between two DAC specificities, are plotted in a graph against the cut-off pairs or against the variables used in selecting the cut-off. For example, the difference between the DAC specificity for Test 2 and the DAC specificity for Test 1 can be plotted against the associated cut-off pairs. If the criterion of equal sensitivity has been used for selecting the cut-off values, this difference of the DAC specificities can also be plotted against sensitivity. A curve obtained in this way has the property that the respective function value belonging to a-cut~off pair depends only on patients who have the measure^ values in the local surroundings" of the cut-off values. These surroundings are prescribed by the cases which lie in the quadrants Q2 and Q4 belonging to the cut-off pairs. This characteristic is particularly important when there are two tests which are strongly correlated. This can be the case when different subforms of the same marker are used in the two tests. By contrast with the ROC analysis, it is thereby possible to specify in a simple way free from distortion the diagnostic quality for a restricted area around the cut-off values, doing so such that it is rendered possible for the first time to compare different measurement methods or different data compilations.
Ascertaining diagnostic quality in conjunction with patient characteristics TPE)
The method according to the invention can be used to submit patient characteristics (PE) to analysis in the same way as the test results. The expectations of the patient characteristic PE (for example mean value of age), for example, are ascertained for this purpose within the quadrants Q2
and Q4, respectively. This procedure is repeated again for all cut-off pairs, and a characteristic is produced as described above. By comparing the characteristics for the test results with the characteristics for the patient characteristics PE, with the aid of statistical methods such as, for example, contingency table analyses, correlation analyses or variance analyses, it is possible to uncover relationships between these elected patient characteristics PE and diagnostic quality.
Analysis with the aid of only one diagnostic test
The method according to the invention can also be applied when the diagnostic quality of only one diagnostic test is to be ascertained and evaluated. In this case, the second test is replaced by a mathematical description of a random test result in the same measured value distribution as that of the first test.
Figures and examples
The figures show:
Figure 1 : shows a scatter plot for two different tests relating to the same illness,
Figure 2: shows an evaluation of the data of Lein et al. (2003) relating to the diagnostics of a prostate carcinoma by means of the DAC method,
Figure 3 : shows an evaluation of age for patients with prostate carcinoma from the study by
Lein et al. (2003) with the aid of the DAC method,
Figure 4: shows an evaluation of the data from the study by Keller et al. (1998) relating to the. significance of the tumour markers CYFRA 21-1 and CEA for distinguishing ""between squamous epithelium carcinoma and glandular carciRoma,
Figure 5: shows an evaluation of the data from the study by Keller et al. (1998) relating to the significance of the tumour markers CYFRA 21-1 and NSE for the diagnostics of a bronchial carcinoma by comparison with benign illnesses,
Figure 6: shows an evaluation of the tumour type for patients having bronchial carcinoma from the study by Keller et al. (1998) with the aid of the DAC method,
Figure 7a: shows an application of the DAC method to a single marker: NSE for the diagnostics of bronchial carcinoma for small-cell tumours, and
Figure 7b: shows an application of the DAC method to a single marker: NSE for the diagnostics of bronchial carcinoma for non-small-cell tumours.
Figure 2 shows a scatter plot of the measurement results of two different diagnostic tests, that is to say with the aid of two different diagnostic marks for patients with ( A. ) and without (X) illness. Four quadrants are yielded by drawing in the respective threshold values (cut-offs) COj and CO2 for each marker. In accordance with the method according to the invention, only those patients who are to be assigned to the quadrants Q2 or Q4 are included per cut-off pair.
Example 1
A comparison was undertaken of the diagnostic quality of PSA (tPSA) and complex PSA (cPSA) in diagnostics of prostate carcinoma.
In the publication Lein M, Kwiatkowski M, Semjonow A, Luboldt H-J3 Hammerer P, Stephan C, et al. A multicenter clinical trial on the use of complexed prostate specific antigen in low prostate specific antigen concentrations. J Urol 2003; 170: 1175-9 (Lein et al., 2003) data collected in a multicentre study were investigated with reference to the question of whether cPSA has a higher diagnostic quality than tPSA. In the study, in which only patients having a (previously independently measured) tPSA measured value <4 μg/L were included, the results of ROC analyses were specified in conjunction with sensitivity/specificity value pairs. The results were not unambiguous. AUCCPSA > AUQPSA was determined for the tPSA value range 2.5:4 μg/L. By contrast, specificity differences found on the basis of 80%, 85%, 90%, and 95% were not significant.
The method according to the invention was applied to the data from Lein et al. (2003), the difference of the DAC specificities being used as parameter.
The curve from Figure 2 for the difference DAC-SPECOPSA-DAC-SPEQPSA was yielded on the basis of the DAC specificities of the two markers.
The two abscissa axes specify cut-off pairs which were ascertained in accordance with the criterion of equal sensitivity. cPSA is superior if the DAC-SPEC difference > 0, otherwise it is tPSA. The curve (thick line) is specified together with its pointwise 95% confidence limits. The curves were smoothed for the representation.
It emerges in accordance with the data of Lein et al. (2003) for the tPSA-cut-o'ff-range from 2.5 to 5.5 μg/L and the associated cPSA cut-off values in the range of 2-4.5 μg/L that an improved diagnostic quality of the cPSA can be assumed. It becomes clear that the various, in part contradictory statements in the publication of Lein et al. (2003) can be combined in one figure by means of the DAC method and thereby be explained.
The confidence interval additionally indicated in Figure 2 indicates approximately the range in which it is possible to assume a significantly better diagnostic quality of the cPSA.
The results of the DAC method can be used, inter alia, for the purpose of undertaking the definition of cut-off values and of then assessing these. The finding is that it is precisely the frequently recommended tPSA cut-off value of 4 μg/L cPSA which is particularly suitable for diagnostic statements.
Example 2
Age was known for the patients of the study Lein et al. (2003) named in Example 1. For Example 2, age was incorporated into an evaluation in accordance with the DAC method.
Figure 3 shows the result. The mean value of the cPSA Test Positive (continuous line) calculated in relation to the respective cut-off pairs (abscissa axes) is plotted against tPSA Test Positives (dashed line). It is to be observed that the PCA patients indicated as correctly positive by means of cPSA are younger on average than the patients indicated as correctly positive by means of tPSA.
Conclusions on advantageous conditions of use for the diagnostic markers, for example the cPSA with younger patients may be drawn on the basis of this Figure 2.
Example 3
Starting from the study Keller T, Bitterlich N, Hilfenhaus S, Bigl H, Lδser T, Leonhardt P (1998): Tumour markers in the diagnosis of bronchial carcinoma: new options using fuzzy logic based tumour marker profiles. J Cancer Res Clin Oncol 124: 565-574 (Keller et al. (1998)) relating to the diagnostics of bronchial carcinoma by means of tumour markers, the tumour markers CYFRA 21-1 and CEA were considered from the point of view as to which marker is more capable of distinguishing between a squamous epithelium carcinoma and a glandular carcinoma. The analysis was based on the fact that high CYFRA 21-1 values are more readily associated with a squamous epithelium carcinoma and high CEA values are more readily associated with a glandular carcinoma.
The detection of the glandular carcinoma was evaluated as "positive" (squamous epithelium: '^negative"). The-DAC specificities for the two markers are given in Figure -4 as a function of the sensitivity which was used in selecting the cut-off pairs.
It can be assumed that CEA is more suitable than CYFRA 21-1 for differentiating between a squamous epithelium carcinoma and a glandular carcinoma. This ability is largely uniform with the entire possible range of possible cut-off pairs.
Example 4
The diagnostic value of CYFRA 21-1 and NSE in the diagnostics of bronchial carcinoma was investigated. The difference of the DAC sensitivities for the two markers is plotted in Figure 5 as a function of the specificity which was used in selecting the cut-off pairs. The result was a better diagnostic value for CYFRA 21-1 over the entire range of possible cut-off pairs.
An investigation was made in a second DAC analysis as to how far different forms of bronchial carcinoma (non-small-cell (NSCLC) or small-cell (SCLC) bronchial carcinoma) relate to the result of the better diagnostic value for CYFRA 21-1. In this case, a patient was coded with 1 when he had an SCLC, and 2 in the case of an NSCLC.
The mean values of this coding for the CYFRA 21-1 test positives (continuous, thick line) and NSE test positives (dashed line) in accordance with the DAC methodology are plotted in Figure 6 against the specificity belonging to the respective cut-off pairs. It is to be observed that PCA patients indicated as correctly positive by CYFRA 21-1 are NSCLC patients for the most part, while it is chiefly SCLC patients that were indicated by NSE.
It may be concluded from this DAC analysis that NSE and CYFRA 21-1 respectively specifically detect one tumour type.
Example 5
The DAC method was applied to evaluate the diagnostic quality of a single test.
The diagnostic value of the tumour marker NSE in the detection of small-cell bronchial carcinomas is to be demonstrated by comparison with the value of NSE in the detection of non-small-cell bronchial carcinoma, use being made of data from the study Keller et al. (1998).
Figure 7a shows results for NSE in the diagnostics of bronchial carcinoma for small-cell tumours, and Figure 7b shows results for NSE in diagnostics of bronchial carcinoma for non-small-cell tumours. The difference of the positive predictive value pVw of the NSE in relation to the positive predictive value of a test, suitably described in mathematical terms, with random results is plotted. The difference-is "specified together with its 95% confidence limit. The abscissa describes the sensitivity which was used in ascertaining the cut-off pairs.
Clearly NSE detects SCLC tumours with a particularly high quality, since the pVw curve in Figure 7a is much higher, and is even at 1 over a wide range.
Claims
1. Method for determining the diagnostic quality of diagnostic tests with the aid of measured values, including the steps of
(a) providing measured values from two diagnostic tests (Test 1 and Test 2) measured 5 on patients with (positive) and without (negative) the illness to be diagnosed,
(b) finding cut-off pairs (COi, CO2X in accordance with the criterion of equal sensitivity for the measured values of the two tests,
(c) selecting for each cut-off pair (COi, CO2)Ic all measured values which either
i. lie above the associated cut-off (COi) for Test 1, or below the associated 0 cut-off (CO2) for Test 2, or
ii. lie' below the associated cut-off (COi) for Test 1 and above the associated cut-off(C02) for Test 2,
(d) determining for each cut-off pair (COi, CO2)k and for the measured values selected in (b) the numbers (rpi>ki fp;ikj rni,k, fhyO for those measured values which belong to 5 patients who correctly tested positive (rp;ik), falsely tested positive (fpi>k), correctly tested negative (rn;ik) and falsely tested negative (fn^k), where i=(Testl, Test 2),
(e) calculating DAC specificities DAC-SPEC;,k for two diagnostic tests from the numbers (rp;iki fpjikj rnjik, fhjjk) for each cut-off pair (COi, CO2X in accordance with DAC-SPECi,k=rni,k/(rni,k+fpi,k).
"0
2. Method according to Claim I1 characterized in that the DAC specificities DAC-SPEC;jk for the two diagnostic tests are plotted against the cut-off pairs (COi, C02)k or the associated sensitivities and compared.
3. Method according to Claim 1, characterized in that the definition of cut-off pairs (COi, C02)k in step (b) is performed in accordance with the criterion of equal specificity of the 5 measured values of the two tests, and in that in step (e) calculation of DAC sensitivities
DAC-SENSj,k is performed on the two diagnostic tests from the numbers (rp;ikι fp^, rn^ ftij,k) for each cut-off pair (CO], C02)k in accordance with DAC-SENSiik=rpiik/(rpiιk+fniik).
4. Method according to Claim 3, characterized in that the DAC sensitivities DAC SENS;ik for the two diagnostic tests are plotted against the cut-off pairs (COi, CO2)k or the associated 0 specificities and compared.
5. Method according to one of Claims 1 to 4, characterized in that the cut-off pairs are ascertained, in step (b) using a regression method.
6. Method according to one of Claims 1 to 4, characterized in that the cut-off pairs are ascertained in step (b) such that the selection of the measured values in step (c) i. and ii. lead to an equal number of measured values in the two subgroups i. and ii.
8. Method according to Claim 7, characterized in that the positive predictive value pVw;jk or the negative predictive value nVwi k are plotted against the cut-off pairs (COi, CO2X or the associated specificities or sensitivities.
9. Method according- to one the Claim 1, characterized in that other parameters such as efficiency or Youden index are determined from the numbers (rp;^ fp^k, rn^k, fn^k) for each cut-off pair (COi, CO2X, and are plotted against the cut-off pairs (COi, CO2X or the associated specificities or sensitivities.
10. Method according to Claim 1, characterized in that in step (d) it is not the numbers (rp^, φi,k, frii,k) which are determined for each cut-off pair (COi, CO2X and the measured values selected in (c), but the values PEIy5 are ascertained in accordance with one or more patient characteristics (PE), and in that in step (e) statistical characteristics such as expectation E^ or measured dispersion S^ for the measured values of these patient groups are calculated from these values.
11. Method according to Claim 10, characterized in that the statistical characteristics are plotted against the cut-off pairs (COi, CO2X or the associated specificities or 'sensitivities.
12. Method according to one of Claims 1 to 11, characterized in that the second test in step (a) is replaced by a mathematical description of a random test result in the same measured value distribution as that of the first test.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP06701307A EP1842139A1 (en) | 2005-01-28 | 2006-01-26 | Selection and evaluation of diagnostic tests by means of discordance analysis characteristics (dac) |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE200510004294 DE102005004294A1 (en) | 2005-01-28 | 2005-01-28 | Determining diagnostic quality of diagnostic tests using measured values involves calculating discordance analysis characteristics specificities for two diagnostic tests for each cut-off pair |
DE102005004294.5 | 2005-01-28 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2006079530A1 true WO2006079530A1 (en) | 2006-08-03 |
Family
ID=36190578
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/EP2006/000670 WO2006079530A1 (en) | 2005-01-28 | 2006-01-26 | Selection and evaluation of diagnostic tests by means of discordance analysis characteristics (dac) |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP1842139A1 (en) |
DE (1) | DE102005004294A1 (en) |
WO (1) | WO2006079530A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8538778B2 (en) | 2008-05-15 | 2013-09-17 | Soar Biodynamics, Ltd. | Methods and systems for integrated health systems |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030233197A1 (en) * | 2002-03-19 | 2003-12-18 | Padilla Carlos E. | Discrete bayesian analysis of data |
US20040096915A1 (en) * | 2000-10-27 | 2004-05-20 | Diamandis Eleftherios P. | Methods for detecting ovarian cancer |
US20040236723A1 (en) * | 2001-08-30 | 2004-11-25 | Reymond Marc Andre | Method and system for data evaluation, corresponding computer program product, and corresponding computer-readable storage medium |
-
2005
- 2005-01-28 DE DE200510004294 patent/DE102005004294A1/en not_active Withdrawn
-
2006
- 2006-01-26 EP EP06701307A patent/EP1842139A1/en not_active Withdrawn
- 2006-01-26 WO PCT/EP2006/000670 patent/WO2006079530A1/en active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040096915A1 (en) * | 2000-10-27 | 2004-05-20 | Diamandis Eleftherios P. | Methods for detecting ovarian cancer |
US20040236723A1 (en) * | 2001-08-30 | 2004-11-25 | Reymond Marc Andre | Method and system for data evaluation, corresponding computer program product, and corresponding computer-readable storage medium |
US20030233197A1 (en) * | 2002-03-19 | 2003-12-18 | Padilla Carlos E. | Discrete bayesian analysis of data |
Non-Patent Citations (2)
Title |
---|
E. TAILLARD, PH. WÄLTI, J. ZUBER: "Un Nouveau Test Statistique pour la Comparaison de Proportions", ACTES DE FRANCORO IV, 2004, Fribourg, Suisse, XP002378737, Retrieved from the Internet <URL:http://ina2.eivd.ch/Collaborateurs/etd/presentations.dir/francoro_test.pdf> [retrieved on 20060426] * |
J.A. HANLEY, B.J. MCNEIL: "The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve", RADIOLOGY, vol. 143, 1982, USA, pages 29 - 36, XP002378736 * |
Also Published As
Publication number | Publication date |
---|---|
EP1842139A1 (en) | 2007-10-10 |
DE102005004294A1 (en) | 2006-08-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Boyd | Mathematical tools for demonstrating the clinical usefulness of biochemical markers | |
Hill et al. | IgA antibodies to human tissue transglutaminase: audit of routine practice confirms high diagnostic accuracy | |
JP6415546B2 (en) | How to support differential diagnosis of stroke | |
JP2012531581A (en) | Methods and systems for cancer detection | |
JP2012531581A5 (en) | ||
Knychalski et al. | The evaluation of diagnostic value of the tumor markers: CCSA-2 and CEA in colorectal cancer | |
Mahé et al. | Screening test accuracy studies: how valid are our conclusions? Application to visual inspection methods for cervical screening | |
Magnussen et al. | Noninvasive methods to measure airway inflammation: future considerations | |
EP1842139A1 (en) | Selection and evaluation of diagnostic tests by means of discordance analysis characteristics (dac) | |
US10234382B2 (en) | Method for diagnosing lung diseases | |
Pakhale et al. | Comparison of per cent predicted and percentile values for pulmonary function test interpretation | |
US20190072559A1 (en) | Automated Method For Assessing Cancer Risk Using Tissue Samples, And System Therefor | |
CN111583994A (en) | Tumor marker cutoff value combined model and application thereof | |
Leithold et al. | Diagnostic work-up for the detection of malnutrition in hospitalized cancer patients | |
Introcaso et al. | Short-term increases of plasma cardiac troponin I are better evaluated by comparison with the reference change value | |
CN104237526B (en) | A kind of system detecting Alzheimer disease risk | |
CN111965241A (en) | Products, uses and methods for ovarian cancer-related screening and assessment | |
CN111965238A (en) | Products, uses and methods for non-small cell lung cancer-related screening and assessment | |
JP6885581B2 (en) | How to estimate the cutoff value in gastrin measurement | |
CN111613327B (en) | System for developing multiple myeloma diagnosis model based on logistic regression and application thereof | |
CN111965235A (en) | Products, uses and methods for pancreatic cancer-related screening and assessment | |
Bergón et al. | The predictive power of serum κ/λ ratios for discrimination between monoclonal gammopathy of undetermined significance and multiple myeloma | |
JP7452922B2 (en) | Operating method of cancer discrimination device, discrimination device and program | |
Sriamporn et al. | Consistency of cytology diagnosis for cervical cancer between two laboratories | |
Agrafiotis et al. | A simplified quantitative acid–base approach for patients with acute respiratory diseases |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWE | Wipo information: entry into national phase |
Ref document number: 2006701307 Country of ref document: EP |
|
WWP | Wipo information: published in national office |
Ref document number: 2006701307 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 12093855 Country of ref document: US |