Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
  • G06Q50/10—Services
  • G06Q50/20—Education
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
  • G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
  • G09B23/02—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for mathematics
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
  • G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers

Definitions

• the present invention provides a method of doing cognitive, medical and psychiatric, and diagnosis in general of latent properties of objects that are usually people using binary scored probing of the objects.
• test is only used either to rank examinees among themselves or, if mastery standards are set, to establish examinee levels of overall mastery of the content domain of the test.
• it is not used to produce a finely grained profile of examinee "cognitive attributes" within a single content domain. That is, an algebra test can be used to assess John's overall algebra skill level relative to others or relative to the standard for algebra mastery but it cannot determine cognitive attribute mastery, such as whether John factors polynomials well, understands the rules of exponents, understands the quadratic formula, etc., even though such fine grained analyses are clearly to be desired by instructor, student, parent, institution, and government agency, alike.
• cognitive diagnosis refers to providing fine-grained profiles of examinee cognitive attribute mastery/non-mastery.
• the cognitive diagnostic algorithm that forms the core of the invention is a particular statistical method.
• a statistical method or analysis combines collected data and an appropriate probability model of the real world setting producing the data to make inferences (draw conclusions). Such inferences often lead to actual decision-making.
• the cognitive diagnosis indicating that Tanya is deficient on her mastery of the quadratic formula can be followed up by providing remediation to improve her understanding of the quadratic formula.
• Example 1 A drug with unknown cure probability p (a number between 0 and 1) is administered to 40 ill patients. The result is that 30 are cured.
• the scale for examinee ability is such that ability less than -2 indicates very low ability examinees (the lowest 2.5%), 0 indicates an average ability examinee and above 2 indicates very high ability examinees (the highest 2.5%).
• IRT based statistical methods are currently heavily used in educational measurement to statistically assess (infer from test data and the IRT model) examinee latent ability levels.
• Educational measurement is the applied statistical science that uses probability models and statistical methods to analyze educational data (often test data) to provide information about learning processes and about various educational settings and to evaluate individual level and group level (state, school district, nation, etc.) intellectual performance.
• Cognitive diagnosis refers to a relatively fine-grained analysis that evaluates examinees in terms of which specific skills (generically called "attributes") in a general subject area each examinee possesses or lacks (see Frederiksen, N., Glaser, R. ; Lesgold, A., and Schafto, M.,1990, Diagnostic Monitoring of Skill and Knowledge Acquisition.
• Example 2 A need for cognitive diagnosis.
• One of the inventors an instructor of a college level introductory statistics course, gave an exam on the first three chapters of the text. The items were constructed to represent the distinct concepts taught in the three chapters. It was deserved to evaluate the students by more than their score on the exam; specifically how well they understand the concepts that were taught.
• a list of the eight concepts, or attributes was compiled: (1) histogram, (2) median/quartile, (3) average/mean, (4) standard deviation, (5) regression prediction, (6) correlation, (7) regression line, and (8) regression fit. As expected, some items involved more than one attribute per item. On the forty- item exam, each attribute appeared in an average of six items.
• Towanda may be judged a master of the rules of exponents but may apply her understanding of exponents to an item incorrectly because the needed competency concerning the rules of exponents is exceptionally high for the item Towanda is trying to solve and in fact is higher than that possessed by Towanda, even though she is a master of the attribute rules of exponents.
• a variation of the simple coin tossing illustration discussed earlier may help illustrate the over- fitting issue. If a possibly unfair coin is tossed four times and comes up as four heads, the most simplistic over-fitted deterministic approach might conclude that the coin will always comes up heads, thus predicting that the pattern to be expected for new coin tossing will be to always get heads. Whereas, the probabilistic statistical approach merely concludes that all that can be inferred is that the unknown probability of heads lies in the interval (0.4,1). From this appropriately cautious perspective, it is thus quite possible the coin is actually fair!
• the UM upon which the present invention is in part based, is statistical and hence, as is crucial, avoids over-fitting of the data by predicting attribute masteries and non-masteries for examinees only when there is strong evidence to support such predictions.
• the flow chart of Fig. 2 illustrates the UM Cognitive Diagnostic (UMCD) procedure as proposed in DiBello et al. Some of its elements are common to the current UMCD algorithm of the present invention.
• the result of administering the test is the examinee responses data matrix
• x is an n by N matrix of 0s (0 denoting an incorrect response for an item/examinee combination) and Is (1 denoting a correct response for the item/examinee combination). Theyth column represents the responses to the n test items for a particular examinee , . For example if two examinees took a three item test, then x might be
• & parameter of a scientific model in general and of a probability model in particular is an unknown quantity in the model that must be statistically determined from data for each particular application of the model, with the value of this parameter varying from application to application.
• the parameters of the n item, N examinee UM, generically denoted by ⁇ are given by
• ⁇ (a, ⁇ ; r, ⁇ , c)
• an "attribute” is a general term for any bundle of knowledge that can be judged as mastered or not mastered.
• the selected attributes (Block 201 of Fig. 2) to be used to build the item/attribute incidence matrix (Block 205 of Fig. 2) are defined by the user of the algorithm and can be anything the user wishes. Indeed the freedom of the user to choose attributes unconstrained by any particular cognitive theory of learning and/or mental processing is a real strength of the UM.
• the UM allows the user to select any attributes based on any conceptualization of learning, mental functioning, or cognition, even a highly informal structure that would be accessible to an instructor of a typical classroom course.
• Each of the N examinees has K attributes and hence the ⁇ component of ⁇ is a matrix of dimension N by K.
• each row of ⁇ corresponds to a single examinee and has K elements (0's and l's). An 0 indicates examinee nonmastery and a 1 indicates examinee mastery.
• a UM model based cognitive diagnosis is to use the available test data x that results from administering the test (Block 207 of Fig. 2) to infer (Block 213 of Fig. 2) for each examinee which of the K attributes there is strong evidence that she has mastered and which there is strong evidence that she has not mastered (noting that for each examinee there will likely be certain attributes for which there is not strong evidence of either mastery or non-mastery).
• the required input data to initiate the proposed UM algorithm consists of two data files that are relatively easy to understand and produce without the user needing a sophisticated understanding of cognitive science, this an advantage of the UMCD relative to other prior art.
• a list of the attributes required to be simultaneously mastered in order to correctly solve the item is selected (Block 201 of Fig. 2).
• the user/practitioner first decides which attributes to cognitively diagnose in the particular educational setting and then constructs the needed test items (Block 203 of Fig. 2). Sometimes the user constructs the test items first and then selects the attributes to be diagnosed.
• the user typically chooses the relevant atfributes and designs the questions to measure these atfributes (in either order), and then decides which of the chosen atfributes are required for the correct solution of each item.
• This relatively easy user activity may be assisted by consultants with personal knowledge of UMCD or by referencing a UMCD tutorial presenting the basic principles of good cognitive diagnosis item manufacture, attribute definition, and incidence matrix construction for use with the UMCD program.
• Item 1 requires Attributes 2 and 3
• Item 2 requires Attribute 1
• Item 3 requires Attributes 3 and 4.
• the examinee response data consists of a record for each examinee of which items were answered correctly and which items were incorrectly answered. Notationally, this is expressed as follows:
• Examinee 2 responses 1 0 0 1 This shows Examinee 1 got Items 3 and 4 right, and Examinee 2 got Items 1 and 4 right. As already indicated, all of these Xy responses are collected together to form the matrix of responses test data examinee responses x.
• ⁇ denotes the (unknown) latent vector of length K indicating for each of the K atfributes examinee mastery (denoted by a 1) and examinee nonmastery (denoted by a 0).
• K (1,0,1,1,0) means that Examinee j has mastered atfributes 1, 3, and 4 and has not mastered attributes 2 or 5. Inferring what ⁇ is for each examinee is the goal of cognitive diagnosis.
• ⁇ ( ⁇ , ⁇ , r, ⁇ , c) denotes the item and examinee parameters of the n item by N examinee model.
• the UM uses the notion of an item response function (IRF), as do all IRT- based models.
• IRF item response function
• An IRF is an increasing S-shaped curve bounded by 0 below and 1 above. In the usual IRT model setting this provides the probability of getting an item correct as a function of a continuous latent ability such as statistics ability, traditionally denoted by ⁇ .
• graphically, such an IRF is represented in Fig. 1.
• the notation P( ⁇ ) refers to the probability of getting the item correct for an examinee of latent ability ⁇ .
• the formulas for the UM depend on using the Fig. 1 ERF.
• the probability model for one examinee responding to one item is given next.
• the symbol 17X1 indicates taking the product over the range of i andy, namely over the outer product asy ranges from 1 to N and over the inner product as i ranges from 1 to n.
• X i ⁇ responses for different examinees and for different items that allows the double product in the basic UM IRT model given by Equation 2.
• Xy denotes the i,j ⁇ member of x and is either a 1 or a 0 according as Item i is answered correctly or incorrectly by Examinee j .
• Equation 1 The Core UM Concepts of Positivity and Completeness
• S The first factor, S , of Equation 1 models positivity
• V( ⁇ j +c,) the second factor
• the second factor P ( ⁇ , +c,) of Equation 1 is considered, which models the degree of completeness for Item i and the prescribed attributes of the UM.
• the parameter c which varies from item to item, is the completeness parameter.
• one core aspect of the UM is that in order to keep the number of parameters per item to a reasonable and hence statistically tractable number relative to the size of the available data set, intentionally trying to explicitly model the role of many minor yet influential latent atfributes is omitted.
• An influential attribute means that attribute mastery versus non-mastery changes the probability of answering the item correctly.
• c quantifies the relative combined influence of these omitted attributes as compared with the combined influence of the explicitly modeled atfributes ⁇ upon examinee responding to Item i.
• the accurate and complete (in the sense of including all the attributes that in fact influence examinee item performance) cognitive diagnostic model for the university statistics examination discussed above includes 200 atfributes.
• the model is restricted to explicitly having 8 attributes in the UM's incidence matrix.
• 8 atfributes are selected which are believed to be important in determining examinee test performance, including all the attributes the instructor wishes to cognitively diagnose.
• the role of c- for an item is to proportion out the relative importance of the major included atfributes ⁇ - versus the excluded minor but still influential attributes as built into the UM through ⁇ j in determining examinee item performance.
• ⁇ is a standard normal random variable (the well- known "bell-shaped" curve), as shown in Fig. 4.
• Equation 3 the item/attribute incidence matrix is needed input into determining ⁇ X
• ⁇ ik Prob (Attribute k applied correctly to Item i given that the examinee has mastered Attribute k).
• High positivity holds for an item when its r's are reasonably close to 0 and its ⁇ 's are reasonably close to 1. That is, with high probability an examinee applies the atfributes required for the item according to the item/attribute incidence matrix correctly if and only if the examinee has mastered these attributes. For example, when high positivity holds, an examinee lacking at least one of the required atfributes for the item is very likely to get the item wrong.
• an examinee who has mastered both the required atfributes is likely to get the item right, provided also that the ⁇ + c is large indicating that the examinee likely will either use the (possibly many) required atfributes needed for the item but excluded from the model correctly as well (i.e., the examinee's ⁇ is large) or that the excluded atfributes will play only a minor role (i.e., the item's c is large).
• the selected attributes forming the incidence matrix being a subset of a larger group of atfributes influencing examinee test item performance with the remainder of the larger group of atfributes being accounted for in the UM by a residential ability parameter, namely completeness, is common to the 1995 UM procedure and the UMCD of the present invention.
• Positivity namely the model including parameters describing how the test items depend on the selected set of atfributes by accounting for a probability that each examinee for each individual test item may achieve mastery of all the atfributes from the subset of the selected set of attributes required for the individual test item but fail to apply at least one such required and mastered attribute correctly to the individual test item thereby responding to the test item incorrectly is common to the 1995 UM procedure and the UMCD of the present invention.
• each examinee for each individual test item may have failed to achieve mastery of at least one specified attribute required for the item and nevertheless apply these required specified atfributes for which mastery was not achieved correctly to the item and also apply the remaining required and mastered atfributes from the selected set of atfributes correctly to the item thereby responding to the test item incorrectly.
• the 1995 UM item parameters were not identifiable whereas the parameters of the UM of the present invention are. Also in common is the administering of the test (Block 207)
• Model calibration refers to the use of the available data, which is viewed as generated by the model, to statistically estimate the unknown model parameters. It must be understood that without calibration, probability models are useless for doing real world inference such as cognitive diagnosis.
• Bayes Modeling Example Although the notion of a Bayes probability model is a complex and sophisticated concept, a simple example will clarify the basic idea of what a Bayes probability model is and how its statistical analysis proceeds.
• Example 3 Example 1 (modified).
• the Bayes approach quantifies such probabilistic knowledge possessed by the investigator about the likelihood of various values of the parameters of the model by assigning a prior probability distribution to the parameter p. That is, a Bayes model puts a probability distribution on the model's parameter(s), where this distribution reflects how likely the user believes (based on prior knowledge and/or previous experience) various values of the unknown parameter are likely to be.
• the prior distribution for p is given as a "density" in the Fig. 5.
• the probability of the unknown parameter p falling in the interval (0.7, 0.8) is much higher than the probability of the unknown parameter p falling in the interval (0.8, 0.9), a fact which will influence our use of the data to estimate p. More generally, the values of p become much more unlikely as p moves away from 0.7 towards either 0.5 or 0.9.
• Block 801 the observed data
• Block 803 the Bayes probability model, which is the combination of the prior disfribution f ⁇ co ) on the model parameters (Block 803) and the likelihood probabililty disfribution f ⁇ X I co ) (Block 805). Note that both and co are likely to be high dimensional in practice. Then the posterior disfribution of parameters (indicated in Block 809) given the data is computed as follows
• ( ⁇ ) ⁇ 0 is the prior disfribution ( " (co) referred to as a density) on the parameters specially created for the Bayes model.
• the choice of the prior is up to the Bayes practitioner and is indicated in Block 803.
• ⁇ ) is the usual likelihood probability disfribution (see Block 805; the notion of a likelihood explained below) that is also at the heart of a non Bayesian statistical inference about ⁇ for observed data X.
• co) ⁇ 0 tells that the random mechanism by which each particular parameter value ⁇ produces the observed data X, whereas the prior distribution f( ⁇ ) tells how likely the practitioner believes each of the various parameter values is.
• Equation 4 f( ⁇
• X ) via' Equation 4, as indicated in Block 811. For example, the inference t aXp 0.72 in Example 3 was the result of finding the value ofp that maximizes the posterior f(p
• Bayesian Statistical Methods Using Markov Chain Monte Carlo The use of complex Bayes models with many parameters has become a reasonable foundation for practical statistical inference because of the rapidly maturing MCMC simulation-based computational approach.
• MCMC is an excellent computational tool to statistically analyze data sets assumed to have been produced by such Bayes models because it allows bypassing computing the complicated posterior distribution of the parameters (Equation 4) required in analytical computational approaches.
• the specific MCMC algorithm used in the invention allows bypassing computing the complex integral in the denominator (see Equation 4) of typical Bayesian approaches (via the Metropolis-Hastings algorithm) and simplifies computing the numerator (see Equation 4) of typical Bayesian approaches (via the Gibbs sampling algorithm).
• MCMC complex Bayes models were usually only useful in theory, regardless of whether the practitioner took a non-Bayesian or a Bayesian approach.
• the heart of a likelihood-based inference is a function describing for each possible value of the parameter being estimated how likely the data was to have been produced by that value.
• the value of the parameter that maximizes this likelihood function or likelihood probability disfribution (which is f(X
• ⁇ ) is best thought of as the probability disfribution for the given parameter(s)co.
• Bayesian Likelihood Based Statistical Inference This is merely likelihood-based inference as modified by the prior belief or information (as expressed by the prior probability distribution, examples of such a prior shown in Figs. 5,6, and 7) of the likelihood of various parameter values, as illustrated in Fig. 10.
• Prior refers to information available before (and in addition to) information coming from the collected data itself.
• the posterior probability distribution is the function showing the Bayes likelihood disfribution of a parameter resulting from "merging" the likelihood function for the actually observed data and the Bayes prior. For instance, in Example 3 with the triangular prior disfribution for p of Fig. 5 as before, Fig.
• Equation 10 simultaneously shows the likelihood function for p, the triangular prior for p, and the Bayes posterior distribution (also called the Bayes likelihood disfribution) for p resulting from this prior and having observed 30 cures out of 40 frials in the data.
• equation 4 gives a formula for the needed posterior distribution function for a given prior and likelihood probability function. Note from the posterior disfribution in Fig. 10 that the estimate ofp obtained by maximizing the posterior distribution is approximately 0.72 as opposed to 0.75 that results from using the maximum likelihood estimate that maximizes the likelihood function.
• MCMC is a tool to simulate the posterior distribution needed to carry out a Bayesian inference in many otherwise infractable Bayesian problems.
• a "simulation" is something that substitutes for direct observation of the real thing; in our situation the substitution is for the direct computation of the Bayes posterior disfribution. Then, by observing the results of the simulation it is possible to approximate the results from direct observation of the real thing.
• MCMC simulation may be used, thereby avoiding the complex intractable integral needed to solve for the posterior disfribution in a Bayes statistical analysis.
• MCMC simulation estimates the posterior disfribution of several statistical cognitive diagnostic models. Each such MCMC uses as input the Bayesian structure of the model (UM or other) and the observed data, as the basic Bayes formula of Equation 4. Recall that the Bayesian structure of the model refers to the prior distribution and the likelihood probability disfribution together
• Non-UM Prior Art Examples Now that the necessary conceptual background of statistical concepts and data computational techniques (especially Bayes probability modeling and MCMC) have been explained and illustrated, the relevant prior art is described (in addition to the UM), consisting of certain other proposed or implemented cognitive diagnostic procedures.
• the model simplifying role played by 0 and the positivity parameters ⁇ 's and r's in UM methodology, thus making the UM model used in the invention tractable, is instead replaced in the Bayes net approach by graph-theoretic techniques to reduce the parametric complexity of the Bayes net's probability tree of conditional probabilities linking latent attribute mastery states with examinee responses to items. These techniques are in fact difficult for a non graph-theoretic expert (as is true of most cognitive diagnostic users) to use effectively.
• ETS Educational Testing Service
• This approach is not easy for practitioners to be able to use on their own, for reasons already stated.
• exporting the approach for reliably independent use outside of ETS has been difficult and requires serious training of the user, unlike the Bayes UM methodology of the present invention.
• it may not have the statistical inference power that the present UM invention possesses, especially because of the important role played by each of positivity, incompleteness with the introduction of ⁇ , and the positive correlational structure that the Bayes UM of the present invention places on the attributes (the importance of which is explained below in the Description of the Preferred Embodiments section).
• Blocks 201, 203, and 207 of the Bayes net of Fig. 12 approach are in common with the DiBello et al 1995 approach (recall Fig. 2).
• Block 1201 is just Block 807 of Fig. 8 of the genereal Bayes inference approach specialized to the Bayes net model.
• Block 1203 is a special case of computing the Bayes posterior (Block 809 of Fig. 8), in fact using MCMC.
• the cognitive diagnostic step (Block 1205) is just a special case of the Bayes inference step (Block 811).
• Kikumi Tatsuoka's Rule Space Approach Two good references are Tatsuoka, K., 1983, Rule space; an approach for dealing with misconceptions based upon item response theory. Psychomefrika 20, 34-38, and Tatsuoka, Kikumi, 1990, Toward an integration of item response theory and cognitive error diagnosis. Chapter 18 in Diagnostic Monitoring of Skill and Knowledge Acquisition. Mahwah, New Jersey, Lawrence Erlbaum. A schematic of the Rule Space approach is shown in Fig. 13. The rule space model for the randomness of examinee responding for each possible attribute vector structure is in some ways more primitive and is much different than the Bayes UM of the present invention. It is based entirely on a probability model of random examinee errors, called "slips" by Tatusoka.
• the computational approach taken is typically Bayesian. Its fundamental idea is that an actual response to the test items should be like the "ideal" production rule based deterministic response (called the ideal response pattern) dictated by the, item/attribute incidence matrix and the examinee's true cognitive state as characterized by his/her atfribute vector, except for random slips. Cognitive diagnosis is accomplished by an actual examinee response pattern being assigned to the "closest" ideal response pattern via a simple Bayesian approach.
• the rule space approach is basically a pattern recognition approach.
• a rule space cognitive diagnosis is computationally accomplished by a complex dimensionality reduction of the n dimensional response space (because there are n items) to the two dimensional "rule space" (see Block 1303 and the two Tatsuoka references for details).
• This produces a two dimensional Bayesian model ( Block 1301, which is analogous to the general Bayes model building Block 807 of Fig. 8) .
• This reduction to the low dimensional "two space” allows one to directly carry out the needed Bayes computation (see Block 1305) without having to resort to MCMC.
• the atfribute state that ⁇ that best predicts the assigned ideal response pattern is inferred to be the examinee's cognitive state, thus providing a cognitive diagnosis.
• Block 1307 is to carry out the actual cognitive diagnosis.
• Block 1401 is similar to Block 201 of Fig. 2, except here the attributes are continuous. Blocks 203 and 207 are in common with the other prior art procedures. Block 1405 is analogous to the Fig. 2 UM Block 209, Block 1405 is analogous to the Fig. 2 UM Block 213, and finally in common with all procedures, the last Block 1407 is the carrying out of a cognitive diagnosis.
• Deterministic Cognitive Model Based Procedures There are numerous approaches that use a deterministic cognitive diagnosis approach. The statistical approaches are by their statistical nature superior to any deterministic approaches (that is, rule-based, data mining, artificial intelligence, expert systems, Al, neural-net based, etc.). All deterministic approaches have no deep and valid method for avoiding over-fitting the data and thus erroneously conclude attribute masteries and non-masteries where in fact the supporting evidence for such conclusions is weak
• the goal of diagnostic tools is to provide the practitioner with a short list of disorders that seem plausible as a result of the observed symptoms and personal characteristics (such as gender, ethnicity, age, etc.) of the patient.
• Bayesian posterior probabilities assigned to the set of disorders is analogous to assigning a set of posterior probabilities to a set of cognitive atfributes.
• Bayesian Network Based Systems A Bayesian Network for medical diagnostics represents the probabilistic relationship between disorders and symptoms/characteristics in a graph that joins nodes that are probabilistically dependent on one another with connecting lines.
• a good general reference is Herskovits, E. and Cooper, G., 1991, Algorithms for Bayesian belief-network precomputation ,Meth. Lnf. Med ,30, 81-89.
• a directed graph is created by the Bayes net modeling specialist and leads from the initial set of nodes that represent the set of disorders through an optional set of intermediate nodes to the resulting observed set of symptoms/characteristics.
• the posterior probability of having a certain disorder is calculated using the Bayes approach of Equation 4 and possibly MCMC.
• a prior distribution has been assigned to the proposed set of possible disorders, and specifying the conditional probabilities for each node given a predecessor node in the graph specifies the needed likelihood function of Equation 4.
• each line of the graph has a conditional probability associated with it.
• Medical applications of Bayesian Networks originally obtained the required numerical values for the conditional probabilities by consulting the appropriate medical literature, consulting available large data sets, or using expert opinion. Now, estimation techniques for obtaining these conditional probabilities have recently been developed.
• Neural Network and Fuzzy Set Theory Based Systems Both Neural Networks and Fuzzy Set Theory based approaches are graphical networks that design the probability relationships between the symptoms/characteristics and disorders via using networks and then do extensive fraining using large data sets.
• the networks are less rigidly specified in Neural Networks and in Fuzzy Set Theory based networks than in Bayesian Networks.
• the fraining of the networks essentially compares many models that are calibrated by the fraining process to find one that fits reasonably well.
• Fuzzy Set Theory techniques allow for random error to be built into the system. Neural Networks may also build in random error as well, just not in the formal way Fuzzy Set Theory does.
• Deterministic Systems Two deterministic approaches used are Branching Logic Systems and Heuristic Reasoning Systems. As discussed above in the cognitive diagnostic prior art portion, . all deterministic systems have drawbacks in comparison with probability model based approaches like the UM.
• the present invention does diagnosis of unknown states of objects (usually people) based on dichotomizable data generated by the objects.
• Applications of the present invention include, but is not limited to, (1) cognitive diagnosis of student test data in classroom instructional settings, for purposes such as assessing individual and course-wide student cognitive progress to be used such as in guiding instruction-based remediation intervention targeted to address detected cognitive deficits, (2) cognitive diagnosis of student test data in computerized instructional settings such as web-based course delivery systems, for purposes such as assessing individual and course-wide cognitive progress to be used such as to guide computer interactive remediation/intervention that addresses detected cognitive deficits, (3) cognitive diagnosis of large-scale standardized tests, thus assessing cognitively defined group-based cognitive profiles for purposes such as evaluating a school district's instructional effectiveness, and providing cognitive profiles as feedback to individual examinees, and (4) medical and psychiatric diagnosis of medical and mental disorders for purposes such as individual patient/client diagnosis, treatment intervention, and research.
• the scope of application of the present invention includes the diagnosis of any latent (not directly observable) structure (possessed by a population of individual objects, usually humans) using any test-like observed data (that is, multiple dichotomizably scored pieces of data from each object such as the recording of multiple questions scored right/wrong observed for each test taker) that is probabilistically confrolled by the latent structure as modeled by the UM.
• attitude questionnaire data might be diagnosed using the present invention to infer for each of the respondees certain atfributes such as social liberal vs. conservative, fiscal liberal vs. conservative, etc.
• Residual Ability Parameter A low dimensional ( certainly not greater than 6, often unidimensional) set of quantities that together summarize examinee proficiency on the remainder of the larger group of atfributes influencing examinee performance on items
• Dichotomously scored probe Analogous to an item in the cognitive diagnosis setting. Anything that produces a two valued response from the object being evaluated
• Association Any relationship between two variables such as atfributes where the value of one variable being larger makes the other variable probabilistically tend to be larger (positive association) or smaller (negative association). Correlation is a common way of quantifying association. Unobservable dichotomized properties. Analogous to atfributes in cognitive diagnostic setting. Any property of objects that is not observable but either has two states or can be encoded as having two states, one referred to as possessing the property and the other as not possessing the property. Applying property appropriately means enhancing the chance of a positive response to the probes dependent on the property.
• Symptoms/characteristics Analogous to items in the cognitive diagnostic setting. Observable aspects of a patient in a medical or psychiatric setting. Can be evident like gender or the symptom of a sore throat, or can be the result of a medical test or question put to the patient. In current UM applications needs to be dichotomizable
• Probe Analogous to an item in the cognitive diagnostic setting. Something that brings about a two-valued response from an object being diagnosed.
• Positive or negative response to aprobe Analogous to getting an item correct or incorrect in the cognitive diagnostic setting. Positive and negative are merely labels given to the two possible responses to a probe, noting that sometimes a "positive" response is contextually meaningful and sometimes it isn't.
• Fig. 1 shows the standard logistic item response function P( ⁇ ) used as the basic building block of ERT models in general and in the UM in particular.
• Fig. 2 displays the flow chart for the 1995 prior art proposed UM cognitive diagnostic procedure.
• Fig. 3 displays a schematic of the 1995 UM probability model for the random response X y of one examinee to one item, indicating the examinee parameters and item parameters influencing the examinee response X ⁇ .
• Fig.4 displays the standard normal probability density function assumed for the disfribution of examinee residual ability ⁇ in the UM.
• Fig. 7 displays a totally uninformative Bayes prior density f(p) in a statistical drug trial study.
• Fig. 8 displays the components of the basic Bayes probability model statistical inference paradigm.
• Fig. 9 displays the likelihood function f(X
• Fig. 10 displays simultaneously the prior density, the likelihood function, and the posterior distribution for p: f(X
• p) f(30 cures out of 40
• Fig. 11 displays the function e 'x , which is to be integrated via simulation.
• Fig. 12 displays a flow chart of Robert Mislevy's Bayes probability inference network approach to cognitive diagnosis.
• Fig. 13 displays a flow chart of Kikumi Tatsuoka' s Bayesian Rule Space approach to cognitive diagnosis.
• Fig. 14 displays a flow chart of Susan Embretson's GLTM approach to cognitive diagnosis.
• Fig. 15 displays a schematic of the UM likelihood for the random response of one examinee to one item, indicating the examinee parameters and item parameters influencing the examinee response Xy for the reparameterized Unified Model used in the present invention.
• Fig. 16 displays the dependence representation of the identifiable Bayesian version of the reparameterized UM used in the invention including prior disfributions and hyperparameters.
• Fig. 17a displays the flow chart of the UM cognitive diagnosis procedure used in the present invention.
• Fig. 17b displays the flow chart of the UM medical/psychiatric diagnosis procedure used in the present invention.
• Fig. 17c displays the flow chart of the general UM procedure used in the present invention.
• Fig. 18 displays a page of the introductory statistics exam to illustrate items simulated in the UMCD demonstration example.
• Fig. 19 displays an item attribute incidence matrix for the introductory statistics exam simulated in the UMCD demonstration example.
• the present invention is based in part on discoveries of failings of the 1995 DiBello et al UM proposed approach. These were overparameterization that caused parameter nonidentifiability, the failure to set mastery levels that also was a further cause of nonidentifiability and raised substantive issues of interpretation for the user, the lack of a practical and effective calibration procedure, and a failure to model the natural positive correlational structure existing between atfributes to thereby improve cognitive diagnostic accuracy. These failings are discussed first. To do so, more must be understood about parameterization and identifiability.
• Nonidentifiability and Model Reparameterization in Statistical Modeling In statistical modeling, a model with fewer parameters that describes reality reasonably well is much preferred to a model with more parameters that describes reality at best a" bit better. This is especially important if the model with more parameters has nonidentifiable parameters, namely parameters that statistically cannot be separated from one another, that is parameters that cannot be estimated at all from the data.
• a trivial example illustrates the important ideas of nonidentifiabililty and the need for reparameterization.
• model is over-parameterized in that b and c play exactly the same role (a parameter multiplying the variable x) and hence cannot be statistically distinguished from each other.
• model parameters b and c are nonidentifiable and cannot be estimated from available data.
• the not-useful and non-identifiable 1995 UM was reparameterized by reducing the number of parameters through the introduction of a smaller yet substantively meaningful set of parameters and through specifying atfribute mastery levels, thereby producing all identifiable, and hence estimable, parameters.
• the General Approach to Reparameterization Assume a model with a meaningful set of K parameters; i.e., the parameters have useful real-world substantive interpretations (like velocity, mass, acceleration, etc., do in physics models).
• the general method is for k ⁇ K to define new and meaningful parameters a consult a 2 , ... ,a h each a being a different function of the original set of K parameters. It is desirable to choose the functions so that the new set of parameters are both identifiable and substantively meaningful.
• a valid reparameterization is not unique and there thus exist many useful and valid reparameterizations.
• specifying the mastery level defines how proficient an examinee must be in applying an attribute to items in order to be classified as having mastered the atfribute.
• This mastery specification is needed not only to achieve identifiability but also is required so that users are empowered to draw substantively meaningful conclusions from the UM cognitive diagnoses. Indeed, it is a meaningless claim to declare an examinee a master of an atfribute unless the user knows what atfribute mastery actually means in the context of the test items that make up the test. Thus, any cognitive diagnostic model that fails to somehow set mastery levels has a fundamental flaw that will cause serious malfunctioning.
• Equation 5 is analogous to Equation 3 for S-, , which used the original parameterization in terms of r and ⁇ . Both equations for S y give the probability that the included atfributes are applied correctly to the solution of Item i by Examinee j. Equation 5 provides a reparametization of the ⁇ 's and r's in order to achieve substantively meaningful parameters that are identifiable.
• Equation 3 version of S ⁇ is replaced with the Equation 5 version below, noting that both formulas produce the same value for S y .
• r n * for Atfribute 1 is by its definition above the probability of applying the attribute correctly to Item i if not mastered divided by the probability of applying the atfribute correctly if mastered.
• the r*'s for the other attributes are defined similarly.
• a value of r jk * ⁇ 0 for an Atfribute k simply means that there is a big advantage to having mastered the atfribute when trying to answer Item i correctly.
• An r* ik relatively close to 1 simply means there is little advantage to having mastered the Atfribute k over not having mastered Atfribute k when trying to solve item i.
• the required atfributes are referred to as highly positive for Item i. "Highly positive” as before simply means that with high probability an examinee uses the atfributes required for the item correctly if and only if ' the examinee possesses all of the attributes that the model says are needed for the item.
• Bayesian model is a probability model for which the model parameters are also assigned a probability disfribution.
• a Bayesian model with hyperparameters is a Bayesian model in which the prior distributions of the basic parameters of the model are in turn also given parameters each having a prior disfribution. These additional parameters that confrol the prior disfribution of the usual model parameters are referred to as hyperparameters.
• a good reference for Bayes modeling in general and hierarchical Bayes modeling in particular is Gelman et al.
• Fig. 16 schematically displays the hierarchical Bayes model for an examinee responding to an item as modeled by our hierarchical Bayes UM. As such it is an augmentation of the reparameterized likelihood schematic of Fig. 15.
• the model parameters ⁇ *, r*, and c/3 have a prior beta distribution, denoted ⁇ (a,b) for each item i, each such disfribution determined by two parameters (a,b).
• Beta disfributions tend to work well as prior disfributions for parameters that are constrained to lie in the interval (0,1), as indicated and explained in Chapter 2 of the Gelman et al book, and which is true of ⁇ *, r*, and c/3.
• the beta disfribution parameters (a,b) provide a rich family of densities from which just about any choice of shape for the prior may be selected, an atfractive property from the modeling perspective.
• Each (a,b) hyperparameter has been given a uniform disfribution on the interval (0.5,2). This means that each value of the parameter, ⁇ say, within the interval (0.5,2) is equally likely.
• This uniform prior over a wide interval is the kind of suitable relatively non-informative (vague) prior that is effective in hierarchical Bayes models in that it allows the model to fit the data well without the prior having an inappropriately strong influence on the statistical inference. It is noted that these distributional choices (beta, uniform) are fairly standard choices, although a certain amount of judgement is required to construct prior disfributions for the relevant variables.
• the Bayesian structure associated with the examinee latent ability parameters (that is, the incompleteness residual ability 0 and the atfribute mastery/nonmastery components of ⁇ ) is now explained.
• This explanation serves to highlight two important components of the current UM procedure, namely specifying atfribute mastery levels and assuming a positive correlational atfribute structure as part of the Bayes model.
• the examinee atfributes and ⁇ are derived from a multivariate normal disfribution with positive correlations.
• a multivariate normal disfribution is a standard and well-understood disfribution for statisticians. For example if a person's weight and height is measured, then the standard model is a bivariate normal disfribution with weight and height positively correlated. For more information, consult any standard statistics textbook.
• Specifying the prior disfribution of atfributes ⁇ and ⁇ is done in two stages.
• ( ⁇ , ⁇ ') is given a multivariate normal prior, where ⁇ ' is the continuous precursor of the dichotomous valued (0/1 valued) components of ⁇ that specify mastery or nonmastery for each atfribute for each examinee.
• the atfribute pair correlations ⁇ kk . (hyperparameters) for ⁇ ' are assigned a uniform prior disfribution on the interval (0,1) because all that is known about them is that they are positive.
• the positive correlational structure for the component atfribute pairs of ⁇ assumed in the Bayes portion of the UM improves cognitive diagnostic accuracy.
• this positive correlational structure allows the model to capture the all-important fact that examinees that have mastered one atfribute are more likely to have mastered another atfribute; that is, attributes are positively correlated or more simply, positively associated.
• this very important building-in of a positive correlational structure for the atfributes was done by casting the UM in a Bayes framework.
• the present invention is not limited to the Bayesian framework.
• Fig. 16 schematically shows an embodiment of the hierarchical Bayes UM in the UMCD
• the present invention is not limited to the embodiment of the UMCD with its Bayes model and cognitive diagnostic MCMC computational algorithm.
• Fig. 17a provides a flow chart of the method of the present invention.
• the Blocks 201, 203,205, and 207 are identical to the UM based blocks of Fig. 2. This reflects that both take the same approach except for the details of the UM model used. Thus the non-Bayesian approach of Fig. 2 and the Bayes approach of Fig. 17a diverge from Block 205 down.
• both require a likelihood model, as already discussed, reparameterization issues related to the nonidentifiability of the 1995 UM led to the discovery of the reparameterization given in Equation 5 to replace the old parameterization of Equation 3.
• Blocks 209 and 1701 respectively now also requires a "Build UM Bayes prior f( ⁇ )" block (Block 1703), thus producing the Bayes model Block 1705.
• Blocks 1701, 1703 and 1705 of Fig. 17 reflect Equations 5 and 6 as well as the Fig. 16 schematic.
• Blocks 1707,1709, and 1711 are understood as follows. The needed posterior distribution ⁇ ⁇
• cognitive diagnoses for every examinee/attribute combination (Block 1711) is obtained.
• M-H within Gibbs is one of numerous variations of the basic MCMC approach.
• the simulated random numbers of the Markov Chain are probabilistically dependent (like the daily high temperatures on two consecutive days).
• Patz et al and in any other good general reference on doing Bayesian analysis using MCMC, such as in Gelman et al or in Gilks et al
• the MCMC simulation avoids entirely the computing (or even simulating of it) of the integral in the denominator and instead produces a "chain" of random numbers whose steady state probability disfribution is the desired posterior disfribution. In simple and practical terms, this means that if the chain for can be run a long time, then the observed disfribution of its simulated random numbers tells approximately what the required posterior disfribution is, thus bypassing the direct or simulated computation of it.
• MCMC estimates the required posterior disfribution with surprising accuracy because we a large number of random numbers of the chain are generated.
• the procedure of the present invention typically runs a chain of length 15000 with the first 5000 generated simulations of the chain thrown out because they are not yet in the required steady state.
• the MCMC simulation approach is at present the only viable approach for statistically analyzing paramefrically complex Bayes models.
• a computer simulation study is constructed demonstrating the power of the use of the current UMCD to cognitively diagnose student attribute mastery based upon the introductory statistics exam, as referred to earlier in Example 2 (refer also to Fig. 19 for the specific item/attribute structure). This simulation is described by following the flow chart of Fig. 17a.
• FIG. 18 gives a sample set of questions (items) 9-18 of this 40 question exam (Block 203 of Fig. 17a). The eight attributes described earlier were chosen (Block 201). The attribute/item structure is given in the table of the item/attribute incidence matrix given in Fig. 19 (Block 205). The user developed this matrix, in this case the patent applicants.
• the eight statistics knowledge attributes from Example 2 should be recalled: (1) histogram, (2) median/quartile, (3) average/mean, (4) standard deviation, (5) regression prediction, (6) correlation, (7) regression line, and (8) regression fit.
• Item 17 above requires atfributes (1), (3), and (4). It is noted, as in the case in this simulation example, that in a typical application of the UMCD the user will construct the test questions and decide on the major atfributes to be diagnosed (perhaps selecting the atfributes first and then developing questions designed to diagnose these atfributes) and hence made part of ⁇ . Referring to this item/attribute table of Fig.
• “Slight to moderate non-positivity” means examinees lacking any of an item's required atfributes (from among the listed eight attributes) will likely get the item wrong.
• the "slight to moderate non-positivity” was achieved by having the r*'s fairly uniform between 0 and 0. 4 and having the ⁇ *'s fairly uniform between 0.7 and 1. Noting that incompleteness is also slight to moderate as just discussed, it can be seen that an examinee possessing all the item's required attributes will likely get the item right. Also, an examinee lacking at least one required atfribute will likely get the item wrong.
• the goal of this study is to observe how effective the UMCD is in recovering the known cognitive abilities of the examinees (the cognitive abilities are known, recall, because they were generated using a known simulation model fed to the computer).
• assessing the method's effectiveness in a realistic computer simulation is one of the fundamental ways statisticians proceed. Indeed, the fact that the simulation model, and hence its parameters generating the data, is known is very useful in using simulation studies to evaluate the effectiveness of a statistical procedure.
• Blocks 205, 1701, 1703, and 1705 of Fig. 17a constitute the assumed Bayes model, as given by Formulas 1, 2, 5, and 6.
• the simulated examinee response data (a matrix of 0s and Is of dimension 500 by 40 (Block 207) was analyzed using MCMC ( Block 1707) according to the identifiable Bayes UM schematically given in Fig. 16.
• MCMC Block 1707
• this chain of 10000 values estimates the desired posterior disfribution of attribute mastery for each examinee.
• the item parameters were also well estimated (calibrated).
• the average difference between the estimated and true ⁇ * and the estimated and true r* values is 0.03 (the range for both parameter types is from 0 to 1), and the average difference between the estimated and true c is 0.3 (the range is between 0 and 3).
• the values of c were not as well estimated as the ⁇ * values and r* values were estimated because the exam was designed to have a highly cognitive structure (that is, relatively positive and complete) and was designed to test a group of examinees modeled to understand the atfributes well (i.e. many of them are masters and hence can be expected to have relatively high ⁇ values).
• the model is paramefrically complex, it is possible to estimate the key parameters well and hence calibrate the model well. Because of this, there is no risk of being hurt by the variance/bias trade-off, as represented above in the example of data that truly follow a four parameter cubic polynomial model. In that case either the situation could be misrepresented by computing a reliable estimate of the one parameter in the biased linear model, or the situation could be misrepresented by computing unreliable estimates of the four parameters in the unbiased cubic polynomial model. By contrast, here in the UMCD simulation, the parameters of the complex and well-fitting UM are estimated well.
• UMCD is distinguished from and surpasses these other approaches in cognitive diagnostic performance.
• the other approaches use different models than the Bayes UM approach does.
• the UMCD is the only model that is simultaneously statistically tractable, contains identifiable model parameters that are capable of both providing a good model fit of the data and being easily interpreted by the user as having meaningful cognitive interpretations, specifies atfribute mastery levels, incorporates into its cognitive diagnosis the positive association of attributes in the data, and is flexible both in terms of allowing various cognitive science perspectives and in incorporating predicted examinee error to produce suitable cognitive inference caution.
• the other models can be unrealistic (because of their adherence to a particular cognitive modeling approach) in settings where the approach provides a poor description of the actual cognitive reality.
• Medical diagnostic models are useful for aiding the practitioner in coming up with diagnoses consisting of a list of possible disorders that a medical practitioner compiles based on the symptoms presented by a patient, but they are not a replacement for the practitioner. Thus, a good system will give a reasonably complete list of the probable disorders, although with enough patient information the number of disorders should be manageable.
• Fig. 17b is a flow chart of the UM medical/psychiatric diagnostic procedure used in the present invention. It should be compared with the Fig. 17a flow chart that gives the analogous UM procedure for cognitive diagnosis.
• the set of potential disorders replaces the set of atfributes (Block 201 '), and the set of symptoms and other patient characteristics consisting of such things as dichotomized laboratory test values, age, race, sex, etc., replaces the items (Block 203').
• ⁇ is then a latent health or latent quality of life variable that combines all latent health variables and quality of life variables that are not potential disorders explicitly listed in the model. Then the UM is applied in exactly the same way that it is applied in the educational diagnostic setting (Fig. 17a).
• Blocks 201' and 203' symptoms/characteristics and disorders are defined (Blocks 201' and 203'), and then an incidence matrix is constructed to indicate which disorders may be related to the presence a particular symptom/characteristic (Block 205').
• the item parameters of co (as used in Blocks 1701, 1703, 1705, 1707') are now symptom characteristic parameters, and they can actually be accurately estimated if the data set used (Block 207') to calibrate the model includes patients with known disorders. This would improve the accuracy of the symptom/characteristic parameter calibration (Block 1707').
• a particular patient can then be assigned a list of disorders that he/she has a high enough probability of having (Block 1711 '), based on the posterior probabilities calculated from the UM estimation program.
• the report to a practitioner of the potential diagnoses may include the posterior probabilities assigned to each disorder (Block 1709').
• the statistical analyses proceed similarly in both settings (Blocks 1701, 1703,1705, 1707', 1709', 1711').
• the diagnosis is then used support the practitioners' diagnostic efforts (Block 1713').
• Fig. 17c presents the flow chart of the present invention applied in a generic setting.
• Fig. 17c should be compared with the cognitive diagnostic flow chart of the present UMCD invention of Fig. 17a applied in educational settings. The following correspondences are required:
• Test Items Probes (Blocks 203", 205" ,207” 1707")
• FIG. 17a A Semi-qualitative Description of the General Structure of the Equations and Relationships Undergirding the Present Invention Equations l,2,5,and 6 and the definitions of ⁇ *, r*, c, ⁇ , and ⁇ are used to help explain the portions of the specific embodiment of the invention.
• the present invention is flowcharted in Figs. 17a, 17b, and 17c, each flow chart for a different application.
• the terminology of cognitive diagnosis (Fig 17a) will here be used for convenience, noting that the terminology of medical and psychiatric diagnosis (Fig 17b) or the terminology of generic diagnosis (Fig. 17c) would function identically.
• Equations 1,5, and 6 together with their identifiable and hence able to be calibrated parameters r*'s and ⁇ *'s provide one explication of the fact that (i) the probability of getting an item correct is increased by examinee mastery of all the attributes needed for the item as contrasted with lacking one or more needed attributes. Further, (ii) the more needed attributes that are not mastered the lower the probability of getting the item correct.
• the clauses (i) and (ii) above qualitatively describe the concept of positivity of an item, which is expressed in one specific manner in the embodiment of the present invention.
• any set of model equations may be used to capture the notion of positivity in a UM used in the present invention provided the parameters of the equations are identifiable, substantively meaningful to the practitioner, and express both (i) and (ii) stated above or express (i) alone.
• Modeling completeness for the UM is characterized by using one or a low number of latent variables to capture the affect on the probability of getting an item correct caused by all influential atfributes not explicitly listed in the model via the incidence matrix (Blocks 205, 205' and 205 ' ').
• Any expression other than P ( ⁇ ,- +c,) of the present invention that expresses the fact that the attributes other than those explicitly listed in the UM incidence matrix can influence the probability of getting an item correct and that captures this influence parsimoniously with one or a small number of latent variables is an acceptable way to model UM completeness.
• the current embodiment specifies atfribute mastery levels by setting the values of parameters A as shown in the schematic of Fig.
• any ways of explicating the need for identifiable parameters expressing positivity and completeness, specifying attribute mastery levels, building into the model that atfributes tend to be associated either positively in the educational settings or perhaps positively and/or negatively in other settings, and expressing the dependence on each item of a subset of the specified atfributes provides a way of expressing aspects of the UMCD being claimed.
• any model concerning objects, usually people, with two valued latent properties such as atfributes or disorders may utilize the specifying of the level of possession of each property such as specifying the level of mastery or specifying the level of disorder judged to constitute a person having the disorder and further may utilize modeling a positive or negative association between properties such as attributes or disorders thus allowing the calibration and subsequent use of the estimated sizes of the associations to improve accuracy when carrying out diagnoses.

Landscapes

• Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Business, Economics & Management (AREA)
• Theoretical Computer Science (AREA)
• Educational Administration (AREA)
• Educational Technology (AREA)
• Computational Mathematics (AREA)
• Mathematical Analysis (AREA)
• Mathematical Optimization (AREA)
• Mathematical Physics (AREA)
• Pure & Applied Mathematics (AREA)
• Algebra (AREA)
• Tourism & Hospitality (AREA)
• Health & Medical Sciences (AREA)
• General Business, Economics & Management (AREA)
• General Health & Medical Sciences (AREA)
• Human Resources & Organizations (AREA)
• Marketing (AREA)
• Primary Health Care (AREA)
• Strategic Management (AREA)
• Economics (AREA)
• Medical Treatment And Welfare Office Work (AREA)
• Electrically Operated Instructional Devices (AREA)
• Management, Administration, Business Operations System, And Electronic Commerce (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Complex Calculations (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Priority Applications (6)

Applications Claiming Priority (2)

Publications (1)

Family

ID=25276331

Family Applications (1)

Country Status (9)

Cited By (4)

Families Citing this family (76)

Citations (2)

Family Cites Families (24)

• 2001
  • 2001-04-20 US US09/838,129 patent/US6832069B2/en not_active Expired - Fee Related
• 2002
  • 2002-04-19 EP EP02728855A patent/EP1384220A4/en not_active Withdrawn
  • 2002-04-19 JP JP2002584283A patent/JP2004527049A/ja active Pending
  • 2002-04-19 BR BR0209029-5A patent/BR0209029A/pt not_active Application Discontinuation
  • 2002-04-19 CA CA002445618A patent/CA2445618C/en not_active Expired - Fee Related
  • 2002-04-19 KR KR10-2003-7013739A patent/KR20040025672A/ko not_active Application Discontinuation
  • 2002-04-19 CN CNA028122003A patent/CN1516859A/zh active Pending
  • 2002-04-19 WO PCT/US2002/012424 patent/WO2002086841A1/en active Application Filing
  • 2002-04-19 MX MXPA03009634A patent/MXPA03009634A/es not_active Application Discontinuation
• 2004
  • 2004-12-08 US US10/970,795 patent/US7457581B2/en not_active Expired - Fee Related
• 2008
  • 2008-09-05 US US12/205,754 patent/US7974570B2/en not_active Expired - Fee Related

Patent Citations (2)

Non-Patent Citations (8)

Also Published As

Similar Documents

Legal Events