JP2008065836A - Method of selecting medical and biochemical diagnostic tests employing neural network-related application - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To identify important variables for supporting an analysis of a failure or a state. <P>SOLUTION: The computer system for selecting variables involves: (a) providing a first set of "n" candidate variables and a second set of "selected important variables", which initially is empty; (b) taking the candidate variables one at a time and evaluating each by training a decision making support system on the basis of the variable combined with the current set of important variables; (c) selecting the best variables, where the best variable is the one that gives the highest performance of the decision making support system, and if it improves performance in comparison to the performance of the selected important variables, adding it to the "selected important variable" set, removing it from the candidate set and continuing processing at step (b), until the best variable no longer improves the performance. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The subject of the present invention relates to the use of prediction techniques, in particular non-linear prediction techniques, for the development of medical diagnostic aids. In particular, effective training techniques are provided for neural networks and other expert systems that have inputs from patient history information for the development of medical diagnostic tools and methods of diagnosis.

  This application is a continuation-in-part of US patent application No. 08/599275 “METHOD FOR DEVEL0PING MEDICAL AND BI0CHEMICAL DIAGN0STICTESTS USING NEURAL NETW0RKS” filed on Feb. 9, 1996, Jerome Lapointe and Duane DeSieno. 35 U. of Jerome Lapointe and Duane DeSieno, US Provisional Patent Application No. 60/011449, “METHOD AND APPARATUS FOR AIDING IN THE DIAGNOSIS OF ENDOMETRIOSIS USINGA PLURALITY OF PARAMETERS SUITED F0R ANALYSIS THROUGH ANEURAL NETWORK”. S. Claims priority according to C§119 (e). The subject matter of each of the above applications and provisional applications is hereby incorporated by reference in its entirety.

Microfish appendix.
Two computer appendices containing computer program source code for the programs described herein are filed concurrently with this application. This computer appendix is 37C. F. R. It can be converted to a microfiche appendix according to 1.96 (b). Computer appendices, hereinafter referred to as “microfiche appendices”, are each incorporated by reference in their entirety. Accordingly, part of the disclosure of this patent document includes material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by either this patent document or the patent disclosure when it appears in the Patent and Trademark Office patent file or record, but otherwise all copyrights Retain even things.

Data collection, decision support system and neural network.
Some computer decision support systems have the ability to classify information and identify patterns in input data, especially when evaluating complex interactions between data sets and variables with a large number of variables. Useful. Collectively referred to as “data collection” or “knowledge discovery in a database” (decision support system herein), these computer decision making systems include processors, internal and peripheral devices, memory devices and input / output interfaces. Use the same basic hardware components that you have, for example, a personal computer (PC). The distinction between systems occurs in software, and more fundamentally, in the paradigm on which software is based. Paradigms that provide decision support functions include regression methods, decision trees, discriminant analysis, pattern recognition, Bayesian decision theory, and fuzzy logic. One of the more widely used decision support computer systems is an artificial neural network.

  Artificial neural networks or “neural networks” are parallel information processing tools in which individual processing elements, called neurons, are arranged in layers and comprise a number of interconnections between elements in successive layers. The working of the processing element is modeled so that the output of the processing element approximates a biological neuron that is generally determined by a non-linear transfer function. In a typical model of a neural network, in an input layer for elements that receive input, an output layer that includes one or more elements that generate output, and one or more hidden layers of elements between them Processing elements are arranged. The hidden layer provides a means by which nonlinear problems can be solved. During the processing element, the input signal to the element is arithmetically weighted according to a weighting factor associated with each input. The resulting weighted sum is transformed by a selected non-linear transfer function, such as a sigmoidal function, resulting in an output whose value varies from 0 to 1 for each processing element. The learning process is called “training”, and the error that occurs between the output of the neural network and the desired output presented in the training data when a particular processing element is combined with the output of other processing elements. It is a trial and error process that requires a series of interactive adjustments to the weights of the processing elements to give an output that produces minimal results. The adjustment of the element weights is triggered by the error signal. The training data is described as several training examples, each example including a set of input values to be presented to the neural network and an associated set of desired output values.

  A common training method is reverse propagation or “backprop”, which propagates the error signal back through the network. The error signal is used to determine how much the weight of a given element should change and the error slope. Its purpose is to converge to a global minimum of mean square error. The path towards convergence, i.e. the downgrading, is taken in the form of steps. Each step is adjustment of the input weight of the processing element. The size of each step is determined by the learning rate. The slope of the down slope gives a false impression that convergence has been achieved, and includes flat and steep areas with valleys that act as local minima, resulting in inaccurate results.

  Some variations of the backprop incorporate a momentum term in which a portion of the previous weight change value is added to the current value. This adds momentum into the descending slope of the algorithm trajectory. This prevents the algorithm trajectory from being “captured” in the local minimum. One back-propagation method that includes a momentum term is a “quick prop” with an adaptive momentum rate. Quick prop variations are described in Fahlman ("Fast Learning Variations on Back-Propagation: An Empirical Study", Proceedings on the 1988 Connectionist Models Summer School, Pittsburgh, 1988, D. Touretzky et al., Pp. 38-51, Morgan Kaufmann, Co-authored with San Mateo, California and Lebriere, “The Cascade-Correlation Learning Architecture”, Advances in Neural Information Processing Systems 2 (Denver, 1989), edited by D. Touretzky, pp. 524-32, Morgan Kaufmann, San Mateo). Quickprop algorithms are publicly available from the Artificial Intelligence Repository maintained by the School of Computer Science at Carnegie Mellon University and can be downloaded over the Internet. In quick props, the dynamic momentum rate is calculated based on the slope of the gradient. If the slope is smaller than the slope after the previous weight adjustment but has the same sign, the weight change accelerates. The acceleration rate is determined by the magnitude of the continuous difference between the slope values. If the current slope is in the opposite direction to the previous slope, the weight change slows down. The quick prop method improves the convergence speed, gives the steepest possible slope, and helps prevent convergence to a local minimum.

  When a neural network is trained based on sufficient training data, the neural network can be generalized to an accurate solution for a new set of input data that was not part of the training data. Work. Neural networks have been shown to work even in the absence of complete data or in the presence of noise. It has also been observed that network performance for new or test data tends to be lower than performance for training data. The difference in performance relative to the test data indicates the extent to which the network can generalize from the training data. However, neural networks can be retrained and can therefore learn from new data, improving the overall performance of the network.

  Therefore, the neural network has characteristics suitable for a number of various problems, including areas that require prediction, such as medical diagnosis.

Neural networks and diagnostics.
When diagnosing and / or treating a patient, the physician uses the patient's condition, symptoms, and the results of applicable medical diagnostic tests to identify the patient's disease state or condition. The physician must carefully determine the association of symptoms and test results with a particular diagnosis and use judgment based on experience and intuition when making a particular diagnosis. Medical diagnosis requires the integration of information from several sources, including medical history, physical testing, and biochemical testing. Based on the results of the tests and the answers to the tests and questions, the physician uses the person's training, experience, knowledge, and expertise to formalize the diagnosis. The final diagnosis requires subsequent surgical procedures to be validated or formulated. Thus, the diagnostic process requires a combination of decision support, intuition and experience. The effectiveness of a physician's diagnosis depends on the person's experience and ability.

  Due to the predictive and intuitive nature of medical diagnosis, attempts have been made to develop neural networks and other expert systems that aid this process. Application of neural networks to medical diagnosis has been reported. For example, neural networks have been used to help diagnose cardiovascular disorders (eg, Baxt (1991) “Use of an Artificial Neural Network for the Diagnosis of Myocardial Infarction”, Analsof Internal Medicine 115: 843; Baxt (1992) "Improving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks", Neural Computation 4: 772; Baxt (1992) "Analysis of the clinical variables that drivedecision in an artificial neural network trained to identify the presence of myocardial infarction ", Annals of Emergency Medicine 21: 1439; Baxt (1994)" Complexity, chaos and human physiology: the justification for non-linear neural computational analysis ", Cancer Letters 77:85). Other medical diagnostic applications include cancer diagnosis (eg Maclin et al. (1991) “Using Neural Networks to Diagnose Cancer” Journal of Medical Systems 15: 11-9; Rogers et al. (1994) “Artificial Neural Networks for Early Detection. and Diagnosis of Cancer ”Cancer Letters 77: 79-83; Wilding et al. (1994)“ Application of Backpropogation Neural Networks to Diagnosis of Breast and Ovarian Cancer Cancer Letters 77: 145-53 ”, neuromuscular disorders (see Pattichis et al. (1995)) ) "Neural Network Models in EMG Diagnosis", IEEE Transactions on Biomedical Engineering 42: 5: 486-495) and Chronic Fatigue Syndrome (Solms et al. (1996) "A Neural Network Diagnostic Tool for the Chronic Fatigue Syndrome", International Conference on Neural Networks, Paper No. 108). However, these methods cannot handle the critical problems associated with developing practical diagnostic tests for a wide range of conditions and do not handle the selection of input variables.

  MYCIN (Davis et al., Production Systems as a Representation for a Knowledge-based Consultation Program, Artificial Intelligence, 1977, 8: 1: 15-45) and its descendants TEIRESIAS, EMYCIN, PUFF, CENTAUR, VM, GUIDON, SACON Computer decision support methods other than neural networks that can be applied to medical diagnosis, including knowledge-based expert systems including ONCOCIN and ROGET, have been reported. MYCIN is an interactive program that diagnoses several infectious diseases and defines antimicrobial therapy. Such knowledge-based systems include actual knowledge and rules or other methods for using that knowledge. All information and rules are pre-programmed into the memory of a system other than the system that develops its own procedure to reach the desired result based on the input data, as in the case of neural networks. Other computer diagnostic methods are Bayesian networks, also called belief or causal probabilistic networks that classify patterns based on probability density functions from training patterns and a priori information. A Bayesian decision-making system used to interpret mammograms to diagnose breast cancer has been reported (Roberts et al. “Mammo Net: A Bayesian Network diagnosing Breast Cancer”, Midwest artificial Intelligence and Cognitive Science Society Conference, Illinois) Carbonda1e, April 1995) and Hypertension (Blinowska et al. (1993) "Diagnostica-A Bayesian Decision-Aid System-Applied to Hypertension Diagnosis", IEEE Transactions on Biomedical Engineering 40: 230-35). Bayesian decision systems are somewhat limited with respect to the reliability of linear relationships and the number of input data points that can be processed, and are not well suited for decision support that requires nonlinear relationships between variables. Implementing Bayesian methods using neural network processing elements can overcome some of these limitations (eg Penny et al. (1996), “Neural Networks in Clinica1 Medicine”, Medical Decision-support, 1996 16: 4: 386-98). These methods are used to diagnose disorders in which important variables are entered into the system by mimicking a physician. However, it may be important to use these systems to improve existing diagnostic procedures.

Endometriosis.
Endometriosis is the growth of uterine tissue outside the uterus. This affects women of reproductive age of about 15-30 percent. The cause of endometriosis is unknown, but may be due to retromenstrual menopause and the endurance of endometrial tissue and cells (menstrual deposits) from the uterus into the peritoneal cavity. Regressive menstruation is thought to occur in most women or all women, but it is unclear why some women have endometriosis and others do not.

  Not all women with endometriosis show symptoms or bother the disease. The degree or severity of endometriosis does not correlate with symptoms. Women with severe illness are completely asymptomatic, while other women with minimal illness suffer intolerable pain. Symptoms associated with endometriosis, such as infertility, pelvic pain, dysmenorrhea, past occurrence of endometriosis, often occur in women who do not have endometriosis. In other cases, these symptoms appear and the woman has endometriosis. Although a relationship between these symptoms and endometriosis appears to exist, the interaction with these and other factors is complex. Clinicians often perform diagnostic laparoscopy on patients who are considered good candidates with endometriosis based on a combination of the above indications. However, endometriosis is not present in a significant portion of these women. Endometriosis therefore represents an example of a disease state in which a physician must rely on experience to shape a diagnosis using a complex set of information. The effectiveness of the diagnosis is related to the experience and ability of the physician.

  Therefore, it was impossible to determine whether a woman had endometriosis based on symptoms alone. Within the medical community, the diagnosis of endometriosis can only be confirmed by direct visualization of endometrial disorders during surgery. Many physicians often require further restrictions and use histology on endometrial biopsy tissue to validate suspected disorders as endometrial (glands and stroma). Thus, a noninvasive diagnostic test for endometriosis would be quite useful.

JP-A-5-277119. JP-A-5-176932. Japanese Patent Laid-Open No. 6-119291. JP-A-7-84981.

  Accordingly, it is an object of the present invention to provide a noninvasive diagnostic assistance device for endometriosis. It is also an object of the present invention to provide a method for selecting important variables to be used in a decision support system that aids in the diagnosis of endometriosis and other disorders and conditions. It is also an object of the present invention to identify new variables, identify new biochemical tests and markers for disease, and design new diagnostic tests that improve existing diagnostic methods.

  Methods are provided for diagnosing illnesses, disorders, and other medical conditions and using decision support systems that assist in it. The methods provided in the present invention include methods for developing diagnostic tests using patient history data and identification of important variables, methods for identifying important selected variables, methods for designing diagnostic tests, diagnostic tests Including methods for assessing utility, expanding the clinical utility of diagnostic tests, and selecting treatment strategies by predicting the outcome of various possible treatments. Diagnose disorders, including diseases that are difficult to diagnose, such as endometriosis, pregnancy-related events such as the possibility of childbirth during a specific period, and other such disorders related to women's health. Helping disease parameters or variables are provided. Although female disorders are taken as an example herein, it should be understood that the method of the invention can be applied to any disorder or condition.

  Also, use neural network training to guide test development to improve test sensitivity and specificity and select diagnostic tests that improve the overall diagnosis or potential of a disease or medical condition Means are provided. Finally, a method for evaluating the effectiveness of a given diagnostic test is described.

  Accordingly, the present invention provides a method for identifying a variable or set of variables that aids in the diagnosis of a fault or condition. Methods for identifying and selecting important variables and diagnostic generation systems collect patient data or information, typically patient history or clinical data, and identify variables based on this data. For example, the data includes information for each patient regarding the number of pregnancies experienced by each patient. Therefore, the extracted variable is the number of pregnancy. Variables are analyzed by a decision support system and illustrated by a neural network to identify important or related variables.

  Methods are provided for developing medical diagnostic tests using computer-based decision support systems, such as neural networks and other adaptive processing systems (collectively referred to as “data collection tools”). A neural network or other such system is trained based on observations collected from a group of test patients whose patient data and symptoms are known or inferred. The subset or subsets of related variables are identified using a decision support system or a plurality of decision support systems, such as neural networks or neural network consensus. Another set of decision support systems is trained based on the identified subset to generate a consensus decision support system based test, such as a neural network based test for that symptom. Using a consensus system, such as a consensus neural network, minimizes the negative impact of local minima in decision support systems, such as neural network based systems, thereby improving system accuracy.

  Also, patient data can be increased by increasing the number of patients used to improve or improve performance. Biochemical test data and other data can also be included as part of additional examples, or the data can be used as additional variables prior to the variable selection process.

  The resulting system is used as a diagnostic aid. Further, when using the system, patient data can be stored and then used to further train the system and develop a system that fits a particular genetic population. This entry of additional data into the system is performed automatically or manually. By doing so, the system learns continuously and adapts to the specific environment in which they are used. In addition to diagnosis, the resulting system has a number of uses including assessing the severity of a disease or disorder, and predicting the outcome of a selected treatment protocol. This system is also used to evaluate the value of other data during the diagnostic procedure, such as biochemical test data and other such data, and to identify new tests useful for diagnosing specific diseases used.

  Accordingly, methods are also provided for improving existing biochemical tests, identifying related biochemical tests, and developing new biochemical tests that help diagnose disorders and conditions. These methods assess the impact of a specific test or potential new test on the performance of a decision support system based test. Such tests are relevant to diagnosis if the addition of information from the test improves performance.

  Disorders and conditions that are of particular importance in the present invention and to which the method of the present invention can be readily applied include endometriosis, infertility, the prediction of pregnancy related events such as the possibility of childbirth during a specific period, preeclampsia Gynecological conditions and other conditions that affect fertility, including However, it is not limited to these. However, it should be understood that the method of the present invention can be applied to any disorder or condition.

  These methods are described with examples for neural networks, but also use other data collection tools such as expert systems, fuzzy logic, decision trees, and other statistical decision support systems that are generally non-linear. Please understand that you can. The variables provided in the present invention are intended to be used with a decision support system, but after identifying the variables, a person with knowledge of important variables, typically a physician, can use them to make decisions. In the absence of a support system, or a less complex linear analysis system can be used to aid diagnosis.

  As shown herein, variables or combinations thereof that have not been known to be important in assisting diagnosis are identified. Furthermore, without supplementing biochemical test data, patient history data can be used to diagnose a disorder or condition when used with a decision support system, such as a neural network provided in the present invention, or Can help diagnose the condition. Furthermore, the accuracy of a diagnosis using biochemical data or a diagnosis not using biochemical data is sufficient to eliminate the need for invasive surgical diagnostic procedures.

  The present invention also provides a method for identifying and extending the clinical utility of diagnostic tests. Specific test results that have not previously been considered clinically useful for the disorder or condition of interest are combined with variables and used with a decision support system such as a neural network. If the performance of the system, the ability to accurately diagnose a fault, is improved by adding test results, the test will have clinical utility or new utility.

  Similarly, the resulting system can be used to identify new utilities or therapies and to identify specific drug and therapy applications. For example, the system can be used to select a sub-population of patients for whom a particular drug or therapy is effective. Accordingly, a method is provided for expanding drug or therapy instructions and for identifying new drugs and therapies.

  In certain embodiments, neural networks are used to evaluate specific observations and test results, guide the development of biochemical diagnostic tests or other diagnostic tests, and provide decision support functions for testing.

  Also provided is a method for identifying key variables or sets used in a decision support system. Although this method will be described herein with reference to medical diagnosis, it can be widely applied in any field, such as financial analysis, in which important parameters or variables are selected from a plurality.

  In particular, a method is provided for selecting valid combinations of variables. This method consists of (1) giving a set of “n” candidate variables and a set of “selected important variables” that are initially empty, (2) all based on chi-square and sensitivity analysis (3) taking the highest “m” ranked variables (where m is 1 to n) at a time and combining them into the current set of important variables Evaluating each variable by training the consensus of the neural network based on (4) selecting the best variable (the best variable is the variable that gives the highest performance) out of the m variables, and If it improves performance compared to the performance of the selected critical variable, it is added to the “selected critical variable” set, removed from the candidate set, and processed in step (3). Continue, otherwise Step to step (5), (5) If all variables in the candidate set have been evaluated, terminate the process, otherwise take the next highest “m” ranked variables at once Evaluating each variable by training a consensus of the neural network based on the variables coupled to the current set of important selected variables and performing step (4). The final set of important selected variables includes multiple, typically more than three to five variables.

  In certain embodiments, the sensitivity analysis comprises (k) determining an average observation for each variable in the observation data set; (l) selecting a training example and executing the example in a decision support system; Generating a stored output value designated as normal output, (m) selecting the first variable in the selected training example, and replacing the observed value with the average observed value of the first variable Executing the modified example in the decision support system in forward mode and recording the output as a modified output, (n) squaring the difference between the normal output and the modified output, A step of accumulating as a sum for each variable (this sum is designated as the variable sum selected for each variable), (o) a step of repeating step (m) and step (n) for each variable in the example , (P) day Repeating step (n) for each example in the set from step (l) (the total of the selected variables represent the relative contribution of each variable for the determination of the decision support system outputs) a. This sum is used to rank each variable according to its relative contribution to the decision support system output decision.

  As shown herein, computer-based decision support systems, such as neural networks, reveal that several input factors that were not initially considered important can affect the results. This ability of a neural network to account for relevant input factors allows it to be used to guide the design of diagnostic tests. Accordingly, a method for designing a diagnostic test and a method for evaluating the utility of the diagnostic test are also provided. In each case, data from the test or possible test is added to the input of the decision support system. A diagnostic test has clinical utility if results improve when data is included in the input.

  Tests that have not previously been known to be important in the diagnosis of a particular disorder can be identified or new tests can be developed. Neural networks can add to the diagnostic test by reducing the effects of spurious data points and identifying any other data points that can be substituted.

  The network is trained against a set of variables, and then clinical data and / or additional patient information from diagnostic or biochemical test data is added to the input data. Select variables that improve results compared to none. Thus, it can be seen that certain tests that were previously not known to be important in diagnosing certain disorders are relevant. For example, the presence or absence of a specific spot on a Western blot of serum antibodies can be correlated to the disease state. New diagnostic tests can be developed based on the identity of a particular spot (ie antigen).

  An example of how to apply predictive technology to help diagnose a disease, more specifically, how to use neural network techniques with input from various sources to help diagnose a disease endometriosis is provided. The A trained set of neural networks that operate according to the network consensus in the computer system is used to assess certain clinical associations, some of which are generally not associated with disease states, eg, obtained by research. This is demonstrated in the case of the exemplary disease state endometriosis and provides factors that are used to help diagnose endometriosis. Neural network training is based on the correlation between answers and questions, referred to herein as clinical data, provided by a significant number of clinical patient physicians whose disease states have not been surgically verified.

  More than 12 to about 16 multiple factors in a particular trained neural network extracted from a set of 40 or more clinical data factors, specifically a set of 14 factors, is associated with endometriosis. Identified as a primary sign. The following set of parameters: age, parity (number of births), pregnancy (number of pregnancy), number of miscarriages, smoking (box / day), past endometriosis history, dysmenorrhea, pelvic pain, abnormal pap / formation Abnormalities, pelvic surgery history, medication history, pregnancy hypertension, genital warts and diabetes were identified as important. Other similar parameter sets were also identified. A subset of these variables can also be used in diagnosing endometriosis.

A decision support system for diagnosing endometriosis, wherein any subset of selected parameter sets, particularly 14 variable sets, includes one (or more) of the following three variable combinations: Can be used with.
a) Number of births, endometriosis history, pelvic surgery history b) Diabetes, pregnancy hypertension, smoking c) Pregnancy hypertension, abnormal pap stain / dysplasia, endometriosis history d) Age, smoking, intrauterine E) Smoking, endometriosis, dysmenorrhea f) Age, diabetes, endometriosis g) Pregnancy hypertension, number of births, endometriosis h) Smoking, births, intrauterine I) Pregnancy hypertension, endometriosis history, pelvic surgery history j) Number of pregnancy, endometriosis history, pelvic surgery history k) Number of childbirth, abnormal PAP stain / dysplasia, endometriosis history l) Number of births, abnormal PAP stain / dysplasia, dysmenorrhea m) History of endometriosis, pelvic surgery, dysmenorrhea n) Number of pregnancy, history of endometriosis, dysmenorrhea.

  Diagnostic software and exemplary neural networks that use variables to diagnose endometriosis are also provided. This software generates a clinically useful endometriosis index.

  In other embodiments, the performance of a diagnostic neural network system used to test endometriosis is a factor used in network training (referred to herein as biochemical test data. This is a test from an analysis. Improve by including variables based on biochemical test results from relevant biochemical tests (including data such as vital signs, such as pulse and blood pressure). The resulting exemplary network is an augmented neural network that uses 15 input factors, including biochemical test results and 14 clinical parameters. The weight set of the 8 augmented neural networks is different from the weight set of the 8 clinical data neural networks. An exemplary biochemical test uses an immunodiagnostic test format, such as an ELISA diagnostic test format.

  The methods applied to endometriosis exemplified herein include gynecological and female-related disorders, such as infertility, prediction of pregnancy-related events such as the possibility of childbirth during a specific period, preeclampsia, etc. It can be similarly applied and used to identify other failure factors, including but not limited to. Thus, neural networks can be trained to predict disease states based on the identification of factors that are important in predicting disease states and combining them with biochemical data.

  The resulting diagnostic system is suitable for use in diagnosing not only the presence of a condition or disorder, but also the severity of the disorder and as an aid in selecting a treatment strategy.

Definition.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. All patents and documents referred to herein are part of the present invention by reference.

  As used herein, a decision support system, also referred to as a “data collection system” or “knowledge discovery in a data system”, is trained on data to classify input data and then later on the basis of training data. Any system that can be used with new input data to make decisions, generally a computer-based system. These systems include expert systems, fuzzy logic, nonlinear regression analysis, multivariate analysis, decision tree classifiers, Bayesian belief networks, and the neural networks exemplified herein. However, it is not limited to these.

  As used herein, an adaptive machine learning process is any system that uses data to generate a predictive solution. Such processes are those implemented by expert systems, neural networks, and fuzzy logic.

  As used herein, an expert system is a computer-based problem solving and decision support system based on knowledge of the task or logical rules or procedures for using that knowledge. Knowledge and logic from the expertise of specialists is input into the computer.

  As used herein, a neural network, or neural network, is a parallel computing model composed of adaptive processing elements that are closely interconnected. In a neural network, processing elements are organized in an input layer, an output layer, and at least one hidden layer. Suitable neural networks are known to those skilled in the art (eg, US Pat. Nos. 5,251,626, 5,473,537 and 5,331,550, Baxt (1991) “Use of an Artificial Neural Network for the Diagnosis of Myocardial Infarction”, Annals of Internal Medicine 115: 843; Baxt (1992) "Improving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks", Neural Computation 4: 772; Baxt (1992) "Analysis of the clinical variables that drive decision in an artificial neural network trained to identify the presence of myocardialinfarction ”, Annals of Emergency Medicine 21: 1439; Baxt (1994)“ Complexity, chaos and human physiology: the justification for non-linear neural computation analysis ”, Cancer Letters 77:85 reference).

  As used herein, a processing element, also called a perceptron or artificial neuron, is a computational unit that maps input data from multiple inputs into a single binary output according to a transfer function. Each processing element has an input weight corresponding to each input that is multiplied by the signal received at that input to generate a weighted input value. The processing element sums the weighted input values for each input to generate a weighted sum that is then compared to a threshold defined by the transfer function.

  As used herein, a transfer function, also called a threshold function or activation function, is a mathematical function that produces a curve that defines two distinct categories. The transfer function is linear, but when used in a neural network, it is more generally non-linear, including quadratic, polynomial, or sigmoidal functions.

  Back propagation as used herein is a training method for a neural network to correct an error between a target output and an actual output. The error signal is fed back into the processing layer of the neural network, and the actual output approaches the target output due to the change in the weight of the processing element.

  The quick prop used here is the back propagation method proposed, developed and reported by Fahlman ("Fast Learning Variations on Back-Propagation: An Empirical Study", Proceedings on the 1988 Connectionist Models Summer School, Pittsburgh, 1988, D. Touretzky et al., Pp. 38-51, Morgan Kaufmann, San Mateo, CA; co-authored with Lebriere, “The Cascade-Correlation Learning Architecture”, Advances in Neural Information Processing Systems 2, (Denver, 1989 ), D. Touretzky, pp. 524-32, Morgan Kaufmann, San Mateo, California).

  Diagnosis, as used herein, is a predictive process that assesses the presence, absence, severity, or method of treatment of a disease, disorder or other medical condition. As used herein, diagnosis also includes a predictive process that determines the results obtained from the treatment.

  Biochemical test data as used herein is the result of any analytical method including, but not limited to, data from immunoassays, biological assays, chromatography, monitors and imagers, and measurements. And also includes data on vital signs and physical functions such as pulse, body temperature, blood pressure, eg, EKG, ECG, EEG, biorhythm monitor results, and other such information. The analysis can evaluate, for example, analytes, serum markers, antibodies, and other such materials obtained from the patient in the sample.

  The patient medical history data used in this specification is data obtained from a patient by a questionnaire or the like, but generally does not include biochemical test data used in this specification. However, as long as such data is historical data, the desired solution produces a number or result that can produce a diagnosis of the disorder.

  The training examples used herein include observation data for a single diagnosis, generally observation data for a single patient.

  As used herein, parameters identified from patient history data are referred to herein as observation factors or values or variables. For example, patient data includes information regarding individual patient smoking habits. A related variable is smoking.

  As used herein, the dividing means means selecting a portion of the data, such as 80%, using it to train the neural network, and using the remaining portion as test data. Thus, the network is trained based on other than part of the data. This process can then be repeated to train the second network. This process is repeated until all segments are used as test data and training data.

  The method of training by dividing the available data used herein into multiple subsets is commonly referred to as the training “holdout method”. The holdout method is particularly useful when the data available for network training is limited.

  Training as used herein is the process of generating a decision support system using input data. In particular, for neural networks, training is the error that occurs between the output of the neural network and the desired output presented in the training data when a particular processing element is combined with the output of other processing elements. Is a trial and error process that performs a series of interactive adjustments to the weights of the processing elements that provide an output that produces a result that minimizes

  As used herein, the variable selection process is a systematic method of selecting a combination of variables that yields a prediction result from any available set. Selection is performed by maximizing the prediction performance of the subset so that the addition of additional variables does not improve the results. In the preferred method provided herein, variables can be selected without considering all possible combinations.

  Candidate variables as used herein are items selected from observations collected from a group of test patients for diagnostic embodiments or other records such as financial records that can be used with a decision support system. Candidate variables are obtained by collecting data such as patient data and classifying observations as a set of variables.

  As used herein, important selected variables are those that enhance the network performance of the task at hand. Including all variables that can be used does not result in an optimal neural network. When some variables are included during network training, network performance is degraded. A network trained using only relevant parameters results in improved network performance. These variables are also referred to herein as a subset of the relevant variables.

  As used herein, ranking is the process of listing variables in order of selection. The ranking may be arbitrary or preferably organized. The organization is performed, for example, by statistical analysis in which variables are ranked in order of importance for tasks such as diagnosis, or by analysis based on a decision support system. Ranking can also be performed by, for example, an expert, a rules-based system, or any combination of these methods.

  As used herein, neural network consensus is a linear combination of outputs from multiple neural networks where the weight of each output is arbitrarily determined or set to an equal value.

  As used herein, a greedy algorithm is a method of optimizing a data set by determining whether to include or exclude points from a given data set. This set starts with no elements, and if there is a partial solution, the other values that best improve the objective are chosen. select.

  As used herein, a genetic algorithm is a method that starts from an initial distribution of randomly generated neural networks that are executed during a training cycle and ranked according to their performance when reaching a desired target. Networks that do not perform well are removed from the distribution, and more appropriate networks are retained and selected for crossover processes to descendants that retain the desired characteristics of the parent network.

  As used herein, the performance of a system is said to improve or become higher when a result more accurately measures or determines a particular result. It should also be understood that the performance of the system is generally better when using more training examples. Thus, the systems of the present invention improve over time as they are used, and more patient data is accumulated and then added to the system as training data.

  As used herein, sensitivity = TP / (TP + FN), specificity is TN / (TN + FP). However, TP = true positive, TN = true negative, FP = false positive, and FN = false negative. Clinical sensitivity measures how well the test detects patients with disease. Clinical specificity measures how well the test accurately identifies patients with no disease.

  The positive predictive value (PPV) used in this specification is TP / (TP + FP). The negative predicted value (NPV) is TN / (TN + FN). A positive predictive value is the likelihood that a patient with a positive test will actually have the disease. A negative predictive value is the likelihood that a patient with a negative test result will not have the disease.

  As used herein, fuzzy logic is a technique for handling systems that cannot be accurately described. Membership functions (membership in a data set) are not binary in a fuzzy logic system. Instead, the membership function takes a fractional value. Thus, an element can be simultaneously included in two conflicting sets, even though the set membership coefficients are different. This type of approach is therefore useful for answering yes or no-answer questions. This type of logic is therefore suitable for classifying responses from patient history questionnaires where answers are often only one degree.

1. General considerations and general methods.
It has been determined that several techniques can be used to train neural networks that analyze observations such as patient history and / or biochemical information. Depending on the data available and the characteristics of the problem to be analyzed, various neural network training techniques can be used. For example, when a large amount of training input can be used, a method of eliminating redundant training information is adopted.

  As shown here, neural networks also reveal that several input factors that were not initially considered important affect the outcome, and perhaps important factors are not the outcome determinants. Make it clear. The ability of a neural network to account for relevant input factors and unrelated input factors allows the neural network to be used in guiding diagnostic test design. As shown herein, neural networks, and other such data collection tools, are valuable advances in diagnosis and provide an opportunity to increase the sensitivity and specificity of diagnostic tests. Care must be taken to avoid the possibility of an answer with insufficient accuracy due to the phenomenon of local minima, as shown herein. The method of the present invention provides a means to avoid this problem or at least minimize it.

  Several problems are solved when developing diagnostic procedures, particularly diagnostic tests based solely or partly on patient information. For example, there is generally a limited amount of data because there are a limited number of patients for which training data is available. To solve this, patient information is split when training the network, as described below. Also, in general, there are a number of input observation factors that can be used for use in connection with available data, and thus methods have been developed to rank and select observations.

  Also, there are generally a large number of binary (true / false) input factors in available patient data, but these factors are generally sparse in nature (some of the binary input factors in available patient data). A value that is positive or negative only in some cases). There is also a high degree of overlap between the positive and negative factors being diagnosed.

  These other characteristics affect the choice of procedures and methods used to develop diagnostic tests. These problems are addressed and solved in the present invention.

2. Development of patient history diagnostic test.
Diagnostic test.
A method of diagnosis based solely on patient history data is provided. As demonstrated herein, a decision support system that assists in diagnosis can be provided, depending only on patient history information. Thus, the resulting system improves the predictive ability of biochemical test data, identifies new disease markers, develops biochemical tests, tests that have not previously been thought of predicting a specific disorder Can be used to identify

  These methods can also be used to select the appropriate treatment method by predicting the outcome of the selected treatment method and to predict the post-therapy condition. Input variables for training are obtained from electronic patient records showing diagnosis and other usable data, including, for example, selected treatments and outcomes. The resulting decision support system is then used with all data that can be used, for example, to classify women into different classes that respond to different treatments and predict the outcome of a particular treatment. This can maximize the probability of successful treatment or protocol selection.

  Similarly, these systems can be used to identify new benefits of drugs or therapies, and can be used to identify specific drug and therapy applications. For example, these systems can be used to select patient subpopulations for which a particular drug or therapy is effective. Accordingly, methods are provided for expanding drug or therapy support and for identifying new drugs and therapies.

Collect patient data, generate variables, and overview.
To illustrate the method of the present invention, FIG. 1 shows a flow chart for developing a patient history-based diagnostic test method. This process begins with the collection of patient history data (step A). Patient history data or observations are obtained from patient questionnaires, clinical results, possibly diagnostic test results, patient medical records, and provided to a system that operates on a computer in computer readable form. In a digital computer, patient history data is classified into a set of variables in two forms: binary values (such as true / false) and quantitative (continuous) values. A binary variable may contain an answer to the question "Do you smoke". The quantitative variable may be the answer to the question “how many cigarettes do you smoke per day”. Other values such as membership functions are also useful as input means.

  Patient history data also includes targets or desired outcome variables that are considered to indicate the presence, absence, or severity of the medical condition to be diagnosed. This desired result information is useful for neural network training. The selection of data to be included in the training data is made using knowledge or assumptions of the presence, weight or absence of the medical condition to be diagnosed. As indicated herein, diagnosis also includes assessment of progression and / or effectiveness of therapeutic treatment.

  The number of variables that can be defined and therefore generated is cumbersome. Binary variables are generally sparse because the number of positive (or negative) responses is often part of the overall number of responses. Thus, if there are a large number of variables and a small number of patients that can be used in a typical training data environment, a step is taken to separate a subset of variables that are important for diagnosis from the variables that can be used (step B). The particular selection of a subset of variables from the available variables affects the diagnostic performance of the neural network.

  The methods outlined herein have been found to produce a subset of variables that are generally equivalent or superior in sensitivity and reliability compared to a subset of variables selected by a trained professional such as a physician. ing. In some examples, variables are given priority or placed in order of rank or relevance.

  Thereafter, the last neural network to be used in the diagnostic procedure is trained (step C). In the preferred embodiment, network consensus (ie, multiple networks) is trained. The resulting network forms a decision support function for the completed patient history diagnostic test (step D).

How to separate important variables.
A method for separating important variables is provided in the present invention. According to this method, a valid set of variables can be selected by comparing all possible combinations of variables. The important variables are used as input for the decision support system.

Separation of important or related variables-ranking of variables.
FIG. 3 shows a flow diagram of a method for separating important or related variables during a diagnostic test (step E). Such a method is generally implemented using a digital computer system provided with potentially relevant information. In this procedure, two independent methods are used to rank the variables in order of importance, and then select a subset of variables that can be used from the top of the rank. As noted above, those skilled in the art can use other ranking methods instead of chi-square or sensitivity analysis. Further, when x is set up to N (total number of candidate variables), ranking is arbitrary.

  The system trains multiple neural networks based on available data (step I) and then generates a sensitivity analysis on all trained networks, as described below, with each input variable diagnosed To what extent it has been used in the network to implement (step J). A consensus sensitivity analysis for each input variable is determined by averaging the individual sensitivity analysis results for each trained network. Based on the sensitivity, the rank for each variable obtained from the patient medical history information is determined (step K).

Ranking of variables.
In a preferred embodiment, the variables are ranked using a decision support system based analysis such as statistical analysis such as chi-square analysis and / or sensitivity analysis. In the exemplary embodiment, sensitivity analysis and chi-square analysis are used to rank the variables. Other statistical methods and / or decision support system based methods may also be used, including but not limited to regression analysis, discriminant analysis, and other methods known to those skilled in the art. The ranked variables can preferably be used to train the network and can be used in the method of variable selection given in the present invention.

  This method uses a sensitivity analysis that modifies each input and measures the corresponding change in output (also Modai et al. (1993) “Clinical Decisions for Psychiatric Inpatients and Their Evaluation by Trained Neural Networks”, Methods of Information in Medicine 32: 396-99; Wilding et al. (1994) “Application of Backpropogation Neural Networks to Diagnosis of Breast and Ovarian Cancer”, Cancer Letters 77: 145-53; Ruck et al. (1990) “Feature Selection in Feed-Forward Neural Networks ”Neural Network Coputing 20: 40-48; Utans et al. (1993)“ Selecting Neural Network Architectures Via the Prediction Risk: Application to Corporate Bond Rating Prediction ”, Proceedings of the First International Conference on Artificial Intelligence Applications on Wall Street. Washington, DC, IEEE Computer Society Press, pp. 35-41; Penny et al. (1996) “Neural Networks in Clinical Medicine”, Medical Decision-support 4: 386-398). Such a method has not been used so far to select important variables as described herein. For example, it has been reported to develop statistical methods to determine relationships between variables rather than to select important variables using sensitivity analysis (Baxt et al. (1995) “Bootstrapping Confidence Intervals for Clinical Input Variable Effects in a Network Trained to Identify the Presence of Myocardial Infarction ”, Neural Computation 7: 624-38). Such sensitivity analysis can be used as part of the selection of variables that are important as diagnostic aids as described herein.

An overview of sensitivity analysis is shown in Step K of FIG. Each network or a plurality of trained neural networks (networks N ₁ to N _n ) is connected to each training example S _x (input data group whose output is known or inferred. There must be at least two training examples. Run in forward mode. However, “x” is the number of training examples. The output of each network N ₁ -N _n of each training example S _x is recorded, ie stored in memory. A new training example is defined that contains the average value of each input variable in all training examples. One at a time, each input variable of each original training example S _x value is exchanged for its corresponding average value V _{1 (avg)} to V _{y (avg)} . However, “y” is the number of variables.

The modified training example S _x ′ is run again in multiple networks, producing a modified output for each network for each variable. The difference between the output from the original training example S _x and the modified output of each input variable is squared and summed (cumulated) to obtain the individual sums corresponding to each input variable. To illustrate, for example, in the case of 10 separate neural networks N _{1 to} N ₁₀ each having ₁₅ variables V _{1 to} V ₁₅ and 5 different training examples S _{1 to} S ₅ , 5 trainings Each example training example is performed over 10 networks, producing a total of 50 outputs. The variable V ₁ is taken from each training example, and the average value V _{1 (avg)} is calculated. This averaged value V _{1 (avg)} is substituted into each of the 5 training examples to generate modified training examples S ₁ ′ to S ₅ ′, which are again 10 Executed in the network. 50 modified output values are generated by the networks N ₁ -N ₁₀ and 5 training examples. The correction is the result of using the mean value variable V _{1 (avg)} . Calculating the difference between the respective 50 pieces of the original output value and the modified output value, i.e. the original output OUT from the training S ₄ in the network _{N 6 (S 4 N 6)} , in the network N ₆ Subtract from the modified output OUT (S ₄ 'N ₆ ) from training example S ₄ . The difference value is squared [OUT (S ₄ 'N ₆ ) −OUT (S ₄ N ₆ )] ² _V1. This value is summed with the difference value squared for all combinations of network and training examples for the iteration in which the variable V ₁ is substituted with its mean value V _{1 (avg)} . That is, the following equation is obtained.

  The process is then repeated for variable # 2 to determine the difference between the original output and the modified output for each combination of network and training example, squared, and then sum the differences. This process is repeated for each variable until all 15 variables have been completed.

  Each resulting sum is then normalized so that the normalized value is 1.0 if all variables contributed equally to a single resulting output. After the previous example, the total squared difference is summed for each variable to obtain the total total squared difference for all variables. Divide the value of each variable by the total squared difference and normalize the contribution from each variable. From this information, the normalized values of each variable can be ranked in order of importance. A higher relative number indicates that the corresponding variable has a greater effect on the output. Use input variable sensitivity analysis to show which variables have played the most role in generating the network output.

  In the present invention, it has been found that performing sensitivity analysis using a consensus network improves the variable selection process. For example, if two variables are highly correlated, a single neural network trained on the data may use only one of the two variables to create a diagnosis. If the variables are highly correlated, very little is gained by including both variables, and the choice of which variable to include depends on the initial starting conditions of the network being trained. Sensitivity analysis using a single network may indicate that only one or only the other is important. Sensitivity analysis obtained from consensus of multiple networks trained using different initial conditions may reveal that both highly correlated variables are important. By averaging the sensitivity analysis over a set of neural networks, a consensus is formed that minimizes the effects of initial conditions.

Chi-square contingency table.
When processing sparse binary data, a positive response to a given variable may be highly correlated with the condition being diagnosed, but rarely occurs in training data, so a neural network The importance of variables shown by sensitivity analysis can be very low. To capture these occurrences, a chi-square contingency table is used as a secondary ranking process. A 2 × 2 contingency table chi-square test is performed on the binary variable, which is the frequency at which each cell of the table is observed for a combination of two variables (FIG. 3, step F). A 2 × 2 contingency table chi-square test is performed on continuous variables using the optimal threshold (which may be determined empirically) (step G). Rank binary and continuous variables based on chi-square analysis (step H).

  Using a standard chi-square 2x2 contingency table (step F) that operates on binary variables, determined by comparing specific binary input variables (with training data against known single output results Determine the importance of the relationship between the desired output. Variables with small chi-square values are generally not related to the desired output.

  In the case of variables having continuous values, a 2 × 2 contingency table can be constructed by comparing the continuous variables with threshold values (step G). The threshold is empirically modified to yield the largest possible chi-square value.

  The continuous variable chi-square value and the binary variable chi-square value can then be combined for common ranking (step H). A second level ranking can then be performed that combines the chi-square ranking variables with the sensitivity analysis ranking variables (step L). This combination of ranking allows sparse variables (ie, values that are positive or negative in some cases) to be included in the set of important variables that are highly related to the output. Otherwise, important information in such non-linear systems can be easily overlooked.

Select important variables from among the ranked variables.
As described above, important variables are selected from among the identified variables. The selection is preferably performed after ranking the variables when the second level ranking process is invoked. A method of identifying important variables (parameters) or sets used in a decision support system is also provided. This method is described here using medical diagnostics as an example, but it is widely applied in any field, such as financial analysis or other attempts to make statistical-based predictions that select important parameters or variables from a plurality. it can.

  In particular, a method is provided for selecting valid combinations of variables. Providing a set of “n” candidate variables and a set of “selected important variables” that are initially empty, and all candidates based on chi-square and sensitivity analysis, as described above After step (2) of ranking variables, the method takes up to “m” ranked variables at a time (where m is 1 to n) and is combined with the current set of important variables. Step (3) of evaluating each variable by training the consensus of the neural network based on the determined variable, and selecting the best variable among the m variables (the best variable is the variable that improves the performance most) ), If it improves performance, add it to the “selected critical variables” set, remove it from the candidate set, continue processing in step (3), otherwise step Go to (5) Step (4), if all variables on the candidate set have been evaluated, the process ends, otherwise the next largest “m” ranked variables at a time. And (5) training a neural network consensus based on the variables combined with the current set of important selected variables and evaluating each variable by performing step (4).

  In particular, the second level ranking process (step L) begins by adding the highest ranked variable from the sensitivity analysis (step K) to the key variable set (step H). Alternatively, the second level ranking process starts with an empty set and then tests the top few (x) variables from each of the two sets of ranking. In this second level ranking process, a set of neural networks is trained using a network training procedure (Step I) against the currently selected segment or subset of variables from the available data. . The ranking process uses the current set of “significant” variables (generally the first empty) as well as the current variables being ranked or being tested for ranking, and using the greedy algorithm A network training procedure that optimizes the input set by myopically optimizing the input set based on the important variables identified to identify the remaining variables that most improve the output.

This training process is shown in FIG. The number of inputs used by the neural network is controlled by eliminating inputs that are found to not contribute significantly to the desired output, ie, the known target output of the training data. Commercial computer programs such as ThinksPro ^TM Neural Network (or TrainDos ^TM DOS version) for Windows ^TM by La Jolla Logical Designs Consulting, California, USA, and other such programs that can be developed by those skilled in the art will change the input. Can be used to train the network.

Brain Scientific ^tm , sold by Nevada Adaptive Solutions, California Scientific Software, Beaverton, Oregon, USA, Neural Network Utility / 2 ^tm , sold by NeuralWare, Pittsburgh, PA, and Ward, Frederick, Maryland, USA Several other commercially available neural network computer programs can be used to perform any of the above operations, including NeuroShell ^tm and NeuroWindows ^tm sold by Systems Group. Other types of data collection tools that provide variable selection and network optimization functions, i.e., decision support systems, can be designed, and other commercially available systems can be used. For example, Neuro Genetic Optimizer ^TM sold by BioComp Systems in Redmond, Washington, USA, and Neuro Forecaster / GENETICA sold by New Wave Intelligent Business Systems (NIB5), Singapore, are models based on natural selection. Using optimized genetic algorithms to eliminate poorly performing nodes in the network distribution, and to send the best performing speed to the descendant nodes to “grow” the optimized network and greatly increase the results Eliminate input variables that do not contribute. Networks based on genetic algorithms use mutations to avoid trapping in local minima and use crossover processes to introduce new structures into the distribution.

Knowledge discovery in data (KDD) is another data collection tool, decision support system designed to identify important relationships that exist in variables, and is useful when there are many possible relationships It is. Including Darwin ^tm sold by Thinking Machines Corporation in the United States Massachusetts base Ddofodo, California, USA Mineset ^tm that Mountain sold by Silicon Graphics, Inc. of view, the Ultragem Data Mining Corporation of Eikoplex ^tm of San Francisco, California, USA, Several KDD systems are on the market. (Eikoplex ^tm is used to give a classification rule that determines the probability of the presence of heart disease). Other systems can be developed by those skilled in the art.

  Continuing the ranking procedure, for example, if x is set to 2, the upper two variables from each of the two ranking pairs are tested by the process (FIG. 3, steps L and S) and the test results Inspect the results to see if shows an improvement (step T). If there is an improvement, add the single best performing variable to the “significant” set of variables and then remove that variable from the two ranks for other tests (step S) (FIG. 3, Step U). If no improvement exists, the process is repeated for the next x variables from each set until an improvement is found or until all variables from the two sets are tested. This process is used until the source set is empty, i.e. all relevant or important variables are included in the last network, or all remaining variables in the set to be tested are Iterate until it is found that it is below the performance of the list. This removal process greatly reduces the number of available subsets of variables that must be tested to determine the set of important variables. In the worst case, with 10 available variables, the process tests only 34 subsets when x = 2 and only 19 subsets of 1024 possible combinations when x = 1. Thus, if there are 100 usable variables, only 394 subsets are tested when x = 2. Therefore, the variables from the network with the best test performance are identified for use (FIG. 3, step V).

  The last set of networks is then trained to perform the diagnosis (FIG. 4, steps M, N, Q, R). In general, some last neural networks are trained to perform a diagnosis. This set of neural networks can be the basis for products that can be supplied to end users. It is useful to determine consensus because different initial conditions (initial weights) can produce different outputs for a given network. (Different initial weights are used to avoid errors being trapped in the local minimum). A consensus is formed by averaging the output of each network in the trained network, which then becomes the single output of the diagnostic test.

Train network consensus.
FIG. 4 shows a procedure for training a neural network consensus. First, it is determined whether the current training cycle is the last training step (step M). If yes, put all available data into the training data set (ie P = 1) (step N). If no, the available data is divided into P equal sized sections and the data is randomly selected for each section (step O). In the illustrated embodiment, for example, five segments, eg, P ₁ -P _5, are generated from a full set of training data that can be used. Next, two configurations are started (step P). First, one or more sections are copied to a test file and the remaining sections are copied to a training file. Continuing with the exemplary embodiment of five partitions, copy _one of the partitions representing 20% of the total data set, eg, P1, to the test file. The remaining four files P _{2 to} P ₄ are identified as training data. Training groups of N neural networks are trained using training segments. Each network has a different starting weight (step Q). Thus, in the exemplary embodiment, there are 20 networks (N = 20) with starting weights randomly selected using 20 different random number seeds. After completing the training for each of the 20 networks, the output values of all 20 networks are averaged to give the average performance of the trained network test data. The data in the test file (section P ₁ ) is then executed over the trained network to give an estimate of the trained network performance. This performance is generally determined as the mean square error or misclassification rate of the prediction. A final performance estimate is generated by averaging the individual performance estimates for each network to create a completed consensus network (step R). This method of training by dividing the available data into multiple subsets is commonly referred to as the training “holdout method”. The holdout method is particularly useful when the data available for network training is limited.

Test set performance can be empirically maximized by performing various experiments that identify network parameters that maximize test set performance. The parameters that can be modified in this set of experiments are 1) the number of hidden processing elements, 2) the amount of noise added to the input, 3) the amount of error tolerance, 4) the choice of learning algorithm, 5) the weight decay Quantity, 6) including the number of variables. A complete search for all possible combinations is generally impractical due to the amount of processing time required. Thus, test networks are trained using training parameters selected empirically via computer programs, such as ThinksPro ^™ or user-developed programs, or generated by others working in the field of interest Trained from the results of existing test results. After the “best” configuration is determined, the final set of networks can be trained based on the completed data set.

3. Development of biochemical diagnostic tests.
Similar techniques for isolating variables can be used to build or validate biochemical diagnostic tests, and biochemical diagnostic test data can be combined with patient history diagnostic tests to increase the reliability of medical diagnostics .

  Selected biochemical tests include any test from which useful diagnostic information is obtained in connection with the patient and / or the patient's symptoms. This test is instrument-based or non-instrument-based and includes analysis of changes in biological samples, patient signs, patient status, and / or these factors. Any of a number of analytical methods can be used, including immunoassays, biological assays, chromatography, monitors and imagers. However, it is not limited to these. This analysis can evaluate analytes, serum markers, antibodies, and those obtained from patients in the sample. In addition, information about the patient can be provided in connection with the test. Such information includes age, weight, blood pressure, genetic history, and other such parameters or variables. However, it is not limited to these.

  The exemplary biochemical test developed in this embodiment uses a standardized test format, such as the Enzyme Linked Immunosorbent Assay or ELISA test, but the information provided herein is based on other biochemical or diagnostic tests. (For example, see Atassi et al., “Molecular Immunology: A Textbook”, Marcel Dekker Inc., New York and Basel 1984 for an explanation of the ELISA test). Important information for the development of ELISA tests is obtained during the Western blot test, a test format that characterizes antibody profiles and determines the reactivity of antibodies to proteins in order to extract antibody properties.

  Western blots are used to identify these antigens, for example, by separating specific antigens in a mixture on a polyacrylamide gel, blotting onto nitrocellulose, and detecting the labeled antibody as a probe. The technique used. (See, eg, Western blots, edited by Stites and Terr, “Basic and Clinical Immunology”, Seventh Edition, Appleton and Large 1991). However, it is sometimes undesirable to use the Western blot test as a diagnostic tool. Alternatively, a range of molecular weights containing information relevant to diagnosis can be identified in advance, and then this information can be “coded” during an equivalent ELISA test.

  In this example, the development of an effective biochemical diagnostic test depends on the availability of Western blot data for patients whose disease symptoms are known or suspected. Referring to FIG. 5, using Western blot data as a source (Step W), the first step in processing Western blot data is to pre-process the Western blot data used by the neural network ( Step X). The image is digitized by performing spline interpolation and image normalization using a computer and converted to a fixed size training record. In order to use data from multiple Western blot tests, the images need to be matched on a given gel based solely on the information in the images. Each input of the neural network needs to accurately represent a specific molecular weight or range of molecular weights. Typically, each gel that is generated contains a standard image for calibration. The included proteins are of known molecular weight, so standard images can also be used to match images included in the same Western blot. For example, a standard curve can be used to estimate the molecular weight range of other images on the same Western blot, thereby matching the nitrocellulose strip.

  A method for aligning images is cubic spline interpolation. This is a way to ensure a smooth transition at the data points represented by the standard. In order to avoid possible performance problems due to extrapolation, termination conditions are set so that the extrapolation is linear. A computer matching step minimizes the variation in molecular weight estimates for a given band on the output of the Western blot.

  To normalize the density of the image by scaling the density so that the darkest band has a scaled density of 1.0 and the brightest band is scaled to 0.0 The obtained scanned image is processed. This image is then processed into a number of fixed length vectors that will be input to the neural network that must first be trained as described below.

  A training example is created in the same process as described above for training the results generated from the processing of Western blot data (step Y). To minimize the perceived weight dependency, redundancy in the interdependent variables, and the perceived problem of desensitization resulting from overtraining the network, a set based on the data from the partitioning method discussed earlier. It is useful to train a neural network (consensus).

  From the sensitivity analysis of the training run on the processed Western blot data, a region of significant molecular weight (MW) can be determined and identified (step AA). As part of the separation step, it is preferable to combine the inputs in adjacent regions into “bins” as long as the sign of the correlation between the input and the desired output is the same. This process reduces the typical 100 plus input created by Western blots, and other inputs, to a much more manipulable number of less than about 20 inputs.

  In certain embodiments, it can be seen that the multiple ranges of molecular weight correlate with a desired output indicative of the condition being diagnosed. The correlation is positive or negative. A reduced input representation is generated using a Gaussian region centered on each peak of the peaks found during Western blot training. The standard deviation is determined such that the Gaussian value is 0.5 or less at the edge of the region.

  In certain embodiments, the basic operation of generating a neural network input is to perform a convolution between a Gaussian image and a Western blot image using a molecular weight log for calculation.

  Data can be tested using the holdout method as described above. For example, five sections may be used, and in each section 80% of the data is used for training and 20% of the data is used for testing. The data is shuffled so that each section may have an example from each gel.

  After identifying the molecular weight regions important for diagnosis (step AA), one or more tests of the selected one or more regions of molecular weight are constructed (step AB). The ELISA biochemical test is an example. The selected region or regions of molecular weight identified as important for diagnosis are then physically identified and used as a component of an ELISA biochemical test. Regions of the same correlation code may or may not be combined during a single ELISA test, but regions of different correlation codes must not be combined during a single test. The value of such a biochemical test is then determined by comparing the biochemical test results to a known or suspected medical condition.

  In this example, biochemical diagnostic test development is enhanced by combining patient and biochemical data in the process shown in FIG. Under these conditions, the patient history diagnostic test is the basis of a biochemical diagnostic test. As described herein, the variables identified as key variables were obtained from Western blot data to train a set of neural networks to be used to identify molecular weight regions important for diagnosis. Combined with data.

  Referring to FIG. 2, Western blot data is used as a source (step W) and preprocessed for use by the neural network as described above (step X). Combine key variables from patient history data and results generated from processing Western blot data and train using the combined data (Step Y) Create training examples in a process similar to the process described above To do. In parallel, the network is trained based on patient history data as described above (step Z).

  A set of neural networks based on the data by the partitioning method to minimize the perceived problem of the starting weight, the redundancy between the interdependent variables, and the desensitization caused by overtraining the network It has been found preferable to train (consensus set).

  From the sensitivity analysis of training execution based only on patient history data, regions of molecular weight that contribute significantly can be determined and identified as described above (step AA). As another step in the separation process, a set of networks is then trained using as input the patient history and bin information combined to separate key bins for Western blot data. “Important bins” represent an important area of molecular weight relevant to diagnosis that takes into account the contribution of patient history information. These bins correlate positively or negatively with the desired output of the diagnosis.

  After identifying molecular weight regions that are important for diagnosis (step AA), one or more tests for the selected region or regions are created and confirmed as described above (step AB). The designed ELISA test is then generated and used to generate ELISA data for each patient in the database (step AC). A set of networks is trained using the segmentation technique described above (step AE) using the ELISA data and key patient history data as inputs. Using the split method gives an estimate of the lower limit of the biochemical test. Final training (step AE) of a set of networks, ie networks to be used as available products, is performed using all available data as part of the training data. If necessary, the new data can be used to confirm the performance of the diagnostic test (step AF). The performance of all training data is the upper limit of the biochemical test performance estimate. The network consensus represents the intended diagnostic test output (AG). This last set of neural networks can then be used for diagnosis.

4). Improved neural network performance.
An important feature of the decision support system described by taking a neural network as an example and the method provided in the present invention is the ability to improve performance. The training method outlined above is repeated as more information becomes available. In operation, all input and output variables are recorded, increasing training data during future training sessions. In this way, the diagnostic neural network can adapt to gradual changes in individual populations and population characteristics.

  If the training neural network is included in a device that allows the user to input the required information and output the neural network score to the user, the process of improving performance in use is automated. Each entry and corresponding output is held in memory. Since the step of retraining the network can be coded into the device, the network can be retrained at any time using population specific data.

5. A method to evaluate the effectiveness of diagnostic test treatment methods.
In general, the effectiveness or usefulness of a diagnostic test is determined by comparing it to a patient medical condition for which the diagnostic test results are known or suspected. A diagnostic test is considered effective when there is a good correlation between the diagnostic test results and the patient medical condition. The better the correlation between the diagnostic test results and the patient medical status, the higher the rating placed on the effectiveness of the diagnostic test. If there is no such correlation, the diagnostic test is considered not very effective. The system provided in the present invention provides a means to assess the effectiveness of a biochemical test by determining whether the variable corresponding to the test is an important selected variable. Tests that identify data that improves the performance of the system are identified.

  The following describes a method (FIG. 6) by which the effectiveness of a diagnostic test can be determined regardless of the correlation between the diagnostic test result and the patient medical condition. Similar methods can be used to assess the effectiveness of a particular treatment.

  In one embodiment, the method uses a combined neural network trained based on combining patient history data and biochemical test data, such as ELISA data, with the performance of a patient history diagnostic neural network trained based only on patient data. Compare with performance. Patient history data is used to isolate important diagnostic variables (step AH) and train the final neural network (step AJ), all as described above. In parallel, biochemical test results are given for all patients or subsets whose patient data is known (step AK), and as described above, first all important diagnostic variables are separated (step AL) and then Training based on the patient and biochemical data combined with the diagnostic neural network by training the last neural network (step AM).

  Then, in step AN, the performance of the patient history diagnostic neural network obtained from step AJ is compared with the performance of the combined diagnostic neural network obtained from step AM. The performance of a diagnostic neural network can be measured by any number of means. In one example, the correlation between each diagnostic neural network output and the patient's known or suspected medical condition is compared. In that case, performance can be measured as a function of this correlation. There are many other ways to measure performance. In this example, the improved performance of the combined diagnostic neural network obtained from step AM over that obtained from step AJ is used as a measure of the effectiveness of the biochemical test.

  The biochemical test in this example, which lacks sufficient correlation between the test results and the known or suspected medical condition, and general diagnostic tests are usually considered to have limited utility. Such a test has been shown to have several uses by the methods described above, thus increasing the effectiveness of that test, which may otherwise be considered unprofitable. The methods described herein serve two purposes: providing a means for assessing the usefulness of a diagnostic test and also providing a means for enhancing the effectiveness of a diagnostic test.

6). Application of these methods to the identification of diagnostic variables and the development of diagnostic tests.
The methods and networks provided in the present invention provide, for example, a means to identify important variables, improve existing biochemical tests, develop new tests, evaluate therapy courses, and identify new disease markers To do. To illustrate these benefits, the provided methods have been applied to endometriosis and pregnancy related events, such as the possibility of labor and childbirth during specific periods.

Endometriosis.
The methods described herein provide a means to develop non-invasive methods for the diagnosis of endometriosis. Furthermore, the method of the present invention provides a means to develop biochemical tests that provide data indicative of endometriosis and to identify and develop new biochemical tests.

  The method of variable selection and use of the decision support system has been applied to endometriosis. A decision support system, in this example, a neural network consensus, has been developed for the diagnosis of endometriosis. In the course of this development, detailed in “Examples”, it was found that a neural network can be developed that can aid in the diagnosis of endometriosis using only patient history data, ie, data obtained from patients via a questionnaire format. . It has been found that biochemical test data can be used to enhance the performance of a particular network, but was not important for its value as a diagnostic tool. Variable selection protocols and neural networks provide a means of selecting a set of variables that can be input into a decision support system that provides a means of diagnosing endometriosis. Some of the identified variables include those previously associated with endometriosis, while others are not. Furthermore, as noted above, variables such as pelvic pain and dysmenorrhea associated with endometriosis do not linearly correlate with it so that it can be diagnosed.

  An exemplary decision support system is described in the example. For example, one neural network indicated by pat07 in this specification will be described in Example 14. Comparison of the pat07 network output with the probability of having endometriosis results in a positive correlation (see Table 1). The pat07 network can predict the likelihood of a woman with endometriosis based on the woman's pat07 score. For example, if a woman has a pat07 score of 0.6, she has a 90% probability of having endometriosis. If the pat07 score is 0.4, she has a 10% chance of having endometriosis. The dynamic range of the pat07 output when applied to the database was about 0.3 to about 0.7. Theoretically, the output value can vary from 0 to 1, but no value less than 0.3 or more than 0.7 was observed. Using the pat07 network, we evaluated over 800 women and their performance can be summarized as follows.

  The pat07 network score is interpreted as a possibility of having endometriosis, not whether a woman is diagnosed as having endometriosis. The likelihood is based on the relative incidence of endometriosis found in each score group. For example, in a group of women with a pat07 network score of 0.6 or greater, 90% of these women have endometriosis and 10% of these women do not have endometriosis. This possibility is related to the female population in the infertility department. Software programs including the pat07 network have been developed.

  adezacrf. One program called exe provides a single screen window interface that allows the user to obtain a female pat07 network score. The user enters the values of all 14 variables and calculates the pat07 network score after every keystroke. adzcrf2. Another program called exe is adezacrf. It is almost exactly the same as exe, but one additional input can be entered, namely the value of the ELISA test. This program and network are specific examples of ways to extend the clinical utility of diagnostic tests. ELISA test results did not correlate with endometriosis. By itself, the ELISA test has no clinical utility. As another input parameter, the ELISA test has improved network performance so that incorporating the ELISA results as input for network analysis expands the clinical utility of the ELISA test. adzcrf2. Another program called exe (described in Appendix II herein) provides a multi-screen window interface that allows the user to obtain a female pat07 network score. The multiple data entry screen guides the user to enter all patient history data and not only the parameters required as input for pat07. After the user enters all the data and finds it accurate, the pat07 score is calculated. This program also has *. Data entered in an fdb file can be stored, the data imported, the pat07 score on the imported data calculated, and the data exported. The user can edit previously entered data. All three programs above serve as specific examples of diagnostic software for endometriosis.

FIG. 11 shows an exemplary interface screen used in the diagnostic software. A display 1100 provided as a MicroSoft Windows ^™ type display provides a template for entering a numerical value for each of the important variables determined for the diagnosis of endometriosis. Input of data for performing the test is performed using only a conventional keyboard or in combination with a computer mouse, trackball or joystick. In this specification, a combination of a mouse and a keyboard is used. Each text box 1101-1106 contains important variables: age (box 1101), number of pregnancy (box 1102), number of births (box 1103), number of miscarriages (box 1104), number of cigarettes smoked per day (box 1105). ), For inputting a numerical value representing the ELISA test result (box 1106). To enter the age of the subject patient, the user moves the mouse so that the pointer on the screen enters box 1101, and then clicks at that location. Use the keyboard to enter a number that represents the age of the patient. Point to the selected box and click to access the remaining boxes.

  Boxes 1107-1115 are important selected variables whose data is binary, ie “yes” or “no”. Boxes and variables correlate as follows:

[Table 1]
――――――――――――――――――――――――――――――
Box variable
1107 Past history of endometriosis
1108 Dysmenorrhea
1109 Hypertension during pregnancy
1110 Pelvic pain
1111 Abnormal PAP / Dysplasia
1112 History of pelvic surgery
1113 History of drug treatment
1114 Genital warts
1115 Diabetes mellitus ――――――――――――――――――――――――――――――

  The “yes” for any of these variables can be displayed by pointing to the corresponding box and clicking the mouse button to indicate “X” in the box.

  The network automatically processes the data after every keystroke, so changes appear in the output values displayed in text boxes 1118-1120 after every entry into template 1100. A text box 1118 labeled “Endo” provides a consensus network output for the presence of endometriosis. A text box 1119 labeled “No Endo” provides a consensus network output for the absence of endometriosis. Text box 1120 provides a relative score that indicates whether the patient has endometriosis. Note that the score in text box 1120 is an artificial number obtained from boxes 1118 and 1119 that make it easier for the physician to interpret the results. As noted above, values in this box in the positive range up to 25 indicate having endometriosis, and values in the negative range up to -25 indicate no endometriosis. Show. The selected transformation allows the physician to more easily interpret the pat07 output.

  As explained in the example, pat07 is not the only network predicting endometriosis. Other networks indicated by pat08 to pat23a have been developed. These also predict endometriosis. All these networks operate in exactly the same way and can easily be used instead of pat07. Therefore, other similarly functioning neural networks can be developed and developed according to the method used to develop pat07. pat08 and pat09 are most similar to pat07. These networks were developed according to the protocol outlined above and were able to select key variables from the same set used for the development of pat07.

  It was found that the initial weighting of the variables affects the outcome of the variable selection procedure, but not in the final diagnostic results. pat08 and pat09 derived disease-related parameters using the same database of patient data as pat07. Pat10 to pat23a were originally training runs designed to reveal several parameters: endometriosis history, pelvic surgery history, dysmenorrhea history, pelvic pain importance. To develop these, the importance of the variable was evaluated by subtracting the variable from the variable selection process. It has been found that training the variable selection process and the final consensus network does not significantly degrade the network performance.

  Thus, while a particular variable or set of variables was considered important in predicting endometriosis, networks trained in the absence of such variables predict endometriosis Does not have significantly reduced ability. These results demonstrate (1) the effectiveness of the method for variable selection and consensus network training, and (2) generally the suitability of the network. In the absence of one data type, the network has found other variables from which to retrieve that information. In the absence of a single variable, the network selected a different variable at its predetermined location to maintain performance.

  Patients suspected of having endometriosis generally must undergo diagnostic surgery to diagnose the disease. The ability to reliably diagnose this disorder using patient history information and optionally biochemical test data such as Western blot data provides a highly desirable alternative to surgery. The methods and identified variables of the present invention provide a means to do so.

  Data related to the diagnosis of endometriosis disease has been collected. This data includes patient history data, Western blot data, ELISA data. The application of the method of the invention shown in the “Examples” has demonstrated that only patient history data can predict endometriosis.

  To evaluate the performance of the variable selection protocol and verify that the 14 variable network (pat07) is ranked (in terms of performance) compared to all possible combinations of 14 variables Training was based on all possible combinations (16,384 combinations). A variable selection protocol was applied to a set of 14 variables. Of the 14 variables, 5 variables were selected. These are pregnancy hypertension, number of births, abnormal PAP / dysplasia, endometriosis history, and pelvic surgery history. This combination was ranked as the 68th best performing combination out of 16,384 possible combinations (99.6 percentile), thereby demonstrating the effectiveness of the variable selection protocol. Also, the combinations including all 14 variables were ranked 718 out of 16,384 possible combinations (95.6 percentile).

  These results also show that a subset of 14 variables is useful. In particular, any subset of a selected set of one (or more) parameters of the next combination of three variables, particularly a set of 14 variables, together with a decision support system for the diagnosis of endometriosis Can be used.

[Table 2]
――――――――――――――――――――――――――――――――――――
a) Number of births, endometriosis history, pelvic surgery history b) Diabetes, pregnancy hypertension, smoking c) Pregnancy hypertension, abnormal pap stain / dysplasia, endometriosis history d) Age, smoking, intrauterine E) Smoking, endometriosis, dysmenorrhea f) Age, diabetes, endometriosis g) Pregnancy hypertension, number of births, endometriosis h) Smoking, births, intrauterine I) Pregnancy hypertension, endometriosis history, pelvic surgery history j) Number of pregnancy, endometriosis history, pelvic surgery history k) Number of childbirth, abnormal PAP stain / dysplasia, endometriosis history l) Number of births, abnormal PAP stain / dysplasia, dysmenorrhea m) History of endometriosis, pelvic surgery, dysmenorrhea n) Number of pregnancy, history of endometriosis, dysmenorrhea.
――――――――――――――――――――――――――――――――――――

  As shown in the example, another set of important selected variables is obtained that works the same as the 14 listed variables. Other smaller subsets can also be identified.

Prediction of pregnancy-related events such as the possibility of childbirth during a specific period.
The method of the invention can be applied to any disorder or condition, and is particularly suitable for situations where diagnostic tests can be adequately correlated or biochemical tests or convenient biochemical tests are not available. For example, the method of the present invention has been applied to the prediction of pregnancy related events, such as the possibility of childbirth during a specific period.

  The impending birth decision is important, for example, to increase the number of newborn infants born by the 34th week. The presence of fetal fibronectin in the vaginal cavity or cervical secretion sample from pregnant patients after week 20 of gestation is associated with labor risk and birth risk before 34 weeks. Methods and devices have been marketed for screening fetal fibronectin in vaginal or cervical secretion samples from pregnant patients after week 20 of pregnancy (US Pat. Nos. 5,516,702, 5,468,619, and 5,281,522, and No. 5,096,830. See also US Pat. Nos. 5,236,846, 5,223,440, and 5,185,270).

  The correlation between the presence of fetal fibronectin in these secretions and labor and delivery before 34 weeks is not perfect. There are significant false positive and false negative rates. Thus, in order to address the need for a method of assessing the likelihood of labor and childbirth before 34 weeks and to improve the predictability of tests that can be used, the method of the present invention has the potential for several pregnancy-related events. It has been applied to the development of a decision support system that evaluates In particular, neural networks have been developed to predict childbirth 34 weeks before (or after) pregnancy. The developed neural networks and other decision support systems described herein can improve fetal fibronectin (fFN) performance by reducing the number of false positives. The results shown in Example 13 demonstrate that using the method of the present invention can improve the diagnostic utility of existing tests because of improved prediction performance.

  As described above, these methods can be used to identify tests that were not previously thought to be associated with a disease, condition, or disorder, to design new tests, and to identify new disease markers.

  The following examples are given for illustrative purposes only and are not intended to limit the scope of the invention.

<Example 1>
Evaluation of related variables in patient history data.
This example illustrates the selection of candidate variables.

Requirements.
The patient medical history is evaluated to determine which variables are relevant to the diagnosis. This example is performed by performing a sensitivity analysis for each variable used in the diagnosis. Two methods can be used to perform this analysis. The first method is to train the network for all information and determine the influence of each input on the network output from the network weight. The second method is a method of comparing the performance of two networks, a network trained including variables and a second network trained excluding variables. This training will be conducted for each variable that is considered relevant. Those that do not contribute to performance will be excluded. These operations are performed to reduce the dimension of input to the network. When training with a limited amount of data, the low input dimension increases the generalization ability of the network.

Data analysis.
The data used in this example included 510 patient histories. Each record contains 120 text and number fields. Forty-five of these fields were known before surgery and were always identified as containing information. These fields were used as basic variables available for network analysis and training. The outline of the variables used in this example is as follows.

The method used.
The most commonly used method for determining the importance of a variable is to train a neural network on data containing all variables. Perform sensitivity analysis on the network and training data using the trained network as a basis. In each training example, the network is run in forward mode (no training). Recorded network output. For each input variable, the network is re-executed, replacing the variable with the average value of the variable over the training example. The difference between output values is squared and accumulated. This process is repeated for each training example. The resulting sum is then normalized so that the sum of the normalized values is equal to the number of variables. Thus, if all variables contribute equally to the output, their normalized value should be 1.0. The normalized values can then be ranked in order of importance.

  There are several problems with the above approach. First, it depends on the neural network solution found. Using different network starting weights, different rankings can be found. Secondly, if the correlation between the two variables is high, any of them will contain sufficient information. Depending on the network training execution, only one variable cannot be identified as important. The third problem is that a network that has been trained too much may distort the true importance of variables.

  In order to minimize the impact of the above issues, several networks were trained on the data. The training process was refined to produce the best test set performance possible, and the network was learning the basic relationship between inputs and desired outputs. By the end of this process, a good set of networks will be available and a training configuration for the last trained network will be established. A sensitivity analysis was performed on each trained network and the normalized values were averaged. In this example, the training run included 15 networks trained on five segments of available data using the holdout method.

  After variable ranking was established, test runs were performed to determine the effect of variable exclusion on test set performance. Excluding variables with small contributions decreases the performance of the test set. If overtraining becomes a problem due to limited training data, the performance of the test set can actually be improved by eliminating variables. To save processing time, ranking-based tests can eliminate groups of variables.

result.
The rankings or variables are as follows and these are reported for the network trained in the execution of pat05.

[Table 3]
――――――――――――――――――――――――――――――
01.35. Medication history 02.33. Endo's past medical history 03.11. Number of births 04.37. Pelvic pain 05.40. Dysmenorrhea 06.34. Pelvic surgery history 07.1. Age (preproc)
08.13. Infertility history 09.8. Box / day 10.36. Current exogenous hormone 11.42. Infertility 12.18. Induced hormone 13.15. Anovulation 14.14. Ovulation 15.43. Appendiceal mass / hypertrophy 16.45. Other symptoms 17.30. Abnormal PAP / dysplasia 18.26. Ectopic pregnancy 19.19. Herpes 20.39. Menstrual abnormality 21.12. Number of miscarriages 22.41. Sexual pain 23.24. Uterine / fallopian tube abnormality 24.31. Women's cancer 25.32. Other medical history 26.10. Number of pregnancy 27.28. Ovarian cyst 28.25. Fibroid 29.22. Vaginal infection 30.16. Unknown 31.27. Functional uterine bleeding 32.38. Abnormal pain 33.5. Pregnancy hyperplasia 34.9. Drug use 35.20. Genital warts 36.3. Pregnancy DM
37.4. Hypertension 38.21. Other STD
39.23. PID
40.44. Undecided 41.2. Diabetes 42.17. Poor ovulation 43.6. Autoimmune disease 44.29. Polycystic ovary syndrome 45.7. Transplanting ――――――――――――――――――――――――――――――

A subset of variables was tested and the last set of 14 variables was used to train the network of pat07 (see Examples 13 and 14). Some variables not in the top 14 above were also used. This improved the performance of the test set.
The ranking of the pat07 network is as follows.

[Table 4]
――――――――――――――――――――――――――――――
01.10. Endo's past medical history 02.6. Number of births 03.14. Dysmenorrhea 04.1. Age (preproc)
05.13. Pelvic pain 06.11. Pelvic surgery history 07.4. Box / day 08.12. History of drug treatment 09.5. Number of pregnancy 10.7. Miscarriage count 11.9. Abnormal PAP / dysplasia 12.3. Pregnancy hyperplasia 13.8. Genital warts 14.2. Diabetes mellitus------------------------------

Conclusion.
The set of variables identified in this example is considered valid based on tests and information.

<Example 2>
Network training on patient history data.
This example demonstrates how to set and optimize various parameters using the above 14 variables.

Requirements.
When the above example is complete, the network set is trained on the reduced patient history and their performance recorded. Experiments were performed to determine the best configuration and parameters for network training. Performance analysis was performed to determine false positives and false negative numbers to see if a given subset of patients could be diagnosed reliably. Since the data is limited, the estimated performance was determined by excluding a small portion (25%) of the database for testing and training on the remaining data. This method was repeated until all data was used as test data in one of the networks. The combined result for the test data is then a performance estimate. The last network trained using all available data as training data.

The method used.
When dealing with a small number of training examples, the holdout method is effective in providing test information useful for determining network configuration and parameter settings. In order to maximize the data available for training without significantly increasing the processing time, a 20% holdout was used instead of the proposed 25%. This resulted in five data categories instead of four, and 80% of the data was for training in each category.

  To minimize the effects of random starting weights, multiple networks were trained with all training runs. In this implementation, the three networks were trained from different random starts in each of the five sections of data. The network outputs are averaged to form a consensus result with a lower variance obtained from a single network.

  Several experiments were conducted to discover network parameters that maximize the performance of the test set. The parameters modified by this process are as follows:

[Table 5]
――――――――――――――――――――――――――――――
1. Number of hidden processing elements 2. The amount of noise added to the input 3. Error tolerance amount 4. Learning algorithm to be used 5. Amount of weight attenuation to use Number of input variables to be used ――――――――――――――――――――――――――――――

  Full searching for all possible combinations of 45 variables is not easy due to the amount of CPU time required for testing. The test network was trained with parameters selected based on parameters known to those skilled in the art as important in this area and based on the results of previous tests. Other variable sets are also appropriate. Also, combinations of all 14 selected variables were tested as shown elsewhere in this specification. After the best configuration was determined, the last set of networks was trained on a complete data set of 510 patients. In the last set of networks, a consensus of eight networks was created and final statistics were generated.

result.
The final holdout training run was pat06 with 14 variables. The performance for the test data was 68.23%. All training runs were pat07 with the same network configuration as pat06. The performance for the training data was 72.9%. Statistics for the last training run were generated based on the use of a network output value cutoff. This example is not considered when the network output is below the cutoff. The following table summarizes the results for the consensus of the eight networks in pat07. A test program called adzcrf was generated to reveal this final training.

PPV = positive prediction value, NPV = negative prediction value

<Example 3>
Western blot data pre-processing and input.

Requirements.
The antigen data from the Western blot for the patient, first sent to the logic design, provided information about only the peak molecular weights and the intensity with which they were associated. Analysis of this data, and the original image from which this data was taken, shows that the digitized original image can be used so that more information can be provided to the neural network. When examining the original image for two experiments, preprocessing the image data reduces the variability of the position of a particular molecular weight in the image. In this preprocessing, a corrected image is generated using a polynomial suitable for the standard image. Image pre-processing will also include normalizing the background level and contrast of the image.

  After the pre-processing is completed, the image data can be used as it is or the peak molecular weight can be extracted. An input to the neural network is generated from the obtained image. Since a normal image is about 1000 pixels long, methods to reduce the number of inputs will be investigated. Since the images are encoded directly into the network input using images of all or reduced dimensions (resolution), the neural network is trained in surveillance learning to help determine the range of molecular weights relevant to disease determination. Will do. This example focuses on using the image as a whole in input to the network.

The method used.
Correlation techniques were used to match similar features on Western blot images to generate correlation plots. From these plots it was concluded that the reconciliation variation for the two sample correlation plots was too large to match the samples accurately. Since each input of the network needs to accurately represent the molecular weight value, it was decided to use only information from the standard image for image matching.

  A second order fit is performed on the standard image to generate a means to translate relative mobility information into molecular weight. After plotting the relative mobility curve against the logarithm of molecular weight and examining the RSQR value, it was concluded that the quadratic fit was not accurate enough to perform this translation. The molecular weight calculated for the standard molecule using a quadratic fit varies from gel to gel.

  Several methods were tried to improve the translation to relative mobility molecular weight. A cubic spline interpolation method was selected. This method ensures a smooth transition at the data points and is calculated quickly. The only important thing is how this method is implemented for relative mobility values outside the interval covered by the standard. If the termination conditions are set appropriately, the extrapolation problem can be avoided. This is the method of choice.

  A spline interpolation method was used to convert the image into a constant size training record. At this point, normalization of image intensity must be considered. Two options are possible. The first is that no normalization is performed. The second is to process the image so that the maximum value over the image is set to 1.0 and the minimum value is set to 0.0. We trained the network for each option and compared the results. If no noise was added to the input, the preprocessed image network had 97% training example performance and 79% performance without preprocessing. When noise was added, the two options gave similar results. We chose to use preprocessed images for further training runs. This selection ensured that a given network input was consistently associated with a particular molecular weight within the tolerances that could be achieved using Western blotting.

  Using the above selections, a series of eight neural networks were trained to provide information about the importance of various molecular weights based on the prediction of Endo presence variables. Only a single hidden processing element was used for training to allow analysis of the direction of correlation. Sensitivity analysis was performed for each network and the resulting consensus was plotted using Excel.

  The network weights were then averaged to generate a consensus value for each weight. Since the weight of the interconnection from the hidden element to the output can be positive or negative, these weights were modified so that all output connections have the same sign. The weights were then averaged and the results were plotted using Excel.

result.
Analysis of Western blot data was performed using cubic spline interpolation for image alignment to network inputs and Max / Min image preprocessing. If a certain amount of variability can be expected in the accuracy of image alignment by Western blotting, this approach is considered to give better results that the polynomial fit was first used.

  Sensitivity analysis and weight plots for the final consensus network showed that there is a region of the Western blot that can aid in disease prediction and diagnosis. The width of the positive and negative correlation regions seen in the network weights also indicates that the results shown are significant. If the peak is very narrow, it must be concluded that the peak is an artifact of a training process similar to overtraining and does not form the basic process to be learned. The areas that are considered important are:

[Table 6]
――――――――――――――――――――――――――――――
Positive correlation
31503.98-34452.12
62548.87-65735.97
84279.36-89458.49
Negative correlation
19165.9-20142.47
50263.36-53352.14
67725.77-78614.77
――――――――――――――――――――――――――――――

  There are several positive and negative peaks, but these are most likely to be included in the two ELISA tests. One test focuses on the positive area and the other focuses on the negative area. The two values obtained can then be combined with patient history data as input to the neural network.

Conclusion.
Neural networks were able to find areas that correlate with the presence of disease based on Western blots.

<Example 4>
Investigation of constant input dimensions for Western blot data.

Requirements.
Using the peak molecular weight extracted from the pre-processed image, we investigated how to convert the changing dimensions of the Western blot data for the patient to a constant dimension for the neural network. This approach is desirable because it requires significantly less network input than the full image approach. The basic problem is that the test generates molecular weight variables that can be interrelated. A comparison of the example and the results of this example indicates that there is a pattern of molecular weights or whether their molecular weights are not relevant. Since there is some variability in molecular weight data, even if classification is performed on a neural network, the technique for processing this data is similar to the fuzzy membership function.

Additional requirements.
A portion is identified from the Western blot data. Since the product of these parts is reproducible, the effectiveness of using this information is determined by processing the Western blot image data into bins corresponding to the molecular weight of these parts.

The method used.
From the results of Example 4, it is determined that some range of molecular weight is correlated with disease. By using a Gaussian region centered on each peak seen in Example 5, a reduced input representation was generated. The Gaussian standard deviation was determined so that the Gaussian value was 0.5 or less at the edge of the region. The basic operation performed to generate the neural network input is a convolution between Gaussian and Western blot images. All calculations were performed using the logarithm of molecular weight.

  Separate software programs were generated. This program performed a convolution on the molecular weight and intensity for the normalized image. The parameters for calculating the network input are included in a table in the binproc program. In binproc, the mean and standard deviation are stored in this table. The program is recompiled when the table values change. The program has a test mode that produces an output file that can plot Gaussian to comparable Western blot images using Excel. Region plots are included in the documentation.

  When processing 36 sub-portions, binproc.c. So as to translate the position of the sub-portions into the values in the binproc table. c was corrected again. This modified program is called fproc. Call it d. Its purpose is to perform the spline interpolation necessary to normalize molecular weight values based on standards. binproc to binproc2. c, the average deviation table and the standard deviation table are min. corresponding to the end points of the small portions in the supplied file. Table and max. Replaced with a table.

  To test any data file generated from the above program, a holdout method was used, with 80% of the data used for training and the remaining 20% used for testing. After training data was generated from the Western blot data, a random number sequence and a patient ID sequence were added to the Excel spreadsheet. The data was then sorted on the random number sequence. This actually shuffles the data. In this way, each section is likely to have an example from each gel. At these rates, five separate training and test files are generated so that network performance can be estimated from the combined test set results.

By using ThinksPro ^™ , the number of inputs used by the network can be varied by eliminating the inputs. Excluded inputs are not presented to the network during training. Use sensitivity analysis as a guide to remove unimportant inputs. Reducing the input space dimension becomes even more important when the number of training examples is small. This method is the same as that used to remove variables during patient history training. Currently, this process is manual.

result.
In Example 5, a network trained on all data was used to determine the range of molecular weights important for the classification process. In this example, the holdout method was used to train the network so that the performance of the test set can be estimated. The first test set is based on the area identified in Example 5. A second test set was created using the small pieces identified in the four ishgel files.

  The performance of the first consensus run based on the top six regions seen in Example 5 is low (50%). Analysis of the generated input data showed that the area used to generate the input data was too narrow to capture important information from the image data. The width of the region was expanded to include the top 10 regions from Example 5, rather than the top six. Tests on 10 wide areas showed slightly better performance. Using sensitivity analysis, three of the ten regions were removed and a complete test was performed. The performance of 6 out of 10 widened areas was improved to 54.5%.

  As the number of inputs to the network further decreases, the performance of the test set (estimated by the holdout method) continues to increase. The best performance was achieved when using only one region with a molecular weight in the range of 66392.65 to 78614.74. The estimated performance for the test data using the holdout method was 58.5%.

  This process was reapplied using 36 regions based on the identified sub-parts as starting points. There was a large amount of duplication in the 36 small parts. The top seven sub-parts were determined from 36 using sensitivity analysis. Similar performance of 58% was achieved using a small subset.

Conclusion.
The test did not produce very high results. The main reason for this is likely that the amount of training data available in this example was limited. The results obtained from previous examples showed that the performance on the validity data decreased as the number of patients in the training sample decreased. This relationship is shown in the table below.

  Even when the number of patients was reduced, better results were achieved for Elisa / patient history data when Elisa variables were included. This indicates the value of the ELISA variable.

  It is clear that several areas can be determined to be important for disease classification. A significantly different set of regions yields similar results, indicating that there may be patterns in the Western blot data indicating the presence of disease. If the patient database is small, it is more difficult to separate these patterns.

  Clearly, increasing the size of the database for Western blot data will improve the performance of the network trained on this data. Combining Western blot data with patient history data will increase the input size of the network. Increasing input dimensions usually requires more training examples to ensure generalization.

<Example 5>
Training network using western blot data.
The purpose of this example is to train a set of networks to use only Western blot data to determine performance estimates for diagnosis. Experiments were performed to determine the best configuration and parameters for network training. The method described in Example 2 above is used for this performance estimation. The last network trained using all available data as training data. The output of this trained network (antigen index) was used as input to the network generated during the combined data phase.

The method used.
Several methods were used to find the most commonly implemented set of training data available. From previous examples, it was found that using sensitivity analysis yielded good results in identifying the importance of each input variable. That number of networks were trained on combinations of variables determined manually by sensitivity analysis.

  In preparing the automation procedure, a 2 × 2 contingency table chi-square analysis of the variables was used to provide an alternative ranking of the importance of the variables. Since the inputs are continuous, a threshold was used for each input to generate the information needed for the contingency table. The chi-square value changes depending on the threshold setting. The threshold used to rank the variables was chosen to maximize the chi-square statistic.

  Training runs performed during the development of the automation procedure are selected from these rankings. When the training run is performed, the automated procedure has not been formalized. Only one segment of training data is used to save overall processing time. The well-performed variable combinations in the first segment of training and test data were then tested for the remaining segments.

  One way to find the best set of inputs proposed in this document is to use a genetic algorithm to determine the most commonly implemented set of inputs. Genetic algorithms usually require thousands of iterations to converge to a good answer. In processing Western blot data, this corresponds to a large amount of computer time even if the training example size is small. For 10 variables, 1024 training runs are required to enumerate all combinations. An alternative method of genetic algorithm was tried. In this alternative method, the neural network was trained to predict the RMS error of the test set based on the selected set of inputs. The training example used in this experiment is the result of a training run on the first section of the Western blot data. The prediction network is then tested with all combinations to determine the predicted minimum combination. The combination of inputs is then used to train the network for Western blot data. The main drawback of this method and the genetic algorithm approach is that sensitivity analysis information that has proven to be very effective is ignored in this process.

result.
The basic ranking for the 10 bins in the Western blot data is based on a consensus of 8 networks trained on a full database of 200 examples. The results are as follows.

[Table 7]
――――――――――――――――――――――――――――――
7: 1.182073
9: 1.055611
3: 1.053245
8: 1.039028
6: 1.027239
10: 1.023135
4: 0.978769
5: 0.952821
2: 0.899936
1: 0.788143
――――――――――――――――――――――――――――――

  The ranking of the 10 variables based on the chi-square analysis is as follows.

[Table 8]
――――――――――――――――――――――――――――――
3: 4.380517
9: 3.751625
7: 3.372731
2: 3.058437
6: 3.022164
5: 2.787982
10: 1.614931
4: 1.225725
1: 0.975502
8: 0.711958
――――――――――――――――――――――――――――――

  During the analysis of western blot data, several networks were trained on one or more first segments of training data. The test results are ranked as follows, indicating that the variable is included in the training run.

                               variable

() Indicates the combination generated by the predictive network training process

  Referring to the test execution above, it is clear that the more important variables in the ranking contribute to the lower test set error, and that the more variables included, the lower the test set results. This demonstrates the importance of selecting the best subset of variables in the development of high performance neural networks.

  Several combinations of variables were used to train the network for all segments of the training data. The results of these runs are shown below.

[Table 9]
――――――――――――――――――――――――――――――
Variable Time set performance
3 57.5%
3, 9 53.5%
3, 7, 9 53.0%
4, 6, 9, 10 57.0%
――――――――――――――――――――――――――――――

  Both rankings of the variables indicate that 3, 7, and 9 are important, so if there is sufficient training data, this combination is likely to exceed 57.5%. The training example performance for this combination is 63.9%, indicating the level of overtraining that occurred. Some of the first segment networks shown above have a combination of variables selected by a neural network trained to predict test performance. These networks are indicated by the number in the last column. This number indicates the sequence in which the test is performed. Combinations without numbers were manually selected from ranking. If this process continues, the prediction network will eventually find the best combination. Since there are many factors that can affect the performance of the test set, there is a high probability that there will be a lot of “noise” in the results of the test set. In order for this method to work better, a consensus approach may be required to generate training values for predicted test set errors. This problem is also seen when using consensus techniques.

Conclusion.
The process of using variable sensitivity and contingency table ranking is an effective and efficient technique for choosing a set of variables to maximize the performance of a neural network. The top three variables under both rankings are the same, indicating that these methods are well implemented. Although this method apparently processes Western blot data, it works well for any form of data, making it a general purpose neural network technique that can also be applied to patient history data.

  The above results show that the performance level improves with more data. Sensitivity analysis shows little variation in the relative values of the variables. Most variables contribute to the answer. This is expected because bins are selected based on an analysis of the weight of the neural network trained on the full Western blot image. However, by using all or most of the variables, the neural network quickly becomes overtrained. This can be avoided by adding data to the training example.

  Tests that are guided by a neural network to select variables are found to be less effective than ranking methods. It is clear that the ranking method is the most effective, but the neural network guidance method can finally find the best set of variables. Since this is a more direct method than the genetic algorithm, it is likely that similar data will be better implemented than the genetic algorithm. The main drawback of this method is that it does not use sensitivity analysis information to assist in searching.

<Example 6>
Combine patient history and ELISA data.

Requirements.
The process developed in the above example is used to train a set of networks for a combination of patient history data and ELISA data. An index generated from an ELISA test based on the use of the entire set of antigens will be used to determine the performance improvement achieved by combining this information with patient history data.

Additional requirements.
In addition to the above requirements, data from multiple ELISAs, comparisons between ELISA100 and ELISA200 data and ELISA2 data, as well as analysis of the interrelationships of variables, to help determine the variables to which the original ELISA test relates did.

The method used.
In order to determine the improved diagnostic test performance achieved by including the results of the ELISA test, some training was performed using the holdout method described in Example 2.

  Data sections were created so that 80% of the data was used for training in each section and the remaining 20% was used for testing.

  In order to minimize the effects of random starting weights, some networks train with full training runs. In such an implementation, the three networks were trained on each of the five segments of data, each from a different random start. The network outputs were averaged to form a consensus result with lower variability obtained from a single network. Since 325 patients have access to all forms of ELISA data, a new training run with the original 14 variables will be performed and an accurate average comparing the impact of ELISA data on disease diagnosis will be obtained. Provided. Analysis of the ELISA2 data showed a wide range of values for the test. A plot showing the relationship of ELISA2 to ELISA100 data shows that the logarithm of ELISA2 data may be better than the raw value.

  The comparative training execution is configured as follows.

Run 1: ELISA 100, ELISA 200, Logarithm (ELISA2) and original 14 variables Run 2: (ELISA2) and original 14 variables Run 3: Original 14 variables

  After making these comparison runs, the last set of networks was trained on a complete data set of 325 patients. The final set of networks created a consensus of eight networks and generated the final statistics. The last run statistic is reported only for training data and represents the true performance limit. The result of the last holdout execution represents a possible lower bound on performance.

  From the training data, each of the 65 variables, including those that are not available for diagnosis, are incorporated into the training examples among the 325 training examples. The TrainDos training program was modified to automate network generation and provide relationships between variables. In each of the 65 networks, one variable is predicted by the remaining 64. Sensitivity analysis was performed on each network to show the importance of each variable in making the prediction.

result.
The consensus results for the three comparison runs are as follows:

[Table 10]
――――――――――――――――――――――――――――――――――――
Run 1: all ELISA variables (CRFE: 1) 66.46%
Execution 2: Logarithm of ELISA2 (CRFEL2) 66.77%
Execution 3: No ELISA variable (CRFEL0) 62.76%
――――――――――――――――――――――――――――――――――――

  Comparing Run 1 and Run 2, it can be seen that there is no effect of adding the ELISA 100 and ELISA 200 data to the ELISA 2 data. Therefore, the ELISA 100 and ELISA 200 variables can be removed.

  Comparing Run 2 and Run 3, it can be seen that the input based on the ELISA test improved the diagnosis of the disease.

Comparing run 3 with pat06, it can be seen that the performance of the test is reduced by 5.47%. This is simply due to a decrease in the number of patients available for training. This also means that an increase in training data over 500 is likely to have a significant impact on the performance of the neural network for test data.

  Based on these results, the last network was trained. The eight networks trained on 325 patients. The performance for this training data was 72.31%. This is the same result as the pat07 execution, but it is clear that the improvement with ELISA2 data is offset by the reduced amount of training data available.

  The results of the sensitivity analysis indicate that the ELISA2 variable ranked 7th among the 15 variables was used.

  A plot of hidden processing element output was created from log files of eight trained networks. The average was determined so that the desired output could be shown on the plot. A comparison of the eight networks reveals that each performs the task in a different way. Some clustering of data points can be seen in some plots. Since this does not happen consistently, no ligation can be derived.

  Statistics were generated for the last training run based on the use of the network output value cutoff. This example is not considered when the network output is below the cutoff.

  The following table summarizes the results for the consensus of the eight networks in CRFLE2.

In general, these results are better than those for pat07.

  As a demonstration of this last training, adzcrf2. A test program called exe (see Appendix II) was generated. This program allows the execution of pat07 and CRFEL2 based on the value input in the ELISA field. If the value in this field is 0, pat07 is used.

  An analysis of variable relationships was performed. Based on an analysis of this relationship, variables that indicated Endo presence as a contributing factor were compared to variables used in the predicted Endo. The training results of the two networks (PATVARSA and PATVARS3) show that in the case of Endo, the relationship is not symmetric as with correlation. To summarize the results, CRFVARSA. XLS was constructed from the results of sensitivity analysis. These results show the non-linear characteristics of the relationship. The importance of variables is affected by other variables during the training run. This means that a means for automatically removing unimportant variables may be required to increase the convenience of this analysis.

  Analysis of the variable relationships (CRFVAR00 to CRFVAR64) shows that in most cases the logarithm of the ELISA2 test is more effective than the raw ELISA2 value. This is especially true for logarithmic values ranked higher for both the predicted Endo presence and AFS Stage.

Conclusion.

  The ELISA2 test adds the predictive power of neural networks. The ELISA2 test eliminates the need for the original ELISA test. Based on this result, the results of processing the Western blot data are likely to further improve the diagnostic test capability of the neural network.

  The effect of increasing training data is clearly seen in the comparison of run 3 and pat06. This difference in performance means that the performance of the neural network is greatly improved by increasing the training data. From this comparison, it is clear that doubling the data will improve performance by 10 to 15%. If the data is increased from 8 to 10 times, the performance may be improved by 75 to 80%.

<Example 7>
Patient history Stage / AFS Score training.

Requirements.
The method developed in the above example is used to identify relevant variables for either disease stage or AFS Score. The selection of the target output variable to use is determined by comparing the performance of the test set from the training run using the phase 1 list of key patient history variables. After selecting a list of important variables, we will train a consensus of eight neural networks on the 510 patient database.

The method used.
Training examples were constructed for the desired output for Stage and the desired output for AFS score. There were 7 patients with missing Stage information and 28 patients with missing Score information. For the stage variable, an average value of 2.09 was used when data was missing. For score, the missing data was replaced with a value that depends on the value of the stage variable. In stage 1, 3 was used as the score. In stage 2, 10.5 was used. In stage 3, 28 was used, and in stage 4, the value 55 was used. Stage and score were reprocessed so that the desired output was in the range of 0.0 to 1.0. The stage was translated linearly. Two methods were used for score. The first method is the square root of score divided by 12.5. The second method is a number obtained by dividing the logarithm of score + 1 by the logarithm of 150.

  The holdout method was used to train the network for stage, square root of score, and logarithm of score. These networks were trained using 45 variables. The results were compared to determine which variables and treatments were used for the remainder of this example. The logarithm of score was selected.

  At this point, a procedure was started to isolate a set of important variables. Eight networks were trained on all training examples and a consensus sensitivity analysis was generated to generate the first ranking of variables. A chi-square contingency table was then generated to produce a second ranking of variables. The procedure for isolating important variables was started manually but was found to be too time consuming. This procedure was implemented as a computer program and was run on the computer for about a week.

  From the variable selection results, a set of eight networks was trained for all training examples. Consensus results were analyzed and compared to the results of Endo presence.

result.
A sensitivity analysis of all 45 variables gave the following variable rankings:

  From the chi-square analysis, the following variable rankings were given:

  The variables selected during the variable selection procedure are as follows, which indicates the ranking from the last sensitivity analysis.

  The comparison between the score network and the Endo presence network can be performed by providing a threshold to the desired score output and creating an Endo presence comparison. The results for the score and pat07 networks are shown below.

Conclusion.
The set of variables identified in this example is considered reasonable.

  The automated variable selection method seems to work properly. The selection of variables is well predicted by sensitivity analysis.

  Since there are two methods for predicting illness, the reliability of prediction can be improved by combining the Endo presence network and the Score network.

<Example 8>
Patient history Adhesions training.

Requirements.
The method outlined in Example 7 is used to identify relevant variables for Adhesions target output variables. This target output variable will be implemented using a phase 1 list of important patient history variables. This also allows the new output to be compared to the Endo presence target variable used in Phase 1. After selecting a list of important variables, we will train a consensus of eight neural networks on the 510 patient database.

The method used.
Training data for the adhesions variable was generated in the same manner as in Example 7. The adhesions variable generated two output variables in the same way as used when Endo was present. At this point, a procedure was started to isolate a set of important variables. A set of eight networks was trained for all training examples, and a consensus sensitivity analysis was generated to produce the first ranking of variables. A chi-square contingency table was then generated to produce a second ranking of variables. The procedure for isolating important variables was started manually but was found to be too time consuming. This procedure was implemented as a computer program and was run on the computer for about a week to complete.

  From the variable selection results, a set of eight networks was trained for all training examples. Consensus results were analyzed and compared to the results of Endo presence.

result.
A sensitivity analysis of all 45 variables gave the following variable rankings:

  The chi-square analysis gave the following variable rankings:

  The variables selected during the variable selection procedure are as follows, which indicates the ranking from the last sensitivity analysis.

  The comparison between the Score network and the Endo presence network can be performed by providing a threshold to the desired score output and creating an Endo presence comparison. The results for the score and pat07 networks are shown below.

Conclusion.
The set of variables identified in this example is considered reasonable. The automated variable selection method seems to work properly. The selection of variables is well predicted by sensitivity analysis.

<Example 9>
This example shows the reproducibility of the process provided herein.

The method used.
The software used to select key variables for Adhesions and Score was modified to handle the desired output of Endo presence. The software was further modified to run in the general case without having to recompile for each specific test.

Similar to the execution for Adhesion and score, the execution was performed for the Endo presence variable. This involves using a consensus of four networks during the variable selection process. The training data was divided into five sections during the training process, generating a total of 20 networks, each evaluating the current set of variables to be tested.

  The results of runs with different random seeds indicated that the number of networks in consensus needs to be increased.

  Two additional variable selection runs were performed using a consensus of 10 networks in the process. In this case, a total of 50 networks are trained to evaluate a single combination of variables. Two separate runs were done in the same way, changing only the random starting seed.

  From these last two variable selection runs, a set of eight networks is trained for each variable set (pat08, pat09) and their performance on new data (not included in the original 510 record database). Makes it possible to evaluate Statistics on the performance of these networks are generated so that they can be compared to the original pat07 consensus net.

result.
In each case using different random number seeds, the variable selection process found various sets of important variables. As the number of networks in consensus increases to 10, common variables increase in different runs.

  Many of the original 14 variables used for pat07 have been identified as important in variable selection execution using a 10 consensus net. The last run performed on the selected variable is called pat08 and pat09.

  The variables used in the pat08 and pat09 consensus networks are listed below along with their sensitivity analysis ranking.

Conclusion.
The variable selection process worked well and produced two alternative networks that worked as well or better than the pat07 net. The reason for this conclusion is that performance statistics generated for training data only appear slightly better for pat07 than for pat08 and pat09. Since the variable selection process carefully selects variables based on the performance of the test set, it is unlikely that the associated network is overtrained. A typical feature when the network is overtrained is that the performance of the training example is improved and the performance of the test set is reduced. Thus, higher performance of pat07 may result in slightly overtraining.

  The variable selection process clearly produced two alternative selections for the same training data, but the performance of the two selections appears to be very similar. This is based on the performance of the last variable selection test set for the two runs. When the relative performance of the two variables is close, it has become clear that random factors can affect their relative ranking. The random factor during variable selection includes a random starting point and the use of noise added to the input during training. Random noise has been found to help better generalization (translation: test set performance). As the number of networks in consensus increases, the degree of random influence decreases.

  The determination of the set of variables that produces a high quality network is considered to be handled by the variable selection process. As more combinations of variables that work well are enumerated, it becomes apparent that a particular variable or combination of variables is essential for good performance.

<Example 10>
Evaluation of the diagnostic performance of exclusion of past history of endometriosis and history of pelvic surgery.
The purpose of this example is to determine the significance of the “historical history of endometriosis” and “history of previous pelvic surgery” variables in assessing the patient's risk of having endometriosis and its conclusions. Is to provide an alternative means of measuring the importance of any given variable in predicting (different from sensitivity analysis).

task:
1. The variable selection process is applied except for “historical history of endometriosis”.
2. Task (1) is repeated using various random seed variables for the variable selection process.
3. The consensus network training process is completed for both sets of “endometriosis related variables” identified in tasks (1) and (2) above.
4). The above tasks (1), (2), and (3) are repeated by removing the “past pelvic surgery history” variable from the endometriosis database.
5. The above tasks (1), (2), and (3) are repeated except for both the “historical history of endometriosis” variable and the “historical history of pelvic surgery” variable from the endometriosis database.

The method used.
Using the variable selection software developed in Example 9 as a basis, the results for each of Example 10 were generated. The software was modified to allow the user to identify variables that would be excluded from consideration based on the requirements of Example 10. The software was also modified to report the classification performance for each set of variables to be tested so that the effects of removed variables can be more easily understood.

  For each execution of variable selection, the parameters of the variable selection process were set as follows.

[Table 11]
――――――――――――――――――――――――――――――
Number of categories: 5
Consensus network: 10
Training example size: 510
Number of passes: 999
――――――――――――――――――――――――――――――

  The ordering of database variables during the variable selection process is based on sensitivity analysis and chi-square analysis. This ordering is similar to that used in pat08 and pat09.

  The network trained for this example is identified as follows (the two nets have different random seeds).

[Table 12]
――――――――――――――――――――――――――――――
Remove Endo's past medical history: pat10, pat11
Remove previous pelvic surgery history: pat12, pat13
Remove both variables: pat14, pat15
――――――――――――――――――――――――――――――

  After the variable selection process was completed for each variable and random seed combination, a set of eight networks was trained using the identified selected variables. Each of these networks is trained on a complete 510 record database. From these training runs, output consensus can be generated in an Excel spreadsheet to evaluate the performance of each network.

result.
The normal performance of the network consensus was estimated using the holdout method in 5 sections. When all variables were available as in pat08 and pat09, the classification performance was estimated at 65.23%.

  When variables from past history of endometriosis were removed from consideration (pat10 and pat11), the performance was estimated to be 62.47%. This corresponds to a reduction of 2.76%.

  When past pelvic surgery history variables were removed from consideration (pat12 and pat13), the performance was estimated at 64.52%. This corresponds to a reduction of only 0.72%.

  When both variables were removed from consideration (pat 14 and pat 15), the performance was estimated at 62.43%. This corresponds to a reduction of 2.80%. This is only slightly worse than if the past history of endometriosis was removed, and would not be inconsistent with other results based on the assumption that the variables were independent (no correlation).

Conclusion.
Neural networks use pelvic surgery history if available, but the effect of removing this variable was minimal. It is believed that the neural network can compensate for the removal of this variable by using other information.

  Removal of the past history of endometriosis is significant. This variable is always at the top of the list for any sensitivity analysis. The removal caused about 2.76% performance degradation above average when all variables were available. If the average performance is estimated to be 65.23% and 50% can be achieved by chance, this corresponds to an effective reduction of 18.12%.

  If both variables are removed, no significant performance degradation appears, indicating that there is no interaction between these two variables. This process of removing variables and performing a variable selection process is considered a good technique for determining the true value of a given variable. Note that there are two variables that are important for diagnosis but highly correlated, and removing only one has little effect because the network compensates for this by using the other. Their values are only apparent when both are removed.

<Example 11>
Evaluation of diagnostic performance for removal of pelvic pain and dysmenorrhea.

Requirements.
the purpose:
1. To determine the importance of the “pelvic pain” and “dysmenorrhea” variables in assessing the patient's risk of having endometriosis.
2. To provide a separate mechanism (different from sensitivity analysis) that measures the importance of any given variable in predicting its conclusion.
task:
1. Apply the variable selection process described herein.
2. Task (1) is repeated using various random seed variables for the variable selection process.
3. The consensus network training process is completed for both sets of “endometriosis related variables” identified in tasks (1) and (2) above.
4). The above tasks (1), (2), and (3) are repeated by removing the “dysmenorrhea” variable from the endometriosis database.
5. The above tasks (1), (2), and (3) are repeated except for both the “pelvic pain” variable and the “dysmenorrhea” variable from the endometriosis database.

The method used.
Using the variable selection software developed in Example 9 as a basis, results were generated for each of these tasks.

  For each execution of variable selection, the parameters of the variable selection process were set as follows.

[Table 13]
――――――――――――――――――――――――――――――
Number of categories: 5
Consensus network: 10
Training example size: 510
Number of passes: 999
――――――――――――――――――――――――――――――

  The ordering of database variables during the variable selection process is based on sensitivity analysis and chi-square analysis. This ordering is similar to that used in pat08 and pat09. The networks trained for this task are identified as follows (the two nets have different random seeds):

[Table 14]
――――――――――――――――――――――――――――――
Removes pelvic pain: pat16, pat17, pat17A
Remove dysmenorrhea: pat18, pat19
Remove both variables: pat20, pat21
Four variables (EXs.11 and 12): pat22, pat23, pat23A
――――――――――――――――――――――――――――――

  After the variable selection process was completed for each variable and random seed combination, a set of eight networks was trained using the identified selected variables. Each of these networks is trained on a complete 510 record database. From these training runs, output consensus can be generated in an Excel spreadsheet to evaluate the performance of each network.

result.
The normal performance of the network consensus was estimated using the holdout method in 5 sections. When all variables were available as in pat08 and pat09, the classification performance was estimated at 65.23%.

  When the pelvic pain variable was removed from consideration (pat 16 and pat 17), the performance was estimated at 61.03%. This corresponds to a reduction of 4.20%.

  When the dysmenorrhea variable was removed from consideration (pat 18 and pat 19), the performance was estimated at 63.44%. This corresponds to a reduction of only 1.79%.

  When both variables were removed from consideration (pat20 and pat21), the performance was estimated at 61.22%. This corresponds to a reduction of 4.00%. This is better than removing only pelvic pain. This means that the performance degradation for pelvic pain is exaggerated. The best implemented network without pelvic pain has a performance of 62.29%, which gives a drop of 2.94%. This is a more reasonable estimate if performance is given when both are removed.

Conclusion.
Testing four variables and ranking the variables in order of importance yields:

[Table 15]
――――――――――――――――――――――――――――――
Pelvic pain 2.94 to 4.20% decrease Endo's past history 2.76% decrease Dysmenorrhea 1.79% decrease Past pelvic surgery history 0.72% decrease ――――――――――――――――― ―――――――――――――

  This process of removing variables and performing a variable selection process is a good way to determine the value of a given variable. Note that there are two variables that are important for diagnosis but highly correlated, and removing only one has little effect because the network compensates for this by using the other. Their true value is only apparent when both are removed.

<Example 12>
Train the neural network to distinguish between mild and severe endometriosis.
the purpose:
1. Train a network consensus to distinguish between minimal / mild endometriosis and moderate / severe endometriosis.
task:
1. Train the network to AFS score as follows.
Positive = Endo Stage III or IV
Negative = No Endo, Endo Stage I or II
2. Apply the variable selection process described in How to develop medical and biochemical tests using a neural network of endometriosis database.
3. Task (2) is repeated using various random seed variables for the variable selection process.
4). Compare the variables selected in (2) and (3) above before proceeding. If the set of selected variables is significantly different, task (2) is repeated using various random seed weights.
5. Train the final consensus network for the variables selected in (2) and (3) above.
6). Repeat steps (2) to (5) using only a subset of the endometriosis database where Endo was present in the patient.

The method used.
The variable selection software developed in Example 10 and modified in Example 11 was used as a basis to generate results for each of the tasks in this example.

  For each execution of variable selection, the parameters of the variable selection process were set as follows.

[Table 16]
――――――――――――――――――――――――――――――
Number of categories: 5
Consensus network: 20
Training example size: 510 (290 in step (6))
Number of passes: 999
――――――――――――――――――――――――――――――

  The ordering of database variables during the variable selection process is based on sensitivity analysis and chi-square analysis performed specifically on the new target output described in Example 1. The network trained for this example is identified as follows (the two nets have different random seeds).

[Table 17]
――――――――――――――――――――――――――――――――――――
Net trained on all databases: AFS01 and AFS02
Nets trained on the Endo presence subset: AFSEP1 and AFSEP2
――――――――――――――――――――――――――――――――――――

  After the variable selection process was completed for each variable and random seed combination, a set of eight networks was trained using the identified selected variables. Each of these networks for AFS01 and AFS02 variables are trained on a complete 510 recording database. Each of the networks for the AFSEP1 and AFSEP2 variables is trained for 291 records where the endo presence variable is positive. From these training runs, output consensus can be generated in an Excel spreadsheet to evaluate the performance of each network.

result.
The count of variables found in the reduced subset run is lower than in the run for all training examples. The normal performance of the network consensus was estimated using a five-part holdout method. Normal classification performance for AFS execution using all training examples was 77.22549%. The normal classification performance for the endo presence subset was 63.008621%. If all examples are classified as negative, the performance for all training examples should be 78.82% and the subset should be 65.29%. By changing the cutoff values for positive and negative classifications, better performance presented by these numbers can be achieved.

Conclusion.
The results of variable selection runs for all training examples and a subset of endo presence examples show that the size of training examples is important in determining important variables. Clearly, as the size of the training example increases, more variables will be considered important. This result can also be interpreted as an indication that more training data will improve the overall performance of the variable selection process and the consensus network used to build the diagnostic test.

<Example 13>
Improved variable selection, development of neural networks to predict pregnancy-related events, and fetal fibronectin testing performance.
Data were collected from over 700 test patients included in the clinical trials of the assay described in US Pat. No. 5,468,619. Variable selection was performed without fetal fibronectin (fFN) test data. The last network, designated EGA1 to EGA4, is trained with the variables shown in the table below.

  EGA1 to EGA4 represent neural networks used for variable selection. In EGA1, the variable selection protocol is implemented in a network architecture with eight inputs in the input layer, three processing elements in the hidden layer, and one output in the output layer. EGA2 is the same as EGA1 except that there are nine inputs in the input layer. EGA3 has seven inputs in the input layer, three processing elements in the hidden layer, and one output in the output layer. EGA4 is similar to EGA1 except that there are eight inputs in the input layer of EGA1.

  The selected variables are as follows.

  EGA = estimated gestational age.

Final consensus network performance.

EGA = estimated gestational age (less than 34 weeks); TP = true positive; TN = true negative; FP = false positive; FN = false negative; SN = sensitivity; SP = specificity; PPV = positive predictive value NPV = negative predictive value; OR = odds ratio (correct total / total correct answer); fFN = result from ELISA assay for fFN

  This result shows that network EGA4, a neural network that predicts parturition in less than 34 weeks, includes seven patient variables and the fFN ELISA assay, has much fewer false positives than the fFN ELISA assay. Furthermore, the false positive number dropped by 50%. Incorporating the fFN test into the neural network improved the performance of the fFN ELISA assay. All neural networks performed better than a single fFN test. Thus, the methods herein can be used to develop neural networks and other decision support systems that can be used to predict pregnancy related events.

<Example 14>
Train a consensus neural network for a specific subset of pat07 variables.
This example shows the results of a task designed to survey the contribution of the pat07 variable to pat07 performance and develop an endometriosis network using the minimum number of pat07 variables.

task:
1. Train the final consensus network using a combination of the following pat07 variables.
a. Subtract Endo history from all 14 (total of 13 variables)
b. Draw pelvic pain from all 14 (total of 13 variables)
c. Subtract dysmenorrhea from all 14 (total of 13 variables)
d. Subtract pelvic surgery from all 14 (total of 13 variables)
2. Train the final consensus network using other combinations of pat07 variables.
a. Endo history, pelvic pain, and dysmenorrhea b. 2. Endo history, pelvic pain, dysmenorrhea, and pelvic surgery history. Train the final consensus network using other combinations of pat07 variables indicated by the above results.

The method used.
Using the original patient database, training examples were generated for each combination of variables to be evaluated. These training examples include only those variables necessary for a given consensus run. TrainDos ^™ was used in batch mode to train a set of eight neural networks for each combination of variables to be evaluated. The network was trained using parameters similar to the pat07 training run. The only difference is the setting of a random number seed for each network. Each network was trained on a total of 510 recording databases. From these training runs, output consensus can be generated in an Excel spreadsheet to evaluate the performance of each network.

result.
Since these runs are the last training runs, the effect of removing the variable is seen, but does not give a clear indication that can be achieved by the holdout method.

Conclusion.
The result of variable selection execution for all training examples aimed at determining the contribution of a given set of variables is not as good as the evaluation method used in the variable selection process. The “holdout” method for evaluation with 5 divisions and 20 net consensus gives significantly better statistics for comparing variables.

<Example 15>
A method and apparatus for assisting in the diagnosis of endometriosis using a plurality of parameters suitable for analysis via a neural network (pat07).
FIG. 7 is a schematic diagram illustrating one embodiment of one type of neural network 10 trained on the form of clinical data used in a consensus network of multiple neural networks (FIG. 10). This structure is stored in digital form along with weight values and data processed by a digital computer. This first type neural network 10 includes three layers: an input layer 12, a hidden layer 14, and an output layer 16. The input layer 12 has 14 input preprocessors 17-30, each with a normalizer (not shown) that generates mean and standard deviation values and weights the clinical factors input to the input layer. Mean and standard deviation values are specific to network training data. The input layer preprocessors 17-30 are coupled to the first and second processing elements 48, 50 of the hidden layer 14 via paths 51-64 and 65-78, respectively, and the hidden layer processing elements 48, Each 50 receives a value or signal from each input preprocessor 17-30. Each path has a unique weight based on the training results for the training data. The unique weights 80-93 and 95-108 are non-linearly related to the output and are unique for each network structure and initial value of the training data. The final value of the weight is based on the initialization value assigned to network training. The combination of weights resulting from the training includes a functional device whose description expressed in weights produces the desired solution, or more specifically, a tentative indicator of the diagnosis of endometriosis.

  In the endometriosis test provided herein, the factors based on output used to train the neural network are: past history of disease, number of births, dysmenorrhea, age, pelvic pain, pelvis History of surgery, amount of smoking per day, history of drug treatment, number of pregnancy, number of miscarriages, abnormal PAP / dysplasia, pregnancy hypertension, genital warts, diabetes. These 14 factors have been determined to be the most influential (maximum sensitivity) set of the original set of over 40 clinical factors. (Other sets of influential factors have also been derived, see examples above).

  The hidden layers are biased by bias weights 94, 119 provided to processing elements 48 and 50 via paths 164 and 179. The output layer 16 includes two output processing elements 120 and 122. The output layer 16 receives input from both hidden layer processing elements 48, 50 via paths 123, 124, and 125, 126. The output layer processing elements 120, 122 are weighted by weights 110, 112 and 114, 116. Output layer 16 is biased by bias weights 128, 130 provided to processing elements 120 and 122 via paths 129 and 131.

  A provisional indicator of the presence or severity of endometriosis is the output pair of values A and B from the two processing elements 120,122. These values are always positive between 0 and 1. One indicator indicates the presence of endometriosis. The other indicator indicates the absence of endometriosis. The output pair A, B generally provides an effective disease index, but the consensus network of trained neural networks provides a more reliable index.

  Referring to FIG. 10, the last index pair C, D is based on an analysis of consensus of provisional index pairs from multiple, specifically eight, trained neural networks 10A-10H (FIG. 10). Each interim indicator pair A, B is fed to one of the two consensus processors 150, 152 via paths 133-140 and 141-148. The first consensus processor 150 processes all positive indicators. The second consensus processor 152 processes all negative indicators. Each consensus processor 150, 152 is an averaging device, ie simply forms a linear combination, such as an average, of a set of similar provisional indicator pairs A, B. The resulting reliability index pair is the desired result, and the input is a set of clinical factors for the test patient.

  FIG. 9 shows an exemplary processor element 120. Similar processors 48 and 50 have more input elements and the processor element 122 is nearly identical. A representative processor element 120 includes a plurality of weight multipliers 110, 114, 128 on each input path (herein, numbered 15, 16, or 3 as a whole and shown as part of the processor element 120). including. The weighted value from the weight multiplier is coupled to adder 156. The output of adder 156 is coupled to an activation function 158 such as an S-shaped transfer function or arctangent transfer function. The processor element can be implemented as dedicated hardware or in a software function.

  Sensitivity analysis can be performed to determine the relative importance of clinical factors. Sensitivity analysis is performed on a digital computer as follows. The trained neural network is run in forward mode (no training) for each training example (input data group for which true output is known or inferred). The network output for each training example is then recorded. Thereafter, each input variable is replaced with the average value of the input variables over all training examples, and the network is re-executed. The difference between the values of each output is then squared and summed (accumulated) to obtain an individual sum.

  This sensitivity analysis process is performed for each training example. Each result sum is then normalized according to a conventional process so that the normalized value is 1.0 if all variables contribute equally to a single result output. From this information, the normalized values can be ranked in order of importance.

  In the analysis of clinical data, the order of sensitivity of factors for this neural network system is: past history of illness, number of births, dysmenorrhea, age, pelvic pain, pelvic surgery history, amount of smoking per day, medication History, number of gestations, number of miscarriages, abnormal PAP / dysplasia, pregnancy hypertension, genital warts, diabetes.

  Certain neural network systems have been trained and found to be effective diagnostic tools. The neural network system shown in FIGS. 7 and 10 is described as follows.

[Table 18]
――――――――――――――――――――――――――――――
0. Bias 1. Age 2. Diabetes 3. 3. Pregnancy hypertension 4. Smoking amount per day Number of pregnancy 6. Number of births 7. Miscarriage 8 Genital warts9. Abnormal PAP / dysplasia 10. 11. History of endometriosis 11. Pelvic surgery history 13. History of drug treatment Pelvic pain14. Dysmenorrhea ――――――――――――――――――――――――――――――

  The weights in the order of identification instead of the order of sensitivity as described above are as follows for each of the eight first-type neural networks 10.

First neural network A

First neural network B

First neural network C

First neural network D

First neural network E

First neural network F

First neural network G

First neural network H

Normalized observations for the first type of neural network

  In addition, as provided herein, the results of biochemical tests, such as tests with an ELISA format test, are used to generate a trained increased neural network system that is relatively reliable with high sensitivity and specificity. A level can be created. Such a second type of neural network is shown in FIG. The numbers are the same as in FIG. 7 except that the node 31 of the input layer 12 and a pair of weights 109 and 111 are added. However, all weights in the network change when training with additional biochemical results. The exact weight set depends on the specific biochemical test training example.

  The training system provided herein can be used. Alternative training techniques can also be used (for example, “Use of an Artificial Neural Network for the Diagnosis of Myocardial Infarction” by Baxt, Annals of Internal Medicine 115, p.843 (December 1, 1991); “Improving the Accuracy of an Artificia 1 Neural Network Using Multiple Differently Trained Networks, Neural Computation 4, p. 772 (1992)).

  When evaluating test results, a high score correlates with the presence of a disease, a low score correlates with the absence of a disease, an extreme score increases reliability, while a moderate score indicates reliability. Noted that will reduce. The presence of endometriosis is indicated by an output of 0.6 or higher, and the absence of it is indicated by 0.4 or lower. Note also that a high relative score correlates with a high relative weight of the disease. The methods herein minimize the number of patients that require further procedures, often surgery, to establish the presence or severity of a disease state.

  Since modifications will be apparent to those skilled in the art, the present invention is intended to be limited only by the scope of the appended claims.

2 is a flow diagram for developing a patient history-based diagnostic test process. 2 is a flowchart for developing a biochemical diagnostic test. 2 is a flow diagram of a process for separating important variables. 2 is a flow diagram of a process for training one or a set of neural networks including variable partitioning. 2 is a flowchart for developing a biochemical diagnostic test. 3 is a flow chart for determining the effectiveness of a biochemical diagnostic test. FIG. 6 is a schematic diagram of a neural network trained based on a form of clinical data used for a consensus network of multiple neural networks. FIG. 4 is a schematic diagram of a second embodiment of a neural network trained on clinical data augmented with form test result data used for consensus of eight neural networks. It is the schematic of the processing element of each node of a neural network. FIG. 6 is a schematic diagram of a consensus network of eight neural networks using the first or second embodiment of the neural network. FIG. 3 is an exemplary interface screen of a user interface during a diagnostic endometriosis index.

Claims (48)

(A) means for providing a first set of n candidate variables and a second set of important selected variables that are initially empty;
(B) means for evaluating each variable by taking candidate variables one at a time and training the decision support system based on the variables coupled to the current set of important selected variables;
(C) When the best variable that gives the best performance of the decision support system is selected from the candidate variables, and the best candidate variable improves the performance compared with the performance of the important selected variable Means to add it to the set of important selected variables, remove it from the candidate set and continue the evaluation using means (b) above until the best candidate variable does not improve performance A computer system for variable selection.   The computer system according to claim 1, wherein said means (a) uses candidate variables obtained from a patient and comprising history data and / or biochemical data. A computer system that generates a test to aid diagnosis,
Means for selecting a set of important selected variables according to the computer system of claim 1;
A computer system comprising means for training a decision support system using a selected final set of important selected variables to generate a diagnostic test.   A computer system that generates a test to assist in the diagnosis evaluates the likelihood of a medical condition or disorder, evaluates the likelihood that a particular condition is ongoing or will occur in the future, or provides predetermined treatment The computer system according to claim 3, wherein the treatment according to the unit is selected or the effectiveness of the treatment is determined.   The computer system according to claim 4, wherein the condition is a condition related to pregnancy or endometriosis.   4. The computer system for generating a test to assist in the diagnosis assesses the presence or severity of a medical condition or determines a possible outcome from treatment for a given therapeutic unit. Computer system. A computer system that improves the effectiveness of diagnostic biochemical tests,
Means for selecting a set of important selected variables according to the computer system of claim 1;
A computer system with means to train a decision support system using important selected variables and a selected final set of biochemical test data to generate tests that are more effective than biochemical tests alone . A computer system that identifies biochemical tests to assist in the diagnosis of a fault or condition,
(A) means for selecting a set of important selected variables according to the computer system of claim 1;
(B) Identifying a set of biochemical test data and making a decision using a selected final set of important selected variables coupled to each element of the set of biochemical test data A means of training the support system and evaluating the performance of the resulting system;
(C) means for repeating training and evaluation until all elements have been used for training for each element of the set of biochemical test data;
(D) a computer system comprising means for selecting elements of a set of biochemical data resulting in a system operating at best performance. (A) means for providing a first set of n candidate variables and a second set of important selected variables that are initially empty; and (b) any candidate variables arbitrarily, or A means of ranking in order;
(C) Take m largest ranked variables one at a time, where m is from 1 to n, and make a decision based on the variables combined with the current set of important selected variables Means to evaluate each variable by training the support system;
(D) Of the m variables, select the best variable that gives the best performance of the decision support system and improve the performance compared to the performance of the selected variable where the best variable is important. If so, add it to the set of important selected variables, remove it from the candidate set, continue to evaluate by means (c), and compare the variables with the performance of the important selected variables. Means for evaluating by using means (e),
(E) a computer system for variable selection comprising means for determining whether all variables of the candidate set have been evaluated.   The computer system according to claim 3 or 9, wherein the candidate variable includes biochemical test data.   10. The computer system of claim 9, wherein the ranking means is based on sensitivity analysis or based on analysis including other decision support system based analysis.   The computer system of claim 9, wherein the ranking means is based on a process that includes statistical analysis.   10. The computer system according to claim 9, wherein the ranking means is based on a process including chi-square, regression analysis or discriminant analysis.   The computer system of claim 9, wherein the means for ranking is determined by a process that uses an expert, a rule-based system, a sensitivity analysis or a combination evaluation. The above sensitivity analysis
(I) means for determining an average observation value for each variable in the observation data set;
(Ii) means for selecting a training example and executing the example via a decision support system to generate an output value designated and stored as a normal output;
(Iii) Select the first variable in the selected training example, replace the observed value with the average observed value of the first variable, and execute the modified example in the forward mode in the decision support system Means for recording the output as a modified output;
(Iv) means for squaring the difference between the normal output and the modified output and accumulating the sum as a sum, wherein the sum for each variable is specified as a variable sum selected for each variable. The computer system according to 11 or 14.   The computer system according to claim 1, wherein the decision support system includes neural network consensus.   10. A computer system as claimed in claim 1 or 9, wherein the set of n candidate variables and the set of important selected variables are each stored in a computer.   4. The method of claim 3, further comprising means for training a final decision support system based on a completed set of important selected variables to generate a decision support system based test for the condition. Computer system.   The computer system according to claim 3, wherein the state is a state related to gynecology.   21. The computer system of claim 19, wherein the condition is selected from among infertility, pregnancy related events, and pre-eclampsia. A computer system that develops a decision support system based test to assist in diagnosing a medical condition, disease or disorder,
(A) means for obtaining observations from a group of test patients whose medical condition is known;
(B) means for classifying the observation results obtained by means (a) into a set of candidate variables having observation values and storing the observation values in a computer as an observation data set;
(C) means for selecting a subset of important selected variables from a set of candidate variables using the computer system of claim 1 or 9;
(D) using observation data corresponding to a subset of important selected variables, a second decision support system based system constitutes a decision support based diagnostic test for a condition, disease or disorder And a means for training the second decision support system. After collecting observations from a group of test patients and before training a second decision support based system,
Obtain test results collected from biochemical tests on at least some of the test patients whose status is known or suspected and classify them into a set of candidate variables, which are then candidate variables 24. The computer system of claim 21, further comprising means for adding to the first set.   Further comprising means for identifying one or more biochemical test data variables ending with a final subset of important selected variables, whereby the identified one or more biochemical test data variables are 23. The computer system of claim 22, useful as an indicator of a failure or condition.   24. A computer system according to any of claims 21-23, wherein the test assesses the presence or weight or treatment unit of a disease, disorder or other medical condition.   24. A computer system according to any of claims 21-23, wherein the test assists in determining a result resulting from the selected treatment.   24. A computer system according to any of claims 21-23, wherein the decision support system comprises a neural network and the final set comprises a consensus of neural networks.   24. A computer system according to any of claims 21-23, wherein a first subset of important selected variables is identified using a sensitivity analysis performed on a decision support based system or its consensus. .   24. A computer system according to any of claims 21 to 23, wherein the first decision support system comprises at least one neural network.   24. A computer system according to any of claims 21 to 23, wherein the second decision support system comprises at least one neural network.   24. The computer system of claim 23, further comprising means for developing a diagnostic biochemical test for one or more identified biochemical test data variables.   24. The method of claim 21-23, further comprising means for collecting additional observations from the patient and classifying them into a set of candidate variables, the candidate variables being then added to the first set of candidate variables. A computer system according to any one of the above. A computer system for developing new biochemical tests or identifying new disease markers,
24. The computer system of claim 23;
A means of identifying biochemical data variables that are important selected variables;
A computer system comprising: biochemical data from which variables are obtained or means for developing tests to detect disease markers.   The computer system according to claim 21 or 22, wherein the candidate variable includes biochemical test data.   The computer system of claim 21, wherein the ranking means is based on an analysis including sensitivity analysis or other decision support system based analysis.   The computer system of claim 21, wherein the means for ranking is based on a process that includes statistical analysis.   The computer system of claim 21, wherein the ranking means is based on a process including chi-square, regression analysis, or discriminant analysis.   The computer system of claim 21, wherein the means for ranking is determined by a process that uses an expert, rule-based system, sensitivity analysis or a combination of evaluations. Sensitivity analysis
(I) means for determining an average observation value for each variable in the observation data set;
(Ii) means for selecting a training example and executing the example via a decision support system to generate an output value designated and stored as a normal output;
(Iii) Select the first variable in the selected training example, replace the observed value with the average observed value of the first variable, execute the modified example in the forward mode in the decision support system, and output the output Means for recording as modified output;
(Iv) means for squaring the difference between the normal output and the corrected output and accumulating the sum as a sum, the sum for each variable being designated as the sum of the variables selected for each variable,
(V) means for using means (iii) and (iv) for each variable in the example;
(Vi) means for using examples (ii)-(v) for each example in the data set, wherein each sum of the selected variables represents the relative contribution of each variable to the decision support system output determination. 38. A computer system as claimed in claim 34 or 37.   40. The computer system of claim 38, further comprising: (vii) means for ranking the variables according to their relative contribution to the determination of the decision support system output.   The means for training the second decision support system is to execute a set of previously unused observation data through the second decision support system after training to provide a performance estimate for the medical condition indicator. 24. A computer system according to any one of claims 21 to 23, wherein a set of previously unused observation data is collected from a patient whose medical condition is known, comprising validation means for providing .   The means for training the second decision support system comprises means for dividing the observation data set into a plurality of sections including at least one test data section and a plurality of training data sections. The test data partition is used to provide a final performance estimate for the second decision support system after the training data partition is executed. 24. The computer system according to claim 21-23.   42. The computer system of claim 41, wherein the second decision support system comprises a plurality of neural networks each having a set of unique starting weights and a performance rating value.   43. The computer system of claim 42, wherein the final performance estimate is generated by averaging performance rating values for a plurality of neural networks.   24. A computer system according to any one of claims 21 to 23, wherein the observed values are obtained from results of patient medical history data and / or biochemical test results.   24. A computer system according to any of claims 21-23, wherein the condition is a pregnancy related condition or endometriosis. The disorder is endometriosis,
Candidate variables are
(I) Past endometriosis history, number of births, dysmenorrhea, age, pelvic pain, pelvic surgery history, smoking amount per day, medication treatment history, pregnancy count, miscarriage count, abnormal PAP / dysplasia Pregnancy, hypertension, genital warts and diabetes, or (ii) age, number of births, number of pregnancy, number of miscarriages, amount of smoking per day, past endometriosis history, dysmenorrhea, pelvic pain, abnormal 33. The computer system of claim 32, comprising at least four variables selected from PAP, pelvic surgery history, medication history, pregnancy hypertension, genital warts, and diabetes.   The computer system according to claim 46, wherein the decision support system comprises a neural network or a consensus of neural networks.   47. The computer system of claim 46, wherein at least five variables are selected.

Images