Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
  • G06Q40/08—Insurance

Definitions

• the .present invention relates to a system and process for underwriting insurance applications, and more particularly to a system and process for underwriting insurance applications based on the detection and identification of outlier applications.
• Classification is the process of assigning an input pattern to one of a predefined set of classes.
• Classification problems exist in many real-world applications, such as medical diagnosis, machine fault diagnosis, handwriting character recognition, fingerprint recognition, and credit scoring, to name a few. Broadly speaking, classification problems can be categorized into two types: dichotomous classification, and polychotomous classification. Dichotomous classification deals with two-class classification problems, while polychotomous classification deals with classification problems that have more than two classes.
• Classification consists of developing a functional relationship between the input features and the target classes. Accurately estimating such a relationship is key to the success of a classifier.
• Insurance underwriting is one of these classification problems.
• the underwriting process consists of assigning a given insurance application, described by its medical and demographic records, to one of the risk categories (also referred to as rate classes).
• a trained individual or individuals traditionally perform insurance underwriting.
• a given application for insurance (also referred to as an "insurance application”) may be compared against a plurality of underwriting standards set by an insurance company.
• the insurance application may be classified into one of a plurality of risk categories available for a type of insurance coverage requested by an applicant.
• the risk categories then affect the premium paid by the applicant, e.g., the higher the risk category, higher the premium.
• a decision to accept or reject the application for insurance may also be part of this risk classification, as risks above a certain tolerance level set by the insurance company may simply be rejected.
• Insurance underwriting often involves the use of a large number of features in the decision-making process.
• the features typically include the physical conditions, medical information, and family history of the applicant.
• insurance underwriting frequently has large number of risk categories (rate classes).
• the risk category of an insurance application is traditionally determined by using a number of rules/standards, which have the form of, for example, "if the value of feature x exceeds a, then the application can't be rate class C, i.e., the application has to be lower than C".
• rate class C i.e., the application has to be lower than C.
• underwriting standards cannot cover all possible cases and variations of an application for insurance.
• the underwriting standards may even be self-contradictory or ambiguous, leading to an uncertain application of the standards.
• the subjective judgment of the underwriter will almost always play a role in the process. Variation in factors such as underwriter training and experience, and a multitude of other effects can cause different underwriters to issue different, inconsistent decisions. Sometimes these decisions can be in disagreement with the established underwriting standards of the insurance company, while sometimes they can fall into a "gray area" not explicitly covered by the underwriting standards.
• a process for preparing an insurance application for underwriting based on a plurality of previous insurance application underwriting decisions includes receiving a request to underwrite the insurance application, assigning a risk classification to the insurance application, defining a set comprising at least one of the plurality of previous insurance application underwriting decisions, comparing the insurance application to the set, and designating the insurance application based at least in part on the comparison between the insurance application and the set and the risk classification assigned to the insurance application, where the designation is one of designating the insurance application as an outlier insurance application and designating the insurance application for underwriting.
• a process for preparing an insurance application for underwriting based on a plurality of previous insurance application underwriting decisions includes receiving a request to underwrite the insurance application, wherein the insurance application comprises at least one feature, assigning a risk classification to the insurance application, and defining a set comprising at least one of the plurality of previous insurance application underwriting decisions.
• the process further includes comparing the insurance application to the set, where comparing comprises comparing at least one feature of the insurance application to a corresponding feature in the at least one of the plurality of previous insurance application underwriting decisions in the set, comparing the classification assignment of the insurance application to the classification assignment of the at least one of the plurality of previous insurance application underwriting decisions in the set, designating the insurance application based at least in part on the comparison between the insurance application and the set and the risk classification assigned to the insurance application, where the designation is one of designating the insurance application as an outlier insurance application and designating the insurance application for underwriting.
• a computer readable medium having code for causing a processor to prepare an insurance application for underwriting based on a plurality of previous insurance application underwriting decisions.
• the medium includes code for receiving a request to underwrite the insurance application, code for assigning a risk classification to the insurance application, code for defining a set comprising at least one of the plurality of previous insurance application underwriting decisions, code for comparing the insurance application to the set, and code for designating the insurance application based at least in part on the comparison between the insurance application and the set and the risk classification assigned to the insurance application, where the designation is one of designating the insurance application as an outlier insurance application and designating the insurance application for underwriting.
• a computer readable medium having code for causing a processor to prepare an insurance application for underwriting based on a plurality of previous insurance application underwriting decisions includes code for receiving a request to underwrite the insurance application, wherein the insurance application comprises at least one feature, code for assigning a risk classification to the insurance application, code for defining a set comprising at least one of the plurality of previous insurance application underwriting decisions, and code for comparing the insurance application to the set, where the code for comparing comprises code for comparing at least one feature of the insurance application to a corresponding feature in the at least one of the plurality of previous insurance application underwriting decisions in the set, code for comparing the classification assignment of the insurance application to the classification assignment of the at least one of the plurality of previous insurance application underwriting decisions in the set.
• the exemplary embodiment further includes code for designating the insurance application based at least in part on the comparison between the insurance application and the set and the risk classification assigned to the insurance application, where the designation is one of designating the insurance application as an outlier insurance application and designating the insurance application for underwriting.
• a system for preparing an insurance application for underwriting based on a plurality of previous insurance application underwriting decisions includes means for receiving a request to underwrite the insurance application, means for assigning a risk classification to the insurance application, means for defining a set comprising at least one of the plurality of previous insurance application underwriting decisions, means for comparing the insurance application to the set, and means for designating the insurance application based at least in part on the comparison between the insurance application and the set and the risk classification assigned to the insurance application, where the designation is one of designating the insurance application as an outlier insurance application, and designating the insurance application for underwriting.
• a system for preparing an insurance application for underwriting based on a plurality of previous insurance application underwriting decisions includes means for receiving a request to underwrite the insurance application, wherein the insurance application comprises at least one feature, means for assigning a risk classification to the insurance application, and means for defining a set comprising at least one of the plurality of previous insurance application underwriting decisions.
• the system also includes means for comparing the insurance application to the set, where the code for comparing comprises means for comparing at least one feature of the insurance application to a corresponding feature in the at least one of the plurality of previous insurance application underwriting decisions in the set, means for comparing the classification assignment of the insurance application to the classification assignment of the at least one of the plurality of previous insurance application underwriting decisions in the set, and means for designating the insurance application based at least in part on the comparison between the insurance application and the set and the risk classification assigned to the insurance application, where the designation is one of designating the insurance application as an outlier insurance application, and designating the insurance application for underwriting.
• a system for preparing an insurance application for underwriting based on a plurality of previous insurance application underwriting decisions includes a receiver for receiving a request to underwrite the insurance application, a classifier module assigning a risk classification to the insurance application, a definition module for defining a set comprising at least one of the plurality of previous insurance application underwriting decisions, a comparison module comparing the insurance application to the set, and a designation module for designating the insurance application based at least in part on the comparison between the insurance application and the set and the risk classification assigned to the insurance application, where the designation is one of designating the insurance application as an outlier insurance application and designating the insurance application for underwriting.
• a system for preparing an insurance application for underwriting based on a plurality of previous insurance application underwriting decisions includes a receiver for receiving a request to underwrite the insurance application, wherein the insurance application comprises at least one feature, a classifier module for assigning a risk classification to the insurance application, a definition module for defining a set comprising at least one of the plurality of previous insurance application underwriting decisions, a comparison module for comparing the insurance application to the set, comparing at least one feature of the insurance application to a corresponding feature in the at least one of the plurality of previous insurance application underwriting decisions in the set, comparing the classification assignment of the insurance application to the classification assignment of the at least one of the plurality of previous insurance application underwriting decisions in the set, and a designation module for designating the insurance application based at least in part on the comparison between the insurance application and the set and the risk classification assigned to the insurance application, where the designation is one of designating the insurance application as an outlier insurance application and designating the insurance application for underwriting
• Figure 1 illustrates the architecture of a quality assurance system based on the fusion of multiple classifiers according to an embodiment of the invention.
• Figure 2 illustrates a table of an outer product using the function T(x,y) according to an embodiment of the invention.
• FIG. 3 illustrates the disjointed rate classes within the universe of rate classes according to an embodiment of the invention.
• Figure 4 illustrates the results of the intersections of the rate classes and the universe according to an embodiment of the invention.
• FIGS 5-9 illustrate the results of T-norm operators according to an embodiment of the invention.
• FIGS 10-14 illustrate the normalized results of T-norm operators according to an embodiment of the invention.
• Figure 15 illustrates a summary of the fusion of two classifiers according to an embodiment of the invention.
• Figure 16 illustrates a penalty matrix for a fusion module according to an embodiment of the invention.
• Figure 17 illustrates a summary of the fusion of two classifiers with disagreement according to an embodiment of the invention.
• Figure 18 illustrates a summary of the fusion of two classifiers with agreement and discounting according to an embodiment of the invention.
• FIGS 19-23 illustrate the results of T-norm operators according to an embodiment of the invention.
• Figures 24-28 illustrate the normalized results of T-norm operators according to an embodiment of the invention.
• Figure 29 illustrates a Dempster-Schaefer penalty matrix according to an embodiment of the invention.
• Figure 30 illustrates a comparison matrix according to an embodiment of the invention.
• Figure 31 illustrates fusion as a function of a confidence threshold for non-nicotine cases according to an embodiment of the invention.
• Figure 32 illustrates fusion as a function of a confidence threshold for nicotine cases according to an embodiment of the invention.
• Figure 33 illustrates a Venn diagram for fusion for non-nicotine cases according to an embodiment of the invention.
• Figure 34 illustrates a Venn diagram for fusion for nicotine cases according to an embodiment of the invention.
• Figure 35 is a flowchart that illustrates an outlier detector according to an embodiment of the invention.
• Figure 36 illustrates an outlier detector used in quality assurance according to an embodiment of the invention.
• Figure 37 illustrates a plot of two features for insurance applications according to an embodiment of the invention.
• Figure 38 is a flowchart that illustrates a tuning process according to an embodiment of the invention.
• Figure 39 is a flowchart that illustrates a classification process according to an embodiment of the invention.
• Figure 40 illustrates a comparison matrix according to an embodiment of the invention.
• Figure 41 illustrates a comparison matrix for a modified process according to an embodiment of the invention.
• FIG 42 is a flowchart that illustrates a multi-variate adaptive regression splines ("MARS") process according to an embodiment of the invention.
• MARS multi-variate adaptive regression splines
• Figure 43 is a histogram that illustrates decision boundaries according to an embodiment of the invention.
• Figure 44 illustrates a parallel network implementation according to an embodiment of the invention.
• Figure 45 illustrates a comparison matrix according to an embodiment of the invention.
• Figure 46 illustrates an annotated comparison matrix according to an embodiment of the invention.
• Figure 47 illustrates a performance of MARS models using five partitions according to an embodiment of the invention.
• Figure 48 illustrates minimum, maximum, and average performances of a network of MARS models according to an embodiment of the invention.
• Figure 49 illustrates a piecewise-continuous classification boundary in a feature space according to an embodiment of the invention.
• Figure 50 illustrates a multi-class neural network decomposed into multiple binary classifiers according to an embodiment of the invention.
• Figure 51 illustrates an architecture for a neural network classifier according to an embodiment of the invention.
• Figure 52 illustrates a confusion matrix before post-processing according to an embodiment of the invention.
• Figure 53 illustrates a confusion matrix after post-processing according to an embodiment of the invention.
• Figure 54 illustrates performance before post-processing according to an embodiment of the invention.
• Figure 55 illustrates performance after post-processing according to an embodiment of the invention.
• a system and process for underwriting of insurance applications that is suitable for use by a computer rather than by human intervention is described.
• the system and process make use of existing risk assignments made by human underwriters to categorize new applications in terms of the risk involved.
• One technical effect of the invention is to provide an automated process for consistent and accurate underwriting decisions for insurance applications. Various aspects and components of this system and process are described below.
• An aspect of the invention provides a system and process for fusing a collection of classifiers used for an automated insurance underwriting system and/or its quality assurance. While the design method is demonstrated for quality assurance of automated insurance underwriting, it is broadly applicable to diverse decision-making applications in business, commercial, and manufacturing processes.
• a process of fusing the outputs of a collection of classifiers is provided. The fusion can compensate for the potential correlation among the classifiers.
• the reliability of each classifier can be represented by a static or dynamic discounting factor, which will reflect the expected accuracy of the classifier.
• a static discounting factor represents a prior expectation about the classifier's reliability, e.g., it might be based on the average past accuracy of the model.
• a dynamic discounting represents a conditional assessment of the classifier's reliability, e.g., whenever a classifier bases its output on an insufficient number of points, the result is not reliable. Hence, this factor could be determined from the post-processing, stage in each model.
• the fusion of the data will typically result in some amount of consensus and some amount of conflict among the classifiers.
• the consensus will be measured and used to estimate a degree of confidence in the fused decisions.
• a fusion module (also referred to as a fusion engine) combines the outputs of several decision engines (also referred to as classifiers or components of the fusion module) to determine the correct rate class for an insurance application.
• a fusion module function may be part of a quality assurance ("QA") process to test and monitor a production decision engine (“PDE”) that makes the rate class assignment in real-time.
• QA quality assurance
• PDE production decision engine
• the fusion module and its components may review the decisions made by the PDE during the previous week. The output of this review will be an assessment of the PDE performance over that week, as well as the identification of cases with different level of decision quality.
• the fusion module may permit the identification of the best cases of application classification, e.g., those with high-confidence, high-consensus decisions. These best cases in turn may be likely candidates to be added to the set of test cases used to tune the PDE. Further, the fusion module may permit the identification of the worst cases of application classification, e.g., those with low-confidence, low-consensus decisions. These worst cases may be likely candidates to be selected for a review by an auditing staff and/or by senior underwriters.
• a fusion module may also permit the identification of unusual cases of application classification, e.g., those with unknown confidence in their decisions, for which the models in the fusion module could not make any strong commitment or avoided the decision by routing the insurance application to a human underwriter. These cases may be candidates for a blind review by senior underwriters.
• a fusion module may also permit an assessment of the performance of the PDE, by monitoring the PDE accuracy and variability over time, such as monitoring the statistics of low, borderline and high quality cases as well as the occurrence of unusual cases. These statistics can be used as indicators for risk management.
• a fusion module may leverage the fact that except for the unusual situation where all the components (e.g., models) contain the same information (e.g., an extreme case of positive correlation), each component should provide additional information. This information may either corroborate or refute the output of the other modules, thereby supporting either a measure of consensus, or a measure of conflict. These measures may define a confidence in the result of the fusion. In general, the fusion of the components' decisions may provide a more accurate assessment than the decision of each individual component.
• the fusion module is described in relation to various types of decision engines, including a case-based decision engine, a dominance-based decision engine, a multi- variate adaptive regression splines engine, and a neural network decision engine respectively.
• the fusion module may use any type of decision engine.
• the fusion module will support a quality assurance process for a production decision engine.
• the fusion module could be used for a quality assurance process for any other decision making process, including a human underwriter.
• a general method for the fusion process which can be used with classifiers that may exhibit any kind of (positive, neutral, or negative) correlation with each other, may be based on the concept of triangular norms ("T-norm"), a multi-valued logic generalization of the Boolean intersection operator.
• T-norm triangular norms
• the fusion of multiple decisions, produced by multiple sources, regarding objects (e.g., classes) defined in a common framework (e.g., the universe of discourse) consists of determining the underlying of degree of consensus for each object (e.g., class) under consideration, i.e., the intersections of their decisions. With the intersections of multiple decisions, possible correlation among the sources needs to be taken into account to avoid under-estimates or over-estimates. This is done by the proper selection of a T-norm operator.
• each model is assumed to be solving the same classification problem. Therefore, the output of each classifier is a weight assignment that represents the degree to which a given class is selected.
• the set of all possible classes referred to as U, represents the common universe of all answers that can be considered by the classifiers.
• the assignment of weights to this universe represents the classifier's ignorance (i.e., lack of commitment to a specific decision). This is a discounting mechanism that can be used to represent the classifier's reliability.
• the outputs of the classifiers may be combined by selecting the generalized intersection operator (e.g., the T-norm) that better represents the possible correlation between the classifiers.
• the generalized intersection operator e.g., the T-norm
• the assignments of the classifiers are intersected and a derived measure of consensus is computed.
• This fusion may be performed in an associative manner, e.g., the output of the fusion of the first two classifiers is combined with the output of the third classifier, and so on, until all available classifiers have been considered.
• the final output may be normalized (e.g., showing the degree of selection as a percentage). Further, the strongest selection of the fusion may be identified and qualified with its associated degree of confidence.
• a fusion module only considers weight assignments made either to disjoint subsets that contain a singleton (e.g., a rate class) or to the entire universe of classes U (e.g., the entire set of rate classes), as will be described in greater detail below.
• the degree of confidence C is computed among the classifiers and used to qualify the decision obtained from the fusion.
• the confidence measure and the agreement or disagreement of the fusion module's decision is used with the production engine's decision to assess the quality of the production engine.
• the application cases may be labeled in terms of the decision confidence. Thus, cases with low, high, or unknown confidence may be used in different ways to maintain and update the production engine.
• aggregation could be used, but would need to be associative, compensate for correlation, accommodate the discounting of classifiers, and generate a confidence measure of the combined decision, properties that are not directly satisfied.
• a particular case may be a Dempster-Shafer ("DS") fusion rule.
• the DS fusion rule requires the classifiers to be evidentially independent, i.e., the errors of one classifier must be uncorrelated with those of another one.
• the DS paradigm does not allow us to represent the ordering among the classes, typical of the insurance underwriting process. This ordering implies that there could be minor differences (such as the selection of two adjacent classes) and major differences (such as the selection of different classes at the extreme of their range). Therefore, the conflict between two sources is a gradual one, rather than a binary one (hit/miss).
• the classifiers' outputs are considered probability assignments.
• T-norms and T-conorms are the most general families of binary functions that satisfy the requirements of the conjunction and disjunction operators, respectively.
• T-norms T(x,y) and T-conorms S(x,y) are two-place functions that map the unit square into the unit interval, i.e., T(x,y): [0,l]x[0,l] - [0,1] and S(x,y): [0,l]x[0,l] -> [0,1]. They are monotonic, commutative and associative functions.
• T-norms and their dual T- conorms may be used.
• one family was selected due to its complete coverage of the T-norm space and its numerical stability.
• This family has a parameter p. By selecting different values of p, T-norms with different properties can be instantiated, and thus may be used in the fusion of possibly correlated classifiers.
• Arthur Dempster (A. P. Dempster, "Upper and lower probabilities induced by a multivalued mapping," Annals of Mathematical Statistics, 38:325—339, 1967, the contents of which are incorporated herein by reference) describes a calculus based on lower and upper probability bounds. Dempster's rule of combination describes the pooling of sources under the assumption of evidential independence. Glenn Shafer (G. Shafer, "A Mathematical Theory of Evidence", Princeton University Press, Princeton, New Jersey, 1976, the contents of which are incorporated herein by reference) describes the same calculus discovered by Dempster, but starting from a set of super-additive belief functions that are essentially lower bounds. Shafer derives the same rule of combination as Dempster.
• Enrique Ruspini E. Ruspini, "Epistemic logic, probability, and the calculus of evidence. Proc. Tenth Intern. Joint Conf. on Artificial Intelligence, Milan, Italy, 1987, the contents of which are incorporated herein by reference) goes on to describe a possible-world semantics for Dempster- Shafer theory.
• Fig. 1 illustrates the architecture of a quality assurance system based on the fusion of multiple classifiers according to an embodiment of the invention.
• These classifiers may include case-based reasoning model (described in U.S. Patent Application Serial Nos. 10/170,471 and 10/171,190, the contents of which are incorporated herein by reference), a multivariate adaptive regression splines model (hereinafter also referred to as "MARS”), a neural network model and a dominance-based model.
• MARS multivariate adaptive regression splines model
• the MARS, neural networks, and dominance-based models are all described in greater detail below.
• System 100 includes a number of quality assurance decision engines 110.
• the quality assurance decision engines 110 comprise a case-based reasoning decision engine 112, a MARS decision engine 114, a neural network decision engine 116, and a dominance-based decision engine 118. It is understood, however, that other types of quality assurance decision engines 110 could be used in addition to and/or as substitutes for those listed in the embodiment of the invention illustrated in Fig. 1.
• Post processing modules 122, 124, 126, and 128 receive the outputs from the various quality assurance decision engines 120 and perform processing on the outputs.
• the results of the post-processing are input into a multi-classifier fusion module 130.
• the multi-classifier fusion module 130 then outputs a fusion rate class decision 135 and a fusion confidence measure 140, which are input into comparison module 150.
• a fuzzy logic rule-based production engine 145 outputs a production rate class decision 147 and a production confidence measure 149, which are then input into comparison module 150. After a comparison has been made between the production rate class decision 147 and the fusion rate class decision 135, and the production confidence measure 149 and the fusion confidence measure 140, a compared rate class decision 151 and a compared confidence measure 153 are output by comparison module 150.
• An evaluation module 155 evaluates the case confidence and consensus regarding the compared rate class 151 and the compared confidence measure 153. Those cases evaluated as "worst cases" are stored in case database 160, and may be candidates for auditing. Those cases evaluated as "unusual cases” are stored in case database 165, and may be candidates for standard underwriting.
• case database 170 Those cases evaluated as "best cases" are stored in case database 170, and may be candidates for using with the test sets.
• the outlier detector and filter 180 may ensure that any new addition to the best-case database 170 will be consistent (in the dominance sense described below) with the existing cases, preventing logical outliers from being used.
• System 100 of Fig. 1 will now be described in greater detail below.
• the fusion process as disclosed in Fig. 1 includes four general steps. These steps are: (1) collection, discounting and postprocessing of modules' outputs; (2) determination of a combined decision via the associative fusion of the modules' outputs; (3) determination of degree of confidence; and (4) identification of cases that are candidates for test set, auditing, or standard reference decision process, via the comparison module 150. These steps will now be described in greater detail below.
• Object knowledge is the level at which each classifier is functioning, e.g., mapping input vectors into decision bins. Meta-knowledge is reasoning about the classifiers' performance over time. Discounting could be static or dynamic. Static discounting may be used a priori to reflect historical (accuracy) performance of each classifier. Dynamic discounting may be determined by evaluating a set of rules, whose Left Hand Side (“LHS") defines a situation, characterized by a conjunct of conditions, and whose Right Hand Side (“RHS”) defines the amount by which to discount whichever output is generated by the classifier.
• postprocessing may be used to detect lack of confidence in a source. When this happens, all the weights may be allocated to the universe of discourse, i.e., refrain from making any decision.
• each decision engine model will independently perform a post-processing step.
• the post processing used for the neural network model will be described.
• some post-processing techniques may be applied to the outputs of the individual networks, prior to the fusion process. For example, if the distribution of the outputs did not meet certain pre-defined criteria, no decision needs to be made by the classifier. Rather, the case will be completely discounted by allocating all of the weights to the entire universe of discourse U. The rationale for this particular example is that if a correct decision cannot be made, it would be better not to make any decision rather than making a wrong decision.
• the outputs as discrete membership grades for all rate classes, the four features that characterize the membership grades may be defined as follows, where N is the number of rate classes and / the membership function, i.e., the output of the classifier.
• Step 1 C ⁇ ⁇ ⁇ OR C > ⁇ - OR E > ⁇ -
• Step 2 D ⁇ ⁇ 4 AND S ⁇ 1 where ⁇ , ⁇ - , ⁇ - , and ⁇ 4 are the thresholds.
• the value of the thresholds is typically dataset dependent. However, in some embodiments, the value of the thresholds may be independent of the dataset.
• the value of the thresholds may be first empirically estimated and then fine-tuned by a global optimizer, such as an evolutionary algorithm. As part of this example, the final numbers are shown below in Table 1. Other optimization methods may also be used to obtain the thresholds.
• post-processing may be used to identify those cases for which the module's output is likely to be unreliable.
• the model assignment of normalized weights to rate classes may be discounted by assigning some or all of those weights to the universe of discourse U.
• the fusion module 150 may perform the step of determining a combined decision via the associative fusion of the decision engine models' outputs.
• any general method that can be used to fuse the output of several classifiers may be used.
• the fusion method may also be associative, meaning that given three or more classifiers, any two of the classifiers may be fused, then fusing the results with the third classifier, and so on, regardless of the order.
• the un-normalized fusion of the outputs of two classifiers Si and S 2 is further defined as:
• each element A(i,j) is the result of applying the operator T to the corresponding vector elements, namely J (i) and / (j), e.g.,
• Matrix 200 illustrates classes 202 and values 204 for vector I 1 and classes 206 and values 208 for vector I 2 .
• Intersection 210 illustrates one intersection between the vector I and vector I . Other intersections and representations may also be used.
• Triangular Norms (also referred to as "T-norms") are general families of binary functions that satisfy the requirements of the intersection operators. T-norms are functions that map the unit square into the unit interval, i.e., T: [0,l]x[0,l] -> [0,1]. T-norms are monotonic, commutative and associative. Their corresponding boundary conditions, i.e., the evaluation of the T-norms at the extremes of the [0,1] interval, satisfy the truth tables of the logical AND operator. As there appear to be an infinite number of T-norms, the five most representative T- norms for some practical values of information granularity may be selected. According to an embodiment of the invention, the five T-norms selected are:
• T (x, y) max(0, x+y- ⁇ )
• Extreme case of negative correlation T l 5 (x,y) max(0,jc ⁇ 5 +y° '5 - ⁇ f
• Partial case of negative correlation T 2 (x,y) x*y
• T3 the minimum operator
• T2 may be selected when the classifiers are uncorrelated (e.g., similar to the evidential independence in Dempster-Shafer).
• Tl may be used if the classifiers are mutually exclusive (e.g., extreme case of negative correlation).
• the operators T ⁇ .5 and T 2 . 5 may be selected when the classifiers show intermediate stages of negative or positive correlation, respectively.
• T-norms may also be used.
• these five T-norms provide a good representation of the infinite number of functions that satisfy the T-norm properties.
• T-norms are associative, so is the fusion operator, i.e.,
• Each element A(i,j) represents the fused assignment of the two classifiers to the intersection of rate classes n and rj.
• Fig. 3 illustrates that each rate class is disjointed and that U 300, is the universe of all (rate) classes.
• rate classes ri 302, r 2 304 to rarni 306 are shown. Given that the rate classes are disjoint, there are five possible situations:
• Fig. 4 depicts a chart 400 that illustrates the result of the intersections of the rate classes and the universe U, according to an embodiment of the invention.
• the chart demonstrates the intersection according to those situations set forth above, such that when situation (a) occurs, the results are tabulated in the main diagonal identified as 410 in Fig. 4. Further, when situation (b) occurs, the results are tabulated in the appropriate areas identified as 420 in Fig. 4. When situation (c) occurs, the results are tabulated in the appropriate areas identified as 430, while when situations (d) or (e) occur, the results are tabulated in the appropriate areas identified as 440 in Fig. 4.
• the intersection may be tabulated at 450, where the column for rl and the row for r2 intersect.
• the intersection of rl and r2 is the empty set ⁇ .
• the decisions for each rate class can be gathered by adding up all the weights assigned to them. According to the four possible situations described above, weights may be assigned to a specific rate class only in situation a) and d), as illustrated in Fig. 4. Thus, there will be:
• Weight (r) A(i,i)+ A(i,N+i)+ A(N+ ⁇ ,i)
• Chart 1500 illustrates the five classes 1510, the five T-norms 1520, and the fused intersection results 1530.
• the confidence in the fusion may be calculated by defining a measure of the scattering around the main diagonal. The more the weights are assigned to elements outside the main diagonal, the less is the measure of the consensus among the classifiers.
• the penalty function is used because the conflict may be gradual, as the (rate) classes have an ordering. Therefore, the penalty function captures the fact that the discrepancy between rate classes rj and r 2 is smaller than then the discrepancy between ri and rs .
• the shape of the penalty matrix P in Fig. 16 captures this concept, as I 600 shows that the confidence decreases non-linearly with the distance from the main diagonal.
• a measure of the normalized confidence C is the sum of element- wise products between A and P 1600, e.g. :
• A is the normalized fusion matrix.
• this fact may be captured by the confidence measure. For instance, consider a situation different from the assignment illustrated in Figs. 5-14, in which the classifiers agreed to select the first rate class. Now e.g., assume that the two classifiers are showing strong preferences for different rate classes, the first classifier is selecting the second rate class, while the second classifier is favoring the first class:
• the second classifier (S2) in the first example may be discounted:
• Fig. 18 The results of the fusion of I and I are summarized in Fig. 18 below.
• Summarization chart 1800 illustrates the classes 1810, T-norms 1820, the fused intersection results 1830 and the confidence measure 1840.
• the rate classes have a slightly lower weight (for T3, T2.5, T2), but the normalized confidence is higher than with respect to Fig. 15, as there is less conflict.
• Fusion matrices A are shown in the tables of Figs. 19-23, while the tables of Figs. 24-28 illustrate matrices A.
• a fusion rule based on Dempster-Shafer corresponds to the selection of:
• T-norm operator T(x,y) x*y
• Constraint b) implies the penalty matrix P 2900 illustrated in Fig. 29. Therefore, the two additional constraints a) and b) required by Dempster-Shafer theory (also referred to as "DS") imply that the classifiers to be fused must be uncorrelated (e.g., evidentially independent) and that there is no ordering over the classes, and any kind of disagreement (e.g., weights assigned to elements off the main diagonal) can only contribute to a measure of conflict and not, at least to a partial degree, to a measure of confidence. In DS, the measure of conflict K is the sum of weights assigned to the empty set. This corresponds to the elements with a 0 in the penalty matrix P 2900 illustrated in Figure 29.
• the normalized confidence C described above may be used as a measure of confidence, i. e. :
• the confidence factor C may be interpreted as the weighted cardinality of the normalized assignments around the main diagonal, after all the classifiers have been fused.
• An additional feature of the present invention is the identification of cases that are candidates for a test set, auditing, or standard reference decision process via the comparison module.
• the comparison module has four inputs. These inputs include the decision of the production engine, which according to an embodiment of the invention, is one of five possible rate classes or a no-decision (e.g., "send the case to a human underwriter"), i.e. :
• An additional input may comprise the decision of the fusion module, which according to an embodiment of the invention, is also one of five possible rate classes or a no- decision (e.g. , "send the case to a human underwriter"), i. e. :
• An additional input may comprise the degree of confidence in the production engine decision.
• the computation of the confidence measure is described in the U.S. Patent
• An additional input may comprise the degree of confidence in the fusion process.
• the normalized confidence measure C is C(FUS).
• the first test performed is to compare the two decisions, i.e., D(FLE) and D(FUS).
• Fig. 30 illustrates all the possible comparisons between the decision of the production engine and the fusion module.
• Label C shows that D(FLE) ⁇ D(FUS) and that both D(FLE) and D(FUS) indicate a specific, distinct rate class.
• label D shows that D(FLE) ⁇ D(FUS), and in particular, that the FLE made no automated decision and suggested to send the application to a human underwriter, while the Fusion module selected a specific rate class.
• a second test may be done by using this information in conjunction with the measures of confidence C(FLE) and C(FUS) associated with the two decisions.
• the performance of the decision engine may be assessed over time by monitoring the time statistics of these labels, and the frequencies of cases with a low degree of confidence.
• a stable or increasing number of label A's would be an indicator of good, stable operations.
• An increase in the number of label B's would be an indicator that the fusion module (with its models) needs to be retrained.
• An increase in the frequency of label C's or of cases with low confidence could be a leading indicator of increased classification risk and might warrant further scrutiny (e.g., auditing, retraining of the fusion models, re-tuning of the production engine).
• An increase in label D's may demonstrate that either the production engine needs re-tuning and/or the fusion modules needs retraining.
• An increase in label E's may demonstrate an increase in unusual, more complex cases, possibly requiring the scrutiny of senior underwriters.
• the candidates for the auditing process will be the ones exhibiting a low degree of confidence (C(FUS) ⁇ Tl), regardless of their agreement with the FLE and the ones for which the Fusion and the Production engine disagree, i.e., the ones labeled C.
• the candidates for the standard reference decision process are the cases for which the fusion module shows no decisions (labeled B or E).
• the fusion module may be implemented using software code on a processor.
• Fig. 31 illustrates the effect of changing the threshold Tl on the measure of confidence C , were 0 ⁇ C ⁇ l.
• Table 3100 display decisions 3110, confidence thresholds 3120 and the case distributions 3130 based on the confidence threshold 3120. Each column shows the number of cases whose measure of confidence is > Tl. As the threshold is raised, the number of "No Fusion Decision” increases. A "No Fusion Decision" occurs when the results of the fusion are deemed too weak to be used. When the threshold T is 1, no case is rejected on the basis of the measure of conflict. This leaves 36 cases for which no decision could be made. As the threshold is decreased, decisions with a high degree of conflict are rejected, and the number of "No Fusion Decisions" increases.
• the set B (4.5%) illustrates a lack of commitment and is a candidate for a review to assign an SRD.
• the set A may be a starting point to identify the cases that could go to the test set. However, set A may need further filtering by removing all cases that were borderline according to the FLE (i.e., C(FLE) ⁇ T2), as well as removing those cases whose fusion confidence was too low (i.e., C(FUS) ⁇ 1). Again T2 will be determined empirically, from the data.
• One component of a fusion module may be determining outlier applications.
• it may be desirable to detect all classification assignments to applications, such as insurance applications, that are inconsistent and therefore potentially incorrect.
• Applications that are assigned these inconsistent labels may be defined as outliers.
• the concept of outliers may extend beyond the realm of insurance underwriting and be intrinsic to all risk classification processes, of which the determination of the proper premium to cover a given risk (i.e., insurance underwriting) is just an example. Therefore, the ultimate domain of this invention may be considered risk classification, with a focus on insurance underwriting.
• the existing risk structure of the risk classification problem is exploited from the risk assignments made by the underwriters, similar to the dominance-based classifier described in greater detail below. But whereas the dominance based classifier uses the risk structure to produce a risk assignment for an unlabeled application, the outlier detector examines the risk structure to find any applications that might have been potentially assigned an incorrect risk assignment by the underwriter.
• the outlier detector may add to the rationality of the overall underwriting process by detecting globally inconsistent labels and bringing it to the attention of human experts.
• Many papers in the decision sciences demonstrate that in the presence of information overload, humans tend to be boundedly rational and often, unintentionally, violate compelling principles of rationality like dominance and transitivity.
• the outlier detector may attempt to counter these drawbacks exhibited by human decision-makers and make the decision-making process more rational.
• the risk assignments can be expected to be more optimal and consistent.
• the system may gain knowledge about exceptional decision rules, or additional features that are implicitly used by experts and which may be left unmentioned during the initial design stages of an automated system. This additional knowledge may be used to improve the performance of any automated system.
• the outlier detector may also act as a knowledge-eliciting module.
• the detection of outliers may further improve the performance and simplicity of other supervised classification systems, such as neural networks and decision-tree classifiers when used as the primary automated system. This is because the presence of global inconsistencies may add to the "non separability" of the feature space, which will often lead to either inferior learning, or very complicated architectures. As the outlier detector reduces the number of global inconsistencies, a cleaner, more consistent training set may be expected to result in a better learning, and by a simpler system. Hence, the outlier detector may improve the classification accuracy, and simplicity of other automated systems.
• the outlier detector uses the principle of dominance to capture the risk structure of the problem, the outlier detector has explanation capability to account for its results. This is because dominance is a compelling principle of rationality and thus the outliers detected by the system are rationally defensible.
• the functionality of the outlier detection system may be generic, so that it can be used to detect outliers for any preference-based problem where the candidates in question are assigned preferences based on the values that they take along a common set of features, and the preference of a candidate is a monotonic function of its feature-values. Therefore, the applicability of an outlier detection system transcends the problem of insurance underwriting, and can be easily extended to any risk classification process.
• the set of entities that have already been scored are stored as precedents, cases, or reference data points for use in future scoring or comparison with new candidates.
• the outlier detector can help in ensuring that any new candidate case that goes into the reference dataset will always lead to a globally consistent dataset, thereby ensuring that the reference dataset is more reliable.
• an outlier detector may exploit the existing risk structure of a decision problem to discover risk assignments that are globally inconsistent.
• the technique may work on a set of candidates for which risk categories have already been assigned (e.g., in the case of insurance underwriting, for example, this would pertain to the premium class assigned to an application).
• the system may find all such pairs of applications belonging to different risk categories, which violate the principle of dominance.
• the outlier detector attempts to match the risk ordering of the applications with the ordering imposed by dominance, and use any mismatch during this process to identify applications that were potentially assigned incorrect risk categories.
• automating an insurance underwriting process may involve trying to emulate the reasoning used by the human expert while assigning premium classes to insurance applications, and finding computable functions that capture those reasoning principles.
• the risk category of an application depends upon the values taken by the application along various dimensions, such as Body Mass Index (“BMI"), Cholesterol Level, and Smoking History. The values of the dimensions are then used to assign risk categories to insurance applications.
• BMI Body Mass Index
• An automated system would operate on these same features while trying to emulate the underwriter.
• the risk associated with an application changes with changes to the magnitude of the individual features. For example, assuming that all other features remaining the same, if the BMI of an applicant increases, the application becomes riskier.
• the outlier detector uses this knowledge to detect all such applications that do not satisfy the principle of dominance.
• Mortality risk is monotonically non-deceasing with respect to the first four variables, meaning that such risk can increase (or remain the same) as the values of the four variables increase.
• higher values in the fifth variable have a positive effect, as they decrease the mortality risk. Therefore, the fifth variable needs to be transformed into another variable.
• Other relationships between all the feature- values may also be used.
• application A dominates application B if and only if application A is at least as good as application B along all the features and there is at least one feature along which application A is strictly better than application B.
• the dominates relation may be based on the above definition of dominance. It is a trichotomous relation, meaning that given two applications A and B either application A dominates application B, application B dominates application A, or neither dominates the other. In the case where neither applicant dominates the other, each application may be better than its counterpart along different features. In such a case, application A and application B may be said to be dominance-tied. For example, as illustrated in Table 3 below, assume there are three applicants A, B, and C with the following feature values:
• application C dominates both application A and application B, since application C is at least as good (e.g., as low) as application A and application B along each feature, and moreover there is at least one feature along which application C is strictly better (e.g., strictly lower) than both application A and application B.
• application A and application B are dominance-tied since each is better (e.g., lower) than the other along some feature (application A has better cholesterol value while application B has better BMI value).
• the relation No_RiskierJThan(A,B) is true if the risk associated with applicant A (say ⁇ A) is no higher than that associated with applicant B (say ⁇ B ), i.e., No_RiskierJThan(A,B) ⁇ - ( ⁇ A ⁇ r B ).
• the dominates relation is a sufficiency condition for the No_Riskier_ Than relation. That is:
• An application may be considered an outlier based on one or more characteristics.
• application X and application Y are marked as outliers if application X dominates application Y, and application X is assigned a risk category that associates greater risk with application X compared to application Y.
• application X and application Y are marked as outliers if application Y dominates application X, and application Y is assigned a risk category that associates greater risk with application Y compared to application X.
• both application X and application Y are labeled as outliers, e.g., applications that have inconsistent assignments, and therefore potentially incorrect risk categories.
• outliers e.g., applications that have inconsistent assignments, and therefore potentially incorrect risk categories.
• An outlier module operates on a set A of applications, each of which has been assigned a risk category from one of the i possible categories.
• the system may be thought of as operating on a set of tuples ⁇ (A j ,x) ⁇ where x is the risk category assigned by the underwriter to application Aj.
• the process for outlier detection may be implemented in pseudocode as set forth below:
• outliers are pairs of tuples (A p ,x), (A q ,y) where A p dominates A q but r y ⁇ r x .
• Fig. 35 illustrates a flowchart for detecting outliers given a set of labeled applications.
• a tuple (Aj,x) is identified.
• a tuple (Aj,y) is identified at step 3520, where the rate class r y for tuple (Aj,y) is greater than the rate class r x .
• step 3560 If there is no other tuple (Aj,y) where r y > r x , a determination is made at step 3560 whether there is another tuple (Aj,x). If yes, the process returns to step 3510, while if not, the system ends at 3570.
• an outlier detector may be implemented in software code, and tested against a database of cases.
• an outlier detector may be tested against a database of approximately 2,900 cases.
• the outlier detector identified more than a dozen of subsets containing at least one inconsistency.
• Table 4 The results produced by the outlier detector in this example are shown in Table 4 below, along with a few relevant feature values.
• each row represents an insurance application for which the risk classification had already been determined, as shown in the first column.
• the risk class "BEST” is a lower risk class compared to the risk class "PREF.”
• a person classified in the "BEST” risk class will have to pay a lower premium than a person classified in the "PREF” class.
• the application indicated in the row first of Table 4 dominates the application of the second row.
• the risk classifications for the applications were reversed. This simple example illustrates the use of an outlier detector to obtain more consistent risk assignments.
• outlier detector 180 is shown after the fusion to insure that any new addition to the best-cases database would be dominance-consistent with the existing cases.
• Another potential use for the outlier detector is its application to the training-cases database used to train each of the decision engines used by the fusion module. This is a Quality Assurance step for the training data to insure that the training cases do not contain outliers (e.g., inconsistent cases in the dominance sense) so as to improve the learning phase of the four models illustrated (CBR, NN, MARS, Dominance) before they are used as run-time classifiers for the Quality Assurance process of the production engine.
• outliers e.g., inconsistent cases in the dominance sense
• an outlier detector 3610 and a training case-base 3620 may be positioned for quality assurance for CBR DE 3630, MARS DE 3640, NN DE 3650 and DOM DE 3660, the output of which is fed into a fusion module (not shown).
• the risk structure of an underlying problem may also be exploited to produce a risk category label for a given application, such as an insurance application.
• This risk classification can be assured to be accurate with a high degree of confidence.
• the application of a dominance classifier may also provide risk assignments having a high confidence measure.
• the relative accuracy of the system approaches 100%, thus minimizing the degree of mismatch between the risk assignment made by a human underwriter and the automated rate class decisions.
• a dominance classifier may have many of the advantages of the outlier detector. The principle of dominance is a compelling principle of rationality and thus the classification produced by the technique is rationally defensible.
• the output of this dominance-based classifier can be combined in a fusion module with the output(s) generated by other classifiers.
• a fusion process may be used for quality assurance of a production decision engine, to provide a stronger degree of confidence in the decision of the engine, in the case of consensus among the classifiers, or to suggest manual audit of the application, in the case of dissent among the classifiers.
• automating an insurance application underwriting process may essentially involve trying to emulate the reasoning used by a human expert while assigning premium classes to insurance applications, and finding computable functions that capture those reasoning principles.
• the risk category of an application depends upon the values taken by the application along various dimensions, such as, but not limited to, body mass index (BMI), cholesterol level, and smoking history.
• BMI body mass index
• An underwriter makes use of these values to assign risk categories to the applications.
• an automated system should operate on these same features while trying to emulate the underwriter.
• the manner in which the risk associated with an insurance application changes with changes to the magnitude of the individual features is also known. For example, when all other features in an insurance application remain the same, if the BMI of an applicant increases, the application becomes riskier.
• a dominance-based risk classification may use this knowledge to generate a risk category for a given application, such as an insurance application.
• a given application such as an insurance application.
• an assumption may be made that there is a monotonic non-decreasing relationship between all the feature-values and the associated risk (i.e., higher values imply equal-or-higher risk).
• a mirror image may be substituted, which will then satisfy this condition that lower values correspond to lower risk. This can be seen with reference to Table 3 regarding the outlier detector.
• Bounded within(B, ⁇ A,C ⁇ ) may be used when application B is bounded_within application A and application C, if and only if application A dominates application B and application B dominates application C, i.e.,
• LHS Left Hand Side
• the best, non-dominated subset for a given risk category may be defined as the one that contains all such applications that are not dominated by another application within that risk category. This may also be referred to as the Pareto-best subset.
• the worst, non-dominating subset for a given risk category may be defined as the one that contains all those applications that do not dominate even a single application in that risk category. This may also be referred to as the Pareto-worst subset.
• Fig. 37 may be referred to, which shows a plot of features fl 3710 and f2 3720 for 1000 insurance applications.
• the insurance applications are plotted as points in the 2-dimensional feature space. For simplicity, assume that these are the only two features used while assigning a risk category to the applications, and that the lower values along a feature correspond to a lower risk.
• circles denote the Pareto-best subset 3730 while the squares denote the Pareto-worst subset 3740.
• the circles take the lowest (e.g., the most desirable) values along both features while the squares take on the highest (e.g., the least desirable) values.
• each of the remaining insurance applications is such that at least one application represented by a circle dominates it, and it dominates at least one application represented by a square.
• there is at least one square S and one circle C such that Bounded_within(X, ⁇ C,S ⁇ ) is true.
• the production of the two subsets O and P is identical to the production of the dominance subset in discrete alternative decision problems.
• articles by Kung, Luccio, and Preparata (1975), and Calpine and Golding (1976), the contents of which are incorporated herein by reference present algorithms which can create these subsets in 0(n . log m ⁇ 1 (n)) time, where n is the number of candidates involved and m is the number of features along which the dominance comparisons are being done.
• an algorithm may produce the Dominance subset for a given set of alternatives X(n,m) where n is the number of candidates and m is the number of features used.
• Dominance(X,k) may be used to indicate the application of such an algorithm to the set X(n,m), where k is either +1 or -1, depending upon whether higher or lower feature values are desired to be considered as better during dominance comparisons.
• two principal modules, the tuning module and the classification module may be used.
• the tuning module may compute the Pareto-best and Pareto- worst subsets for each risk category.
• the Classification module may use the results of the tuning to classify new applications.
• the tuning module may use the Dominance algorithm to compute the Pareto-best and the Pareto-worst sets for each risk category. Given a set of applications A, such as insurance applications that have been partitioned into i different risk categories by the underwriter, tuning may use the pseudocode set forth below:
• Fig. 38 is a flowchart illustrating the steps involved in the tuning process according to an embodiment of the invention.
• each separate risk category is determined.
• a set of applications A is divided into the different risk categories.
• the Pareto-best subset of the applications within each risk category is computed.
• the Pareto-best subset is stored.
• the Pareto-worst subset of the applications within each risk category is computed.
• the Pareto-worst subset is stored, completing the tuning process at step 3812.
• the classification module may use the sets O and P from the tuning process to assign risk classifications to new applications.
• the classification module assigns a risk category to any new application by checking if a given application satisfies the Bounded within relation with respect to a Pareto-best, and another Pareto-worst application for a given rate class.
• each application in U is assigned a risk category. Assignment of a risk category may be carried out according to the pseudocode set forth below using the principle of dominance based risk classification:
• step 3906 if application Z is bounded, risk category r k is assigned to application Z at step 3914. The process then moves on to step 3912 to determine if there is another application Z.
• an application may be regarded as ambiguous by the system. No risk category is assigned to the application and the application is marked as unresolved.
• the comparison matrix 4000 illustrated in Fig. 40 provides an example of the performance of the system for a particular set of applicants.
• the system initially used the tuning set in order to compute the Pareto-best and the Pareto-worst subsets for each of the risk categories, which in this case are eight risk categories.
• the system may then classify a set of applications that were not in the tuning set. For these applications, risk assignments were also obtained from the human underwriters. This allows a comparison of the performance of the system with that of the experts using the comparison matrix.
• the classifier is 100% accurate, but may have a lower coverage, meaning that it does not provide a decision for a large number of cases.
• a different tradeoff may be achieved between relative accuracy and coverage of the system by allowing a minor relaxation of the classification rule used in the extreme rate classes (e.g., the best and worst rate class).
• the extreme rate classes e.g., the best and worst rate class.
• one type of modification makes use of the fact that since the risk categories are totally ordered, the principle of dominance-based risk classification can be relaxed for the best and the worst risk categories. This relaxation may therefore be expected to improve the coverage of the automated system.
• application X can be no riskier than application A which implies that:
• the relaxed principle of dominance based risk classification for the worst risk category can be seen by noting that if application A dominates application X such that the risk category assigned to application A is the worst risk category, say r wors t 5 then the risk category of application X is also r wors t; z.e.:
• the steps for classification remain the same except that during the rj-loop in Fig. 39, the application at hand is tested for the relaxed conditions described above respectively, and assigned the risk category accordingly if one of the conditions is satisfied.
• the comparison matrix 4100 shown in Fig. 41 illustrates performance of the dominance based risk classifier used after incorporating the relaxed conditions defined above, during classification of an applicant and tested against a case base of approximately 541 cases. Coverage of the classifier has improved, since 68 applicants that were initially marked as unresolved by the classifier are now assigned a risk category. Whereas the relative accuracy of the new classifier is not 100% like its counterpart, the number of misclassifications is relatively few. In other words, for a large gain in coverage the overall drop in accuracy obtained by the use of the modified classifier may be relatively minor. Thus, the relaxation conditions may permit a tradeoff between accuracy and coverage of the dominance based risk classifier.
• the earlier version of the classifier may be used.
• a network of multivariate adaptive regression splines (“MARS”) based regression models may be used to automate decisions in business, commercial, or manufacturing process.
• MARS multivariate adaptive regression splines
• such a method and system may be used to automate the process of underwriting an application as applicable to the insurance business.
• a MARS based system may be used as an alternative to a rules-based engine ("RBE").
• RBE rules-based engine
• a MARS model may not be as transparent as other decision engines (e.g., "RBE"), but may achieve better accuracy. Therefore, MARS may be used as an alternative approach for a quality assurance tool to monitor the accuracy of the production decision engine, and flag possible borderline cases for auditing and quality assurance analysis.
• a MARS module may be a regression-based decision system, which may provide the simplicity of implementation of the model since it is based on a mathematical equation that can be efficiently computed.
• a MARS module may facilitate the automation of the "clean case” (e.g., those cases with no medical complications) underwriting decision process for insurance products.
• a MARS module may be used for other applications as well.
• a MARS module may be used to achieve a high degree of accuracy to minimize mismatches in rate class assignment between that of an expert human underwriter and the automated system.
• the development of a parallel network of MARS models may use a set of MARS models as a classifier in a multi-class problem.
• the MARS module is described in the context of a method and system for automating the decision-making process used in underwriting of insurance applications. However, it is understood that the method and system may be broadly applicable to diverse decision-making applications in business, commercial, and manufacturing processes. Specifically, a structured methodology based on a multi-model parallel network of MARS models may be used to identify the relevant set of variables and their parameters, and build a framework capable of providing automated decisions. The parameters of the MARS-based decision system are estimated from a database consisting of a set of applications with reference decisions against each application. Cross-validation and development/hold-out may be used in combination with resampling techniques to build a robust set of models that minimize the error between the automated system's decision and the expert human underwriter.
• Fig. 42 is a flowchart illustrating a process for building a MARS module according to an embodiment of the invention.
• one or more applications are digitized. Digitization may include assuring that the key application fields required by the model to make a decision are captured in digital form by data entry.
• a case base is formed. Creating a case base may include assuring that the records corresponding to each application (e.g., case) are stored in a Case Base (CB) to be used for model construction, testing, and validation.
• preprocessing of cases occurs. Preprocessing may include one or more sub-steps. By way of example, preprocessing may involve location translation and truncation 4216, such as focusing on values of interest for each field. Further, preprocessing may involve range normalization 4217, such as normalizing values to allow for comparison along several fields. Preprocessing may also involve tag encoding 4218, where tag encoding includes augmenting a record with an indicator, which embodies domain- knowledge in the record by evaluating coarse constraints into the record itself.
• step 4220 partitioning and re-sampling occurs.
• five-fold partitioning may be used, with a stratified sampling within each rate class used to create five disjoint partitions in the CB.
• step 4225 generation of a development and validation set occurs. Each partition may be used once as a validation set, with the remaining four used as training sets. This may occur five times to achieve reliable statistics on the model performance and robustness.
• one or more model building experiments occur.
• Experiments with modeling may involve modeling techniques such as global regression and classification and regression trees ("CART") to determine rate classes from a case description. This may result with the selection of MARS as the modeling paradigm.
• CART global regression and classification and regression trees
• a parallel network of MARS models is implemented.
• implementation of networks of MARS models may be used to improve classification accuracy.
• the MARS model(s) described may be used as an input to a fusion module. Fusion of multiple classifiers based on MARS, Case-based Reasoning, Neural Networks, etc., may be used to improve classification reliability, as described above. The steps of the process illustrated in Fig. 42 will now be described in greater detail.
• a MARS model framework starts from a database of applications with the corresponding response variable (e.g., rate class decisions) provided for each. This may be done via cooperative case evaluation sessions with experienced underwriters, or may be accomplished via the reuse of previously certified cases.
• This database of applications is hereby referred to as a "Certified Case Base” or a "Case Base”.
• the characteristics of the certified case base closely match those of incoming insurance applications received in a reasonable time window i.e., they form a "representative sample.”
• the Case Base may form the basis of all MARS model development.
• pre-processing occurs.
• one of the first steps in the model development process is to study the data and its various characteristics. This process may ensure that adequate attention is given to the understanding of the problem space. Later, appropriate pre-processing steps may be taken to extract the maximum information out of the available data via a choice of a set of explanatory variables that have the maximum discriminatory power.
• one of the early findings was the fact that for most of the candidate variables that were chosen on the basis of experience and judgment of the human underwriting experts the decision boundary regions as indicated by the human experts start at the tail-end of the variable distribution.
• the decision problem may be to classify each applicant into risk classes, which are typically increasing in risk.
• the attribute denoted by the level of cholesterol in the blood of an individual may be considered. It is a known fact that a cholesterol level below 220 can be treated as almost normal. This suggests that in cases where the cholesterol level is at a certain level, such as up to about 240 at demarcation 4302, the human expert does not perceive a significant risk due to this factor. Thus, all cases with a cholesterol reading below this threshold can be grouped into a single class, e.g., "Class 1," 4304 and the members in this class would not consequently impact the response variable (e.g., the rate class decision). As shown, a cholesterol level value of 240 is close to the 75 quantile 4306 of the distribution, while the value of 270 is in the 90 th quantile range 4308.
• One of the sub-steps may include location transformation and truncation 4216.
• a location transformation may be considered for all variables that exhibit the above property.
• Each variable may be transformed by subtracting out its normal value. This is realized by combining the knowledge of human experts as well, since for the majority of the attributes that are health related, there are well-documented and published normal thresholds.
• the values of the variable may be saturated after a location transformation.
• the positive values may be considered, e.g. :
• Another sub-step may involve "tag"-encoding 4218.
• a specialized set of variable encoding may also be used to extract the maximum information out of the decision space.
• This encoding may be referred to as the "tag.”
• the tag is essentially an ordinal categorical variable developed from a collection of indicators for the various decision boundaries as defined by human experts. These indicators are evaluated for each relevant variable in the collection. The maximum of the individual indicators over the collection of variables results in the final "tag.” For example, assume that there are four key variables (out of a larger number of fields in the case) that are highlighted by actuarial studies to determine mortality risk. Since the same studies indicate the critical thresholds that impact such risk, there is no reason to re-learn those thresholds.
• Table 5 illustrates four variables: Nicotine History (NH), Body Mass Index (BMI), Cholesterol Ratio (Choi. Rat.), and Cholesterol Level (Choi. Lev.), and four groups of rules, one for each variable.
• Nicotine History NH
• Body Mass Index BMI
• Cholesterol Ratio Choi. Rat.
• Cholesterol Level Choi. Lev.
• the value of the tag starts with a default of 1 and is modified by each applicable rule set. A running maximum of the tag value is returned at the end, as the final result of tag.
• C) Tag is determined by the MAX of the values determined by each of the four rule sets
• a tag may provide a utilization of the available human expert knowledge to obtain a boost in accuracy.
• the models were built with and without the inclusion of the specialized "tag" variable and found that inclusion of the tag results in an improvement in accuracy by about 1-2% on average.
• a stratified sampling methodology may be used to partition the data set into five equal parts. The stratification was done along the various rate classes to ensure a consistent representation in each partitioned sample. Further, a simple re-sampling technique may be used based on reusing each partition by taking out one part (done five times without replacement) as a holdout and recombining the remaining four and using it as a development sample to build a complete set of MARS models. This may be done five times, as mentioned earlier.
• model-building experiments are performed.
• a variety of exploratory regression models may be built and trained on the CB development sets. Further, their classification accuracy may be tested and validated on the CB validation sets.
• a parallel-network of MARS models may evolve and develop from a global regression model and a classification and regression trees ("CART") model, and allows the use of MARS in the framework of a multi-class classification problem.
• CART classification and regression trees
• the response variable is a polychotomous categorical variable, i.e., a variable that can take values from a set of labels (e.g., "Preferred Best,” “Preferred,” “Select,” “Standard Plus,” “Standard”).
• a risk metric may be obtained such as from an actuarial department of the insurance company. This allows the mapping of the categorical values to numerical values (e.g., reflecting mortality risk) and treating the response variable as a continuous one in order to fit a global multivariate linear regression. Using this method, a moderate fit to the data is obtained. However, the maximum accuracy achieved was about 60%, far from the desired accuracy level of above 90%.
• a CART based model may be built using the data. To maintain robustness and to avoid the possibility of overfitting the model, it may be necessary to minimize the structural complexity of the CART model. This approach yielded a CART tree with about 30 terminal nodes. Its corresponding accuracy level was substantially better than the global regression and was about 85%. Increasing the accuracy for the training sets would have resulted in deeper, more complex trees, with larger number of terminal nodes. Such trees would exhibit overfitting tendencies and poor generalization capabilities, leading to low accuracy and robustness when evaluated on the validation sets.
• a parallel-network of MARS models is implemented.
• one issue involved the difficulty of global models to incorporate the jumps in decision boundaries of majority of the variables in an extremely small bounded range.
• the decision boundaries begin only after the 75 quantile value of the explanatory variable, the shift over all other decision variables usually occur by the 95 quantile.
• This issue may be addressed in a number of ways. According to one approach, "tag" encoding as explained above helps the MARS search algorithm to find the "knots" in the right place.
• a parallel network arrangement is a collection of MARS models, each of which solves a binary, or two-class problem.
• This may take advantage of the fact that the response variable is ordinal e.g., the decision classes being risk categories are increasing in risk.
• the approaches to these issues should not be considered as limitations of the methodology presented here, but rather a property explored in order to achieve better results.
• the above case generalizes to handle problems where the response may not be ordinal.
• An advantage of the order of the response variable may be taken by building two models each for every rate class, except the boundary classes, with one model for each side. For easier reference, the two models may be referred to as the left model and the right model. Fig.
• a population 4402 is divided into non-smoking applications 4404, non-underwritten applications 4406 and nicotine applications 4408.
• the "Preferred" class has been broken down into a "Preferred Left” model 4410 and "Preferred Right” model 4412.
• the results are then input into the aggregation module 4416, which aggregates all results from the binary classifiers and selects the rate class that best fits a given application. For example, for the rate class "Preferred,” two models are built which estimate class membership value.
• the "Left” model distinguishes all preferred cases from cases of classes, which are to the left of preferred while the "Right” model does the opposite.
• the final class membership value may be the minimum of these two membership values obtained. Further, in the general case where there is no known order amongst classes, the Left/Right models may collapse into a single model providing with one estimated membership value.
• the MARS methodology may be adapted to handle logistic regression problems in the classical sense. Such an adaptation would need an adjustment of the lack-of-fit ("LOF") criteria to be changed from least squares to logistic.
• LEF lack-of-fit
• logistic regression procedure is in itself a likelihood maximization problem that is typically solved by using an iteratively re- weighted least squares (“IRLS”) algorithm or its counterparts.
• IRLS iteratively re- weighted least squares
• the viability of MARS may depend on the fast update criteria of the least squares LOF function, which an IRLS logistic estimation would generally prohibit.
• an approximation may be made to use the final set of MARS variables back into a SAS logistic routine and refit.
• this is an approximation because if one could ideally use logistic LOF function, then one could have derived the optimal set of logistic candidate variable transforms.
• a re-fit process may still achieve the same degree of fit and provide model parsimony in some of the subset models built.
• the logistic function is a (0,1) map, this gives class membership values that can be treated as probabilities.
• a MARS module may be implemented with software code in SAS and using MARS, where the code has been trained and tested using the five-fold partitions method described above.
• Fig. 45 illustrates a comparison matrix 4500 (with a dimensionality of kx k), whose k columns contain the set of possible decisions available to the classifier, and whose k rows contain the correct corresponding standard reference decision, can describe a classifier's performance on a given data set, is illustrated in Fig. 45.
• 4602 refers to the total number of agreements between the classifier and the standard reference decisions for non-smokers
• 4608 refers to the total number of agreements between the classifier and the standard reference decisions for smokers.
• the notations 4604 and 4606 refer to the total number of disagreements between the classifier and the standard reference decisions for non- smokers
• 4610 and 4612 refer to the total number of disagreements between the classifier and the standard reference decisions for smokers.
• 4614 refers to the total number of agreements not to make a decision and send the case to UW (e.g., underwriter) and notations 4616 and 4618 refer to the total number of disagreements not to make a decision and send to UW.
• UW e.g., underwriter
• the matrix depicted in Fig. 46 may be used to illustrate the performance measures used in the evaluation of the classifiers.
• ⁇ ml + m2 + m3 + m4 + m5 + m6 + m7 + m8 + m9.
• Three measures of performance for the classifier may be used, where M(i,j) is a cell in the matrix shown in Fig. 45:
• coverage may be redefined as:
• An addition performance measure may include:
• Relative Accuracy the total number of correct decisions made by the classifier as a percentage of the total number of decisions made, i.e. :
• Relative Accuracy ⁇ M(i , / ⁇ ⁇ ⁇ ' ⁇
• a confidence metric for the classifier output, one could adjust a confidence threshold to achieve various tradeoffs between accuracy and coverage. At one extreme, one could have a very low tradeoff, accepting any output (this would yield 100% coverage but very low accuracy). At the other extreme, one could have very high confidence thresholds. This would drastically reduce coverage but increase relative accuracy.
• the results of networks of MARS (or Neural Networks, as described below) models could also be post-processed to establish an alternative confidence metric that could be used to achieve other tradeoffs between accuracy and coverage.
• the tables set forth in Fig. 47 describe the performance of the network of MARS models on each of the five partitions. For each partition, the global and relative accuracy is listed, with the corresponding coverage. The results are shown with and without post-processing.
• Partition 1 Partition 1, 4710, Partition 2, 4720, Partition 3, 4730,
• Partition 4, 4740 and Partition 5, 4750 shows the performance results of the network of MARS models applied to 80% of the data used to build the model (training set 4760) and 20% of the data that was withheld from the model construction (validation set 4770).
• the tables in Fig. 48 summarize the minimum 4810, maximum 4820, and average 4830 results of applying the network of MARS models to the five partitions.
• MARS may include overfit and cost-complexity pruning, cross- validation, and multi-collinearity.
• MARS is essentially a recursive-partitioning procedure. The partitioning is done at points of the various explanatory variables defined as "knots" and overall optimization is achieved by performing knot optimization over the lack-of-fit criteria.
• MARS employs a two-sided power basis function of the form:
• 't' is the knot around which the basis is formed. It may be important to use an optimal number of basis functions to guard against possible overfit.
• an experiment may be performed with one dataset by starting from a small number of maximal basis functions and building it up to a medium size number and use the cost-complexity notion developed in CART methodology and deployed in MARS to prune back and find a balance in terms of optimality which provides an adequate fit.
• the use of cost-complexity pruning revealed that 25-30 basis functions were sufficient.
• Another important criteria which affects the pruning is the estimated degrees of freedom allowed. This may be done by using ten-fold cross validation from the data set for each model.
• MARS multi-collinearity
• MARS does provide a parameter that penalizes the separate choice of correlated variables in a downstream partition. MARS then works with the original parent instead of choosing other alternates. According to an embodiment of the invention, a medium penalty may be used to take care of this problem.
• EA evolutionary algorithms
• Neural networks may be advantageous, as they can approximate any complex nonlinear function with arbitrary accuracy (e.g., they are universal functional approximators). Neural networks are generally non- parametric and data-driven. That is, they approximate the underlying nonlinear relationship through learning from examples with few a priori assumptions about the model. In addition, neural networks are able to provide estimates of posterior probabilities. Such posterior probability values may be useful for obtaining the highest possible decision accuracy in the classifier fusion or other decision-making processes.
• neural networks can be broadly categorized into two main classes, i.e., feed-forward and recurrent (also called feed back) neural networks. Among all these types, multiple-layer feed-forward neural networks are often used for classification. Neural networks can be directly applied to solve both dichotomous and polychotomous classification problems. However, it is generally more accurate and efficient when neural networks are used for two-class (e.g., dichotomous) classification problems. As the number of classes increases, direct use of multi-class neural networks may encounter difficulties in training and in achieving the desired performance.
• insurance underwriting problems may often involve the use of large numbers of features in the decision-making process.
• the features typically include the physical conditions, medical information, and family history of the applicant.
• insurance underwriting frequently has a large number of risk categories (e.g., rate classes).
• the risk category of an application is traditionally determined by using a number of rules/standards, which often have the form of "if the value of feature x exceeds a, then the application can't be rate class C, i.e., has to be lower than C".
• These types of decision rules, 4930 and 4940 in Fig 49 "clip" the decision surface.
• Decision rules interpreted and used by a human underwriter may form an overall piecewise-continuous decision boundary, as shown in the graph of Fig. 49.
• a neural network may need to deal with a large number of features and target classes.
• the large number of features and high number of target classes call for a high degree of complexity of neural network (“NN") structure (e.g., more nodes and more parameters to learn, i.e. higher Degrees of Freedom (DOF).
• NN neural network
• DOF Degrees of Freedom
• Such complex NN structures may require more training data for properly framing the network and achieving reasonable generality (performance). However, sufficient data may be difficult to obtain. Even with sufficient data, the complex neural network structure requires enormous training time and computational resources.
• complex NN structures tend to have more local minima, and thus, training is prone to fall into local minima and fails to achieve global minimization. As a result, it usually difficult to achieve a desired performance for a neural network with complex structure.
• Another issue to be addressed involves incorporating domain knowledge into the neural network classification process.
• the discrete rules that human underwriters use for risk category assignment form an overall piecewise- continuous decision boundary in the feature space and neural networks may have difficulty learning the decision boundary due to the insufficient data points being available.
• One way to alleviate the difficulty and improve the performance of the neural network may be to directly incorporate the rules into the neural network model and use these rules as additional information to "guide" network learning.
• One aspect of the present invention is related to a method and system of improving the performance of neural network classifiers, so that the neural network classifier can perform automated insurance underwriting and its quality assurance with a level of accuracy and reliability that is comparable to the rule-based production decision engine.
• this invention improves the performance of classifiers by decomposing a multi-class classification problem into a series of binary classification problems.
• Each of the binary classifiers may classify one individual class from the other classes and the final class assignment for an unknown input will be decided based on the outputs of all of the individual binary classifiers.
• this invention incorporates the domain knowledge of the human underwriter into a neural network design.
• the domain knowledge represented by a number of rules, may be integrated into a classifier by using an auxiliary feature, the value of which is determined by the rules.
• this invention may also analyze the outputs of the individual binary classifiers to identify the difficult cases for which the classifier cannot make a solid decision. To reduce misclassification rate, these difficult cases may then be sent to a human underwriter for further analysis.
• a single neural network contains multiple output nodes.
• decomposing the multi-class classifier into multiple binary classifiers may solve a multi-class classification problem.
• a hypothetical life insurance company has risk categories "Catl”, “Cat2”, “Cat3”, “Cat4", and "Cat5".
• a rating of "Catl” is the best risk, while "Cat5" is the worst.
• the concept of the multi-class classifier decomposition used in this invention can be illustrated in the example of Fig. 50.
• Each binary classifier (5010, 5020, 5030) is for one class and is trained to classify the specific class (the "class") and the rest of the classes combined (the "others').
• the training set is relabeled "1" for the data points in the "class” group and "0" for the data points in the "others” group.
• each of the binary classifiers determines the probability that the new case belongs to the class for which the binary classifier is responsible. Therefore, the output of the neural network is a number in the [0,1] interval.
• the final class for the new input case is assigned by the MAX decision rule 5040. For example, an application may receive a "0.6 and a 1" in the Cat3 and Cat4 categories, respectively, and a "0" in the Catl, Cat2, and Cat5 risk categories. The MAX decision rule 5040 may then select the Cat4 risk category.
• the neural network is multiple-layer feed-forward in type and has one hidden layer.
• using different neural network types with more than one hidden layer may be explored for obtaining better performance.
• the current invention is not limited to one hidden layer feed-forward neural networks. Instead, the method may work equally well for multiple numbers of hidden layers.
• domain knowledge may be integrated into neural network learning by representing the knowledge with an auxiliary feature. The domain knowledge may be first represented by a series of rules.
• a typical rule has the following format (once again using the afore-mentioned five hypothetical rate classes): "If the applicant's cholesterol level exceeds 252, he does not qualify for rate class CI, i.e., the best rate class for him is C2".
• this rule can be expressed in a general IF-THEN rule as follows.
• the classifier design process for a neural network classifier may comprise data preprocessing, classifier design and optimization, and post-processing. These three aspects are described in greater detail below.
• Data preprocessing may include range normalization and feature extraction and selection.
• range normalization is a process of mapping data from the original range to a new range. Normalization may be generally problem specific. However, it is often done either for convenience or for satisfying the input requirements of the algorithm(s) under consideration.
• one purpose of normalization is to scale all features the classifier is using to a common range so that effects due to arbitrary feature representation (e.g., different units) can be eliminated.
• some classifiers, such as neural networks require a range of input to be normalized.
• range normalization One way to normalize data is range normalization.
• the feature value is divided by its range, i.e., the difference between the maximum and the minimum of the feature value.
• the normalized values v will be in the range of [0, 1].
• the range normalization requires knowing the minimum and the maximum values of the data. The greatest advantage of this normalization is that it introduces no distortion to the variable distribution, as the instance values and their corresponding normalized values have a linear relationship. That is, given two instance values with the first being twice the second, when they are normalized the first normalized value will still be twice the second normalized value. This is why range normalization is also called linear scaling or linear transformation.
• Another type of data preprocessing may involve feature extraction/selection. For example, raw data is placed within a 20-column spreadsheet. The first column is the applicant ID number and the second column is the rate class. Columns 3 through 20 are the attributes/variables/features for the applicant. Instead of directly using the 18 original features, two new features are derived. The first derived feature is the body mass index ("BMI"). Underwriter experience has shown that the BMI has more discriminating power in classification. The second derived feature, tag, is used to represent the domain knowledge in neural network training. The two derived features are further described below.
• BMI body mass index
• BMI is defined as ratio of weight in kilogram and the height squared in meters. Let wt he the weight in pounds and Ht be the height in inches. BMI can be expressed as:
• domain knowledge is incorporated into the neural network classifier by using an artificial feature, such as tag.
• the tag feature may take different values based on a set of rules that represent the domain knowledge.
• the five family history features are condensed and represented by two features, FH1 and FH2. While the FH1 feature has the binary values of 0 or 1, FH2 has the triple values of 0,1, and 2.
• the values of FH1 and FH2 are determined by the following rules, where the terms age_sib_card_canc_diag, age_moth_card_canc_diag, age_fath_card_canc_diag, age _moih_card death, agejath_card_death respectively correspond to the age when a sibling of the applicant was diagnosed with a cardiac or cancer disease, the age when the mother of the applicant was diagnosed with a cardiac or cancer disease, the age when the father of the applicant was diagnosed with a cardiac or cancer disease, the age when the mother of the applicant died due to a cardiac disease, and the age when the father of the applicant died due to a cardiac disease. For a given applicant, one or more of these terms may be not applicable.
• THEN FH ! is 1.
• domain knowledge may be represented by a set of rules.
• a typical rule may have the following format (once again using the afore-mentioned five hypothetical rate classes): "If the applicant's cholesterol level exceeds 252, he does not qualify for rate class CI, i.e., the best rate class for him is C2".
• this rule can be expressed in a general IF-THEN rule as follows:
• x,. is the i"' feature
• t tJ is the j"' threshold of the i' h feature
• C . is the j' b rate class.
• a vector with binary number "0" or "1" may be used to represent the consequent part of the IF-THEN rule. For example, [0, 1, 1, 1, 1] means the best rate class of C 2 while [0, 0, 0, 1, 1] means the best rate class of C 4 .
• the auxiliary feature takes integer numbers ranging from one
• the value of the auxiliary feature the number of ones in the vector V. END of all data points
• the auxiliary feature may be treated as a regular feature and included into the final feature set.
• the neural network may then be trained and tested with the final feature set. Because of the additional information provided by the auxiliary feature, the neural network may be "guided" during learning to more quickly find the piecewise continuous decision boundary, which not only reduces the training time and efforts, but may also improve the classification performance of neural network classifier.
• Additional features that may be used for neural network classifier design include, but are not limited to, tag, BMI, diastolic and/or systolic blood pressure readings, cholesterol level, cholesterol ratio, various liver enzymes, such as SGOT (Serum Glutamic Oxaloacetic Transaminase), SGPT (Serum Glutamic Pyruvic Transaminase), GGT (Galactan Galactosyl Transferase), nicotine use history, and various aspects of family history.
• SGOT Serum Glutamic Oxaloacetic Transaminase
• SGPT Serum Glutamic Pyruvic Transaminase
• GGT Galactan Galactosyl Transferase
• a three-layer feed-forward neural network with back propagation learning may be used.
• Two separate models may be used for nicotine and non-nicotine cases, respectively.
• rate classes e.g., "Preferredj ⁇ ic,” “Standardplus_nic,” and Standardjriic
• non-nicotine cases may have five rate classes, e.g., "Best,” “Preferred,” “Select,” “Standardplus,” and “Standard.”
• Both models are multiple-class classifiers.
• a neural network with multiple output nodes may be a typical design for multiple-class classifiers where each of the neutral network output nodes corresponds to each class.
• neural networks with multiple output nodes may have a large number of weights and biases, and thus require a large training data set and more training time for properly training the network.
• multiple binary neural networks may be used to perform the multiple-class classification. Using multiple binary-networks may reduce the complexity of the network, thus reducing the training time, but also may improve the classification performance.
• An example of the architecture of a neural network classifier is illustrated in Fig. 51.
• the non-nicotine model 5110 has five binary classifiers 5120 while the nicotine model 5130 has three binary classifiers 5140.
• Each model 5110, 5130 has a MAX function 5150 and 5160.
• Applications in the non- nicotine model 5110 are then assigned to the appropriate rate class 5170, while applications in the nicotine model 5130 are assigned to the appropriate rate class 5180.
• each binary network has the structure of 12-5-1, e.g., twelve input nodes, five hidden neurons, and one output node.
• Activation functions for both hidden and output neurons may be logistic sigmoidal functions.
• the range of target values may scaled to [0.1 0.9] to prevent saturation during training process.
• the Levenberg-Marquardt numerical optimization technique may be used as the backpropagation-learning algorithm to achieve second-order training speed.
• Each binary network represents an individual rate class and is trained with the targets of one-vs-other.
• each network provides the probability of the unknown case belonging to the class it represents.
• the final rate class of the unknown case is determined by the MAX decision rule, e.g., given a vector whose entry values are in the interval [0,1], the MAX rule will return the value of the position of the largest entry.
• the distribution of the outputs is characterized. If the distribution of the outputs does not meet certain predefined criteria, no decision needs to be made by the classifier. Rather, the case will be sent to human underwriter for evaluation. The rationale here is that if a correct decision cannot be made, it would be preferable that the classifier makes no decision rather than the wrong decision.
• the four features that characterize the membership grades may be the same as those set forth above with respect to the fusion module discussed above, i.e., cardinality, entropy, the difference between the highest and the second high values of outputs, and the separation between rank orders of the highest and the second highest values of outputs.
• Step 1 C ⁇ ⁇ ⁇ OR C > r 2 OR E > ⁇ -
• Step 2 D ⁇ ⁇ 4 AND S ⁇ l
• ⁇ ⁇ , ⁇ - , ⁇ - , and r 4 are the thresholds.
• the value of the thresholds is typically data set dependent. In this embodiment, the value of the thresholds are first empirically estimated and then fine-tuned by evolutionary algorithms (EA). The final numbers for all five-fold data sets are illustrated in Table 7 below:
• a neural network classifier may be implemented using software code, and tested against a case base.
• a software implementation of a neural network may use a case base of 2,879 cases. After removal of 173 UW cases, the remaining 2,706 cases were used for training and testing the neural network classifier. Five-fold cross-validation was used to estimate the performance of the classifier.
• the combined confusion matrices of the five-fold runs are illustrated in Fig. 52.
• the combined confusion matrices for the five-fold runs after postprocessing are illustrated in Fig. 53.
• the performance for this example before postprocessing is provided in Fig. 54, while the performance for this example after postprocessing is provided in Fig. 55.
• the systems and processes described in this invention may be implemented on any general purpose computational device, either as a standalone application or applications, or even across several general purpose computational devices connected over a network and as a group operating in a client-server mode.
• a computer- usable and writeable medium having a plurality of computer readable program code stored therein may be provided for practicing the process of the present invention.
• the process and system of the present invention may be implemented within a variety of operating systems, such as a Windows® operating system, various versions of a Unix-based operating system (e.g., a Hewlett Packard, a Red Hat, or a Linux version of a Unix-based operating system), or various versions of an AS/400-based operating system.
• the computer-usable and writeable medium may be comprised of a CD ROM, a floppy disk, a hard disk, or any other computer-usable medium.
• One or more of the components of the system or systems embodying the present invention may comprise computer readable program code in the form of functional instructions stored in the computer-usable medium such that when the computer-usable medium is installed on the system or systems, those components cause the system to perform the functions described.
• the computer readable program code for the present invention may also be bundled with other computer readable program software. Also, only some of the components may be provided in computer-readable code.
• the computer may be a standard computer comprising an input device, an output device, a processor device, and a data storage device.
• various components may be computers in different departments within the same corporation or entity. Other computer configurations may also be used.
• various components may be separate entities such as corporations or limited liability companies. Other embodiments, in compliance with applicable laws and regulations, may also be used.
• the system may comprise components of a software system.
• the system may operate on a network and may be connected to other systems sharing a common database.
• Other hardware arrangements may also be provided.

Applications Claiming Priority (2)

Publications (2)

Family

ID=33309735

Family Applications (1)

Country Status (2)

Cited By (6)

Families Citing this family (39)

Citations (4)

Family Cites Families (364)

• 2003
  • 2003-04-30 US US10/425,722 patent/US7801748B2/en not_active Expired - Fee Related
• 2004
  • 2004-03-18 WO PCT/US2004/008256 patent/WO2004099943A2/en active Application Filing

Patent Citations (4)

Also Published As

Similar Documents

Legal Events