JP6239486B2 - Prediction model creation method - Google Patents

Description

  Embodiments described herein relate generally to a prediction model creation method.

  The importance of utilizing large-scale data is widely recognized, and data analysis techniques and prediction models are used in various situations such as analysis of large amounts of customer information and detection of anomalies based on device sensor data.

  Also in the field of software development management, with the development of various tools and infrastructure, a large amount of software development data has been accumulated, and control of development projects by building predictive models has become an important issue.

  When considering the use of a prediction model, as a characteristic desired for a prediction model, in addition to high prediction accuracy, interpretation of the prediction results, such as the investigation of the cause of “why did such a prediction occur?” This is also an important factor.

JP 2010-272004 JP 2012-194894 JP 2010-9177

  Various prediction models have been proposed so far, but it has been difficult to achieve both high prediction accuracy and ease of interpretation of prediction results. Simple prediction models such as naive Bayes and decision trees are superior in terms of ease of interpretation of prediction results, but are inferior to more complex group learning methods such as random forest in terms of prediction accuracy. On the other hand, group learning methods such as random forest are excellent in terms of prediction accuracy, but the prediction results tend to be black boxes, and are inferior in the ease of interpretation of the prediction results.

  An object of the present invention is to create a prediction model that improves the ease of interpretation of prediction results while maintaining prediction accuracy.

  According to the embodiment, the prediction model creation method includes an evaluation step, a repetition step, an integration step, and an aggregation step. The evaluation step is data accumulated in the past, discretizes the learning data that is used to create the prediction model, and calculates the lift value, which is a coefficient for calculating the posterior probability from the prior probability for each discrete region. , Generate a unitary prediction model that represents the lift value for each discrete region, apply the unitary prediction model to learning data, compare the prediction results with the actual results, and calculate the prediction accuracy and necessary parameter values Evaluate the model. The iteration step changes the discrete region, operates the evaluation step a plurality of times, generates a plurality of unit prediction models, and evaluates the plurality of unit prediction models. The integration step converts the evaluation results of a plurality of unit prediction models into one integrated evaluation result. The aggregation step generates an aggregate prediction model by linearly approximating the nonlinear part of the integrated evaluation result.

It is a block diagram which shows an example of a structure of the prediction model production apparatus of embodiment. It is a flowchart which shows an example of the prediction model creation method of embodiment. An example of the operation of creating weighted learning data by bootstrap sampling will be described. An example of discretization of explanatory variables when entropy is used as an evaluation function is shown. An example of a unit prediction model is shown together with an example of calculating a lift value. An example of logarithmic score calculation is shown. An example of calculating the maximum correct answer rate is shown. An example of a ROC curve and AUC is shown. An example of the evaluation result of a unit prediction model is shown. It is a flowchart which shows an example of the update of a weight. An example of updating weights is shown. An example of the parameter for unit prediction model integration is shown. The example of calculation of an integrated score is shown. An example of an aggregate prediction model is shown. An example of the relationship between a single prediction model and an aggregate prediction model is shown. An example of the comparison result with the aggregated prediction model of embodiment, an integrated prediction model (an integrated score is utilized for direct prediction), and a random forest prediction model is shown.

  Before describing the embodiment, terms related to the description of the embodiment will be described.

・ Scale level
A standard that categorizes variables and their measured data mathematically and statistically based on the nature of the information they represent. In order from the lowest, there are four scale levels: nominal scale, ordinal scale, interval scale, and proportional scale, with the higher level including the properties of lower levels.

・ Learning data
A collection of data used for learning predictive models. Each data is expressed as a vector composed of one or more explanatory variables and one objective variable. The goal is to learn the prediction model so that the objective variable becomes a predetermined value for all combinations of the values of the explanatory variables. Data obtained by adding weights to individual pieces of learning data is referred to as weighted learning data.

·test data
A collection of data used to evaluate (test) a predictive model. The structure of the data set itself is the same as the learning data.

・ Bootstrap sampling
A method of generating a new sample by performing restoration extraction as many times as the number of data from one data set, and analyzing the characteristics of the population, model estimation error, and the like by repetition. The restored sample is called a bootstrap sample.

-OOB (Out-of-Bag) data
In bootstrap sampling, when some data is selected redundantly in the restoration extraction, data that is not selected is generated. A collection of these data is called OOB (Out-of-Bag) data.

-Correct answer rate
One of the evaluation indices of the prediction model. Percentage of correct answers in all prediction target data (for example, in positive and negative two-class discrimination, predicted as positive and actually positive, or predicted as negative and actually negative). When the threshold value for discrimination is changed, the correct answer rate also changes accordingly. The maximum value is called the maximum correct answer rate. Further, misclassified data at the maximum correct answer rate is referred to as misclassified data.

・ ROC curve
Graph for performance evaluation of prediction model. The data is sorted in descending order of the output of the prediction model, the horizontal axis represents the false positive rate (for example, the rate of positive and negative in the two-class discrimination), and the vertical axis represents the true positive rate. A plot of the percentage that was actually positive as expected. The area under the ROC curve is called AUC (Area Under the ROC Curve), which is one of the evaluation indices of the prediction model and takes a value between 0 and 1. It can be said that the closer the AUC is to 1, the higher the prediction accuracy.

・ Bias-variance decomposition
The error of the prediction model can be decomposed into an error (bias) due to the deviation of the prediction model from the true model, an error (variance) due to sample variation of the learning data, and an error that cannot be essentially reduced. There is generally a trade-off between bias and variance.

・ Cross-validation (K-division cross-validation)
An experimental method for evaluating the performance of predictive models. The data set used for learning / evaluation of the prediction model is divided into K (K = 5, 10 etc. are often used), (K-1) set is used as learning data, and the remaining one set is used as test data. Repeat K times.

・ Prediction model
The model is a model that can learn the relationship between factors, inherent trends and patterns from the observed and measured data, and calculate various quantities. When the variable (objective variable) to be predicted is a continuous value, it is called a regression model, and when it is a discrete value, it is called a discriminant model. Typical prediction models for discrimination include, for example, decision trees and naive bayes.

  An example of target observation data is customer data, and an example of measurement data is sensor data of various devices. The customer data includes past customer pattern information. For example, a structured model of customer characteristics and purchase patterns is a prediction model. By creating a prediction model, when a new customer arrives, you can make a prediction that if you have such a product, you will buy it. A simple prediction model can be represented by a function that inputs raw data and outputs a prediction result.

・ Naive Bayes
A kind of prediction model. A posteriori probability for objective variable discrimination calculated based on Bayes' theorem. However, independence of explanatory variables is assumed in the calculation of posterior probabilities. The posterior probability is calculated by multiplying the prior probability by a kind of correction coefficient called a lift value. The logarithm of the model output (estimated posterior probability) is called the logarithmic score.

・ Group learning
A learning method that improves accuracy by integrating and combining multiple prediction models that are not so accurate alone. As a method of integrating / combining a plurality of results, majority vote, average, or the like is used. Typical group learning algorithms include bagging (evaluating the prediction result of bootlap sampled learning data by majority vote), boosting (changing the weight of the learning data based on the prediction result), random forest, and the like. Here, individual prediction models that are components of group learning are unit prediction models, evaluation results of multiple unit prediction models are integrated and combined, integrated evaluation results, and models that combine and combine multiple unit prediction models are combined. This is called an integrated prediction model. When the unit prediction models are integrated / combined, the weight assigned to each unit prediction model is called an integration weight. In addition, an operation for converting the integrated prediction model into one single prediction model is called aggregation.

・ Random Forest
A kind of group learning algorithm using decision trees. It is an improved version of bagging and succeeds in improving the prediction accuracy of the integrated results by randomizing the selection of explanatory variables in the decision tree and increasing the variation of individual predictions.

Hereinafter, embodiments will be described with reference to the drawings.
FIG. 1 is a functional block diagram illustrating a configuration example of a system that executes a prediction model creation method. A broken line means an input to the model creation processing unit 70, and a solid line means an output from the model creation processing unit 70. The system includes an input data management unit 10, an output data management unit 40, and a model creation processing unit 70. The input data management unit 10 manages the bootstrap maximum number of times data 12, the boosting maximum number of times data 14, the learning data 16, and the test data 18. The output data management unit 40 stores the weighted learning data 42, the unit prediction model 44, the logarithmic score 46, the maximum correct answer rate data 48, the misclassification data 50, the integrated weight 52, the integrated score 54, the aggregated prediction model 56, and the evaluation result data 58. to manage. The model creation processing unit 70 includes a bootstrap / sample generation unit 72, a naive Bayes prediction model generation unit 74, a boosting processing unit 76, a prediction model integration unit 78, a prediction model aggregation unit 80, and a prediction model evaluation unit 82.

  The bootstrap maximum number of times data 12 is input to the bootstrap sample generation unit 72. The boosting maximum frequency data 14 is input to the boosting processing unit 76. The learning data 16 is input to the bootstrap sample generation unit 72. The test data 18 is input to the evaluation result data 58.

  The bootstrap sample generation unit 72 outputs the weighted learning data 42. The weighted learning data 42 is input to the naive Bayes prediction model generation unit 74 and the boosting processing unit 76. The naive Bayes prediction model generation unit 74 outputs a single prediction model 44, a logarithmic score 46, maximum correct answer rate data 48, misclassification data 50, and an integrated weight 52. The maximum correct answer rate data 48 and the erroneous determination data 50 are input to the boosting processing unit 76. The boosting processing unit 76 also outputs weighted learning data 42. The weighted learning data 42, the logarithmic score 46, and the integrated weight 52 are input to the prediction model integration unit 78. The prediction model integration unit 78 outputs an integrated score 54. The integrated score 54 is input to the prediction model aggregation unit 80, and the prediction model aggregation unit 80 outputs the aggregated prediction model 56. The integrated score 54 and the aggregated prediction model 56 are input to the prediction model evaluation unit 82, and the prediction model evaluation unit 82 outputs evaluation result data 58.

In the embodiment, a prediction model with high prediction accuracy and easy interpretation of a result is created.
In order to improve the prediction accuracy, bootstrap sampling and boosting are combined to randomize the training data and apply it to the naive Bayes prediction model. Here, the weighted average of the evaluation results of each prediction model is integrated with the weight (integrated weight) calculated from the AUC of the OOB data, the evaluation results are integrated, and a prediction model is generated based on the result, thereby achieving high prediction accuracy. Realized. When creating a prediction model from learning data that is past data, the goal is to increase the prediction accuracy of test data that is new data as much as possible. However, if you fit too much to learning data (overfitting), test data The prediction accuracy may be deteriorated. By using OOB data, that is, data that was not selected by random sampling that allowed duplication in bootstrap sampling, the degree of overfitting or new Predictability of data can be estimated in advance. If the prediction accuracy for OOB data is low, the prediction accuracy for test data may not be high even if the prediction accuracy for learning data is high. By using such information as a weight when integrating the evaluation results, high prediction accuracy is realized as a total.

  Since the naive Bayes prediction model is generally a relatively stable prediction model with a large bias and a small variance, it has conventionally not been very compatible with the group learning algorithm. In group learning, an unstable prediction model with a small bias and a large variance, such as a decision tree, is more effective. Therefore, by applying a combination of bootstrap sampling and boosting, the dispersion of the Naive Bayes prediction results is increased, and by using weights (integrated weights) calculated from the AUC of OOB data, extreme predictions are made. The influence of the results can be reduced and the prediction accuracy can be increased.

  In order to improve the interpretability of the results, the nonlinear part of the prediction model by collective learning is linearly approximated and aggregated into an equivalent single naive Bayes prediction model while maintaining the prediction accuracy in general. A model that is easy to interpret was realized. The aggregated prediction model is equivalent to a single naive Bayes model, so it is only necessary to examine a single model, and the influence of each variable can be taken into account independently, making it easy to interpret the prediction results .

FIG. 2 is a flowchart illustrating an example of a prediction model creation method.
At block 102, the learning data 16 is input to the bootstrap sample generator 72. In the bootstrap sample generation unit 72, the bootstrap maximum number of times data 12 is set. The learning data 16 is raw data for creating a prediction model, such as customer data.

  In step 104, the bootstrap sample generation unit 72 performs random extraction (referred to as bootstrap sampling) with the same number of times as the number of data on the learning data 16 to obtain the weighted learning data 42. Output.

  FIG. 3 shows an example of bootstrap sampling. The learning data 16 includes a large number, for example, 20 data sets, and the data set includes an identifier of data, explanatory variables X1, X2, and X3, and an objective variable Y (binary). The goal of learning the prediction model is to determine, for example, the objective variable Y = 1. The weighted learning data 42 obtained by bootstrap sampling is also composed of 20 data sets, and the data set includes data identifiers, explanatory variables X1, X2, and X3, an objective variable Y (binary), and weights. The sum of the weights of all data sets is the number of data sets (= 20). Data with identifier ID = 1, 2 has not been extracted once, and data with identifier ID = 3 has been extracted twice.

  In step 106, the bootstrap sample generation counter is incremented (+1).

  In step 108, the naive Bayes prediction model generation unit 74 discretizes the explanatory variable X of the weighted learning data 42. When the explanatory variables X1, X2, and X3 of the weighted learning data 42 are order scales (or interval scales and proportional scales), the variables are discretized based on a predetermined evaluation function. The main evaluation function for discretization includes entropy, expected lift value (Japanese Patent Application No. 2013-118091), and any of them may be used, and the evaluation function is not particularly limited.

  As an example, FIG. 4 shows the discretization of the explanatory variable X1 when entropy is used as the evaluation function. Several discretization candidates are prepared for each variable, and the candidate having the smallest evaluation function is determined. Since candidate 2 has the smallest entropy, discretization of variable X1 is determined as candidate 2. The variable X2 is also discretized in the same way. Since the variable X3 is originally a discrete variable of nominal scale, there is no need for discretization.

In step 110, the naive Bayes prediction model generation unit 74 calculates lift values of the discrete regions for the explanatory variables X1 and X2. The lift value is a kind of correction coefficient for calculating the posterior probability P (A | D) from the prior probability P (A) in the Naive Bayes method, and is defined by the following equation obtained by transforming the Bayes' theorem. .
P (A | D) = P (A) × P (D | A) / P (D)
The posterior probability P (A | D) is a probability that the variable Y = A when the variable X enters the discrete region D. Prior probability P (A) is a probability that variable Y = A when there is no condition. The numerator of lift value P (D | A) / P (D): P (D | A) is the probability that variable X enters discrete region D under the condition of variable Y = A, and denominator: P (D) Is the probability that the variable X enters the discrete region D. A set of (variable, discrete region, lift value) obtained as a result is a unit prediction model 44.

  An example of the unit prediction model 44 is shown in FIG. 5 together with a lift value calculation example. The simplex prediction model also includes an initial probability (a value obtained by dividing the total weight of Y = 1 by the total number of data).

  In step 112, the naive Bayes prediction model generation unit 74 evaluates the single prediction model 44 using the weighted learning data 42. The weighted learning data 42 is applied to the unit prediction model 44 and evaluated based on the output. The evaluation result of the unit prediction model 44 is composed of a logarithmic score (logarithm of posterior probability) 46, maximum correct answer rate data 48, misclassification data 50, and AUC of OOB data.

The logarithmic score 46 is calculated by applying a naive Bayes prediction model to each element (ID = 1, 2,...) Of the target data. Logarithmic score 46, the initial probability, each explanatory variable X1, X2, ... corresponding by multiplying the lift value of the discrete regions of, obtained by taking the last Log _10.

A calculation example of the logarithmic score 46 is shown in FIG. The variable X1 is included in the discrete region _D1,1 , and the variable X2 is included in the discrete region _D2,2 . Data with identifier ID = 1, 2 is not applicable because weight = 0. The logarithmic score of the data with the identifier ID = 3 is Log ₁₀ (0.4 × 0.659 × 2.133 ×...) = 0.747. Here, 0.4 is an initial probability, 0.659 is a lift value of X1εD _1,1 , and 2.133 is a lift value of X2εD _2,2 .

  It is expected that the higher the log score of each data obtained by applying the naive Bayes prediction model, the higher the probability that the data is Y = 1. The threshold value of the logarithmic score is determined, and the prediction result is determined by regarding the data having the logarithmic score equal to or higher than the threshold value as Y = 1. Here, the correct answer rate of the prediction model can be calculated by comparing the prediction result with the actual result.

  When the threshold value of the logarithmic score is moved, the correct answer rate also changes accordingly. The maximum value of the correct answer rate is called maximum correct answer rate data 48. In addition, a set of data with different predictions and actual results at the maximum correct answer rate is referred to as misclassification data 50.

  A calculation example of the maximum correct answer rate is shown in FIG. When the threshold value is 1, Y = 1 is predicted, and the total weight of data for which the actual result is Y = 1 is 5, and Y = 0 is predicted, and the total weight of the data for which the actual result is also Y = 0 is 12. Since the total weight of all data is 20, the correct answer rate = (5 + 12) /20=0.85. When the threshold value is 2, the correct answer rate = (7 + 7) /20=0.7 is calculated in the same manner. Therefore, the maximum correct answer rate = 0.85, and misclassification data = {4, 11}.

  As a general evaluation method of the prediction result, there is an ROC curve in addition to the correct answer rate. The area under the ROC curve is called AUC (Area Under the ROC Curve).

  In the weighted learning data 42, data in which the weight is inverted (data of weight> 0 is changed to weight = 0 and data of weight = 0 is changed to weight = 1) is referred to as OOB data.

From the value auc_oob obtained by calculating AUC with respect to the OOB data, an integrated weight (w) 52 that is a weight of each single prediction model 44 at the time of integrated score calculation (step 124) is obtained.
w = log _e (auc_oob / (1-auc_oob))
However, when auc_oob <= 0.5, w = 0.

  An example of the ROC curve and AUC is shown in FIG. The ROC curve is a graph in which data is sorted in descending order of logarithmic score, and the false positive rate is plotted on the horizontal axis and the true positive rate is plotted on the vertical axis. The true positive rate on the vertical axis is the rate where Y = 1 was actually predicted, and the false positive rate on the horizontal axis was the rate where Y = 1 was actually predicted when Y = 1 was predicted. . The area under the ROC curve is AUC, and in this example, AUC = 0.875.

  The maximum correct answer rate data 48 and the erroneous determination data 50 output in step 112 are used in “update the weight of weighted learning data” in step 116. The logarithmic score 46 and the integration weight (w) 52 output in step 112 are used in “calculation of integration score” in step 124.

  FIG. 9 shows an example of the evaluation result of the unit prediction model. From weighted learning data (weight = 0 removed), logarithmic score (0.749 for data with identifier ID = 3), maximum correct answer rate (= 0.85), misclassification data 50 (= {4, 11}) ) Is obtained. From the OOB data, an integrated weight w (= 1.946) is obtained from auc_oob = 0.875.

  In step 114, the boosting processing unit 76 determines whether or not to end the boosting process. The boosting process is a process of adjusting the weight of the learning data based on the evaluation result of the prediction model based on the learning data. Typical boosting techniques include AdaBoost and LogitBoost, but any of them may be used, and the boosting technique is not particularly limited.

The end determination can be based on the number of boosting processes, the maximum correct answer rate, or the maximum correct answer rate. For example, the number of boosting processes exceeds the boosting maximum number data 14 (value is N _bst ), the maximum correct answer rate = 1 (misidentification data set = φ), or the maximum correct answer rate is 0.5. If it falls below, it will break out of the boosting loop.

  If the end determination result is negative, the boosting processing unit 76 updates the weight of the weighted learning data 42 in step 116 (boosting processing). The boosting processing unit 76 inputs the weighted learning data 42, the maximum correct answer rate data 48, and the misclassification data 50, recalculates the weight for each data, and updates the weighted learning data 42.

  An example of updating the weight is shown in FIG. In updating the weights, the weight of misclassified data is increased and the whole is normalized. In step 1001, b = a / (1-a) is calculated. Here, a is the maximum correct answer rate. In step 1002, the weight of misclassification data is multiplied by b (W_temp = bxW_old). The weight of correctly determined data is not changed (W_temp = W_old). In step 1003, a ratio of weights of all data before and after the change is obtained (k = ΣW_temp / ΣW_old). In step 1004, the changed weight is divided by k and normalized (W_new = W_temp / k). Σ is an integration over the number of data.

FIG. 11 shows an example of updated weighted learning data.
As shown in FIG. 9, since the maximum correct answer rate = 0.85, b is calculated as follows (step 1001).
b = 0.85 / (1−0.85) = 5.666
Since the data with ID = 3 is correctly identified, the weight is not changed (step 1002).
W _{3_temp} = W _{3_old} = 2
Since the data with ID = 4 has been erroneously determined, the weight is multiplied by b (step 1002).
_W 4_temp = bxW 4_old = 11.333
Similarly, the weight is updated according to whether the determination is correct or incorrect.
ΣW _{i_temp} = 31
ΣW _{i_old} = 20
k = ΣW _{i_temp} / ΣW _{i_old} = 1.55
_{W3_new} = _{W3_temp} / _k = 1.29
_{W4_new} = _{W4_temp} / _k = 7.31
Thereafter, the counter of the number of boosting processes is incremented (+1) in step 118, the process returns to step 108, and the boosting process is continued.

If the result of the boosting end determination is OK, in step 120, the bootstrap sample generation unit 72 determines whether or not the bootstrap sample processing should be ended. The termination determination can be based on the number of bootstrap sample generations. For example, if the bootstrap sample generation count exceeds the bootstrap maximum count data 12 (value is N _bsp ), the process exits the bootstrap sample generation loop.

  If the end determination is negative, the process returns to step 104 and the bootstrap sample generation process is continued.

In the process so far, the weighted learning changed to various weights as a result of repeating the two loops of the bootstrap sampling process and boosting process in a nested manner (with the boosting group inside and the bootstrap sampling loop outside) The data 42, the corresponding unit prediction model 44, the logarithmic score 46, and the integrated weight 52 are stored in the output data management unit 40. Hereinafter, each loop is referred to as Run = 1, 2,. For example, when N _bst = 10 and N _bsp = 20, there is a possibility that there are unit prediction models 44 starting from Run = 1 and up to Run = 10 _× 20 = 200 and these data sets.

When the bootstrap sampling end result is OK, the prediction model integration unit 78 integrates a plurality of unit prediction models 44 stored in the output data management unit 40 for each Run. First, in step 122, the prediction model integration unit 78 calculates logistic regression parameters α and β in each Run using the weighted learning data 42 and the logarithmic score 46. Specifically, logistic regression analysis of the following equation is performed using logarithmic score 46 as an explanatory variable (= S) and Y of weighted learning data 42 as an objective variable to obtain logistic regression parameters α and β.
P (Y = 1) = 1 / (1 + e− ^{(α + β × S)} )
Since the log score 46 of the individual data obtained in each Run is an approximate index of the posterior probability assuming the independence of the variables, it is not a probability value in a strict sense (it does not fall within the range of 0 to 1). When integrating a plurality of unit prediction models 44, it was confirmed by experiments that the scales are different and the prediction accuracy is lowered. Therefore, by converting the logarithmic score 46 using a logistic regression equation, the logarithmic score 46 is normalized in the range of 0 to 1, and as a result, the prediction accuracy when integrating a plurality of unit prediction models 44 is improved.

  As a result, as shown in FIG. 12, the integrated weight w, auc_oob data calculated in step 112 and the logistic regression parameters α and β are used together as parameters for unitary prediction model integration.

In step 124, the prediction model integration unit 78 uses the unit prediction models 44 of all Runs and the model integration parameters w, α, and β to calculate an integrated score S _all of an integrated prediction model that is a kind of prediction model for group learning. calculate.

A calculation example of the integrated score S _all is shown in FIG. The prediction model integration unit 78 calculates a logarithmic score S _n (= Σlog (Lift ⁱ _(n) ), Σ is an integration from i = 0 to m) derived from the unit prediction model 44 of all Runs to a logistic regression equation ( P _n = 1 / (1 + e− ^{(αn + βn × Sn)} )) and taking a weighted average with the weight w of each single unit prediction model 44, thereby integrating and combining a plurality of single unit prediction models 44. The integrated score S _all is calculated.
S _all = Σw _i × P _i / Σw _i
Here, the explanatory variables of the unitary prediction model 44 are X ₁ , X ₂ ,... X _m , the initial probability at Run = k is Lift ⁰ _(k) , and the lift value of the variable X _i is Lift ⁱ _(k) . Σ is an integration from i = 0 to m.

In step 126, the prediction model aggregating unit 80 constructs a single unit prediction model (aggregated prediction model 56) in which the integrated prediction models are aggregated. Specifically, as shown in FIG. 14, the nonlinear part of the prediction formula obtained by Taylor expansion of the logistic regression equation ( _Pn = 1 / (1 + e− ^{(αn + βn × Sn)} )) included in the integrated score is obtained. The aggregated prediction model 56 is obtained by performing linear approximation (ignoring non-linear terms and making only linear terms), and performing equation transformation and organizing terms.

  By considering A as an initial probability (log) and B as a lift value (log) of the variable Xj in the aggregate prediction model 56, a single naive Bayes prediction formula can be obtained. As a result, two loops of bootstrap sampling processing and boosting processing are repeated in a nested manner, and a large number of unit prediction models corresponding to weighted learning data of various weights are obtained. can get.

  In step 128, the prediction model evaluation unit 82 evaluates the integrated prediction model and the aggregate prediction model 56 based on the test data. Although the integrated prediction model is not constructed in the embodiment, a virtual model using the integrated score for direct prediction is used as the integrated prediction model. The integrated prediction model and the aggregate prediction model 56 are applied to the test data to obtain evaluation result data 58.

  As shown in FIG. 15, the simplex prediction model 44 includes weights w corresponding to all Runs and logistic regression parameters α and β. On the other hand, the aggregate prediction model 56 is a single naive Bayes prediction model having an aggregated discrete region and a lift value.

  A general method, for example, the above-mentioned AUC etc. can be used for evaluation of a prediction model. Here, as an example, comparison was made with a random forest, an integrated prediction model, and an aggregate prediction model 56 using four data sets of software development data published by NASA. The evaluation results are shown in FIG. Prediction accuracy (AUC) is calculated by a 5-fold cross validation (a method in which all data is divided into 5 and evaluation is performed 5 times with 4 learning data and 1 test data). This was repeated 5 times, and the average value was obtained. As a result, in all four data sets, the prediction accuracy (AUC) of the integrated prediction model and the aggregate prediction model 56 exceeded the random forest. Comparing the integrated prediction model and the aggregated prediction model 56, the prediction accuracy is almost the same in this data set. However, in terms of model complexity, the integrated prediction model is not much different from other collective learning models (such as random forest), but the aggregate prediction model 56 is aggregated into a single model. The structure is simple and the interpretation of the model (such as analysis of the cause that led to the prediction result) is easy.

  As described above, according to the embodiment, the prediction accuracy is improved by randomizing the learning data by combining bootstrap sampling and boosting and applying the learning data to the naive Bayes prediction model. In bootstrap sampling, random extraction is allowed with duplication, so there is unselected data (OOB). By evaluating the prediction accuracy for the OOB data, it is possible to estimate in advance how much the prediction accuracy is likely to be for new data not included in the learning data. High prediction accuracy is realized by integrating the evaluation results by taking the weighted average of the evaluation results of each prediction model using the integrated weight calculated from the AUC of the OOB data, and generating the prediction model based on the result. . In addition, by linearly approximating the nonlinear part of the prediction model by collective learning and consolidating it into an equivalent single naive Bayes prediction model while maintaining the prediction accuracy in general, a model that can easily interpret the prediction result should be realized. Can do.

  Note that the processing of the present embodiment can be realized by a computer program, so that the computer program can be installed and executed on a computer through a computer-readable storage medium storing the computer program, as in the present embodiment. The effect of can be easily realized.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine suitably the component covering different embodiment. For example, the integration of multiple prediction models using bootstrap sampling, boosting, and OOB data can be applied to simpler prediction models other than naive Bayes. Further, the linear approximation of the nonlinear part of the prediction model by collective learning may be applied when a plurality of naive Bayes prediction models generated are not limited to the above-described embodiment.

  DESCRIPTION OF SYMBOLS 16 ... Learning data, 42 ... Weighted learning data, 44 ... Single prediction model, 46 ... Logarithmic score, 52 ... Integrated weight, 54 ... Integrated score, 56 ... Aggregate prediction model, 72 ... Bootstrap sample generation part, 74 ... Naive Bayes prediction model generation unit, 76 ... boosting processing unit, 78 ... prediction model integration unit, 80 ... prediction model aggregation unit, 82 ... prediction model evaluation unit.

Claims (14)

Discretized learning data, which is data accumulated in the past and used to create a prediction model, finds lift values that are coefficients for calculating posterior probabilities from prior probabilities for each discrete region, and A single unit prediction model representing a lift value for each region is generated, the single unit prediction model is applied to the learning data, the prediction result is compared with the actual result, and the single unit prediction model is obtained by calculating the prediction accuracy and the necessary parameter value. An evaluation step for evaluating the predictive model;
Changing the discrete region, operating the evaluation step a plurality of times, generating a plurality of unit prediction models, and repeatedly evaluating the plurality of unit prediction models;
An integration step of converting the evaluation results of the plurality of unit prediction models into one integrated evaluation result;
An aggregation step of generating an aggregate prediction model from the integrated evaluation result;
Comprising
The aggregation step is a prediction model creation method for generating the aggregate prediction model by linearly approximating a nonlinear portion of the integrated evaluation result. The evaluation step obtains a logarithm of the posterior probability of the single unit prediction model and an integrated weight according to the area under the ROC curve of the single unit prediction model;
The prediction according to claim 1, wherein the integration step obtains a logistic regression parameter corresponding to the logarithm, and converts the evaluation results of the plurality of unit prediction models into the integrated evaluation result using the integration weight and the logistic regression parameter. Model creation method.   3. The aggregation step generates the aggregated prediction model by omitting a nonlinear term of a logistic regression equation using the logistic regression parameter included in the integrated evaluation result and leaving a linear term for the integrated evaluation result. The prediction model creation method described. Further comprising a weighted learning data generation step of bootstrap sampling the learning data to generate weighted learning data;
The evaluation step discretizes the weighted learning data, obtains a lift value for each discrete region, generates the unit prediction model, evaluates the unit prediction model,
The repeating step includes
A first iteration of updating the weight of the weighted learning data, operating the evaluation step a first positive integer m times, generating m unit prediction models, and evaluating the m unit prediction models Steps,
After evaluating the m unit prediction models, the operation of the weighted learning data generation step is repeated a second positive integer n times to generate m × n unit prediction models, and the m × n unit prediction models are generated. A second iteration step for evaluating the prediction model;
The prediction model creating method according to claim 1, wherein the integration step converts the evaluation result of the m × n unit prediction models into one integrated evaluation result.   5. The prediction model creation method according to claim 4, wherein in the first iteration step, the evaluation step is operated until m exceeds a predetermined number of times, or a maximum correct answer rate of prediction becomes 1 or falls below a predetermined value.   5. The prediction model creation method according to claim 4, wherein the second iteration step operates the weighted learning data generation step until n exceeds a predetermined number of times. The learning data includes explanatory variables and objective variables,
The weighted learning data further includes a weight indicating the number of times the learning data is extracted by bootstrap sampling,
In the evaluation step, an initial probability obtained by dividing the sum of the weights of data in which the objective variable becomes a target value by the number of data and a single prediction model representing a lift value for each discrete region,
The prediction model creation method according to claim 4, wherein the logarithm of the posterior probability of the single prediction model is obtained by multiplying the initial probability and the lift value for each discrete region. A program executed by a computer, wherein the program is
Discretized learning data, which is data accumulated in the past and used to create a prediction model, finds lift values that are coefficients for calculating posterior probabilities from prior probabilities for each discrete region, and A single unit prediction model representing a lift value for each region is generated, the single unit prediction model is applied to the learning data, the prediction result is compared with the actual result, and the single unit prediction model is obtained by calculating the prediction accuracy and the necessary parameter value. An evaluation step for evaluating the predictive model;
Changing the discrete region, operating the evaluation step a plurality of times, generating a plurality of unit prediction models, and repeatedly evaluating the plurality of unit prediction models;
An integration step of converting the evaluation results of the plurality of unit prediction models into one integrated evaluation result;
An aggregation step of generating an aggregate prediction model from the integrated evaluation result;
Comprising
The aggregation step is a program for generating the aggregate prediction model by linearly approximating a nonlinear part of the integrated evaluation result. The evaluation step obtains a logarithm of the posterior probability of the single unit prediction model and an integrated weight according to the area under the ROC curve of the single unit prediction model;
The program according to claim 8, wherein the integration step obtains a logistic regression parameter corresponding to the logarithm, and converts the evaluation results of the plurality of unit prediction models into the integrated evaluation result using the integration weight and the logistic regression parameter. .   10. The aggregation step generates the aggregate prediction model by omitting a nonlinear term of a logistic regression equation that uses the logistic regression parameter included in the integrated evaluation result and leaving a linear term for the integrated evaluation result. The listed program. Further comprising a weighted learning data generation step of bootstrap sampling the learning data to generate weighted learning data;
The evaluation step discretizes the weighted learning data, obtains a lift value for each discrete region, generates the unit prediction model, evaluates the unit prediction model,
The repeating step includes
A first iteration of updating the weight of the weighted learning data, operating the evaluation step a first positive integer m times, generating m unit prediction models, and evaluating the m unit prediction models Steps,
After evaluating the m unit prediction models, the operation of the weighted learning data generation step is repeated a second positive integer n times to generate m × n unit prediction models, and the m × n unit prediction models are generated. A second iteration step for evaluating the prediction model;
The program according to claim 8, wherein the integration step converts the evaluation result of the m × n unit prediction models into one integrated evaluation result.   The program according to claim 11, wherein the first iteration step operates the evaluation step until m exceeds a predetermined number of times, or a maximum correct answer rate of prediction becomes 1 or falls below a predetermined value.   The program according to claim 11, wherein in the second repetition step, the weighted learning data generation step is operated until n exceeds a predetermined number of times. The learning data includes explanatory variables and objective variables,
The weighted learning data further includes a weight indicating the number of times the learning data is extracted by bootstrap sampling,
In the evaluation step, an initial probability obtained by dividing the sum of the weights of data in which the objective variable becomes a target value by the number of data and a single prediction model representing a lift value for each discrete region,
The program according to claim 11, wherein the logarithm of the posterior probability of the simplex prediction model is obtained by multiplying the initial probability and the lift value for each discrete region.

Images