CN112364298A - Strategy method for automatically adjusting model based on model effect function - Google Patents

Strategy method for automatically adjusting model based on model effect function Download PDF

Info

Publication number
CN112364298A
CN112364298A CN202011234901.9A CN202011234901A CN112364298A CN 112364298 A CN112364298 A CN 112364298A CN 202011234901 A CN202011234901 A CN 202011234901A CN 112364298 A CN112364298 A CN 112364298A
Authority
CN
China
Prior art keywords
model
strategy
psi
sample
effect function
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011234901.9A
Other languages
Chinese (zh)
Inventor
梁协君
卢成伟
周恒�
蒋涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Youshu Finance Information Services Co ltd
Original Assignee
Hangzhou Youshu Finance Information Services Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Youshu Finance Information Services Co ltd filed Critical Hangzhou Youshu Finance Information Services Co ltd
Priority to CN202011234901.9A priority Critical patent/CN112364298A/en
Publication of CN112364298A publication Critical patent/CN112364298A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Pure & Applied Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Computational Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Algebra (AREA)
  • Operations Research (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a strategy method for automatically adjusting a model based on a model effect function, which comprises the following specific steps. And obtaining evaluation index KS statistic and stability index PSI of the model according to the algorithm, monitoring KS and PSI, and setting a triggering automatic parameter adjusting threshold value. And constructing an objective function based on the model effect based on the KS and PSI definitions. And if the monitoring index triggers the threshold value, adjusting the corresponding algorithm parameters by using a TPE algorithm, and recording the parameter value and the model effect function value every time. And finding out the parameter value which enables the model effect function to be maximum to serve as a new parameter of the model, so as to achieve the purpose of automatically improving the effect of the original model according to the sample at the current time period. The strategy method can greatly reduce the workload brought by modeling personnel to continuously adjust the model for solving the failure problem of the old model. The modeling efficiency is greatly improved. The method can be widely applied to online modeling scenes such as customer value mining, credit auditing and the like.

Description

Strategy method for automatically adjusting model based on model effect function
Technical Field
The invention relates to the relevant technology of machine learning, in particular to a strategy method for automatically improving model effect based on machine learning model evaluation indexes and actual business targets.
Background
As the internet continues to deeply penetrate the daily lives of consumers, the demand of on-line modeling scenarios such as inventory customer value mining, credit auditing, etc. faced by financial institutions is increasing. And the customer group data can form a sudden or gradual unstable trend due to various reasons, so that the prediction effect of the original model is gradually reduced and even fails. This will hinder the development of the business and cause a lot of losses to the financial institution. And the modeler is a time-consuming and labor-consuming task to adjust the model from time to time. Especially when the number of models reaches a certain magnitude, the maintenance cost of the models rises exponentially. Future fields of model adaptation will necessarily be able to be automated.
At present, there are many methods for automatically adjusting a model, but basically, the methods are only limited to the optimization of single indexes such as accuracy or stability, and the effect is not stable. The results are far from the results obtained after manual model adjustment.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a strategy method for automatically adjusting a model based on a model effect function, which can effectively improve the model prediction capability and the model stability capability after the model is automatically reconstructed.
In order to solve the technical problem, the invention is solved by the following technical scheme:
firstly, constructing an original model unit. The content of the unit comprises: 1. and recording the related configuration of the original model. Including the time of construction of the original model and the detailed information of the sample set of the training model (specifically, the model-entering features and the detailed data dictionary specifications of the target variable to be learned). 2. And recording the algorithm information of the original model. Including the name of the algorithm used, the values of the particular super parameter set. 3. And recording each evaluation index of the original model. The evaluation indexes include model KS statistics, model stability indexes PSI, AUC, GINI values, and the like. The frequency of evaluation index was recorded every day.
The mathematical definition of the KS statistic (Kolmogorov-Smirnov) mentioned in the first step above is: KS ═ max (TPR-FPR), where TPR denotes true positive rate (rate determined to be true positive as well); FPR indicates the false positive rate (rate of positive cases but not true cases). From this it can be seen that the KS statistic is the maximum of the TPR and FPR differences, which is a measure of the difference between the cumulative distributions of good and bad samples. The risk differentiation capability of the model can be evaluated, and the risk differentiation capability is a typical index for evaluating the prediction capability of the model.
The stability index psi (publication stability index) mentioned in the first step is mathematically defined as:
Figure BDA0002766470620000021
wherein A isiA sample ratio representing a ratio of samples in the i-th group to which the prediction probability (output result of the model) of the samples falls after dividing the n-th group by the prediction probability based on the cross-time samples; eiThe sample ratio is expressed such that the prediction probability of a sample falls in the i-th group after dividing n groups by the prediction probability (output result of the model) based on the training samples. The model PSI is an index for quantifying the difference degree of the two sample distributions and is a typical index for evaluating the stability of the model.
And secondly, constructing a strategy unit. The content of the unit comprises: 1. and setting a strategy trigger value. And triggering the strategy when the evaluation index of the original model is lower than a certain threshold value. 2. A trigger policy time point is determined. 3. Data preparation and recording of detailed information of the sample set. And uploading the new data set to serve as the original data set of the new model training sample and the verification set. 4. And (5) customizing the strategy. The method comprises the steps of setting a model effect function, setting a domain space and setting a function solving strategy.
The set policy trigger value mentioned in the second step above may be defined according to the failure experience of the model index. If KS statistic is <0.2, the model has no discrimination capability; 0.2-0.4, which represents that the model has the distinguishing capability; 0.41-0.5, which shows that the model has better distinguishing capability; 0.51-0.6, which shows that the model has good distinguishing capability; 0.61-0.75, which shows that the model has very good distinguishing capability; KS statistic >0.75, indicating that model anomalies are likely to be problematic. So the policy trigger value may be KS statistic <0.2 or KS statistic > 0.75. The trigger metrics associated with the trigger values herein shall include any metrics that may reveal a failure of the model, such as model KS statistics, model stability metrics PSI, AUC, GINI values, and the like.
The model effect function mentioned in the second step is defined as follows:
Figure BDA0002766470620000031
wherein, KSt(Dt,θt) Sample D representing time ttAnd hyperparametric array thetatA KS index quantity calculated in the t period; PSIt(Dt,θt) Sample D representing time ttAnd hyperparametric array thetatPSI indexes calculated in the period t; | KSt(Dt-1,θt-1)-KSt(Dt,θt) L is used for measuring the stability of the new model and the old model across time;
Figure BDA0002766470620000032
to measure how much a business person needs to pay attention to the model in the face of stability across time.
The model effect function considers the prediction capability and the stability capability of the model, simultaneously uses the PSI change percentage of the original model in the cross-time period as a measurement index of the attention degree of business personnel to the future change of business data, and uses the index to measure the cross-time stability of the new model and the old model. The reason for this setting is based on the assumption that "future data is evolved based on historical data", and the quantitative mode corresponding to the change degree of the historical data is described as the change percentage of the PSI index of the original model. This incorporates the amplitude of the variation of the data layer into the model effect function.
The domain space mentioned in the second step is specifically a value domain space of the super parameter set defining different underlying algorithms of the new model. The algorithm comprises a decision tree, a logistic regression, a neural network, a generalized linear regression, a random forest, an integration model based on a boosting method (boosting), a stacking method (stacking), a self-help aggregation method (bagging) and the like. The method for defining the hyperparameter set value domain space comprises an empirical method and a statistical distribution method by using a model expert.
The function mentioned in the second step solves the strategy. Specifically, the following equation set is solved by using a TPE algorithm to obtain an optimal hyperparameter set thetat：
Figure BDA0002766470620000033
Wherein, F (D)t,θt,Dt-1,θt-1) For model effect function, PSI, in the second stept(Dt,θt)<0.1 denotes the sample D with the t periodtAnd a set of hyper-parameters thetatThe PSI indicator calculated during the period t must not exceed 0.1. 0.1 is based on the failure experience of the PSI index, i.e., less than 0.1 for the PSI index, indicating that the model need not be updated. Adding PSI to a set of equationst(Dt,θt)<0.1, because for a new model of t period, the model effect is at least the sample D of t periodtThe upper part is kept stable. KSt(Dt,θt)>Sample D with time t is denoted 0.2tAnd a set of hyper-parameters thetatThe KS statistic calculated during time t must be no less than 0.2, 0.2 being based on failure experience of the KS statistic, since for a new model for time t there is essentially no predictive capability if the KS statistic does not reach 0.2.
The reason why the effect function of the model is maximized rather than minimized is that it is expected that the higher the prediction capability of the new model is, the better the stability capability of the new model is, the lower the stability of the new model is, the better the stability of the old model across time is, and the conclusion that the effect function of the model needs to be maximized can be deduced by combining the mathematical definition of the effect function of the model.
The TPE algorithm used in the function solution strategy in the second step is substantially an improved algorithm of bayesian optimization, and like bayesian optimization, the TPE algorithm tracks the evaluation results of the past models, and uses these results to form a probability model, and maps the hyper-parameters to the score probability P (score | θ) of the model effect function. And by iterating the score probability function mapped by the hyperparameter set, the hyperparameter set which meets the maximum model effect function can be quickly found according to the standard of an expected improvement function. In the TPE algorithm, the Expected Improvement function (Expected Improvement) is defined as:
Figure BDA0002766470620000041
wherein g (theta) represents P (score | theta) at score>＝score*A function expression of time; l (θ) represents P (score | θ) at score<score*A function expression of time.
Figure BDA0002766470620000042
A criterion is specified for selecting the next set of hyper-parameters from the score probability function P (score | θ).
And thirdly, constructing a new model unit. The content of the unit comprises: 1. the result of the policy unit is recorded. Including recording the new model algorithm name and the optimal hyper-parameter array. 2. And recording the model evaluation index of the new model. Including model KS statistics, model stability indicators PSI, AUC, GINI values, Lift values, and the like. The frequency of evaluation index was recorded every day.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a flowchart of implementation steps of a policy method for automatically adjusting a model based on a model effect function according to an embodiment of the present invention.
FIG. 2 is a diagram of an example of codes for constructing a domain space in the method disclosed by the embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the invention without any inventive step, are within the scope of protection of the invention.
The steps of the whole technical scheme are clearly and completely explained in the following combined with the application.
The first step is as follows: and constructing an original model unit.
1) And recording the related configuration of the original model.
Model build time: year 2020, 1 month and 1 day
Model build sample detail information:
a. sample name: to public customer data 20200101
b. Sample dimension: 10000*142
c. A sample data dictionary:
Figure BDA0002766470620000051
Figure BDA0002766470620000061
2) and recording the algorithm information of the model.
The algorithm name: XGBOOST algorithm
The set of hyperparameters:
parameter Chinese name Parameter English name Parameter value field Parameter value
Learning rate Eta [0,1] 0.3
Leaf number controller Gamma [0,+∞[ 0
Maximum tree depth Max_depth [0,+∞[ 6
Leaf minimum weight sum Min_child_weight [0,+∞[ 1
Sample sampling ratio Subsample ]0,1] 1
Column sample ratio Colsample_bytree ]0,1] 1
L2 regularization parameter Lambda [0,+∞[ 1
L1 regularization parameter Alpha [0,+∞[ 0
Positive and negative balance parameters Scale_pos_weight [0,+∞[ 1
… … … …
3) And recording the evaluation index of the original model.
Date KS statistics PSI AUC GINI …
20200101 0.478 0.025 0.763 0.526 …
20200102 0.477 0.023 0.762 0.525 …
… … … … … …
20200830 0.221 0.162 0.550 0.100 …
20200831 0.198 0.173 0.562 0.124 …
The second step is that: and constructing a strategy unit.
1) And setting a strategy trigger value. The policy is triggered when any of the rules in the following table is satisfied.
Rule sequence number Index (I) Symbol Threshold value
1 KS statistics < 0.2
2 PSI > 0.1
3 AUC < 0.6
4 GINI < 0.2
… … … …
2) A trigger policy time point is determined.
Triggering the strategy time point: 20200830
3) Data preparation and recording of detailed information of the sample set.
a. Sample name: to public customer data 20200830
b. Sample dimension: 10000*142
c. A sample data dictionary:
Figure BDA0002766470620000071
4) and (5) customizing the strategy.
Set model effect function:
Figure BDA0002766470620000072
wherein t is 20200830; t-1 ═ 20200101; dt"Pair public customer data 20200830"; dt-1"Pair public customer data 20200101"; thetatRepresenting a corresponding hyper-parameter set of an algorithm in a new model trained on data of 8, month and 30 days in 2020; thetat-1The corresponding super parameter set of the algorithm in the original model trained on the data of 1 month and 1 day in 2020 is shown.
Set domain space:
a. the algorithm name is as follows: XGBOOST algorithm
b. Domain space configuration table:
Figure BDA0002766470620000081
set function solution strategy.
a. Solving a system of equations:
Figure BDA0002766470620000091
wherein t is 20200830; t-1 ═ 20200101; dt"Pair public customer data 20200830"; dt-1"Pair public customer data 20200101"; thetatRepresenting a corresponding hyper-parameter set of an algorithm in a new model trained on data of 8, month and 30 days in 2020; thetat-1The corresponding super parameter set of the algorithm in the original model trained on the data of 1 month and 1 day in 2020 is shown.
b. Solving an equation set algorithm: TPE algorithm
And thirdly, constructing a new model unit.
1) The result of the policy unit is recorded.
New model algorithm name: XGBOOST algorithm
The optimal hyper-parameter array:
parameter Chinese name Parameter English name Parameter value field Optimum parameter value
Learning rate Eta [0,1] 0.7
Leaf number controller Gamma [0,+∞[ 1
Maximum tree depth Max_depth [0,+∞[ 10
Leaf minimum weight sum Min_child_weight [0,+∞[ 3
Sample sampling ratio Subsample ]0,1] 2
Column sample ratio Colsample_bytree ]0,1] 1
L2 regularization parameter Lambda [0,+∞[ 0.8
L1 regularization parameter Alpha [0,+∞[ 0.1
Positive and negative balance parameters Scale_pos_weight [0,+∞[ 1
… … … …
2) And recording the model evaluation index of the new model.
Date KS statistics PSI AUC GINI …
20200830 0.517 0.007 0.812 0.624 …
20200831 0.516 0.007 0.811 0.622 …
… … … … … …
In this embodiment, the index performance of the newly trained model based on the model effect function even exceeds the index performance of the original model:
Figure BDA0002766470620000101
finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
In summary, the above-mentioned embodiments are only preferred embodiments of the present invention, and all equivalent changes and modifications made in the claims of the present invention should be covered by the claims of the present invention.

Claims (3)

1. A strategy method for automatically adjusting a model based on a model effect function is characterized by comprising the following steps:
s1, constructing an original model unit, wherein the content comprises recording the related configuration of the original model, recording the algorithm information of the original model, and recording each evaluation index of the original model;
s2, constructing a strategy unit, wherein the content comprises setting a strategy trigger value, determining a trigger strategy time point, preparing data and customizing a strategy, and the customizing strategy comprises setting a model effect function, setting a domain space and setting a function solving strategy;
and S3, constructing a new model unit, wherein the content comprises recording the name of the new model algorithm, the optimal hyper-parameter array and the model evaluation index of the new model.
2. The method of claim 1, wherein the method comprises:
the model effect function described in step S2, which is defined as:
Figure FDA0002766470610000011
wherein, KSt(Dt,θt) Sample D representing time ttAnd hyperparametric array thetatA KS index quantity calculated in the t period; PSIt(Dt,θt) Sample D representing time ttAnd hyperparametric array thetatPSI indexes calculated in the period t; | KSt(Dt-1,θt-1)-KSt(Dt,θt) L is used for measuring the stability of the new model and the old model across time;
Figure FDA0002766470610000012
to measure how much stability over time needs attention.
3. The method of claim 1, wherein the method comprises:
the setting function solving policy described in step S2 is: solving the following equation set by using a TPE algorithm to obtain an optimal hyperparameter set thetat
Figure FDA0002766470610000013
Wherein, F (D)t,θt,Dt-1,θt-1) For the model effect function, PSI, in step S2t(Dt,θt)<0.1 denotes the sample D with the t periodtAnd a set of hyper-parameters thetatThe PSI index calculated in the period t must not exceed 0.1; KSt(Dt,θt)>Sample D with time t is denoted 0.2tAnd a set of hyper-parameters thetatThe calculated KS statistic at time t must not be less than 0.2.
CN202011234901.9A 2020-11-08 2020-11-08 Strategy method for automatically adjusting model based on model effect function Pending CN112364298A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011234901.9A CN112364298A (en) 2020-11-08 2020-11-08 Strategy method for automatically adjusting model based on model effect function

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011234901.9A CN112364298A (en) 2020-11-08 2020-11-08 Strategy method for automatically adjusting model based on model effect function

Publications (1)

Publication Number Publication Date
CN112364298A true CN112364298A (en) 2021-02-12

Family

ID=74509741

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011234901.9A Pending CN112364298A (en) 2020-11-08 2020-11-08 Strategy method for automatically adjusting model based on model effect function

Country Status (1)

Country Link
CN (1) CN112364298A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113111568A (en) * 2021-03-05 2021-07-13 苏州热工研究院有限公司 Method for predicting stress corrosion crack propagation rate of nickel-based alloy

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113111568A (en) * 2021-03-05 2021-07-13 苏州热工研究院有限公司 Method for predicting stress corrosion crack propagation rate of nickel-based alloy
CN113111568B (en) * 2021-03-05 2023-05-30 苏州热工研究院有限公司 Nickel-based alloy stress corrosion crack growth rate prediction method

Similar Documents

Publication Publication Date Title
Dumitrescu et al. Machine Learning or Econometrics for Credit Scoring: Let's Get the Best of Both Worlds
CN110866832A (en) Risk control method, system, storage medium and computing device
Chatterjee et al. Understanding wage inequality in Australia
CN112529683A (en) Method and system for evaluating credit risk of customer based on CS-PNN
CN113240527A (en) Bond market default risk early warning method based on interpretable machine learning
CN112364298A (en) Strategy method for automatically adjusting model based on model effect function
CN112085593B (en) Credit data mining method for small and medium enterprises
Wang et al. How magic a bullet is machine learning for credit analysis? An exploration with FinTech lending data
Anderson et al. Portfolio choices with many big models
CN116911994A (en) External trade risk early warning system
CN115146904A (en) Artificial intelligence-based security asset risk monitoring system
CN114565457A (en) Risk data identification method and device, storage medium and electronic equipment
de Castro et al. Forecasting inflation time series using score‐driven dynamic models and combination methods: The case of Brazil
CN118313798B (en) Automatic evaluation method for standard of credit card based on artificial intelligence
CN118247046B (en) Behavior fraud prediction method and device and electronic equipment
CN118014738B (en) Life insurance recommending method and system based on artificial intelligence
KR102576143B1 (en) Method for performing continual learning on credit scoring without reject inference and recording medium recording computer readable program for executing the method
CN118071513B (en) Personalized bank asset management method and system based on artificial intelligence driving
CN112862602B (en) User request determining method, storage medium and electronic device
CN117273919A (en) Internet credit assessment method for individual households and small micro-enterprise e-commerce merchants
Nazari et al. Using the Hybrid Model for Credit Scoring (Case Study: Credit Clients of microloans, Bank Refah-Kargeran of Zanjan, Iran)
CN117934061A (en) User repurchase behavior prediction method and device, electronic equipment and storage medium
CN115619525A (en) Enterprise post-credit early warning method and system based on electricity consumption data
Sifa et al. Prediction of Rupiah Exchange Rate Against US Dollar Using Kernel-Based Time Series Approach
CN114140230A (en) Loan credit line determining method, electronic device and storage medium thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: 310019 room 808, floor 8, building 1, No. 8 and 10, Jiuhe Road, Shangcheng District, Hangzhou, Zhejiang

Applicant after: Zhejiang youshuzhi Technology Co.,Ltd.

Address before: 310000 room 808, 8 / F, building 4, No. 9, Jiuhuan Road, Jianggan District, Hangzhou City, Zhejiang Province

Applicant before: HANGZHOU YOUSHU FINANCE INFORMATION SERVICES CO.,LTD.

CB02 Change of applicant information