CN113919931A

CN113919931A - Loan application scoring model use effect evaluation method and system

Info

Publication number: CN113919931A
Application number: CN202110981825.6A
Authority: CN
Inventors: 肖玉龙
Original assignee: Beijing Ruizhi Tuyuan Technology Co ltd
Current assignee: Beijing Ruizhi Tuyuan Technology Co ltd
Priority date: 2021-08-25
Filing date: 2021-08-25
Publication date: 2022-01-11

Abstract

The invention discloses a use effect evaluation method and a use effect evaluation system of a loan application scoring model, wherein the method comprises the following steps: the method comprises the steps of examining and approving a first client sample and a second client sample in different periods by using a preset loan application scoring model, obtaining a model verification sample set and an external model verification sample set, deducing current evaluation labels of the sample clients with unknown labels of the model verification sample set and the external model verification sample set, associating each sample client with a corresponding front evaluation label, calculating scoring variable values of the model verification sample set and the external model verification sample set and scoring values of each examination index of the model verification sample set and the external model verification sample set according to the current evaluation labels of the sample clients with the unknown labels of the model verification sample set and the external model verification sample set, and analyzing the use effect of the preset loan application scoring model according to the scoring variable values of the model verification sample set and the external model verification sample set and the scoring values of each examination index of the model verification sample set and the external model verification sample set. Model risks can be effectively found, and bad account loss is avoided.

Description

Loan application scoring model use effect evaluation method and system

Technical Field

The invention relates to the technical field of business risk assessment, in particular to a using effect assessment method and system of a loan application scoring model.

Background

A loan application scoring model (application scoring or scoring for short) is an important technology in the field of commercial bank loan risk control, and algorithms such as statistical analysis, machine learning and the like are used for establishing scoring and comprehensively evaluating the credit of a client and giving a credit approval decision for legally collected client authorization information. The credit granting approval is carried out by using the scores, the credit granting approval method is higher in efficiency, lower in cost and more objective than manual approval, the situation that different approval personnel give different decisions can be avoided, the application scores are developed, deployed and on-line are completed, risk decisions are participated, and the using effect of the application scores needs to be continuously tracked and monitored. Through monitoring, the change of the grading effect is timely and effectively found, measures (such as grading iteration or risk strategy adjustment) are taken to avoid model risks, and the method is significant in the field of business bank model risk management and control.

Typically, the monitoring of the application score by the commercial bank can be divided into front-end monitoring and back-end monitoring. The front-end monitoring mainly monitors the stability of application scoring, monitors statistical indexes such as scoring distribution, scoring interval passing rate, scoring stability (PSI) and scoring variable stability (PSI) of newly-added credit application clients, and the back-end monitoring mainly monitors the online use effect of an application scoring model, is the most important ring of bank model risk assessment, and can provide risk decision opinions for off-line scoring, scoring strategy adjustment or scoring iteration. The general practice of the industry is that a certain loan client sample is accumulated for a period of time (the requirements of the performance period after loan and the number of the sample are required to be met) when the loan of a loan client is required to be applied for scoring, and the client is marked with a good or bad label (the labeling logic is consistent with the model development). And calculating an effect index of applying for grading (taking the industry common statistic KS as an example) according to the good or bad label of the paying customer. And comparing the evaluation model with a paying customer sample KS when applying for grading development, if the KS of a paying customer converted by a newly added credit month is attenuated too fast or is lower than an empirical value of 0.20 (a threshold value generally concerned by supervision and bank risk departments), judging that the grading model is close to failure, and adopting corresponding risk evasive measures such as grading offline, grading strategy adjustment or grading iteration and the like.

Even if the flow quality of the channel does not change every month, the paying customer has larger deviation with the credit granting application customer after being interfered by factors such as credit granting wind control, cash withdrawal conversion willingness, cash withdrawal wind control and the like; secondly, because the credit granting wind control and the withdrawal wind control are dynamically adjusted according to bank risk preference, the paying off customers in each new month are also constantly changed, and paying off customers from a new credit granting month are also deviated from paying off customers in an earlier credit granting month. The deviation of the customer group has a great influence on the later stage monitoring of the application scoring, so that the evaluation of the application scoring effect based on the paying customer sample is not comparable according to a time axis, the model effect conclusion obtained based on the evaluation is not accurate, the model risk decision suggestion is not in line with the actual business situation, and the wrong model risk decision may bring loss to a commercial bank.

Disclosure of Invention

In view of the above-mentioned problems, the present invention provides a method and a system for evaluating the usage effect of a loan application scoring model, which are used to solve the problems mentioned in the background art that the model effect conclusion obtained based on manual experience is inaccurate, the given model risk decision suggestion is not in line with the actual business situation, and the loss of a commercial bank may be caused by an incorrect model risk decision.

A loan application scoring model use effect evaluation method comprises the following steps:

examining and approving the first customer sample and the second customer sample at different periods by using a preset loan application scoring model to obtain a model verification sample set and an out-of-model verification sample set;

deducing the current evaluation labels of the sample clients with unknown labels in the model verification sample set and the out-of-model verification sample set, and associating each sample client with the corresponding previous evaluation label;

calculating the grading variable values of the model verification sample set and the model external verification sample set and a first grading value of each approval index of the model verification sample set and the model external verification sample set according to the current evaluation label of the sample client with the unknown label of the model verification sample set and the model external verification sample set;

analyzing a use effect conclusion of the preset loan application scoring model according to the scoring variable values in the model verification sample set and the model external verification sample set and the first scoring value of each examination and approval index of the model verification sample set and the model external verification sample set;

wherein the current evaluation label comprises: good person labels and bad person labels.

Preferably, the examining and approving the first customer sample and the second customer sample at different periods by using the preset loan application scoring model to obtain the model verification sample set and the out-of-model verification sample set includes:

generating a plurality of approval indexes by using the preset loan application scoring model;

the method comprises the steps of obtaining customer data of each sample customer in a first customer sample and a second customer sample, and grading each sample customer by using a preset loan application grading model based on the customer data of each sample customer to obtain a grading result;

correspondingly distributing each sample client in the first client sample and the second client sample to the multiple examination and approval indexes according to the grading result;

and after distribution is finished, counting the number of sample clients of each examination and approval index, and confirming the statistical results and the multiple examination and approval indexes corresponding to the first client sample and the second client sample as the model verification sample set and the out-of-model verification sample set.

Preferably, the inferring a current evaluation label of a sample customer of unknown labels of the model verification sample set and the out-of-model verification sample set, associating each sample customer with its corresponding previous evaluation label, comprises:

screening credit negative overdue variables of a bank, performing extreme qualification on the credit negative overdue variables, and determining a first extreme qualification rule;

inquiring credit investigation variable thresholds of a multi-head bank credit investigation mechanism and a civil credit investigation mechanism, carrying out second-grade identification on the credit investigation variable thresholds, and determining a second-grade identification rule;

performing third-party bad judgment according to a preset effective grading threshold of a third party, and determining a third bad judgment rule;

constructing a better principle according to the first, second and third bad identification rules;

avoiding a preset scoring variable label, determining a target scoring variable label, calculating a scoring variable index IV of the target scoring variable label, and constructing a scoring variable evasive principle;

designing good and bad label inference strategies according to the scoring variable evasion principle and the better principle;

and determining the current evaluation label of each sample client according to the good-bad label inference strategy and associating each sample client with the corresponding previous evaluation label.

Preferably, the calculating the values of the scoring variables of the model verification sample set and the model external verification sample set and the first scoring value of each approval index according to the current evaluation label of the sample client with the unknown label of the model verification sample set and the model external verification sample set includes:

determining an application score of each sample client according to the current evaluation label of each sample client in the first client sample and the second client sample;

calculating a first scoring value of each approval index by using the following formula:

wherein, KS_nExpressed as the first value of the nth approval index, m_nExpressed as the number of sample clients in the nth index, F_GExpressed as the cumulative probability distribution of the estimated good person sample, F_BExpressed as the estimated cumulative probability distribution of the sample of bad persons, Score_iThe application score expressed as the ith sample customer;

calculating the values of the scoring variables of the model verification sample set and the model external verification sample set according to the following formula:

wherein IV is expressed as the value of the grading variable of the model verification sample set/the model external verification sample set, and N is expressed as the value of the model verification sample set/the model external verification sampleNumber of centralized approval indexes, GoodDist_jExpressed as the proportion of the number of sample clients of the good label in the jth examination and approval index in the model verification sample set/out-of-model verification sample set to the number of sample clients of all good labels in the model verification sample set/out-of-model verification sample set, BadDist_jThe method is expressed as the proportion of the number of sample clients of the badness label in the jth examination and approval index in the model verification sample set/the out-of-model verification sample set to the number of sample clients of all badness labels in the model verification sample set/the out-of-model verification sample set, and ln is expressed as a natural logarithm.

Preferably, the analyzing the result of the preset loan application scoring model according to the scoring variable values in the model verification sample set and the verification sample set outside the model and the first scoring value of each approval index of the two sets includes:

adjusting the first scoring value of each examination and approval index in the model verification sample set and the model external verification sample set by using an actual risk strategy adjustment log of a preset loan application scoring model to obtain a second scoring value of each examination and approval index;

comparing the model verification sample set with a second grading value of the same examination and approval index in the model external verification sample set to determine whether the use grading effect of the preset loan application grading model is reduced or not;

if the usage scoring effect of the preset loan application scoring model is determined to be reduced, determining the specific reason for the reduction of the usage scoring effect of the preset loan application scoring model by using the scoring variable values in the model verification sample set and the verification sample set outside the model;

and confirming the specific reason of the reduction of the use scoring effect of the preset loan application scoring model and the use scoring effect of the preset loan application scoring model as the use effect conclusion of the preset loan application scoring model.

Preferably, the method further comprises:

drawing a first grading monitoring report form of a preset loan application grading model in a preset period;

generating a risk decision suggestion of a preset loan application scoring model according to a first scoring monitoring report of the preset loan application scoring model in a preset period;

and uploading the risk decision suggestion of the preset loan application scoring model to a preset server so that a worker can perfect the preset loan application scoring model.

Preferably, the obtaining of the customer data of each sample customer in the first customer sample and the second customer sample, and based on the customer data of each sample customer, scoring by using each sample customer in a preset loan application scoring model to obtain a scoring result includes:

obtaining credit scoring data of each sample client from the client data of the sample client;

confirming whether the credit score data of each sample client meets the evaluation condition of a preset loan application score model, if so, inputting the credit score data of the sample client into the preset loan application score model, otherwise, confirming that the credit score data of the sample client is unreasonable;

evaluating the credit evaluation score of each sample client by using a preset credit evaluation algorithm in the preset loan application scoring model based on the input credit evaluation data of the sample client;

and confirming the credit evaluation score of each sample client as the scoring result of the user.

Preferably, the generating of the risk decision suggestion of the preset loan application scoring model according to the first scoring monitoring report of the preset loan application scoring model in the preset period includes:

determining initial operation parameters of a preset loan application scoring model according to the scoring variable values of the model verification sample set and the first scoring value of each examination and approval index in the model verification sample set;

performing attribute division on the initial operation parameters to obtain division results;

determining a characteristic attribute factor of a preset loan application scoring model according to the dividing result;

constructing a decision tree of a preset loan application scoring model according to the characteristic attribute factors;

substituting the scoring result in the first scoring monitoring report into the decision tree to obtain an analysis result output by the decision tree;

analyzing the analysis result to obtain a plurality of evaluation indexes of a preset loan application scoring model and a weight factor corresponding to each evaluation index;

establishing an evaluation factor weight matrix based on a preset loan application scoring model according to the plurality of evaluation indexes and the weight factor corresponding to each evaluation index;

substituting the analysis result into the evaluation factor weight matrix to obtain an association degree matrix of the first grading monitoring report form associated with the evaluation factor weight matrix;

determining a fluctuation matrix factor of a preset loan application scoring model according to the incidence matrix and the evaluation factor weight matrix;

analyzing each fluctuation matrix factor to obtain a related risk factor, and screening the risk factor corresponding to each fluctuation matrix factor according to the first grading monitoring report to obtain a target risk factor with the maximum correlation of each fluctuation matrix factor;

and generating a solution suggestion corresponding to the target risk factor corresponding to each fluctuation matrix factor, and counting a plurality of solution suggestions to integrate so as to obtain the risk decision suggestion of the preset loan application scoring model.

A system for evaluating the effectiveness of a loan application scoring model, the system comprising:

the examination and approval module is used for examining and approving the first customer sample and the second customer sample in different periods by using a preset loan application scoring model to obtain a model verification sample set and an out-of-model verification sample set;

the inference module is used for inferring the current evaluation labels of the sample clients with unknown labels in the model verification sample set and the model verification sample set, and associating each sample client with the corresponding previous evaluation label;

the calculation module is used for calculating the scoring variable values of the model verification sample set and the model external verification sample set and the first scoring values of each examination and approval index of the model verification sample set and the model external verification sample set according to the current evaluation labels of the sample clients with unknown labels in the model verification sample set and the model external verification sample set;

the analysis module is used for analyzing a use effect conclusion of the preset loan application scoring model according to the scoring variable values in the model verification sample set and the model external verification sample set and the first scoring value of each examination and approval index of the model verification sample set and the model external verification sample set;

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.

The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention.

FIG. 1 is a flowchart illustrating a method for evaluating the effectiveness of a loan application scoring model according to the present invention;

FIG. 2 is another flowchart of the usage evaluation method of the loan application scoring model according to the present invention;

FIG. 3 is a flowchart illustrating a method for evaluating the effectiveness of a loan application scoring model according to the present invention;

FIG. 4 is a flowchart of individual loan transaction translation in an application embodiment;

FIG. 5 is a sample screenshot of a constructed customer;

FIG. 6 is a KS monitoring report screenshot;

FIG. 7 is a KS monthly monitor screenshot;

fig. 8 is a schematic structural diagram of a usage effect evaluation system of a loan application scoring model according to the present invention.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

Even if the flow quality of the channel does not change every month, the paying customer has larger deviation with the credit granting application customer after being interfered by factors such as credit granting wind control, cash withdrawal conversion willingness, cash withdrawal wind control and the like; secondly, because the credit granting wind control and the withdrawal wind control are dynamically adjusted according to bank risk preference, the paying off customers in each new month are also constantly changed, and paying off customers from a new credit granting month are also deviated from paying off customers in an earlier credit granting month. The deviation of the customer group has a great influence on the later stage monitoring of the application scoring, so that the evaluation of the application scoring effect based on the paying customer sample is not comparable according to a time axis, the model effect conclusion obtained based on the evaluation is not accurate, the model risk decision suggestion is not in line with the actual business situation, and the wrong model risk decision may bring loss to a commercial bank. In order to solve the above problem, the present embodiment discloses a usage effect evaluation method of a loan application scoring model.

A method for evaluating the usage effect of a loan application scoring model, as shown in fig. 1, comprises the following steps:

step S101, examining and approving a first customer sample and a second customer sample in different periods by using a preset loan application scoring model, and acquiring a model verification sample set and an out-of-model verification sample set;

step S102, deducing the current evaluation labels of the sample clients with unknown labels in the model verification sample set and the model external verification sample set, and associating each sample client with the corresponding previous evaluation label;

step S103, calculating the grading variable values of the model verification sample set and the model external verification sample set and a first grading value of each examination and approval index of the model verification sample set and the model external verification sample set according to the current evaluation label of the sample client with the unknown label of the model verification sample set and the model external verification sample set;

step S104, analyzing a result of use of the preset loan application scoring model according to the scoring variable values in the model verification sample set and the verification sample set outside the model and the first scoring value of each examination and approval index of the model verification sample set and the verification sample set outside the model;

The working principle of the technical scheme is as follows: examining and approving a first client sample and a second client sample in different periods by using a preset loan application scoring model, acquiring a model verification sample set and an external model verification sample set, deducing current evaluation labels of the sample clients with unknown labels of the model verification sample set and the external model verification sample set, associating each sample client with a corresponding front evaluation label, calculating scoring variable values of the model verification sample set and the external model verification sample set and a first scoring value of each examination index of the model verification sample set and the external model verification sample set according to the current evaluation labels of the sample clients with the unknown labels of the model verification sample set and the external model verification sample set, and analyzing a use effect conclusion of the preset loan application scoring model according to the scoring variable values of the model verification sample set and the external model verification sample set and the first scoring values of each examination index of the model verification sample set and the external model verification sample set.

The beneficial effects of the above technical scheme are: the assessment method has the advantages that the assessment of the scoring values and the scoring variable values of the preset loan application scoring models by the clients in different periods is more reasonable than the assessment of the using effects obtained by the experience of workers in the traditional technology, the model risks can be found earlier and more effectively, the bad account loss of banks caused by the incredible scoring effect conclusions and the model risk decision suggestions is avoided, the time for obtaining the scoring effect conclusions and the model risk decision suggestions is greatly shortened, the problems that the model effect conclusions obtained based on manual experience in the prior art are inaccurate, the model risk decision suggestions given further do not accord with the actual business conditions, and the loss of commercial banks caused by wrong model risk decisions are solved.

In one embodiment, as shown in fig. 2, the examining and approving the first customer sample and the second customer sample at different periods by using the preset loan application scoring model to obtain a model verification sample set and an out-of-model verification sample set includes:

step S201, generating a plurality of approval indexes by using the preset loan application scoring model;

step S202, obtaining customer data of each sample customer in the first customer sample and the second customer sample, and grading each sample customer by using a preset loan application grading model based on the customer data of each sample customer to obtain a grading result;

step S203, correspondingly distributing each sample client in the first client sample and the second client sample to the multiple examination and approval indexes according to the grading result;

and step S204, after distribution is finished, counting the number of sample clients of each examination and approval index, and confirming the counting results and the multiple examination and approval indexes corresponding to the first client sample and the second client sample as the model verification sample set and the out-of-model verification sample set.

The beneficial effects of the above technical scheme are: the client samples corresponding to each examination and approval index after examination and approval can be effectively stored by generating a plurality of examination and approval indexes, so that each sample client can be rapidly scored, all the sample clients are correspondingly stored in the corresponding examination and approval indexes according to the final scoring result, and the working efficiency is improved.

In one embodiment, said inferring a current evaluation label of a sample customer for unknown labels of said set of model validation samples and set of off-model validation samples, associating each sample customer with its corresponding previous evaluation label comprises:

The beneficial effects of the above technical scheme are: the current evaluation label of each sample client is comprehensively evaluated according to the three rules, so that the final evaluation result can be more practical and accurate, and meanwhile, the human label of each sample client can be evaluated from multiple angles, and the objectivity of the evaluation result is guaranteed.

In one embodiment, the calculating the values of the scoring variables of the model verification sample set and the out-of-model verification sample set and the first scoring value of each approval index according to the current evaluation tags of the sample clients with unknown tags of the model verification sample set and the out-of-model verification sample set comprises:

wherein IV is the value of the grading variable of the model verification sample set/the model external verification sample set, N is the number of the examination and approval indexes in the model verification sample set/the model external verification sample set, and GoodDist_jExpressed as the proportion of the number of sample clients of the good label in the jth examination and approval index in the model verification sample set/out-of-model verification sample set to the number of sample clients of all good labels in the model verification sample set/out-of-model verification sample set, BadDist_jThe method is expressed as the proportion of the number of sample clients of the badness label in the jth examination and approval index in the model verification sample set/the out-of-model verification sample set to the number of sample clients of all badness labels in the model verification sample set/the out-of-model verification sample set, and ln is expressed as a natural logarithm.

The beneficial effects of the above technical scheme are: the value of the credit of each examination and approval index can be effectively and reasonably evaluated according to the current evaluation label of each sample client in each examination and approval index by using a formula to calculate the value of the credit of each examination and approval index, and the accuracy of data evaluation is improved.

In one embodiment, the analyzing the conclusion of the usage effect of the preset loan application scoring model according to the scoring variable values in the model verification sample set and the out-of-model verification sample set and the first scoring value of each approval index of the two sets includes:

The beneficial effects of the above technical scheme are: the method has the advantages that the first rating value of each examination and approval index in the model verification sample set and the model external verification sample set is adjusted by utilizing the actual risk strategy adjustment log of the preset loan application scoring model, so that the rating value of each examination and approval index can be acquired more observably, the accuracy of evaluation data is improved, further, the specific reason for the decline of the use scoring effect of the preset loan application scoring model can be accurately determined according to the parameters corresponding to each actual examination and approval index by determining the rating variable values in the model verification sample set and the model external verification sample set, and the objectivity is further improved.

In one embodiment, the method further comprises:

The beneficial effects of the above technical scheme are: the method solves the problem that the evaluation method for the grading effect is not proper after the commercial bank is used on the grading line, and the grading effect on the current day, the current week and the current month can be effectively evaluated. Compared with industrial experience, the evaluation on the scoring effect is more reasonable, the model risk can be found earlier and more effectively, the distrusted bad account loss brought to banks by the incredible scoring effect conclusion and the model risk decision suggestion is avoided, and the time efficiency of obtaining the scoring effect conclusion and the model risk decision suggestion is greatly shortened.

In one embodiment, as shown in fig. 3, the obtaining of the customer profile of each sample customer in the first customer sample and the second customer sample, and scoring by using each sample customer in the preset loan application scoring model based on the customer profile of each sample customer, to obtain the scoring result, includes:

step S301, obtaining credit scoring data of each sample client from the client data of the sample client;

step S302, confirming whether credit score data of each sample client meets the evaluation condition of a preset loan application score model, if so, inputting the credit score data of the sample client into the preset loan application score model, and otherwise, confirming that the credit score data of the sample client is unreasonable;

step S303, evaluating the credit evaluation score of each sample client by using a preset credit evaluation algorithm in the preset loan application scoring model based on the input credit evaluation data of the sample client;

and step S304, confirming the credit evaluation score of each sample client as the scoring result of the user.

The beneficial effects of the above technical scheme are: the scoring result of each sample client can be visually determined according to the credit assessment score by assessing the credit assessment score of each sample client by using a preset credit assessment algorithm in the preset loan application scoring model.

In an embodiment, the generating of the risk decision suggestion of the preset loan application scoring model according to the first scoring monitoring report of the preset loan application scoring model in the preset period includes:

The beneficial effects of the above technical scheme are: by determining the target risk factor of each fluctuation matrix corresponding to the preset loan application scoring model, the corresponding risk decision suggestion can be reasonably determined for improvement based on the index of the preset loan application scoring model, the final decision suggestion is ensured to be in accordance with the actual condition of the preset loan application scoring model, and the objectivity is ensured.

In one embodiment, the commercial bank customer borrowing application conversion process is as follows:

and (5) a credit application is granted. The client has the borrowing demand and applies for borrowing to the bank. The bank evaluates the credit condition of the client through credit risk rules, application scoring, manual examination and the like, passes the examination and approval of the client with good credit condition, gives differentiated credit line and pricing (interest) according to different client risks, and then passes the credit granting. And for the customers who do not meet the risk control requirements, refusing the examination and approval, namely refusing the credit granting.

And (6) submitting a present application. After the bank is approved to obtain the amount, the client can carry out withdrawal application to obtain the fund. The conversion of the customer withdrawal application is accidental, and if the withdrawal time point and the credit granting time point are long, the credit condition of the customer may change, so that the bank can check again when the customer withdrawal application is carried out, and the bank can deposit money to the customer bank card after the withdrawal check is passed (withdrawal is passed). And the client which can not pass the cash withdrawal wind control audit is rejected, namely cash withdrawal rejection.

The specific business transformation process is shown in fig. 4, and from fig. 4 we can see that the paying customer has a large deviation from the credit requesting customer through the business funnel. The clients applying for the credit are full clients applying for the credit, and can represent the whole client applying population in a statistical sense; in general, the credit passing rate varies according to the quality of channel customers, and is 5% -30% different, and the client group has larger deviation compared with the credit application clients. The credit determines whether to apply for conversion of the withdrawal within a future period of time (30 days) by the urgency of the customer to the funds borrowed and the satisfaction of the amount and interest granted, with conversion rates typically varying from 30% to 50% and further excursions in the customer population. When the withdrawal application is carried out, wind control examination is needed again, the clients passing the withdrawal can obtain bank deposit, the withdrawal passing rate is usually 70% to 90%, and the client group further deviates.

Therefore, even if the flow quality of the channel does not change every month, the paying customer has a large deviation with the credit granting application customer after being interfered by factors such as credit granting wind control, cash withdrawal conversion willingness, cash withdrawal wind control and the like; secondly, because the credit granting wind control and the withdrawal wind control are dynamically adjusted according to bank risk preference, the paying off customers in each new month are also constantly changed, and paying off customers from a new credit granting month are also deviated from paying off customers in an earlier credit granting month. The deviation of the customer group has a great influence on the later stage monitoring of the application scoring, so that the evaluation of the application scoring effect based on the paying customer sample is not comparable according to a time axis, the model effect conclusion obtained based on the evaluation is not accurate, the model risk decision suggestion is not in line with the actual business situation, and the wrong model risk decision may bring loss to a commercial bank.

Here, a customer sample matrix is given, as shown in fig. 5, to better illustrate the above customer population bias situation. The modeling sample A uses client data (including a model verification sample set) of an earlier credited month, the out-of-modeling sample B is sample data of a closer month after the application scoring model is deployed and used online, the sample A and the sample B do not have time intersection, and different samples such as A1, A2, A3, A4, B1, B2, B3, B4 and the like can be generated respectively according to business links such as credit application, credit passing, withdrawal application and withdrawal passing (payment). Industry experience, post-application scoring monitoring was based on comparative evaluation of the scoring effectiveness of samples a4 and B4. Samples a4 and B4 were shifted (shift amplitude was determined depending on the magnitude of the risk policy adjustment during the period) due to the application model scoring itself and other risk policy adjustments, respectively, the customer samples had been shifted, the conclusion of the scoring effect based on the B4 sample was unreliable (could not represent the credit application customer population), and was not comparable to the scoring effect based on the a4 sample, i.e., a4 could not infer a1, B4 could not infer B1, and B4 could not infer a 4.

The premise of applying the scoring model iteratively is that the original score fails or fails to achieve the expected effect. The industry typically infers the scoring effect of a4 based on B4, and thus a1 or B1, from which it is not trustworthy to give a scoring effect conclusion and model risk decision suggestion.

First, an untrusted scoring effect concludes that model risk decision suggestions may cause bad account losses to the bank. If the original score fails (the actual original score is still effective) and an iterative suggestion is given, a person who uses the new score to drag back the product has a high risk; secondly, if the original score is given to continue to be valid (the actual original score is close to failure or has failed), there is a high risk of continuing to use the original score. These exposed risks will cause bad account losses to the bank. And blindness is brought to risk strategy adjustment, and the cost of bank manpower is indirectly increased.

Secondly, the timeliness of the obtained scoring effect conclusion and model risk decision suggestion is low. The timeliness of finding model risk and giving recommendations is low because a sufficient number of newly added loan samples with certain credit performance periods are accumulated, and scoring effects can be observed only after 3 to 6 months.

In order to solve the above problem, the present embodiment provides an innovation for monitoring the online usage effect of the application scoring model (i.e. an innovation for monitoring the later stage), which can well monitor and evaluate the usage effect of the scoring, and includes the following steps:

1) and the scoring model is developed and reviewed based on a pay-off client sample A4 (known good and bad label sample), enters a model monitoring link after being deployed online, and monitors a scoring KS index and a scoring variable IV (Information Value) index aiming at the rear-section monitoring. The score effectiveness index (KS _ A4) and IV index on sample A4 were calculated.

The IV index can reflect the contribution and the interpretability of variables to the quality label, and is commonly used for selecting important variables in a scoring model. In the latter monitoring, when the score KS fluctuates greatly, the contribution degree change of a specific variable can be located according to the change, and the IV value calculation formula is as follows:

n is the number of variable bins, i is the ith bin

·GoodDist_iThe good persons in the ith box account for all the good persons

·BadDist_iDistributed for the badges, is the proportion of the badges in the ith box to all the badges

2) And designing a good and bad label inference strategy and archiving. And (4) the client with unknown good or bad labels needs to deduce the good or bad labels so as to calculate the KS index and the variable IV index. The good-bad label inference should satisfy the principles of interpretability, extreme-bad identification, no better quality, evasion of scoring variable and the like, and the probability of making mistakes is reduced as much as possible. Here, a rule-based good/bad label determination method is recommended: firstly, screening out relevant variables of negative overdue credit investigation of the people's bank, and carrying out extreme qualification, such as loan in the last two years, rules of how many days the credit card is overdue and how much more than overdue amount is larger than the overdue amount, the credit card is in a frozen state, and the like; secondly, a people bank credit investigation organization can be used for inquiring the multiple heads, and a civil credit investigation organization can be used for inquiring the multiple head related variable which is greater than a certain threshold (the height is strictly greater than a risk strategy rejection threshold) for identification; finally, the identification may be based on some threshold (highly stringent risk policy rejection threshold) for a valid score developed by the third party or by itself. The designed good and bad label inference strategy must avoid the variables used by the application scoring, and avoid the misjudgment caused by the self-verification of the scoring.

3) According to the good-and-bad label inference strategy designed in (2), the good-and-bad labels of clients with unknown good-and-bad labels on client samples, such as a credit application (A1), a credit pass (A2), a cash application (A3), a credit application (B1), a credit pass (B2), a cash application (B3) and a cash application (B4), are identified, and scoring effect indexes, namely KS _ A1, KS _ A2, KS _ A3, KS _ B1, KS _ B2, KS _ B3 and KS _ B4, are respectively calculated. And calculating the scoring variable IV index corresponding to each sample.

After the line is drawn, the accumulated effect index KS monitoring report forms, as shown in FIG. 6, the modeling sample A (the earlier month) and the modeling sample B (the closer month) have no time intersection. By comparison, the fluctuation of the effect of the accumulated newly added sample B and the model development sample A after the model is online can be analyzed, and the fluctuation (such as the IV fluctuation of specific variables) caused by any reason can be analyzed. Firstly, comparing the effects on A1 and B1, whether the model effect has attenuation or not and the attenuation amplitude can be evaluated; secondly, comparing the effects on B1 and B2, the influence of credit wind control adjustment on the scoring effect can be evaluated; then, comparing the effects on B2 and B3, the influence of customer willingness to suggest on the scoring effect can be evaluated; finally, comparing the effects on B3 and B4, the impact of presenting the wind control on the scoring effect can be assessed. In the example table, the effect of B4 is reduced to 0.25, but the effect of B1 is 0.29 and is substantially equal to that of a1(0.30), and no obvious reduction is caused, and the reason that the scoring effect on B4 is reduced can be basically judged to be the result of multiple interferences of credit wind control, client intention to promote and promote wind control.

Judging whether the scoring effect is improper according to the effect on B4, comparing A1 with B1, evaluating objectively (eliminating the interference of credit/cash-up wind control strategy and cash-up will), comparing samples in the matrix transversely and longitudinally, adjusting the comprehensive evaluation scoring effect of the log by combining with the actual risk strategy, and giving an objective conclusion of the model effect. And the specific reason of the reduction of the grading effect can be positioned by combining the IV index fluctuation condition of the specific variable of each sample. The KS monitoring report is illustrated below (data virtualization).

5) And drawing an effect index KS monitoring report of a newly added month (fixed frequency) after the online. As shown in fig. 7, the report can also be drawn by day or week, and the time for monitoring the scoring effect can be shortened to day or week, so that the timeliness of the scoring effect conclusion and the model risk decision suggestion can be greatly improved.

The above embodiment can achieve the following advantageous effects: the method solves the problem that the commercial bank cannot evaluate the scoring effect on the credit granting application client, can evaluate the application scoring effect by using the credit granting application client (replacing a loan client), and avoids various problems of evaluating the scoring effect empirically by using the loan client, such as unreliable scoring effect index, low timeliness and the like. Meanwhile, the problem that the evaluation method for the grading effect is not appropriate after the commercial bank is used on the grading line is solved, and the grading effect on the current day, the current week and the current month can be effectively evaluated. Compared with industrial experience, the evaluation on the scoring effect is more reasonable, the model risk can be found earlier and more effectively, the distrusted bad account loss brought to banks by the incredible scoring effect conclusion and the model risk decision suggestion is avoided, and the time efficiency of obtaining the scoring effect conclusion and the model risk decision suggestion is greatly shortened.

The embodiment also discloses a usage effect evaluation system of the loan application scoring model, as shown in fig. 8, the system includes:

the approval module 801 is used for approving the first customer sample and the second customer sample at different periods by using a preset loan application scoring model to obtain a model verification sample set and an out-of-model verification sample set;

an inference module 802 for inferring current evaluation labels of sample customers of unknown labels of the model validation sample set and the out-of-model validation sample set, associating each sample customer with its corresponding previous evaluation label;

the calculating module 803 is configured to calculate, according to the model verification sample set and the current evaluation label of the sample client with an unknown label in the model external verification sample set, the scoring variable values of the model verification sample set and the model external verification sample set, and the first scoring value of each approval index of the model verification sample set and the model external verification sample set;

the analysis module 804 is used for analyzing a result of use of the preset loan application scoring model according to the scoring variable values in the model verification sample set and the model external verification sample set and the first scoring value of each examination and approval index of the model verification sample set and the model external verification sample set;

The working principle and the advantageous effects of the above technical solution have been explained in the method claims, and are not described herein again.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A loan application scoring model use effect evaluation method is characterized by comprising the following steps:

2. The method for evaluating the usage effect of the loan application scoring model according to claim 1, wherein the examining and approving the first customer sample and the second customer sample at different periods by using the preset loan application scoring model to obtain the model verification sample set and the out-of-model verification sample set comprises:

3. The method of claim 1, wherein the inferring a current assessment tag of a sample client for which the model verification sample set and the out-of-model verification sample set are unknown tags associates each sample client with its corresponding previous assessment tag comprises:

4. The method for evaluating the effectiveness of a loan application scoring model according to claim 1, wherein the calculating the values of the scoring variables of the model verification sample set and the first scoring value of each approval index according to the current evaluation tags of the sample clients with unknown tags in the model verification sample set and the model verification sample set comprises:

5. The method for evaluating the usage effect of the loan application scoring model according to claim 1, wherein the analyzing the usage effect conclusion of the preset loan application scoring model according to the scoring variable values in the model verification sample set and the model external verification sample set and the first scoring value of each approval index of the two comprises:

6. The method of claim 1, wherein the method further comprises:

7. The method for evaluating the usage effect of the loan application scoring model according to claim 2, wherein the obtaining of the customer profile of each sample customer in the first customer sample and the second customer sample, and the scoring by each sample customer in the preset loan application scoring model based on the customer profile of each sample customer, comprises:

8. The method for evaluating the usage effect of the loan application scoring model according to claim 6, wherein the generating of the risk decision suggestion of the preset loan application scoring model according to the first scoring monitoring statement of the preset loan application scoring model in the preset period comprises:

9. A system for evaluating the effectiveness of a loan application scoring model, the system comprising: