CN117853221A

CN117853221A - Method, device and system for monitoring consumption credit model

Info

Publication number: CN117853221A
Application number: CN202311866535.2A
Authority: CN
Inventors: 梁磊; 蔡苗; 周寅; 温国梁
Original assignee: Postal Savings Bank of China Ltd
Current assignee: Postal Savings Bank of China Ltd
Priority date: 2023-12-29
Filing date: 2023-12-29
Publication date: 2024-04-09

Abstract

The method obtains model performance scores of all months by monitoring the number of positive examples and the number of negative examples of the consumption credit model, takes an average value of the model performance scores of all months in one year as a final model performance score, and finally determines a prompt consumption credit model to be optimized according to the range of the final model performance score, thereby achieving the purpose of monitoring the consumption credit model, avoiding a large amount of manual processing in the middle process, further solving the problems that the monitoring of the consumption credit model in the prior scheme is sent to staff in a short message notification mode, the staff carries out manual treatment, and the treatment period is long, and delays the development of related services to cause loss.

Description

Method, device and system for monitoring consumption credit model

Technical Field

The application relates to the technical field of consumer credit models, in particular to a monitoring method, a monitoring device and a monitoring system of a consumer credit model.

Background

Consumption credit models based on big data and machine learning technology are widely applied in banking industry, data value exertion is promoted, and development of bank credit-eliminating business is assisted. In order to better monitor the effect of the lending model to prevent related risks, banks pay more and more attention to model monitoring and early warning disposal work. The existing model monitoring and early warning treatment method mainly monitors defined model indexes and generates early warning signals, the model monitoring indexes cannot be flexibly configured, the early warning treatment also needs a large amount of manual intervention, and the degree of automation and intellectualization is low, so that the efficiency is low.

In addition, the monitoring of the consumption credit model in the existing scheme is sent to the staff in a short message notification mode, the staff carries out the treatment manually, the treatment period is longer, and accordingly the development of related business is delayed to cause loss.

Disclosure of Invention

The main purpose of the application is to provide a method, a device and a system for monitoring a consumption credit model, so as to at least solve the problems that the monitoring of the consumption credit model in the existing scheme is sent to staff in a short message notification mode, the staff carries out the treatment manually, the treatment period is long, and the development of related business is delayed to cause loss.

To achieve the above object, according to one aspect of the present application, there is provided a method of monitoring a consumed credit model, the method comprising:

obtaining the number of positive cases and the number of negative cases of a consumption credit model in each month, wherein the number of positive cases is the number of examples, which are consistent with the predicted result and the real situation of the consumption credit model, and the number of negative cases is the number of examples, which are inconsistent with the predicted result and the real situation of the consumption credit model, and the consumption credit model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises the number of examples obtained in a historical time period: user credit data and whether the user has repayment capability;

Determining model performance scores of all months according to all the positive cases and the negative cases, and determining that the final model performance score is the average of all the model performance scores;

and determining whether to generate a first early warning instruction according to the range of the final model performance score, wherein the first early warning instruction is used for prompting that the consumed credit model needs to be optimized.

Optionally, determining the model performance score of each month according to all the number of positive examples and the number of negative examples includes: determining the total number of positive examples and the total number of negative examples, and determining the total number of accumulated positive examples and the total number of accumulated negative examples of each segment, wherein the total number of positive examples is the sum of all positive examples, the total number of negative examples is the sum of all negative examples, the total number of accumulated positive examples is the part of the number of positive examples, and the total number of accumulated negative examples is the part of the number of negative examples; determining the initial model performance score of each segment as a difference value of a first ratio and a second ratio, wherein the first ratio is the ratio of the accumulated positive case number to the total positive case number, and the second ratio is the ratio of the accumulated negative case number to the total negative case number; and determining the model performance score as the maximum value of all the initial model performance scores.

Optionally, determining whether to generate the first early warning instruction according to the range of the final model performance score includes: determining to generate the first early warning instruction under the condition that the final model performance score is greater than or equal to a first performance score threshold value; and under the condition that the final model performance score is smaller than a first performance score threshold value, determining that the first early warning instruction is not generated.

Optionally, after determining that the first warning instruction is not generated, the method further comprises: and generating a second early warning instruction to prompt that the consumed credit model needs to be replaced under the condition that the final model performance score is smaller than a second performance score threshold value.

Optionally, the method further comprises: acquiring current credit data and training credit data of a user, and determining a first target feature data duty ratio in the current credit data and a second target feature data duty ratio in the training credit data, wherein the training credit data is used for training the consumption credit model, the first target feature data duty ratio is the duty ratio of the credit data of the target user in the current credit data, and the second target feature data duty ratio is the duty ratio of the credit data of the target user in the training credit data; determining a stability score that characterizes a stability of the consumer credit model from PSI (i) = (Ai-B) ×ln (Ai/B), wherein PSI (i) is the stability score for month i, ai is the first target feature data duty cycle for month i, and B is the second target feature data duty cycle; and carrying out weighted summation on the stability scores to obtain a total stability score, and determining whether to generate a first early warning instruction according to the range of a final stability score, wherein the final stability score is an average value of the total stability score.

Optionally, after acquiring the current credit data and the training credit data of the user, the method further comprises: determining a data null rate and a characteristic repetition rate, wherein the data null rate is the duty ratio of the data quantity with null characteristic values in the current credit data, and the characteristic repetition rate is the duty ratio of the data quantity with same characteristic values in the current credit data.

Optionally, the method further comprises: acquiring current credit data and training credit data of a user, and determining a current feature data standardized value and a training feature data standardized value according to z= (x-u)/s, wherein z is the current feature data standardized value or the training feature data standardized value, x is the current feature data of the current credit data or the training feature data of the training credit data, u is the average value of the current credit data or the average value of the training credit data, and s is the standard deviation of the current credit data or the standard deviation of the training credit data; and determining the characteristic concentrated trend drift degree of the consumption credit model as a ratio of a standardized average value difference value to the standardized value of the training characteristic data, wherein the standardized average value difference value is a difference value between an average value of all the standardized values of the training characteristic data and an average value of all the standardized values of the current characteristic data.

Optionally, after determining whether to generate the first early warning instruction according to the range of the final model performance score, the method further includes: acquiring early warning logs generated by each consumption credit model in real time by adopting a jump, and transmitting all acquired early warning logs to kafka; and consuming data from the kafka by adopting a flink streaming real-time data technology, and analyzing early warning information to obtain an early warning event.

According to another aspect of the present application, there is provided a monitoring apparatus for a consumer credit model, the apparatus comprising:

the first obtaining unit is configured to obtain a number of positive cases and a number of negative cases of the consumption credit model in each month, where the number of positive cases is a number of examples where a prediction result and a real situation of the consumption credit model match, and the number of negative cases is a number of examples where the prediction result and the real situation of the consumption credit model do not match, and the consumption credit model is obtained by training using multiple sets of training data, where each set of training data includes training data obtained in a historical period: user credit data and whether the user has repayment capability;

a first determining unit, configured to determine a model performance score of each month according to all the number of positive examples and the number of negative examples, and determine a final model performance score as an average of all the model performance scores;

And the second determining unit is used for determining whether to generate a first early warning instruction according to the range of the final model performance score, wherein the first early warning instruction is used for prompting that the consumption credit model needs to be optimized.

According to another aspect of the present application, there is provided a monitoring system for a consumer credit model, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising a monitoring method for executing any one of the consumer credit models.

By applying the technical scheme, the model performance score of each month is obtained by monitoring the number of positive examples and the number of negative examples of the consumption credit model, the average value of the model performance scores of all months in one year is used as the final model performance score, and finally, the consumption credit model is prompted to be optimized according to the range of the final model performance score, so that the purpose of monitoring the consumption credit model is achieved, a large amount of manual processing is avoided in the middle process, the problem that the monitoring of the consumption credit model in the traditional scheme is carried out by sending the monitoring to staff in a short message notification mode, the staff carries out manual treatment, the treatment period is long, and the development of related business is delayed to cause loss is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

FIG. 1 illustrates a flow diagram of a method of monitoring a consumer credit model provided in accordance with an embodiment of the present application;

FIG. 2 illustrates a flow diagram of another method of monitoring a consumer credit model provided in accordance with an embodiment of the present application;

fig. 3 shows a block diagram of a monitoring device of a consumer credit model provided according to an embodiment of the present application.

Detailed Description

It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the present application described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

For convenience of description, the following will describe some terms or terms related to the embodiments of the present application:

a flime is an open source data processing tool for collecting, aggregating, and moving large amounts of log data and event streams. It may aggregate data from different sources (e.g., web servers, databases, sensors, etc.) to a central location and then send the data to other systems for processing and analysis. Jume is commonly used to build large-scale data processing and analysis systems, such as log management, real-time monitoring, and data warehouse. The method is an item of Apache software foundation, provides rich plugins and expansion functions, and can meet various data processing requirements.

kafka is a distributed stream processing platform originally developed by linkedln and was open in 2011. It is a high performance, low latency message queue system for handling large scale real-time data streams. The kafka has high reliability, expandability and fault tolerance, and can be used for constructing scenes such as real-time data pipelines, log aggregation, event processing, streaming processing and the like. It is mainly composed of producer, consumer and agency, and can support large-scale data stream processing and storage.

A flink is a streaming data processing framework that provides efficient data streaming capability, is capable of handling large-scale data, and supports event time handling, state management, precise primary semantics, and the like. The link can be used for real-time data analysis, real-time monitoring, real-time recommendation and other scenes. The system supports various data sources and data formats, and has good expansibility and flexibility.

As introduced in the background art, the consumption credit model based on big data and machine learning technology is widely applied in banking industry, and the development of data value exertion and assisting bank credit elimination business is promoted. In order to better monitor the effect of the lending model to prevent related risks, banks pay more and more attention to model monitoring and early warning disposal work. The existing model monitoring and early warning treatment method mainly monitors defined model indexes and generates early warning signals, the model monitoring indexes cannot be flexibly configured, early warning treatment also needs a large amount of manual intervention, the degree of automation and intellectualization is low, the efficiency is low, in order to solve the problems that the monitoring of a consumed credit model in the existing scheme is sent to staff in a short message notification mode, the staff carries out treatment manually, the treatment period is long, and accordingly relevant service development is delayed to cause loss.

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

In this embodiment, a method of monitoring a consumer credit model is provided, it being noted that the steps shown in the flowchart of the figures may be performed in a computer system, such as a set of computer executable instructions, and, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order other than that shown or described herein.

Fig. 1 is a flow diagram of a method of monitoring a consumer credit model provided in accordance with an embodiment of the present application.

As shown in fig. 1, the method comprises the steps of:

step S101, obtaining the number of positive cases and the number of negative cases of a consumption credit model in each month, wherein the number of positive cases is the number of cases where the predicted result and the actual situation of the consumption credit model are consistent, the number of negative cases is the number of cases where the predicted result and the actual situation of the consumption credit model are inconsistent, the consumption credit model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises the number obtained in a historical time period: user credit data and whether the user has repayment capability;

Specifically, the performance of the model may be monitored by monitoring the number of positive examples and the number of negative examples of the consumer credit model. The user has a repayment capability that indicates that the user may clear the previous borrow during the time period.

Step S102, determining model performance scores of each month according to all the number of positive examples and the number of negative examples, and determining that the final model performance score is the average of all the model performance scores;

step S102, determining the model performance score of each month according to all the number of positive examples and the number of negative examples, including: determining a total number of positive examples and a total number of negative examples, and determining a total number of positive examples and a total number of negative examples of each segment, wherein the total number of positive examples is the sum of all positive examples, the total number of negative examples is the sum of all negative examples, the total number of positive examples is the part of the number of positive examples, and the total number of negative examples is the part of the number of negative examples; determining the initial model performance score of each segment as a difference value between a first ratio and a second ratio, wherein the first ratio is the ratio of the accumulated positive number of cases to the total number of positive cases, and the second ratio is the ratio of the accumulated negative number of cases to the total number of negative cases; and determining the model performance score as the maximum value of all the initial model performance scores.

Specifically, for example, the number of accumulated positive examples in the segment 1 (exemplified by one month) is 100, the number of accumulated negative examples in the segment 1 is 50, the total number of positive examples is 300, the total number of negative examples is 500, the initial model performance score of the segment 1 is 100/300-50/500=0.2, the initial model performance scores of the segments 2 to N are sequentially calculated, the maximum value is taken as the model performance score (the model performance score of one month of the model performance scores, the rest months are the same, the segments are respectively carried out, the accumulated positive examples and the accumulated negative examples in each segment are obtained, and thus the model performance score of each month is obtained).

And step S103, determining whether a first early warning instruction is generated according to the range of the final model performance score, wherein the first early warning instruction is used for prompting that the consumption credit model needs to be optimized.

In the above steps, the model performance score of each month is obtained by monitoring the number of positive examples and the number of negative examples of the consumption credit model, the average value of the model performance scores of all months in one year is used as the final model performance score, and finally, the consumption credit model is prompted to be optimized according to the range of the final model performance score, so that the purpose of monitoring the consumption credit model is achieved, a large amount of manual processing is avoided in the middle process, and the problem that the consumption credit model in the prior art is monitored by sending the monitoring result to staff in a short message notification mode, the staff carries out manual treatment, the treatment period is long, and the development of related business is delayed to cause loss is solved.

Step S103, namely determining whether to generate the first early warning command according to the range of the final model performance score, including: determining to generate the first early warning instruction under the condition that the final model performance score is greater than or equal to a first performance score threshold value; and under the condition that the final model performance score is smaller than a first performance score threshold value, determining that the first early warning instruction is not generated.

Specifically, for example, the first performance score threshold is 1/5, and the final model performance score is 1/3, the first early warning instruction is generated, for example, the model performance score is 1/6, and the first early warning instruction is not generated.

In addition, the model service interface is switched to replace the current consumer credit model with the standby model, for example, in the case that the final model performance score is less than 1/5.

The service node A model state is 0 (i.e. stopped), and when other node model services are normal, the model service of the node A is automatically restarted, so as to avoid influencing the use of the subsequent model.

In addition, after determining that the first warning command is not generated, the method further includes: and generating a second early warning instruction to prompt that the consumed credit model needs to be replaced under the condition that the final model performance score is smaller than a second performance score threshold value.

For example, the first performance score threshold is 1/5, the second performance score threshold is 1/6, and the first performance score threshold is 1/7, then a second early warning instruction is generated to prompt that the consumption credit model needs to be replaced, and the consumption credit model is considered to be unsuitable for processing the current user credit data, and the model needs to be replaced.

In one embodiment of the present application, the method further includes: acquiring current credit data and training credit data of a user, and determining a first target feature data duty ratio in the current credit data and a second target feature data duty ratio in the training credit data, wherein the training credit data is used for training the consumption credit model, the first target feature data duty ratio is the duty ratio of the credit data of the target user in the current credit data, and the second target feature data duty ratio is the duty ratio of the credit data of the target user in the training credit data; determining a stability score based on PSI (i) = (Ai-B) ×ln (Ai/B), the stability score representing a score of stability of the consumer credit model, wherein PSI (i) is the stability score of month i, ai is the first target feature data duty cycle of month i, and B is the second target feature data duty cycle; and carrying out weighted summation on the stability scores to obtain a total stability score, and determining whether to generate a first early warning instruction according to the range of a final stability score, wherein the final stability score is an average value of the total stability score.

Specifically, the stability of the consumption credit model can be obtained by selecting the target user, checking the credit data of the target user in the current credit data and training the duty ratio of the credit data, determining whether the consumption credit model can be applicable to the current credit data or not by checking the stability of the consumption credit model, determining that the consumption credit model cannot be applicable to the current credit data if the stability score is greater than or equal to a corresponding threshold value, and replacing the consumption credit model.

For example: in calculating the final stability score of 1 month to 3 months, the stability score of 1 month is 0.1, with a weight of 0.2; the stability score for 2 months was 0.2, with a weight of 0.5; the 3 month stability score was 0.15, weight 0.3, then the 1 month to 3 month final stability score was: 0.1×0.2+0.2×0.5+0.15×0.3=0.165.

In one embodiment of the present application, after acquiring the current credit data and the training credit data of the user, the method further comprises: and determining a data null rate and a characteristic repetition rate, wherein the data null rate is the duty ratio of the data quantity with null characteristic values in the current credit data, and the characteristic repetition rate is the duty ratio of the data quantity with same characteristic values in the current credit data.

In particular, by monitoring the data null rate and the feature repetition rate, the detected data quality can be determined. A smaller empty rate indicates more perfect field information, and a larger empty rate indicates more field information missing. The repetition rate is too high, indicating that this feature is of reduced importance to lose the meaning of moulding.

In one embodiment of the present application, the method further includes: acquiring current credit data and training credit data of a user, and determining a current feature data standardized value and a training feature data standardized value according to z= (x-u)/s, wherein z is the current feature data standardized value or the training feature data standardized value, x is the current feature data of the current credit data or the training feature data of the training credit data, u is the average value of the current credit data or the average value of the training credit data, and s is the standard deviation of the current credit data or the standard deviation of the training credit data; and determining the characteristic central tendency drift degree of the consumption credit model as the ratio of a standardized average value difference value to the standardized value of the training characteristic data, wherein the standardized average value difference value is the difference value between the average value of all the standardized values of the training characteristic data and the average value of all the standardized values of the current characteristic data.

Specifically, the z-score normalization method, that is, z= (x-u)/s, is adopted to determine the current feature data normalization value and the training feature data normalization value, so that the two normalization values are taken into consideration to determine the feature concentration trend drift degree of the consumed credit model, so that the difference between the current credit data and the training credit data can be known, when the difference is greater than the threshold value, the consumed credit model is considered to be replaced, for example, the current feature data normalization values of three months are calculated together, namely, 0.5, 0.6 and 0.7 respectively, and the average value of all the current feature data normalization values is (0.5+0.6+0.7)/3=0.6, and the average value of all the training feature data normalization values is the same and is not repeated herein.

In one embodiment of the present application, after determining whether to generate the first early warning instruction according to the range in which the final model performance score is located, the method further includes: acquiring early warning logs generated by each consumption credit model in real time by adopting a jump, and transmitting all acquired early warning logs to kafka; and consuming data from the kafka by adopting a flink streaming real-time data technology, and analyzing early warning information to obtain an early warning event.

Specifically, a flime is adopted to acquire early warning logs generated by each consumption credit model in real time, the acquired early warning data stream is sent to kafka, and a flink stream type real-time data technology is adopted to analyze early warning information and analyze early warning events so as to achieve the purpose of processing early warning information in real time.

Most of the monitoring is based on a single model, the monitoring of a plurality of models of a plurality of systems can be realized, and the consumed credit model can be deployed in different systems in a bank.

In order to enable those skilled in the art to more clearly understand the technical solutions of the present application, the implementation procedure of the method for monitoring the consumer credit model of the present application will be described in detail below with reference to specific embodiments.

The embodiment relates to a specific method for monitoring a consumed credit model, as shown in fig. 2, comprising the following steps:

step S1: obtaining the number of positive cases and the number of negative cases of the consumption credit model in each month, wherein the number of positive cases is the number of examples of the prediction result and the real situation of the consumption credit model, the number of negative cases is the number of examples of the prediction result and the real situation of the consumption credit model, the consumption credit model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises the number of examples obtained in a historical time period: user credit data and whether the user has repayment capability; acquiring current credit data and training credit data of a user, determining a first target feature data duty ratio in the current credit data and a second target feature data duty ratio in the training credit data, wherein the training credit data are used for training a consumption credit model, the first target feature data duty ratio is the duty ratio of the credit data of the target user in the current credit data, and the second target feature data duty ratio is the duty ratio of the credit data of the target user in the training credit data;

Step S2: determining the total number of positive examples and the total number of negative examples, determining the accumulated positive examples and the accumulated negative examples of each segment, wherein the total number of positive examples is the sum of all positive examples, the total number of negative examples is the sum of all negative examples, the accumulated positive examples are parts of the positive examples, and the accumulated negative examples are parts of the negative examples; determining the initial model performance score of each segment as a difference value of a first ratio and a second ratio, wherein the first ratio is the ratio of the accumulated positive number of cases to the total number of positive cases, and the second ratio is the ratio of the accumulated negative number of cases to the total number of negative cases; determining the model performance score as the maximum value of all initial model performance scores;

step S3: determining to generate a first early warning instruction under the condition that the final model performance score is greater than or equal to a first performance score threshold; determining not to generate a first early warning instruction under the condition that the final model performance score is smaller than a first performance score threshold;

step S4: determining a stability score according to PSI (i) = (Ai-B) x ln (Ai/B), wherein the stability score represents the stability score of the consumption credit model, PSI (i) is the stability score of the ith month, ai is the first target characteristic data duty ratio of the ith month, and B is the second target characteristic data duty ratio; and carrying out weighted summation on the stability scores to obtain a total stability score, determining whether to generate a first early warning instruction according to the range of the final stability score, wherein the final stability score is the average value of the total stability score.

The model performance score of each month is obtained by monitoring the number of positive examples and the number of negative examples of the consumption credit model, the average value of the model performance scores of all months in one year is used as a final model performance score, and finally, the consumption credit model is prompted to be optimized according to the range of the final model performance score, so that the purpose of monitoring the consumption credit model is achieved, a large amount of manual processing is avoided in the middle process, and the problems that the monitoring of the consumption credit model in the prior art is achieved by sending the consumption credit model to staff in a short message notification mode, the staff carries out manual treatment, the treatment period is long, and the development of related business is delayed and loss is caused are solved.

It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.

The embodiment of the application also provides a monitoring device of the consumption credit model, and it is noted that the monitoring device of the consumption credit model of the embodiment of the application can be used for executing the monitoring method for the consumption credit model provided by the embodiment of the application. The device is used for realizing the above embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

The following describes a monitoring device for a consumer credit model provided in an embodiment of the present application.

Fig. 3 is a block diagram of a monitoring device of a consumer credit model provided in accordance with an embodiment of the present application. As shown in fig. 3, the apparatus includes:

a first obtaining unit 31, configured to obtain a number of positive cases and a number of negative cases of each month of a consumption credit model, where the number of positive cases is a number of examples where a predicted result and a true situation of the consumption credit model match, and the number of negative cases is a number of examples where the predicted result and the true situation of the consumption credit model do not match, where the consumption credit model is trained using a plurality of sets of training data, where each set of training data includes training data obtained during a historical period: user credit data and whether the user has repayment capability;

a first determining unit 32 configured to determine a model performance score for each month based on all the number of positive examples and the number of negative examples, and determine a final model performance score as an average of all the model performance scores;

and a second determining unit 33, configured to determine whether to generate a first early warning instruction according to the range of the performance score of the final model, where the first early warning instruction is used to prompt that the consumer credit model needs to be optimized.

In the device, the model performance score of each month is obtained by monitoring the number of positive examples and the number of negative examples of the consumption credit model, the average value of the model performance scores of all months in one year is used as the final model performance score, and finally, the consumption credit model is prompted to be optimized according to the range of the final model performance score, so that the purpose of monitoring the consumption credit model is achieved, a large amount of manual processing is avoided in the middle process, and the problem that the consumption credit model in the prior art is monitored by sending the monitoring result to staff in a short message notification mode, the staff carries out manual treatment, the treatment period is long, and the development of related business is delayed to cause loss is solved.

In one embodiment of the present application, the first determining unit includes a first determining module, a second determining module, and a third determining module, where the first determining module is configured to determine a total number of positive examples and a total number of negative examples, and determine a cumulative positive example number and a cumulative negative example number of each segment, where the total number of positive examples is a sum of all positive examples, the total number of negative examples is a sum of all negative examples, the cumulative positive example number is a portion of the positive example number, and the cumulative negative example number is a portion of the negative example number; the first determining module is configured to determine a difference between a first ratio and a second ratio of the initial model performance score of each segment, where the first ratio is a ratio of the number of accumulated positive examples to the total number of positive examples, and the second ratio is a ratio of the number of accumulated negative examples to the total number of negative examples; the third determining module is configured to determine the model performance score as a maximum of all of the initial model performance scores.

In an embodiment of the present application, the second determining unit includes a fourth determining module and a fifth determining module, where the fourth determining module is configured to determine to generate the first early warning instruction if the performance score of the final model is greater than or equal to a first performance score threshold; and the fifth determining module is used for determining that the first early warning instruction is not generated under the condition that the final model performance score is smaller than a first performance score threshold value.

In an embodiment of the present application, the apparatus further includes a generating unit, after determining that the first early warning instruction is not generated, the generating unit is configured to generate a second early warning instruction to prompt that the consumer credit model needs to be replaced if the performance score of the final model is less than a second performance score threshold.

In one embodiment of the present application, the apparatus further includes a second acquiring unit, a third determining unit, and a fourth determining unit, where the second acquiring unit is configured to acquire current credit data and training credit data of the user, and determine a first target feature data duty ratio in the current credit data and a second target feature data duty ratio in the training credit data, where the training credit data is used to train the consumption credit model, where the first target feature data duty ratio is a duty ratio of credit data of the target user in the current credit data, and where the second target feature data duty ratio is a duty ratio of credit data of the target user in the training credit data; a third determining unit configured to determine a stability score according to PSI (i) = (Ai-B) ×ln (Ai/B), where PSI (i) is the stability score of the i-th month, ai is the first target feature data duty ratio of the i-th month, and B is the second target feature data duty ratio; and the fourth determining unit is used for carrying out weighted summation processing on the stability scores to obtain a total stability score, and determining whether to generate a first early warning instruction according to the range of a final stability score, wherein the final stability score is an average value of the total stability score.

In one embodiment of the present application, the apparatus further includes a fifth determining unit, after acquiring the current credit data and the training credit data of the user, the fifth determining unit is configured to determine a data null rate and a feature repetition rate, where the data null rate is a duty ratio of a data amount with null feature values in the current credit data, and the feature repetition rate is a duty ratio of a data amount with identical feature values in the current credit data.

In one embodiment of the present application, the apparatus further includes a third acquiring unit and a sixth determining unit, where the third acquiring unit is configured to acquire current credit data and training credit data of the user, and determine a current feature data standardized value and a training feature data standardized value according to z= (x-u)/s, where z is the current feature data standardized value or the training feature data standardized value, x is the current feature data of the current credit data or the training feature data of the training credit data, u is an average value of the current credit data or the training credit data, and s is a standard deviation of the current credit data or a standard deviation of the training credit data; and the sixth determining unit is used for determining that the characteristic central tendency drift degree of the consumption credit model is the ratio of a standardized average value difference value to the standardized value of the training characteristic data, wherein the standardized average value difference value is the difference value between the average value of all the standardized values of the training characteristic data and the average value of all the standardized values of the current characteristic data.

In an embodiment of the present application, the apparatus further includes a first processing unit and a second processing unit, where after determining whether to generate a first early warning instruction according to a range where the performance score of the final model is located, the first processing unit is configured to acquire early warning logs generated by each of the consumer credit models in real time by using a flime, and send all acquired early warning logs to kafka; and the second processing unit is used for consuming data from the kafka by adopting a link streaming real-time data technology and analyzing early warning information to obtain an early warning event.

The monitoring device of the consumption credit model comprises a processor and a memory, wherein the first acquisition unit, the first determination unit, the second determination unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions. The modules are all located in the same processor; alternatively, the above modules may be located in different processors in any combination.

The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one, and the problem that the monitoring of the consumption credit model of the existing scheme is sent to the staff in a short message notification mode by adjusting kernel parameters, the staff carries out the treatment manually, the treatment period is longer, and the development of related services is delayed to cause loss is solved.

The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.

The embodiment of the invention provides a computer readable storage medium, which comprises a stored program, wherein the program is used for controlling equipment where the computer readable storage medium is located to execute a monitoring method of a consumed credit model.

The embodiment of the invention provides a processor which is used for running a program, wherein the monitoring method of the consumed credit model is executed when the program runs.

The embodiment of the invention provides equipment, which comprises a processor, a memory and a program stored in the memory and capable of running on the processor, wherein the processor realizes at least the following steps when executing the program: obtaining the number of positive cases and the number of negative cases of the consumption credit model in each month, wherein the number of positive cases is the number of examples of the prediction result and the real situation of the consumption credit model, the number of negative cases is the number of examples of the prediction result and the real situation of the consumption credit model, the consumption credit model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises the number of examples obtained in a historical time period: user credit data and whether the user has repayment capability; determining a model performance score of each month according to all the number of positive examples and the number of negative examples, and determining a final model performance score as an average of all the model performance scores; and determining whether to generate a first early warning instruction according to the range of the final model performance score, wherein the first early warning instruction is used for prompting that the consumed credit model needs to be optimized. The device herein may be a server, PC, PAD, cell phone, etc.

Optionally, determining the model performance score of each month according to all the number of positive examples and the number of negative examples includes: determining a total number of positive examples and a total number of negative examples, and determining a total number of positive examples and a total number of negative examples of each segment, wherein the total number of positive examples is the sum of all positive examples, the total number of negative examples is the sum of all negative examples, the total number of positive examples is the part of the number of positive examples, and the total number of negative examples is the part of the number of negative examples; determining the initial model performance score of each segment as a difference value between a first ratio and a second ratio, wherein the first ratio is the ratio of the accumulated positive number of cases to the total number of positive cases, and the second ratio is the ratio of the accumulated negative number of cases to the total number of negative cases; and determining the model performance score as the maximum value of all the initial model performance scores.

Optionally, after determining that the first warning command is not generated, the method further includes: and generating a second early warning instruction to prompt that the consumed credit model needs to be replaced under the condition that the final model performance score is smaller than a second performance score threshold value.

Optionally, the method further comprises: acquiring current credit data and training credit data of a user, and determining a first target feature data duty ratio in the current credit data and a second target feature data duty ratio in the training credit data, wherein the training credit data is used for training the consumption credit model, the first target feature data duty ratio is the duty ratio of the credit data of the target user in the current credit data, and the second target feature data duty ratio is the duty ratio of the credit data of the target user in the training credit data; determining a stability score based on PSI (i) = (Ai-B) ×ln (Ai/B), the stability score representing a score of stability of the consumer credit model, wherein PSI (i) is the stability score of month i, ai is the first target feature data duty cycle of month i, and B is the second target feature data duty cycle; and carrying out weighted summation on the stability scores to obtain a total stability score, and determining whether to generate a first early warning instruction according to the range of a final stability score, wherein the final stability score is an average value of the total stability score.

Optionally, after acquiring the current credit data and the training credit data of the user, the method further comprises: and determining a data null rate and a characteristic repetition rate, wherein the data null rate is the duty ratio of the data quantity with null characteristic values in the current credit data, and the characteristic repetition rate is the duty ratio of the data quantity with same characteristic values in the current credit data.

Optionally, the method further comprises: acquiring current credit data and training credit data of a user, and determining a current feature data standardized value and a training feature data standardized value according to z= (x-u)/s, wherein z is the current feature data standardized value or the training feature data standardized value, x is the current feature data of the current credit data or the training feature data of the training credit data, u is the average value of the current credit data or the average value of the training credit data, and s is the standard deviation of the current credit data or the standard deviation of the training credit data; and determining the characteristic central tendency drift degree of the consumption credit model as the ratio of a standardized average value difference value to the standardized value of the training characteristic data, wherein the standardized average value difference value is the difference value between the average value of all the standardized values of the training characteristic data and the average value of all the standardized values of the current characteristic data.

The present application also provides a computer program product adapted to perform a program initialized with at least the following method steps when executed on a data processing device: obtaining the number of positive cases and the number of negative cases of the consumption credit model in each month, wherein the number of positive cases is the number of examples of the prediction result and the real situation of the consumption credit model, the number of negative cases is the number of examples of the prediction result and the real situation of the consumption credit model, the consumption credit model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises the number of examples obtained in a historical time period: user credit data and whether the user has repayment capability; determining a model performance score of each month according to all the number of positive examples and the number of negative examples, and determining a final model performance score as an average of all the model performance scores; and determining whether to generate a first early warning instruction according to the range of the final model performance score, wherein the first early warning instruction is used for prompting that the consumed credit model needs to be optimized.

The application also provides a monitoring system of the consumed credit model, which comprises: the system comprises one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising a monitoring method for executing any one of the consumer credit models described above. The model performance score of each month is obtained by monitoring the number of positive examples and the number of negative examples of the consumption credit model, the average value of the model performance scores of all months in one year is used as a final model performance score, and finally, the consumption credit model is prompted to be optimized according to the range of the final model performance score, so that the purpose of monitoring the consumption credit model is achieved, a large amount of manual processing is avoided in the middle process, and the problems that the monitoring of the consumption credit model in the prior art is achieved by sending the consumption credit model to staff in a short message notification mode, the staff carries out manual treatment, the treatment period is long, and the development of related business is delayed and loss is caused are solved.

It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.

From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

1) According to the monitoring method of the consumption credit model, the model performance score of each month is obtained by monitoring the number of positive examples and the number of negative examples of the consumption credit model, the average value of the model performance scores of all months in one year is used as the final model performance score, and finally, the consumption credit model is prompted to be optimized according to the range of the final model performance score, so that the purpose of monitoring the consumption credit model is achieved, a large amount of manual processing is avoided in the middle process, the problem that the monitoring of the consumption credit model in the traditional scheme is sent to staff in a short message notification mode, the staff carries out manual treatment, the treatment period is long, and the problem of loss caused by delay of development of related business is solved.

2) According to the monitoring device for the consumption credit model, the model performance score of each month is obtained by monitoring the number of positive examples and the number of negative examples of the consumption credit model, the average value of the model performance scores of all months in one year is used as the final model performance score, and finally, the consumption credit model is prompted to be optimized according to the range of the final model performance score, so that the purpose of monitoring the consumption credit model is achieved, a large amount of manual processing is avoided in the middle process, the problem that the monitoring of the consumption credit model in the existing scheme is sent to staff in a short message notification mode, the staff carries out manual treatment, the treatment period is long, and accordingly, the development of related business is delayed to cause loss is solved.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the same, but rather, various modifications and variations may be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method of monitoring a consumer credit model, comprising:

2. The method of claim 1, wherein determining a model performance score for each month based on all of the number of positive examples and the number of negative examples comprises:

determining the total number of positive examples and the total number of negative examples, and determining the total number of accumulated positive examples and the total number of accumulated negative examples of each segment, wherein the total number of positive examples is the sum of all positive examples, the total number of negative examples is the sum of all negative examples, the total number of accumulated positive examples is the part of the number of positive examples, and the total number of accumulated negative examples is the part of the number of negative examples;

Determining the initial model performance score of each segment as a difference value of a first ratio and a second ratio, wherein the first ratio is the ratio of the accumulated positive case number to the total positive case number, and the second ratio is the ratio of the accumulated negative case number to the total negative case number;

and determining the model performance score as the maximum value of all the initial model performance scores.

3. The method of claim 1, wherein determining whether to generate the first warning instruction based on the range within which the final model performance score is located comprises:

determining to generate the first early warning instruction under the condition that the final model performance score is greater than or equal to a first performance score threshold value;

and under the condition that the final model performance score is smaller than a first performance score threshold value, determining that the first early warning instruction is not generated.

4. The method of claim 3, wherein after determining that the first warning instruction is not generated, the method further comprises:

and generating a second early warning instruction to prompt that the consumed credit model needs to be replaced under the condition that the final model performance score is smaller than a second performance score threshold value.

5. The method according to claim 1, wherein the method further comprises:

acquiring current credit data and training credit data of a user, and determining a first target feature data duty ratio in the current credit data and a second target feature data duty ratio in the training credit data, wherein the training credit data is used for training the consumption credit model, the first target feature data duty ratio is the duty ratio of the credit data of the target user in the current credit data, and the second target feature data duty ratio is the duty ratio of the credit data of the target user in the training credit data;

determining a stability score that characterizes a stability of the consumer credit model from PSI (i) = (Ai-B) ×ln (Ai/B), wherein PSI (i) is the stability score for month i, ai is the first target feature data duty cycle for month i, and B is the second target feature data duty cycle;

and carrying out weighted summation on the stability scores to obtain a total stability score, and determining whether to generate a first early warning instruction according to the range of a final stability score, wherein the final stability score is an average value of the total stability score.

6. The method of claim 5, wherein after obtaining the current credit data and the training credit data for the user, the method further comprises:

determining a data null rate and a characteristic repetition rate, wherein the data null rate is the duty ratio of the data quantity with null characteristic values in the current credit data, and the characteristic repetition rate is the duty ratio of the data quantity with same characteristic values in the current credit data.

7. The method according to claim 1, wherein the method further comprises:

acquiring current credit data and training credit data of a user, and determining a current feature data standardized value and a training feature data standardized value according to z= (x-u)/s, wherein z is the current feature data standardized value or the training feature data standardized value, x is the current feature data of the current credit data or the training feature data of the training credit data, u is the average value of the current credit data or the average value of the training credit data, and s is the standard deviation of the current credit data or the standard deviation of the training credit data;

And determining the characteristic concentrated trend drift degree of the consumption credit model as a ratio of a standardized average value difference value to the standardized value of the training characteristic data, wherein the standardized average value difference value is a difference value between an average value of all the standardized values of the training characteristic data and an average value of all the standardized values of the current characteristic data.

8. The method according to any one of claims 1 to 7, wherein after determining whether to generate a first warning instruction according to the range in which the final model performance score is located, the method further comprises:

acquiring early warning logs generated by each consumption credit model in real time by adopting a jump, and transmitting all acquired early warning logs to kafka;

and consuming data from the kafka by adopting a flink streaming real-time data technology, and analyzing early warning information to obtain an early warning event.

9. A monitoring device for a consumer credit model, comprising:

10. A monitoring system for a consumer credit model, comprising: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising a monitoring method for executing the consumer credit model of any of claims 1-8.