CN114693127A

CN114693127A - Evaluation information determination method, device, equipment and computer storage medium

Info

Publication number: CN114693127A
Application number: CN202210335807.5A
Authority: CN
Inventors: 朱翔
Original assignee: China Construction Bank Corp; CCB Finetech Co Ltd
Current assignee: China Construction Bank Corp; CCB Finetech Co Ltd
Priority date: 2022-03-31
Filing date: 2022-03-31
Publication date: 2022-07-01

Abstract

The application discloses an evaluation information determination method, an evaluation information determination device, an evaluation information determination equipment and a computer storage medium, which can train a preset neural network through user historical service data to obtain a first risk identification model and first quality evaluation information of the model. And generating service characteristic information by using the target model to be evaluated, and retraining the neural network through the service characteristic information and the historical service data of the user, so that a second risk identification model and corresponding second quality evaluation information can be obtained after retraining by introducing new characteristic information (namely service characteristic information). And then, according to a comparison result obtained by the two quality evaluation information, whether the training degree of the risk identification model obtained by two times of training is improved or not can be reflected, so that whether the target model (namely the item identification model) is effective or not can be quickly verified and evaluated, and the reliability of the item identification model in the risk item identification scene is favorably improved.

Description

Evaluation information determination method, device, equipment and computer storage medium

Technical Field

The application belongs to the technical field of finance, and particularly relates to an evaluation information determination method, an evaluation information determination device, evaluation information determination equipment and a computer storage medium.

Background

Typically, financial institutions identify various types of risk events in a credit management business scenario for a letter. However, the conventional credit risk causes are complicated and complicated, so that a credit management business system generates a large amount of credit risk troubleshooting items, and credit practitioners and credit managers need to manually identify, judge and finally review to confirm the true validity of the risk items. Therefore, if the identification capability of the business system is insufficient, a large number of invalid risk items can be identified, and then a large amount of manpower is consumed for rechecking, so that the labor cost is high, the efficiency is low, and the reliability is low.

Disclosure of Invention

The embodiment of the application provides an evaluation information determination method, an evaluation information determination device, evaluation information determination equipment and a computer storage medium, which can improve the reliability of risk item identification in a credit management scene.

An embodiment of a first aspect of the present application provides an evaluation information determining method, including:

acquiring historical service data of a user;

training a preset neural network through user historical service data to obtain a first risk identification model and first quality evaluation information of the first risk model;

generating service characteristic information by using a target model to be evaluated;

training a neural network through the service characteristic information and the historical service data of the user to obtain a second risk identification model and second quality evaluation information of the second risk model;

and determining target evaluation information of the target model according to the comparison result of the first quality evaluation information and the second quality evaluation information.

In an embodiment of the first aspect of the present application, the user historical service data corresponds to a plurality of feature dimensions, and the first quality evaluation information includes a first accuracy and/or a first recall rate;

training a preset neural network through historical service data of a user to obtain a first risk identification model and first quality evaluation information of the first risk model, wherein the first quality evaluation information comprises the following steps:

inputting historical service data of a user into a neural network;

training a neural network according to a plurality of characteristic dimensions to obtain a first risk model and first output information of the first risk model;

and calculating according to the first output information to obtain a first accuracy rate and/or a first recall rate.

In an embodiment of the first aspect of the present application, the generating the service characteristic information by using the target model to be evaluated as the item identification model includes:

generating business feature information based on at least one transaction feature of the transaction recognition model,

the characteristic dimension corresponding to the service characteristic information is different from a plurality of characteristic dimensions of the historical service data of the user.

In an embodiment of the first aspect of the application, the second quality-assessment information comprises a second accuracy and/or a second recall,

training a neural network through the service characteristic information and the user historical service data to obtain a second risk identification model and second quality evaluation information of the second risk model, wherein the second quality evaluation information comprises:

inputting historical service data of a user into a neural network;

training a neural network according to the characteristic dimensionality corresponding to the business characteristic information and a plurality of characteristic dimensionalities corresponding to the historical business data of the user to obtain a second risk identification model and second output information of the second risk model;

and calculating according to the second output information to obtain a second accuracy and/or a second recall rate.

In an embodiment of the first aspect of the present application, the object model is a transaction identification model, and after the object evaluation information of the object model, the method further includes:

acquiring user service data to be identified under the condition that the target evaluation information meets a preset condition;

and performing feature extraction on user service data to be identified through the item identification model to obtain item identification information.

An embodiment of a second aspect of the present application provides an evaluation information determination apparatus, including:

the first acquisition module is used for acquiring historical service data of a user;

the first training module is used for training a preset neural network through user historical service data to obtain a first risk identification model and first quality evaluation information of the first risk model;

the generating module is used for generating service characteristic information by utilizing a target model to be evaluated;

the second training module is used for training the neural network through the service characteristic information and the historical service data of the user to obtain a second risk identification model and second quality evaluation information of the second risk model;

and the determining module is used for determining the target evaluation information of the target model according to the first quality evaluation information and the second quality evaluation information.

In an embodiment of the second aspect of the present application, the user historical service data corresponds to a plurality of feature dimensions, and the first quality evaluation information includes a first accuracy and/or a first recall rate;

the first training module includes:

the first input submodule is used for inputting the historical service data of the user into the neural network;

the first training submodule is used for training the neural network according to a plurality of characteristic dimensions to obtain a first risk model and first output information of the first risk model;

and the first calculation submodule is used for calculating according to the first output information to obtain a first accuracy and/or a first recall rate.

In an embodiment of the second aspect of the present application, the target model is a transaction identification model, and the generating module is specifically configured to:

In an embodiment of the second aspect of the application, the second quality-assessment information comprises a second accuracy and/or a second recall,

a second training module comprising:

the second input submodule is used for inputting the historical service data of the user into the neural network;

the second training submodule is used for training the neural network according to the characteristic dimensionality corresponding to the business characteristic information and the characteristic dimensionalities corresponding to the historical business data of the user to obtain a second risk identification model and second output information of the second risk model;

and the second calculation submodule is used for calculating according to the second output information to obtain a second accuracy and/or a second recall rate.

In an embodiment of the second aspect of the present application, the target model is a transaction identification model, and the apparatus further includes:

the second acquisition module is used for acquiring user service data to be identified under the condition that the target evaluation information meets the preset condition;

and the extraction module is used for extracting the characteristics of the user service data to be identified through the item identification model to obtain item identification information.

An embodiment of a third aspect of the present application provides a computer apparatus, the apparatus comprising: a processor and a memory storing computer program instructions;

the processor, when executing the computer program instructions, implements the rating information determining method as described in any of the embodiments of the first aspect.

An embodiment of a fourth aspect of the present application provides a computer storage medium having computer program instructions stored thereon, which when executed by a processor, implement the evaluation information determination method according to any one of the embodiments of the first aspect.

An embodiment of a fifth aspect of the present application provides a computer program product, wherein instructions of the computer program product, when executed by a processor of a computer device, cause the computer device to execute the evaluation information determination method according to any one of the embodiments of the first aspect.

According to the evaluation information determining method, the evaluation information determining device, the evaluation information determining equipment and the computer storage medium, the preset neural network can be trained through the historical service data of the user, and the first risk identification model and the first quality evaluation information of the model are obtained. And generating service characteristic information by using the target model to be evaluated, and retraining the neural network through the service characteristic information and the historical service data of the user, so that a second risk identification model and corresponding second quality evaluation information can be obtained after retraining by introducing new characteristic information (namely service characteristic information). And then, according to a comparison result obtained by the two quality evaluation information, whether the training degree of the risk identification model obtained by two times of training is improved or not can be reflected, so that whether the target model (namely the item identification model) is effective or not can be quickly verified and evaluated, and the reliability of the item identification model in the risk item identification scene is favorably improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly described below, and for those skilled in the art, other drawings may be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flowchart of an evaluation information determination method according to an embodiment of the present application;

FIG. 2 is a flowchart illustrating an evaluation information determination method according to an embodiment of the present application

Fig. 3 is a schematic structural diagram of an evaluation information determination apparatus according to another embodiment of the present application;

fig. 4 is a schematic hardware structure diagram of a computer device according to still another embodiment of the present application.

Detailed Description

Features and exemplary embodiments of various aspects of the present application will be described in detail below, and in order to make objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended to be illustrative only and are not intended to be limiting. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present application by illustrating examples thereof.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

A credit management business system of a financial institution generally needs to identify various risk events of various links such as before, during and after credit in a credit scene through a credit management model. The risk events are embodied as credit risk item information in a business system, and in the related technology, because the risk events have various and complex causes, credit management models of various credit fields and various links are different. Risk items are identified for users by each credit management model, identification results are gathered to a credit management service system, and a large amount of generated credit risk troubleshooting items need to be identified, judged and finally audited manually by related personnel, so that the time and energy of the personnel are wasted, the labor cost is increased, and the credit management efficiency is low. Because the risk violation identified by the credit management model needs to manually confirm whether the violation is a real risk, if the credit management model configured by the business system has low effectiveness, a large number of invalid risk items can be identified, and a large amount of manpower is consumed for rechecking.

In order to solve the prior art problems, embodiments of the present application provide an evaluation information determination method, apparatus, device, and computer storage medium. First, a method for determining evaluation information provided in the embodiment of the present application will be described below. It should be noted that, in the technical solution of the present application, the acquisition, storage, use, processing and the like of the related data of the assets, finance and the like all conform to the related regulations of the national laws and regulations.

Fig. 1 is a schematic flowchart illustrating an evaluation information determination method according to an embodiment of the present application. As shown in fig. 1, the method includes steps S101 to S105:

s101, acquiring historical service data of a user;

s102, training a preset neural network through user historical service data to obtain a first risk identification model and first quality evaluation information of the first risk model;

s103, generating service characteristic information by using a target model to be evaluated;

s104, training the neural network through the service characteristic information and the historical service data of the user to obtain a second risk identification model and second quality evaluation information of the second risk model;

s105, determining target evaluation information of the target model according to the first quality evaluation information and the second quality evaluation information.

According to the embodiment of the application, the preset neural network can be trained through the historical service data of the user, and the first risk identification model and the first quality evaluation information of the model are obtained. And generating service characteristic information by using the target model to be evaluated, and retraining the neural network through the service characteristic information and the historical service data of the user, so that a second risk identification model and corresponding second quality evaluation information can be obtained after retraining by introducing new characteristic information (namely service characteristic information). And then, according to a comparison result obtained by the two quality evaluation information, whether the training degree of the risk identification model obtained by two times of training is improved or not can be reflected, so that whether the target model (namely the item identification model) is effective or not can be quickly verified and evaluated, and the reliability of the item identification model in the risk item identification scene is favorably improved.

In some embodiments, the historical traffic data of the user obtained in step S101 may be the traffic data of the user over a historical period (e.g., one year or 6 months).

Illustratively, the user historical traffic data may include: the system comprises more than two kinds of characteristic information of borrower (user) credit investigation information, guarantee slow-release measure information, user early warning information, important credit risk item information, borrower rating information and the like. The user service data may also include other service data capable of reflecting risk events, and this embodiment is not limited.

For example, the characteristic information included in the historical business data of the user can be determined according to expert experience.

In this embodiment, each item of information included in the user historical service data corresponds to one feature dimension. For example, the credit investigation information of the borrower can correspond to credit investigation condition characteristics, the guarantee slow-release measure information can correspond to risk slow-release capability characteristics, the user early warning information can correspond to early warning condition characteristics, the major credit risk item information can correspond to risk condition characteristics, and the borrower rating information can correspond to credit rating characteristics.

Each feature is a dimension, and in order to adapt to the situation that risk causes existing in the credit risk identification scene are complicated, the user historical business data can correspond to a plurality of feature dimensions. In this embodiment, based on these feature dimensions, the historical service data of the user may be input to the neural network for training, and a reference model, that is, a first risk identification model, may be established.

For example, the preset neural network may be a convolutional neural network model built by using a tensflo framework, data related to credit and risk of the user is extracted as features (i.e., the plurality of feature dimensions), input data (such as historical business data) corresponding to the features is obtained, so as to obtain an identification result of whether the user will generate an actual default, and the result is represented by a probability value.

In this embodiment, after the user historical service data is obtained, the step S102 is executed by using the user historical service data as a sample set, and a preset neural network is trained through the user historical service data to obtain a first risk identification model and first quality evaluation information of the first risk model.

The first quality evaluation information is information for evaluating the degree of superiority and inferiority of the first risk identification model. For example, the first quality-assessment information may include a first accuracy rate and/or a first recall rate. Wherein, the accuracy rate is the accuracy degree of the first risk identification model, and the recall rate is the proportion of the samples predicted to be positive in the samples which are actually positive.

The first accuracy and the first recall rate are used as indexes for evaluating a first risk identification model obtained by training of user historical business data so as to evaluate the quality of the model, such as prediction effect, performance and the like. In particular, these indicators may be calculated by the output probability of the first risk identification model. Therefore, in this embodiment, step S102 may specifically include steps S1021 to S1023:

s1021, inputting historical service data of the user into a preset neural network;

s1022, training the neural network according to the multiple feature dimensions to obtain a first risk model and first output information of the first risk model;

s1023, calculating according to the first output information to obtain a first accuracy rate and/or a first recall rate.

In this embodiment, the user historical business data is a large amount of user credit data in a past period, and each data includes information on whether a default is included in addition to the information on the plurality of characteristic dimensions. These user historical business data form a sample set containing positive and negative samples, and a risk identification model is input through step S1021.

The preset neural network is used as a convolutional neural network model, and in the process of training the neural network through step S1022, training may be performed according to a plurality of feature dimensions based on the user historical service data, and a first risk identification model and first output information of the model are obtained. The first output information is the result of the training.

Based on the training result, step S1023 may be performed to calculate a first accuracy rate and/or a first recall rate. Wherein, according to the training result and the real sample set, a confusion matrix can be formed, namely:

TP: the sample is positive, and the output result is positive;

FP: the sample is negative, and the output result is positive;

TN: the sample is negative, and the output result is negative;

FN: the sample is positive and the output is negative.

Thus, the first accuracy is the ratio of the correct number to the total number of samples, and can be expressed by equation (1):

first accuracy rate (TP + TN)/(TP + FP + TN + FN) (1)

The proportion of the samples with positive output results in the samples with true positive first recall rate can be expressed by the following formula (2):

first recall ratio TP/(TP + FN) (2)

In this embodiment, both the first accuracy and the first recall rate may be used to evaluate the result of the model training in this round, that is, evaluate the prediction effect and performance of the first risk identification model. For example, the higher the value of the first accuracy rate is, the higher the prediction accuracy characterizing the first risk identification model is to some extent, and the higher the value of the first recall rate is, the smaller the false positive rate characterizing the first risk identification model can be.

In the credit management scenario, the main function of the credit management model is risk item identification, that is, the credit management model serves as an item identification model, and various identified risks are finally embodied as credit default risks, so in the embodiment of the application, the preset item identification model serves as a target model to be evaluated, and validity (such as available or unavailable) verification is performed on the target model through the established first risk identification model serving as a reference model. Specifically, after training the neural network by using the historical service data of the user, the method may execute step S103, and generate a new feature (i.e., a feature dimension corresponding to the service feature information) by using the target model to be evaluated, for retraining the preset neural network.

In some specific embodiments, in step S103, in an embodiment of the first aspect of the present application, the generating the service characteristic information by using the target model to be evaluated as the item identification model specifically includes:

the characteristic dimension corresponding to the service characteristic information is different from a plurality of characteristic dimensions corresponding to the user historical service data.

In the present embodiment, in order to evaluate the event recognition model, the object model, which is the evaluated event recognition model, may be subjected to feature extraction for the event recognition model, so as to generate corresponding business feature information. The item identification model is used as a credit management model, and is generally configured with one or more items to perform user credit management (such as credit approval) according to the identification result of the item, for example, the item includes "whether to allow the enterprise with risk behaviors," whether to allow the business with head office approval as negative or in future to allow business division approval, "and the like. In order to verify the effectiveness of the event recognition model, in the present embodiment, at least one event feature of the event recognition model, such as the event feature of "whether to admit the enterprise with risky behaviors" as described above, is used as a new feature of the preset neural network, and the preset neural network is retrained. That is, since the event recognition model is abstracted to a new feature dimension as the business feature information and participates in the neural network training, the feature dimension corresponding to the business feature information is different from the plurality of feature dimensions corresponding to the user historical business data. Therefore, if the introduced business feature information can improve the quality of the trained second risk identification model, the item identification model is proved to be effective and can be put into practical use.

For example, in the process of retraining the preset neural network through step S104, a second risk recognition model and second quality evaluation information of the model may be obtained, where the second quality evaluation information is information for evaluating the degree of superiority and inferiority of the second risk recognition model. For example, the second quality-assessment information may include a second accuracy rate and/or a second recall rate. Specifically, in step S104, the neural network is trained through the service feature information and the user historical service data to obtain a second risk identification model and second quality evaluation information of the second risk model, which may specifically include S1041 to S1043:

s1041, inputting historical service data of the user into a neural network;

s1042 trains a neural network according to the characteristic dimensionality corresponding to the business characteristic information and the characteristic dimensionalities corresponding to the historical business data of the user to obtain a second risk identification model and second output information of the second risk model;

and S1043, calculating according to the second output information to obtain a second accuracy rate and/or a second recall rate.

In this embodiment, the training sample set may adopt the same sample set as that in step S102, that is, the user historical service data, and step S1041 is executed to input the training sample set into a preset neural network.

Based on the input historical service data of the user, step S1042 is executed, the neural network is trained according to the feature dimensions corresponding to the service feature information and the feature dimensions corresponding to the historical service data of the user, and second output information of the second risk identification model and the second risk model can be obtained. The second output information is the result of the training.

Based on the training result, step S1043 may be performed to calculate a second accuracy and/or a second recall rate. The calculation method of the second accuracy rate is the same as the calculation method of the first accuracy rate, and the calculation method of the second recall rate is the same as the calculation method of the first recall rate, which is not described herein again.

In this embodiment, both the second accuracy and the second recall rate may be used to evaluate the result of the current round of model training, that is, to evaluate the prediction effect and performance of the second risk identification model.

After obtaining the second quality evaluation information for evaluating the prediction effect and performance of the second risk identification model, S105 may be executed to determine the target evaluation information of the target model according to the first quality evaluation information and the second quality evaluation information. By comparing the first quality evaluation information and the second quality evaluation information during the step S105, if the accuracy rate is improved and/or the recall rate is improved, the added service characteristic information can be proved to be valid. For example, the added service characteristic information is that the 'approval of the general bank is a passing approval of a denied or a continued service division', the obtained second accuracy rate is higher than the first accuracy rate, the second recall rate is higher than the first recall rate, and the 'approval of the general bank is a passing approval of a denied or a continued service division' can be judged to have higher correlation with an actual violation, so that the corresponding item identification model is judged to be effective, and evaluation information describing the effectiveness of the corresponding item identification model is generated. Otherwise, if the accuracy and/or recall rate of the second information is not improved compared with the first information, the item recognition model can be judged to be invalid, and evaluation information describing the invalidity of the item recognition model is generated.

According to the embodiment of the application, the trained risk identification models (including the first risk identification model and the second risk identification model) are used as the measuring standard, the item identification models (namely credit management models) of the credit fields form new features, the new features participate in retraining of the risk identification models, and the effectiveness of the item identification models is judged by improving the effectiveness of the risk identification models. Compared with the traditional method of confirming the authenticity of items generated by a credit management model by adopting a manual rechecking mode, the method and the device have higher efficiency and can greatly reduce the human input cost. Compared with a manual rechecking mode, the embodiment of the application can avoid the limitation of professional abilities of related personnel, and is favorable for obtaining more objective effective target evaluation information.

When the item identification model is validated based on the target evaluation information, the model may be applied to risk item identification. Therefore, optionally, in this embodiment of the application, after determining the target evaluation information of the target model, as shown in fig. 2, the method may further include:

s106, under the condition that the target evaluation information meets the preset condition, acquiring user service data to be identified;

s107, feature extraction is carried out on user service data to be recognized through the item recognition model, and item recognition information is obtained.

The user service data to be recognized may include service data required for the application field of the corresponding item recognition model. For example, in a pre-loan link of a credit loan for public credit, identification of user rating and credit granting needs to be performed according to user service data, and therefore, in a transaction identification model applied in the link, collected user service data to be identified may include data information of corresponding feature dimensions, for example, information of credit investigation of a borrower (i.e., a user), information of guaranteed slow-release measures, information of rating of the borrower, and the like. Correspondingly, the item identification model needs to identify items such as user rating items and whether credit is granted.

It should be understood that, in different links of different fields, the feature dimensions of the user service data to be identified, which are required by the corresponding item identification model, are different, and the items to be identified are different.

Therefore, in this embodiment, user service data to be identified is obtained, and the corresponding item identification model is input to perform feature extraction, so as to obtain item identification information. For example, the item identification model may be a Logistic Regression (LR) model, a Support Vector Machine (SVM) model, a Random Forest (RF) model, or a decision tree model, and the present embodiment is not limited to the above.

According to the embodiment of the application, the reliability identification of the user item risk can be realized after the effectiveness of the item identification model (namely the credit management model) is verified based on the benchmark risk identification model. This may avoid a large number of invalid credit management models being used for actual risk identification, thereby increasing the efficiency of credit management.

The evaluation information determination method according to the embodiment of the present application is described in detail herein with reference to fig. 1 and 2, and the apparatus of the embodiment of the present application will be described in detail below with reference to fig. 3.

Fig. 3 is a schematic structural diagram illustrating an evaluation information determination apparatus according to an embodiment of the present application. As shown in fig. 3, the apparatus includes:

a first obtaining module 301, configured to obtain historical service data of a user;

the first training module 302 is configured to train a neural network through user historical service data to obtain a first risk identification model and first quality evaluation information of the first risk model;

a generating module 303, configured to generate service characteristic information by using a target model to be evaluated;

a second training module 304, configured to train the neural network through the service feature information and the historical service data of the user, to obtain a second risk identification model and second quality evaluation information of the second risk model;

a determining module 305, configured to determine target evaluation information of the target model according to a comparison result of the first quality evaluation information and the second quality evaluation information.

In some embodiments, the historical traffic data of the user obtained by the first obtaining module 301 may be the traffic data of the user in a historical period (e.g., one year or 6 months).

Illustratively, the user historical traffic data may include: the system comprises more than two kinds of characteristic information of borrower (user) credit investigation information, guarantee slow-release measure information, user early warning information, important credit risk item information, borrower rating information and the like. The user service data may also include other service data capable of reflecting risk events, and this embodiment is not limited. In this embodiment, each item of information included in the user historical service data corresponds to one feature dimension.

Each feature is a dimension, and in order to adapt to the situation that risk causes existing in the credit risk identification scene are complicated, the user historical business data can correspond to a plurality of feature dimensions. In this embodiment, based on these feature dimensions, the user historical service data may be input to a neural network for training, and a first risk identification model of the reference may be established.

For example, the preset neural network may be a convolutional neural network model built by using a tensflo framework, data related to credit and risk of the user is extracted as features (namely, the plurality of feature dimensions), input data corresponding to the features is obtained, so as to obtain an identification result of whether the user will generate an actual default, and the result is represented by a probability value.

Optionally, in some embodiments, the historical service data of the user corresponds to a plurality of feature dimensions, and the first quality evaluation information includes a first accuracy and/or a first recall rate; the first training module 302 may include:

The accuracy rate is the accuracy degree, and the recall rate is the proportion of samples predicted to be positive in the real positive samples.

In this embodiment, the user historical business data is a large amount of user credit data in a past period, and each data includes information on whether a default is included in addition to the information on the plurality of characteristic dimensions. The historical business data of the users form a sample set containing positive and negative samples, and a risk identification model is input through the first input submodule.

The preset neural network is used as a convolutional neural network model, training can be performed according to a plurality of characteristic dimensions based on historical service data of a user, and a first risk identification model and first output information of the model are obtained. The first output information is the result of the training.

Based on the training results, the first calculation sub-module calculates a first accuracy rate and/or a first recall rate. In this embodiment, both the first accuracy and the first recall rate may be used to evaluate the result of the model training in this round, that is, evaluate the prediction effect and performance of the first risk identification model. For example, the higher the value of the first accuracy rate is, the higher the prediction accuracy characterizing the first risk identification model is to some extent, and the higher the value of the first recall rate is, the smaller the false positive rate characterizing the first risk identification model can be.

In the credit management scenario, the main function of the credit management model is risk item identification, that is, the credit management model serves as an item identification model, and various identified risks are finally embodied as credit default risks, so in the embodiment of the application, the preset item identification model serves as a target model to be evaluated, and validity (such as available or unavailable) verification is performed on the target model through the established first risk identification model serving as a reference model. Specifically, the generating module 303 may be configured to:

In order to evaluate the event recognition model, the object model, which is the evaluated event recognition model, may be subjected to feature extraction to generate corresponding business feature information. The item identification model is used as a credit management model, and is generally configured with one or more items to perform user credit management (such as credit approval) according to the identification result of the item, for example, the item includes "whether to allow the enterprise with risk behaviors," whether to allow the business with head office approval as negative or in future to allow business division approval, "and the like. In order to verify the effectiveness of the event recognition model, in the present embodiment, at least one event feature of the event recognition model, such as the event feature of "whether to admit the enterprise with risky behaviors" as described above, is used as a new feature of the preset neural network, and the preset neural network is retrained. That is, since the event recognition model is abstracted to a new feature dimension as the business feature information and participates in the neural network training, the feature dimension corresponding to the business feature information is different from the plurality of feature dimensions corresponding to the user historical business data. Therefore, if the introduced business feature information can improve the quality of the trained second risk identification model, the item identification model is proved to be effective and can be put into practical use.

In the process of retraining the preset neural network through the second training module 304, a second risk recognition model and second quality evaluation information of the model can be obtained, where the second quality evaluation information is information for evaluating the degree of superiority and inferiority of the second risk recognition model. For example, the second quality-assessment information may include a second accuracy rate and/or a second recall rate. The second training module 304 may specifically include:

the second input submodule is used for inputting the historical service data of the user into a preset neural network;

and the second calculating submodule is used for calculating according to the second output information to obtain a second accuracy and/or a second recall rate.

In this embodiment, the same sample set as that used by the first training module 302 may be used as the training sample set, that is, the user historical service data is input into the preset neural network through the second input sub-module.

Based on the input user historical service data, the second training submodule trains the neural network according to the characteristic dimensionality corresponding to the service characteristic information and the characteristic dimensionalities corresponding to the user historical service data, and second output information of a second risk identification model and a second risk model can be obtained. The second output information is the result of the training.

Based on the training results, the second calculation submodule calculates a second accuracy and/or a second recall. The calculation method of the second accuracy rate is the same as the calculation method of the first accuracy rate, and the calculation method of the second recall rate is the same as the calculation method of the first recall rate, which is not described herein again.

In this embodiment, both the second accuracy and the second recall rate may be used to evaluate the result of the model training in this round, that is, evaluate the prediction effect and performance of the second risk identification model.

After obtaining the second quality evaluation information for evaluating the prediction effect and performance of the second risk identification model, the determining module 305 may determine the target evaluation information according to the first quality evaluation information and the second quality evaluation information. In the process, by comparing the first quality evaluation information with the second quality evaluation information, if the accuracy rate is improved and/or the recall rate is improved, the added service characteristic information can be proved to be valid, that is, the information has higher correlation with the actual default, so that the corresponding item identification model is judged to be valid, and the evaluation information describing the validity of the corresponding item identification model is generated. Otherwise, if the accuracy and/or recall rate of the second information is not improved compared with the first information, the item recognition model can be judged to be invalid, and evaluation information describing the invalidity of the item recognition model is generated.

When the item identification model is validated based on the target evaluation information, the model may be applied to risk item identification. Therefore, optionally, in this embodiment of the present application, the apparatus 300 may further include:

The user service data to be recognized may include service data required for the application field of the corresponding item recognition model. For example, in a pre-credit link of the credit, the user rating and credit granting needs to be identified according to the user service data, so that the collected user service data to be identified may include data information of corresponding characteristic dimensions, for example, credit information of a borrower (i.e., a user), guarantee slow-release measure information, and borrower rating information. Correspondingly, the item identification model needs to identify items such as user rating items and whether credit is granted.

Therefore, in this embodiment, the user service data to be identified is obtained, the corresponding item identification model is input, and the feature extraction is performed to obtain the item identification information. For example, the item identification model may be a Logistic Regression (LR) model, a Support Vector Machine (SVM) model, a Random Forest (RF) model, or a decision tree model, and the present embodiment is not limited to the above.

It should be noted that all relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and the corresponding technical effect can be achieved, and for brevity, no further description is provided herein.

Fig. 4 shows a hardware structure diagram of a computer device provided in an embodiment of the present application.

The computer device 400 may include a processor 401 and a memory 402 storing computer program instructions.

In particular, the processor 401 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 402 may include mass storage for data or instructions. By way of example, and not limitation, memory 402 may include a Hard Disk Drive (HDD), floppy Disk Drive, flash memory, optical Disk, magneto-optical Disk, tape, or Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 402 may include removable or non-removable (or fixed) media, where appropriate. The memory 402 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 402 is a non-volatile solid-state memory.

The memory may include Read Only Memory (ROM), Random Access Memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions and when the software is executed (e.g., by one or more processors), it is operable to perform operations described with reference to the methods according to an aspect of the application.

The processor 401 reads and executes the computer program instructions stored in the memory 402 to implement any one of the evaluation information determination methods in the above-described embodiments.

In one example, computer device 400 may also include a communication interface 403 and a bus 410. As shown in fig. 4, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 to complete communication therebetween.

The communication interface 403 is mainly used for implementing communication between modules, apparatuses, units and/or devices in the embodiments of the present application.

Bus 410 includes hardware, software, or both to couple the components of computer device 400 to one another. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. Bus 410 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

In addition, in combination with the evaluation information determination method in the foregoing embodiments, the embodiments of the present application may be implemented by providing a computer storage medium. The computer storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement any of the evaluation information determination methods in the above embodiments.

In addition, with the evaluation information determination method in the foregoing embodiments, the embodiments of the present application may be implemented by providing a computer program product. The instructions in the computer program product, when executed by a processor of a computer device, cause the computer device to perform any one of the evaluation information determination methods as in the above embodiments.

It is to be understood that the present application is not limited to the particular arrangements and instrumentality described above and shown in the attached drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions or change the order between the steps after comprehending the spirit of the present application.

The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the present application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.

It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.

Aspects of the present application are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such a processor may be, but is not limited to, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable logic circuit. It will also be understood that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware for performing the specified functions or acts, or combinations of special purpose hardware and computer instructions.

As described above, only the specific embodiments of the present application are provided, and it can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered within the scope of the present application.

Claims

1. An evaluation information determination method, characterized by comprising:

acquiring historical service data of a user;

training the neural network through the service characteristic information and the user historical service data to obtain a second risk identification model and second quality evaluation information of the second risk model;

2. The method according to claim 1, wherein the user historical service data corresponds to a plurality of characteristic dimensions, and the first quality evaluation information comprises a first accuracy and/or a first recall;

the method for training the preset neural network through the historical service data of the user to obtain a first risk identification model and first quality evaluation information of the first risk model comprises the following steps:

inputting the user historical business data into the neural network;

training the neural network according to the multiple feature dimensions to obtain the first risk model and first output information of the first risk model;

and calculating according to the first output information to obtain the first accuracy and/or the first recall rate.

3. The method according to claim 2, wherein the object model is a transaction recognition model, and the generating the service characteristic information by using the object model to be evaluated comprises:

generating the business feature information according to at least one item feature of the item identification model,

and the characteristic dimension corresponding to the service characteristic information is different from a plurality of characteristic dimensions of the user historical service data.

4. The method according to claim 3, wherein the second quality-assessment information comprises a second accuracy rate and/or a second recall rate,

the training of the neural network according to the service characteristic information and the user historical service data to obtain a second risk identification model and second quality evaluation information of the second risk model includes:

inputting the user historical business data into the neural network;

training the neural network according to the feature dimensions corresponding to the service feature information and a plurality of feature dimensions corresponding to the user historical service data to obtain a second risk identification model and second output information of the second risk model;

and calculating according to the second output information to obtain the second accuracy and/or the second recall rate.

5. The method of claim 1, wherein the object model is a transaction recognition model, and wherein after the object model's object evaluation information, the method further comprises:

and extracting the characteristics of the user service data to be identified through the item identification model to obtain item identification information.

6. An evaluation information determination apparatus characterized by comprising:

the generating module is used for generating service characteristic information by using a target model to be evaluated;

the second training module is used for training the neural network through the service characteristic information and the user historical service data to obtain a second risk identification model and second quality evaluation information of the second risk model;

and the determining module is used for determining the target evaluation information of the target model according to the comparison result of the first quality evaluation information and the second quality evaluation information.

7. The apparatus according to claim 6, wherein the user historical service data corresponds to a plurality of feature dimensions, and the first quality evaluation information comprises a first accuracy and/or a first recall;

the first training module comprises:

the first input submodule is used for inputting the historical business data of the user into the neural network;

the first training submodule is used for training the neural network according to the multiple feature dimensions to obtain the first risk model and first output information of the first risk model;

and the first calculation submodule is used for calculating according to the first output information to obtain the first accuracy and/or the first recall rate.

8. The apparatus of claim 7, wherein the object model is a transaction recognition model, and wherein the generation module is specifically configured to:

9. The apparatus of claim 8, wherein the second quality-assessment information comprises a second accuracy rate and/or a second recall rate,

the second training module comprising:

the second input submodule is used for inputting the historical business data of the user into the neural network;

the second training submodule is used for training the neural network according to the feature dimensions corresponding to the business feature information and the feature dimensions corresponding to the historical business data of the user to obtain a second risk identification model and second output information of the second risk model;

and the second calculating submodule is used for calculating according to the second output information to obtain the second accuracy and/or the second recall rate.

10. The apparatus of claim 6, wherein the object model is a transaction recognition model, the apparatus further comprising:

the second acquisition module is used for acquiring user service data to be identified under the condition that the target evaluation information meets a preset condition;

11. A computer device, the device comprising: a processor and a memory storing computer program instructions;

the processor, when executing the computer program instructions, implements the rating information determination method according to any of claims 1-5.

12. A computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement the rating information determining method according to any one of claims 1 to 5.

13. A computer program product, wherein instructions in the computer program product, when executed by a processor of a computer device, cause the computer device to perform the rating information determination method according to any one of claims 1 to 5.