CN115906189A

CN115906189A - Model verification method, device, equipment and computer readable storage medium

Info

Publication number: CN115906189A
Application number: CN202211576256.8A
Authority: CN
Inventors: 蔡宇笙; 李姗姗; 陈伟杰; 郭琰琰; 陆凌; 张星语; 潘瑶
Original assignee: China Construction Bank Corp; CCB Finetech Co Ltd
Current assignee: China Construction Bank Corp; CCB Finetech Co Ltd
Priority date: 2022-12-09
Filing date: 2022-12-09
Publication date: 2023-04-04

Abstract

The application discloses a model verification method, a model verification device, model verification equipment and a computer-readable storage medium. The method comprises the following steps: obtaining a metering model group, wherein the metering model group comprises m metering models; acquiring first verification conclusions corresponding to the m metering models respectively aiming at the first target verification index; aiming at a first target verification index, integrating m first verification conclusions according to a first preset rule, determining a second verification conclusion corresponding to the metering model group, and obtaining n second verification conclusions corresponding to the metering model group and respectively aiming at n verification indexes; acquiring the priority corresponding to the second verification conclusion and a preset corresponding relation between the second verification conclusion and the verification index; and according to the sequence of the priority levels from high to low, integrating the n second verification conclusions according to a preset corresponding relation, and determining the model verification conclusion corresponding to the metering model group. According to the embodiment of the application, the standardization degree of the verification conclusion and the efficiency of model verification can be improved.

Description

Model verification method, device, equipment and computer readable storage medium

Technical Field

The present application belongs to the field of computer technologies, and in particular, to a model verification method, apparatus, device, and computer-readable storage medium.

Background

Typically, verification of metrology models includes data verification, model verification, and variable verification. The data verification, the model verification, and the variable verification may correspond to a plurality of verification indexes, respectively. Therefore, in order to verify the overall performance of the model, multiple times of verification need to be performed on each verification index, and multiple verification conclusions need to be integrated to obtain a unique performance index for evaluating the overall performance of the model.

In the prior art, the verification conclusion is usually integrated manually by a user, and the integration process of the verification conclusion is very complicated. Moreover, in order to ensure the accuracy of the verification conclusion, the verification conclusion needs to be rechecked for many times, so that the efficiency of model verification is low. In addition, the judgment criteria of different users for the verification conclusion may be different, resulting in a lower standardization degree of the verification conclusion.

Disclosure of Invention

The embodiment of the application provides a model verification method, a model verification device, a model verification apparatus, a computer readable storage medium and a computer program product, which can improve the standardization degree of a verification conclusion and improve the efficiency of model verification.

In a first aspect, an embodiment of the present application provides a model verification method, where the method includes:

obtaining a metering model group, wherein the metering model group comprises m metering models, and m is a positive integer;

aiming at a first target verification index, obtaining first verification conclusions corresponding to the m metering models respectively, wherein the first target verification index is any one of n verification indexes, and n is a positive integer;

according to the first target verification index, integrating the m first verification conclusions according to a first preset rule, determining a second verification conclusion corresponding to the metering model group, and obtaining n second verification conclusions corresponding to the metering model group and respectively aiming at the n verification indexes;

acquiring the priority corresponding to the second verification conclusion and a preset corresponding relation between the second verification conclusion and the verification index;

and according to the sequence of the priority from high to low, integrating the n second verification conclusions according to the preset corresponding relation, and determining the model verification conclusion corresponding to the metering model group.

In a possible implementation manner, the obtaining, for a first target verification index, first verification conclusions corresponding to the m metering models respectively includes:

obtaining a third verification conclusion of the target metering model in a target verification period aiming at the first target verification index, and obtaining third verification conclusions corresponding to p verification periods respectively, wherein the target metering model is any one of the m metering models, the target verification period is any one of the p verification periods, and p is a positive integer;

and integrating the p third verification conclusions according to a second preset rule, determining a first verification conclusion corresponding to the target metering model, and obtaining first verification conclusions corresponding to the m metering models respectively.

In a possible implementation manner, the obtaining, for the first target verification index, a third verification conclusion of the target metering model in a target verification period includes:

receiving a first input from a user to determine the target metrology model and the target verification period;

in response to the first input, obtaining metric data of the target metrology model for the first target verification metric over the target verification period;

and determining a third verification conclusion corresponding to the index data according to a third preset rule.

In one possible implementation, the first verification conclusion comprises a first target verification conclusion comprising any one of normal, concern, and improvement;

the integrating m first verification conclusions according to a first preset rule aiming at the first target verification index and determining a second verification conclusion corresponding to the metering model group comprises:

acquiring the conclusion number corresponding to the first target verification conclusion and the total number corresponding to the first verification conclusion;

determining the target proportion of the first target verification conclusion in the first verification conclusion according to the conclusion number and the total number;

and under the condition that the target proportion is not less than a first preset threshold value, determining that the second verification conclusion is the first target verification conclusion.

In one possible implementation, the priority corresponding to the improvement is higher than the priority corresponding to the attention, and the priority corresponding to the attention is higher than the priority corresponding to the normal;

the integrating the n second verification conclusions according to the preset corresponding relation from the high priority to the low priority to determine the model verification conclusion corresponding to the metering model group includes:

searching a second target verification conclusion in the n second verification conclusions, wherein the second target verification conclusion is the improvement;

determining a second target verification index corresponding to the second target verification conclusion;

determining whether the preset corresponding relation exists between the second target verification conclusion and the second target verification index;

and under the condition that the preset corresponding relation exists between the second target verification conclusion and the second target verification index, determining that the model verification conclusion corresponding to the metering model group is to be improved.

In one possible implementation, the method further includes:

and under the condition that the preset corresponding relation does not exist between the second target verification conclusion and the second target verification index, taking the attention as the second target verification conclusion, and returning and executing the second target verification conclusion searched in the n second verification conclusions until the model verification conclusion corresponding to the metering model group is determined to be the attention or the normal.

In a second aspect, an embodiment of the present application provides a model verification apparatus, including:

the device comprises a first acquisition module, a second acquisition module and a calculation module, wherein the first acquisition module is used for acquiring a metering model group, the metering model group comprises m metering models, and m is a positive integer;

a second obtaining module, configured to obtain, for a first target verification index, first verification conclusions corresponding to the m metering models, where the first target verification index is any one of n verification indexes, and n is a positive integer;

a first integration module, configured to integrate, according to a first preset rule, the m first verification conclusions with respect to the first target verification index, determine a second verification conclusion corresponding to the metering model group, and obtain n second verification conclusions corresponding to the metering model group and respectively corresponding to the n verification indexes;

a third obtaining module, configured to obtain a priority corresponding to the second verification conclusion and a preset correspondence between the second verification conclusion and the verification index;

and the second integration module is used for integrating the n second verification conclusions according to the preset corresponding relation from high priority to low priority and determining the model verification conclusion corresponding to the metering model group.

In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory storing computer program instructions;

the processor, when executing the computer program instructions, implements the method of any one of the possible implementation methods of the first aspect described above.

In a fourth aspect, the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when executed by a processor, the computer program instructions implement the method in any one of the possible implementation methods in the first aspect.

In a fifth aspect, the present application provides a computer program product, where instructions of the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the method in any one of the possible implementation methods as described in the first aspect.

According to the model verification method, the model verification device, the model verification equipment, the computer readable storage medium and the computer program product, the first verification conclusions corresponding to the m metering models are obtained according to the first target verification index, and then the m first verification conclusions are integrated, so that the second verification conclusion corresponding to the metering model group and aiming at the first target verification index can be determined. In the same way, n second verification conclusions corresponding to the metering model group and aiming at the n verification indexes respectively can be determined. The model verification conclusion corresponding to the metering model group can be determined by integrating the n second verification conclusions according to the preset corresponding relation between the second verification conclusion and the verification index according to the sequence from high priority to low priority. Therefore, the standardization degree of the verification conclusion can be improved and the efficiency of model verification can be improved by integrating the verification results in stages based on the preset rule.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flowchart of a model verification method provided in an embodiment of the present application;

fig. 2 is a schematic structural diagram of a model verification apparatus according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Features and exemplary embodiments of various aspects of the present application will be described in detail below, and in order to make objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of, and not restrictive on, the present application. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present application by illustrating examples thereof.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of another like element in a process, method, article, or apparatus that comprises the element.

In addition, the data acquisition, storage, use, processing and the like in the technical scheme of the application all conform to relevant regulations of national laws and regulations.

As described in the background section, in order to solve the problems in the prior art, embodiments of the present application provide a model verification method, apparatus, device, computer-readable storage medium, and computer program product.

First, a model verification method provided in the embodiments of the present application is described below.

Fig. 1 shows a schematic flowchart of a model verification method provided in an embodiment of the present application. As shown in fig. 1, the model verification method provided in the embodiment of the present application includes the following steps:

s110, obtaining a metering model group, wherein the metering model group comprises m metering models, and m is a positive integer;

s120, aiming at a first target verification index, obtaining first verification conclusions corresponding to the m metering models respectively, wherein the first target verification index is any one of n verification indexes, and n is a positive integer;

s130, aiming at the first target verification index, integrating the m first verification conclusions according to a first preset rule, determining a second verification conclusion corresponding to the metering model group, and obtaining n second verification conclusions corresponding to the metering model group and respectively aiming at the n verification indexes;

s140, acquiring the priority corresponding to the second verification conclusion and the preset corresponding relation between the second verification conclusion and the verification index;

s150, according to the sequence of the priority levels from high to low, integrating the n second verification conclusions according to a preset corresponding relation, and determining the model verification conclusion corresponding to the metering model group.

According to the model verification method, the first verification conclusions corresponding to the m metering models are obtained according to the first target verification index, and then the m first verification conclusions are integrated, so that the second verification conclusion corresponding to the metering model group and aiming at the first target verification index can be determined. In the same way, n second verification conclusions corresponding to the metering model group and aiming at the n verification indexes respectively can be determined. The model verification conclusion corresponding to the metering model group can be determined by integrating the n second verification conclusions according to the preset corresponding relation between the second verification conclusion and the verification index according to the sequence from high priority to low priority. Therefore, the standardization degree of the verification conclusion can be improved and the efficiency of model verification can be improved by integrating the multiple verification results in a staged manner based on the preset rule.

Specific implementations of the above steps are described below.

In some embodiments, in S110, the metrology model group may include one metrology model or may include a plurality of metrology models, which is not limited herein. For example, the credit risk model may include two metering models, a credit investigation model and a credit non-investigation model.

In some embodiments, in S120, the metrology model group may correspond to n verification metrics, and accordingly, each metrology model in the metrology model group may correspond to the n verification metrics. That is, the model verification conclusion corresponding to the metrology model group may be determined by the verification results of the n verification indicators. A metrology model may have a verification conclusion for a verification index. In this way, for each verification index, m first verification conclusions corresponding to m metrology models, respectively, can be obtained. Wherein the first verification conclusion may include normal, concern and improvement.

In addition, the metering model group can be verified from three aspects of data verification, model verification, variable verification and the like. The data verification can correspond to verification indexes such as consistency verification, integrity verification, comprehensive verification, accuracy verification and the like of data; the model verification can comprise verification indexes such as discrimination verification, judicious verification, accuracy verification, homogeneity verification, heterogeneity verification, stability verification, concentration verification and the like of the model; the variable verification can comprise verification indexes such as group stability verification and discrimination verification. As such, the first target verification index may be any one of the verification indexes described above.

Based on this, in order to improve the accuracy of the first verification conclusion, in some embodiments, the S120 may specifically include:

aiming at the first target verification index, obtaining a third verification conclusion of the target metering model in a target verification period, and obtaining third verification conclusions corresponding to p verification periods respectively, wherein the target metering model is any one of m metering models, the target verification period is any one of p verification periods, and p is a positive integer;

Here, the verification period may be preset, and the verification period may be one month, two months, or three months, which is not limited herein. In order to determine the first verification conclusion corresponding to the target metering model, p verification periods may be selected to respectively verify the first target verification index of the target metering model, and p third verification conclusions corresponding to the verification periods are obtained. Wherein the p value can be preset. Wherein the third verification conclusion may include normal, concern and improvement.

As an example, the priorities corresponding to different third verification conclusions may be the same. That is, the second preset rule may specifically be that the third verification conclusion with a larger occurrence number of the p third verification conclusions is taken as the first verification conclusion corresponding to the target metering model.

As another example, the priorities for different third verification conclusions may be different. For example, the priority to be improved may be higher than the priority of attention, which may be higher than the normal priority. In this way, the second preset rule may specifically be that the third verification conclusion with the highest priority is used as the first verification conclusion corresponding to the target metering model.

Based on this, a specific example is given. If the value of p is 2, the first target verification index is the verification index corresponding to the model discrimination, and both the two third verification conclusions are normal, it can be determined that the first verification conclusion corresponding to the target metering model is normal.

In this way, the p third verification conclusions are integrated according to the second preset rule, the first verification conclusion corresponding to the target metering model is determined, and the accuracy of the first verification conclusion can be improved.

In order to obtain a third verification conclusion, in some embodiments, the obtaining a third verification conclusion of the target metrology model in the target verification period with respect to the first target verification index specifically includes:

receiving a first input from a user for determining a target metrology model and a target verification period;

in response to the first input, acquiring index data of the target metering model for a first target verification index in a target verification period;

Here, the user may select a target metrology model and a target verification period at a reporting interface. The report interface can display a report corresponding to the metering model group. The report can provide the overall situation of the internal rating system and can be used for displaying relevant information and call quantity information of model development. The overall situation of the internal rating system may include model performance, model variable performance, and data quality. The model representation condition can be used for displaying functions such as model distinguishing capacity, model stability, model accuracy, pool model homogeneity and heterogeneity and the like. The model variable performance can be used for displaying the single variable distinguishing capability of the model and the relevant indexes of the single variable stability. The data quality condition can be used for displaying relevant indexes of data accuracy, consistency, completeness and comprehensiveness. That is, the report may include index data of the target metering model for the first target verification index in the target verification period.

As an example, different index data may correspond to different third preset rules. Taking the model discrimination verification as an example, the index data may include an AR value (Accuracy Ratio) and a KS value (Kolmogorov-Smirnov), and the AR value and the KS value may be calculated by monitoring the report. Based on this, the third preset rule may be to determine a third verification conclusion according to the relationship between the AR value and the KS value and the threshold value. For example, if AR ≧ 0.3 and KS ≧ 0.3, the third verification conclusion may be normal, if AR <0.3 and KS <0.3, the third verification conclusion may be due for improvement, drag AR <0.3 or KS <0.3, the third verification conclusion may be attention.

In some embodiments, in S130, for the first target verification index, the first verification conclusions corresponding to the m metering models may be integrated according to a first preset rule, so as to obtain a second verification conclusion corresponding to the metering model group. For the metrology model group, the n verification indicators may correspond to the n second verification conclusions, respectively. Wherein the second verification conclusion may include normal, concern and improvement.

As an example, the priorities for different first verification conclusions may be the same. That is, the first preset rule may specifically be that the first verification conclusion with a larger occurrence number of the m first verification conclusions is taken as the second verification conclusion corresponding to the metrology model group.

As another example, the priorities for different first verification conclusions may be different. For example, the priority to be improved may be higher than the priority of attention, which may be higher than the normal priority. In this way, the first preset rule may specifically be that the first verification conclusion with the highest priority is used as the first verification conclusion corresponding to the metrology model group.

Additionally, in some embodiments, the first verification conclusion may include a first target verification conclusion, which may include any of normality, concern, and improvement. Based on this, the S130 may specifically include:

acquiring the number of conclusions corresponding to the first target verification conclusion and the total number corresponding to the first verification conclusion;

Here, the user may control the second verification conclusion by presetting, modifying, and the like fields of a control value (normal, attention, should be improved), a sign (greater than, less than, greater than or equal to, less than or equal to), and a first preset threshold.

As an example, the user may determine the overall conclusion (i.e., the second verification conclusion) corresponding to the metric model group by controlling the percentage of the number of normal or improved or concerned metric models to the total number of metric models in the entry configuration.

As a more specific example, taking model partition force verification as an example, if the ratio (i.e. the target ratio) of the number of normal metering models to the total number of metering models is greater than 90%, the second verification conclusion corresponding to the metering model group may be normal. If the ratio of the number of metrology models to be improved to the total number of metrology models (i.e., the target ratio) is greater than 10%, the second verification conclusion corresponding to the metrology model set may be to be improved. The rest of the cases may then be attention.

In some embodiments, different second verification conclusions may correspond to different priorities in S140. The priority corresponding to the second verification conclusion may be preset. Wherein, the priority corresponding to improvement may be higher than the priority corresponding to attention, and the priority corresponding to attention may be higher than the priority corresponding to normal. In addition, the authentication indexes may correspond to different priorities. For example, the model validation corresponding validation indexes may have a higher priority than the variable validation corresponding validation indexes, and the variable validation corresponding validation indexes may have a higher priority than the data validation corresponding validation indexes. In the model verification, the variable verification and the data verification, the priorities corresponding to different verification indexes may also be different, and are not described herein again.

Based on this, a preset corresponding relationship may exist between the second verification conclusion and the verification index, and the preset corresponding relationship between the second verification conclusion and the verification index may also be preset. For example, there may be a preset correspondence between the improvement and the model distinguishing force, a preset correspondence between the normality and the data consistency, and the like.

In some embodiments, in S150, in order to determine the model verification conclusion corresponding to the metrology model group, the second verification conclusion may be screened according to the priority, and then the verification index corresponding to the second verification conclusion is determined according to the preset corresponding relationship, so as to determine the model verification conclusion corresponding to the metrology model group.

Based on this, in some embodiments, the S150 may specifically include:

searching a second target verification conclusion in the n second verification conclusions, wherein the second target verification conclusion is to be improved;

determining whether a preset corresponding relation exists between a second target verification conclusion and a second target verification index;

and under the condition that the second target verification conclusion and the second target verification index have a preset corresponding relation, determining that the model verification conclusion corresponding to the metering model group is to be improved.

Here, since the priority to be improved may be highest among the should-be-improved, the attention, and the normal, it may be first searched for whether there is an authentication conclusion to be improved among the n second authentication conclusions. If the second verification conclusion includes the improvement, the determination of the second target verification index corresponding to the improvement can be continued. If a preset corresponding relationship exists between the to-be-improved and the second target verification index, the model verification conclusion corresponding to the metering model group can be determined to be improved. The number of the second target verification indexes may be one or more, and is not limited herein.

As an example, if there is a preset corresponding relationship between the model differentiation amount and the amount to be improved, in the case that the second target verification result is the amount to be improved and the second target verification index is the model differentiation amount, the model verification result corresponding to the metrology model group may be directly determined as the amount to be improved.

Based on this, in some embodiments, in a case that there is no preset correspondence between the second target verification conclusion and the second target verification index, the attention is taken as the second target verification conclusion, and the search for the second target verification conclusion among the n second verification conclusions is performed in return until it is determined that the model verification conclusion corresponding to the metrology model group is attention or normal.

Here, if there is no preset corresponding relationship between the first target verification index and the second target verification index to be improved, the attention may be regarded as the second target verification conclusion, and the verification conclusion that whether there is attention is searched for among the n second verification conclusions may be returned. If the second verification conclusion includes the attention, the second target verification index corresponding to the attention may be continuously determined. If a preset correspondence exists between the attention and the second target verification index, the model verification conclusion corresponding to the metrology model group may be determined to be attention. The number of the second target verification indexes may be one or multiple, and is not limited herein. If no preset corresponding relation exists between the concerned and the second target verification index, the model verification conclusion corresponding to the metering model group can be determined to be normal.

As an example, the user may control the model verification conclusion by presetting fields such as a control value, a symbol, a second preset threshold, a priority, a verification index, and the like. For example, a user may preset a preset corresponding relationship between a model distinguishing force and a force to be improved, a verification index may be set as the model distinguishing force, a control value may be set as the force to be improved, a sign may be set equal to, and a second preset threshold may be set as 1. In this way, if the verification conclusion for the model distinguishing force corresponding to the metrology model group is to be improved, it can be determined that the model verification conclusion corresponding to the metrology model group is to be improved.

Therefore, by flexible rule configuration, model verification personnel can control the model verification conclusion through fields such as control values, symbols, second preset thresholds, priorities, verification indexes and the like, and further can realize rapid configuration and rapid change of the conclusion.

Based on the model verification method provided by the above embodiment, correspondingly, the application also provides a specific implementation manner of the model verification device. Please see the examples below.

As shown in fig. 2, a model verification apparatus 200 provided in the embodiment of the present application includes the following modules:

a first obtaining module 210, configured to obtain a metering model group, where the metering model group includes m metering models, and m is a positive integer;

a second obtaining module 220, configured to obtain, for a first target verification index, first verification conclusions corresponding to the m metering models, where the first target verification index is any one of n verification indexes, and n is a positive integer;

a first integration module 230, configured to integrate the m first verification conclusions according to a first preset rule for the first target verification index, determine second verification conclusions corresponding to the metering model group, and obtain n second verification conclusions corresponding to the metering model group and for the n verification indexes, respectively;

a third obtaining module 240, configured to obtain a priority corresponding to the second verification conclusion and a preset corresponding relationship between the second verification conclusion and the verification index;

and a second integration module 250, configured to integrate the n second verification conclusions according to a preset correspondence relationship in an order from high priority to low priority, and determine a model verification conclusion corresponding to the metering model group.

The model verification apparatus 200 is described in detail below, specifically as follows:

in some embodiments, the second obtaining module 220 may specifically include:

the first obtaining sub-module is used for obtaining a third verification conclusion of the target metering model in a target verification period aiming at the first target verification index, and obtaining third verification conclusions corresponding to p verification periods respectively, wherein the target metering model is any one of m metering models, the target verification period is any one of p verification periods, and p is a positive integer;

and the first integration submodule is used for integrating the p third verification conclusions according to a second preset rule, determining a first verification conclusion corresponding to the target metering model and obtaining first verification conclusions corresponding to the m metering models respectively.

In some embodiments, the first obtaining sub-module may specifically include:

a receiving unit for receiving a first input of a user for determining a target metrology model and a target verification period;

an acquisition unit configured to acquire, in response to a first input, index data of a target metrology model for a first target verification index in a target verification period;

and the determining unit is used for determining a third verification conclusion corresponding to the index data according to a third preset rule.

In some of these embodiments, the first verification conclusion may include a first target verification conclusion, which may include any of normality, concern, and improvement;

based on this, the first integration module 230 may specifically include:

the second obtaining submodule is used for obtaining the number of the conclusions corresponding to the first target verification conclusion and the total number corresponding to the first verification conclusion;

the first determining submodule is used for determining the target proportion of the first target verification conclusion in the first verification conclusion according to the conclusion number and the total number;

and the second determining submodule is used for determining that the second verification conclusion is the first target verification conclusion under the condition that the target proportion is not smaller than the first preset threshold value.

In some of these embodiments, the priority corresponding to the improvement may be higher than the priority corresponding to the attention, and the priority corresponding to the attention may be higher than the priority corresponding to the normal;

based on this, the second integration module 250 may specifically include:

the searching sub-module is used for searching a second target verification conclusion from the n second verification conclusions, and the second target verification conclusion is to be improved;

the third determining submodule is used for determining a second target verification index corresponding to the second target verification conclusion;

the fourth determining submodule is used for determining whether a preset corresponding relation exists between the second target verification conclusion and the second target verification index;

and the fifth determining submodule is used for determining that the model verification conclusion corresponding to the metering model group is to be improved under the condition that the preset corresponding relation exists between the second target verification conclusion and the second target verification index.

In some embodiments, the second integration module 250 may further include:

and the sixth determining submodule is used for taking the attention as a second target verification conclusion under the condition that the preset corresponding relation does not exist between the second target verification conclusion and the second target verification index, and returning and executing the second target verification conclusion searched in the n second verification conclusions until the model verification conclusion corresponding to the metering model group is determined to be attention or normal.

According to the model verification device, the first verification conclusions corresponding to the m metering models are obtained according to the first target verification index, and then the m first verification conclusions are integrated, so that the second verification conclusion corresponding to the metering model group and aiming at the first target verification index can be determined. In the same way, n second verification conclusions corresponding to the metering model group and aiming at the n verification indexes respectively can be determined. The model verification conclusion corresponding to the metering model group can be determined by integrating the n second verification conclusions according to the preset corresponding relation between the second verification conclusion and the verification index according to the sequence from high priority to low priority. Therefore, the standardization degree of the verification conclusion can be improved and the efficiency of model verification can be improved by integrating the verification results in stages based on the preset rule.

Based on the model verification method provided by the above embodiment, the embodiment of the present application further provides a specific implementation manner of the electronic device. Fig. 3 shows a schematic diagram of an electronic device 300 provided in an embodiment of the present application.

The electronic device 300 may include a processor 310 and a memory 320 storing computer program instructions.

In particular, the processor 310 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 320 may include mass storage for data or instructions. By way of example, and not limitation, memory 320 may include a Hard Disk Drive (HDD), a floppy Disk Drive, flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 320 may include removable or non-removable (or fixed) media, where appropriate. The memory 320 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 320 is a non-volatile solid-state memory.

The memory may include Read Only Memory (ROM), random Access Memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software comprising computer-executable instructions and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the application.

The processor 310 may implement any of the model verification methods described in the embodiments above by reading and executing computer program instructions stored in the memory 320.

In one example, electronic device 300 may also include a communication interface 330 and a bus 340. As shown in fig. 3, the processor 310, the memory 320, and the communication interface 330 are connected via a bus 340 to complete communication therebetween.

The communication interface 330 is mainly used for implementing communication between modules, apparatuses, units and/or devices in the embodiments of the present application.

Bus 340 includes hardware, software, or both to couple the components of the electronic device to each other. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. Bus 340 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

Illustratively, the electronic device 300 may be a mobile phone, a tablet computer, a notebook computer, a palm top computer, a vehicle-mounted electronic device, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like.

The electronic device may execute the model verification method in the embodiment of the present application, thereby implementing the model verification method and apparatus described in conjunction with fig. 1 and fig. 2.

In addition, in combination with the model verification method in the foregoing embodiments, the embodiments of the present application may provide a computer storage medium to implement. The computer storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement any of the model validation methods in the embodiments described above.

It is to be understood that the present application is not limited to the particular arrangements and instrumentality described above and shown in the attached drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions or change the order between the steps after comprehending the spirit of the present application.

The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the present application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.

It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.

Aspects of the present application are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such a processor may be, but is not limited to, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable logic circuit. It will also be understood that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware for performing the specified functions or acts, or combinations of special purpose hardware and computer instructions.

As described above, only the specific embodiments of the present application are provided, and it can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered within the scope of the present application.

Claims

1. A method of model validation, comprising:

acquiring first verification conclusions corresponding to the m metering models respectively aiming at a first target verification index, wherein the first target verification index is any one of n verification indexes, and n is a positive integer;

and according to the sequence of the priorities from high to low, integrating the n second verification conclusions according to the preset corresponding relation, and determining the model verification conclusion corresponding to the metering model group.

2. The method according to claim 1, wherein the obtaining, for a first target verification index, first verification conclusions corresponding to the m metrology models, respectively, comprises:

3. The method of claim 2, wherein obtaining a third verification conclusion of the target metrology model for the first target verification metric during the target verification period comprises:

4. The method of claim 1, wherein the first verification conclusion comprises a first target verification conclusion comprising any one of normal, concern, and improvement;

the integrating, according to a first preset rule, the m first verification conclusions to the first target verification index to determine a second verification conclusion corresponding to the measurement model group includes:

and under the condition that the target proportion is not smaller than a first preset threshold value, determining that the second verification conclusion is the first target verification conclusion.

5. The method of claim 1, wherein the improvement-compliant correspondence is higher in priority than the attention-compliant correspondence, which is higher in priority than the normal correspondence;

the step of integrating the n second verification conclusions according to the preset corresponding relation in the order from high priority to low priority to determine the model verification conclusion corresponding to the metering model group includes:

6. The method of claim 5, further comprising:

7. A model validation apparatus, the apparatus comprising:

the system comprises a first obtaining module, a second obtaining module and a third obtaining module, wherein the first obtaining module is used for obtaining a metering model group, the metering model group comprises m metering models, and m is a positive integer;

8. An electronic device, characterized in that the electronic device comprises: a processor and a memory storing computer program instructions;

the processor, when executing the computer program instructions, implements the model validation method of any of claims 1-6.

9. A computer-readable storage medium having computer program instructions stored thereon which, when executed by a processor, implement the model validation method of any of claims 1-6.

10. A computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the model verification method of any one of claims 1-6.