CN117195505A

CN117195505A - Evaluation method and system for informatization evaluation calibration model of electric energy meter

Info

Publication number: CN117195505A
Application number: CN202311048733.8A
Authority: CN
Inventors: 刘婧; 于海波; 陈天阳; 薛冰; 高欣; 谭煌; 陈昊; 陈文礼; 李媛; 刁新平; 乔文俞; 程瑛颖; 苏宇; 李亚杰; 田成明; 谷凯; 郜波; 郑安刚
Original assignee: State Grid Chongqing Electric Power Co Marketing Service Center; Beijing University of Posts and Telecommunications; China Electric Power Research Institute Co Ltd CEPRI
Current assignee: State Grid Chongqing Electric Power Co Marketing Service Center; Beijing University of Posts and Telecommunications; China Electric Power Research Institute Co Ltd CEPRI
Priority date: 2023-08-18
Filing date: 2023-08-18
Publication date: 2023-12-08

Abstract

The invention discloses an evaluation method and an evaluation system for an informationized evaluation calibration model of an electric energy meter, and belongs to the technical field of model evaluation. The method of the invention comprises the following steps: aiming at an electric energy meter informatization evaluation calibration model, determining an evaluation index related to the model accuracy of the electric energy meter informatization evaluation calibration model and an evaluation index related to service requirements; based on an analytic hierarchy process, constructing a key index system for evaluating the electric energy meter informatization evaluation calibration model according to the evaluation index related to the model accuracy and the evaluation index related to the service demand; and carrying out joint verification on key index values under a plurality of working conditions by utilizing a multiple hypothesis test method, mapping a verification result into a model grade, and determining an evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model grade. The reliability of the informationized evaluation calibration model of the electric energy meter can be determined through the evaluation result.

Description

Evaluation method and system for informatization evaluation calibration model of electric energy meter

Technical Field

The invention relates to the technical field of model evaluation, in particular to an evaluation method and an evaluation system for an informationized evaluation calibration model of an electric energy meter.

Background

According to the relevant regulations of the national market administration on forced verification, the traditional mode of the electric energy meter for living of strain leather (hereinafter referred to as electric energy meter) due rotation is gradually pushed to replace due rotation with 'misalignment replacement', and a new mode of deferred use or replacement according to the state of the electric energy meter is constructed. The implementation of the misalignment replacement has important significance in the aspects of maintaining fair and fair trade settlement between an electric power enterprise and an electric power customer, pushing the change of the supervision mode of the electric energy meter legal system, assisting the national double-carbon target and the like. To achieve this objective, there is a need to reform the current calibration mode of the electric energy meter through the transmission of the traditional physical object, and to propose a verification method of the informationized calibration model aiming at the novel remote informationized calibration mode of physical and digital organic fusion, so as to evaluate and verify the informationized calibration model of the electric energy meter from theory and practice, and improve the informationized measurement accuracy and reliability to a new height.

In the aspect of research of model evaluation verification technology, foreign starting is earlier. Rebba discusses several statistical methods for model verification, with emphasis on proposing methods based on interval hypothesis testing and model reliability. Gray et al propose nonlinear model verification techniques based on open loop and inverse simulation for evaluating and improving nonlinear real-time helicopter dynamics models. They represent the unmodeled dynamics by using the relevant flight data as an open loop input to the simulation and simulate a single rigid body state equation in the model revealing the dynamics effect, and then evaluate and refine the model by comparing the results in normal and reverse simulation modes. Sankaramana extends the Bayesian factor method and applies it to model verification where there is inherent uncertainty and cognitive uncertainty. Magdevska et al propose a fuzzy dynamic model verification method for cellular networks based on an enlarged verification dataset. The method eliminates the inaccurate model by adding random initial state disturbance to increase the average error of inaccurate calculation model prediction and simultaneously maintaining the error of accurate model prediction. The method is helpful for detecting the overfitted model and remarkably improving the accuracy of the fuzzy model. Rehman et al propose a model validation method based on factor analysis and the Prooni method. Firstly, replacing a multivariable time sequence of a simulation model and an actual system with a small amount of common factors with physical significance through factor analysis; secondly, extracting the characteristics of each common factor by adopting an improved self-adaptive prooney method, and ensuring the best fitting of non-stationary signals; and then, establishing a complete similarity evaluation model of the simulation system based on the energy proportion, the information entropy and the contribution rate variance. Finally, according to the physical meaning of the extracted features, the error position is identified in the evaluation process. In addition, there are also recent scholars of model verification methods in the power domain. Torkzadeh proposes a practical voltage sag assessment model verification method. The proposed method is applied to the netherlands high voltage, ultra high voltage grid and to medium voltage grid. The validity of the model was studied by means of data from a power quality monitor of three severe voltage drop events in the netherlands grid. At present, foreign research is mainly focused on theory and method research of a model evaluation verification method in other fields, and application of the method in the aspect of electric energy meter evaluation calibration model is not developed.

In the aspect of model evaluation verification technology research, a plurality of verification method researches aiming at theoretical models in different fields exist in China. Aiming at how to more efficiently complete the mathematical model verification work, the high-race et al propose a method for performing mathematical model verification by using a function call selection method, and complete the verification aiming at an embedded martial control software model. Li Jinwei et al constructed mathematical models of ship life predictions and conducted validation studies on the constructed models. And the structural data of the ship under different conditions are acquired in various modes, so that the data source interference is reduced. Two comparison models are introduced in verification research, and life prediction experiments on the same ship are carried out. In order to verify the established dynamic tension theoretical equation for dynamic tension calibration of filament winding, the left Rui et al proposes a set of simulation test platform for the reliability verification of a dynamic tension measurement model, tension calibration simulation is carried out through simulation model adjustment and boundary condition setting, and the evaluation verification of the dynamic tension measurement model is realized through the mutual verification of the theoretical analysis and simulation experiment of the dynamic tension measurement model. Many studies have been made in the field of power systems in foreign countries on the technique of evaluating and verifying the results of models. The method comprises the steps of flexibly analyzing model evaluation and verification theory by Zhang et al of North China electric university, providing a decoupling principle aiming at electric power system model verification, constructing an evaluation system of an electric power system, and verifying an electric power system simulation model. An Jun et al research a dynamic simulation error tracing and credibility verification method of a power system based on WAMS measurement and Thevenin equivalent, and put forward a block decoupling dynamic simulation model verification strategy based on the Thevenin equivalent model, so that the verification difficulty of a complex power system simulation model is reduced. Zhao et al propose a solar photovoltaic power station model verification method based on hybrid data dynamic simulation. Firstly, a mixed data dynamic simulation implementation scheme suitable for DIgSILENT PowerFactory software is provided, and then a solar photovoltaic power station grid-connected analysis model based on an IEEE 9 system is established. And finally, realizing the model verification of the solar photovoltaic power station by adopting a hybrid data dynamic simulation method. The results demonstrate the effectiveness of the proposed method in the model validation of solar photovoltaic power plants. Xu Xianyong et al propose two verification methods for saving electric power and quantity after the parallel reactive power compensation device is put into operation, aiming at the problem that the verification method for saving electric power and quantity by the parallel reactive power compensation device installed in the power supply and distribution system is deficient: the reactive power economic equivalent calculation electricity-saving power quantity verification method and the power factor calculation electricity-saving power quantity verification method improve verification capability aiming at electricity-saving quantity. Liang Jifeng et al provide a multi-criterion fusion verification method suitable for CVT harmonic measurement errors of transformer station field operation for solving the problem of harmonic measurement errors caused by Capacitive Voltage Transformers (CVTs), and an effective way is provided for CVT harmonic error verification of complex transformer station fields. In the aspect of the electric energy meter informatization evaluation calibration model result evaluation verification technology under the multi-working condition, no corresponding method for performing result verification in the field is currently found in China.

Disclosure of Invention

Aiming at the problems, the invention provides an evaluation method for an informationized evaluation calibration model of an electric energy meter, which comprises the following steps:

aiming at an electric energy meter informatization evaluation calibration model, determining an evaluation index related to the model accuracy of the electric energy meter informatization evaluation calibration model and an evaluation index related to service requirements;

based on an analytic hierarchy process, constructing a key index system for evaluating the electric energy meter informatization evaluation calibration model according to the evaluation index related to the model accuracy and the evaluation index related to the service demand;

the method comprises the steps of obtaining an original data set of an informationized evaluation calibration model of the electric energy meter, preprocessing the original data set to obtain a verification sample data set, calculating key index values under different working conditions in the verification sample data set according to a determined index system, carrying out joint verification on the key index values under a plurality of working conditions by utilizing a multiple hypothesis test method, mapping a verification result into a model grade, and determining an evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model grade. Optionally, the model accuracy related evaluation index is a calculation system for calculating an average absolute error, a maximum error and an interpretation Fang Chafen of the electric energy meter informationized evaluation calibration model.

Optionally, the evaluation index related to the service requirement is a calculation system for calculating the detection rate, the false detection rate and the area under the working characteristic curve of the test subject of the electric energy meter informatization evaluation calibration model.

Optionally, based on an analytic hierarchy process, constructing a key index system for evaluating the informationized evaluation calibration model of the electric energy meter according to the evaluation index related to the accuracy of the model and the evaluation index related to the service requirement, including:

based on the analytic hierarchy process, aiming at the evaluation index related to the model accuracy and the evaluation index related to the business requirement, constructing a hierarchical structure of an evaluation index system;

according to the hierarchical level structure, constructing a comparison judgment matrix of a pairwise evaluation index system;

and calculating to obtain normalized relative importance vectors of each evaluation index system to the upper evaluation index system according to the comparison judgment matrix, and determining a key index system according to the normalized relative importance vectors of each evaluation index system to the upper evaluation index system.

Optionally, before calculating the normalized relative importance vector of each evaluation index system to the upper layer evaluation index system according to the comparison and judgment matrix, the method further includes:

And carrying out consistency check on the comparison and judgment matrix, and calculating to obtain normalized relative importance vectors of each evaluation index system to the upper evaluation index system according to the comparison and judgment matrix after the consistency check is passed.

Optionally, acquiring an original data set of the informationized evaluation calibration model of the electric energy meter, and preprocessing the original data set to obtain a verification sample data set, including:

carrying out normalization processing on the original data set to obtain a normalized original data set;

and determining key factors of the normalized original data set, and carrying out normalization processing on the normalized original data set based on the key factors to obtain a normalized verification sample data set.

Optionally, the key factor is determined based on a pearson correlation analysis method and a maximum information coefficient method.

Optionally, performing joint verification on the value of the key indicator includes:

performing single hypothesis testing on the values of the key indexes corresponding to the verification sample data sets under a single working condition in the verification sample data sets, and performing multiple hypothesis testing on the values of the key indexes corresponding to the verification sample data sets under a plurality of working conditions in the verification sample data sets after the single hypothesis testing is completed;

The single hypothesis test adopts Bayesian factor test;

the multiple hypothesis testing includes: control overall error rate test, control error discovery rate test, and control positive error discovery rate test.

Optionally, mapping the verification result to a model level, and determining the evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model level includes:

based on the verification result, establishing a score map of an informationized evaluation calibration model of the electric energy meter;

the score map comprises: box-cox transformation score mapping, logistic regression score mapping or integrated tree model score mapping;

determining the credibility score of the informationized evaluation calibration model of the electric energy meter based on the score mapping;

and grading the credibility score based on a chi-square box grading method or a business-based grading method to obtain a model grade, and determining an evaluation result according to the model grade. In still another aspect, the present invention further provides an evaluation system for an informationized evaluation calibration model of an electric energy meter, including:

the index system determining unit is used for determining an evaluation index related to the model accuracy of the electric energy meter informatization evaluation calibration model and an evaluation index related to the service requirement aiming at the electric energy meter informatization evaluation calibration model;

The key index system determining unit is used for constructing a key index system for evaluating the informatization evaluation calibration model of the electric energy meter according to the evaluation index related to the accuracy of the model and the evaluation index related to the service demand based on an analytic hierarchy process;

and the evaluation unit is used for acquiring an original data set of the informationized evaluation calibration model of the electric energy meter, preprocessing the original data set to obtain an authentication sample data set, calculating key index values under different working conditions in the authentication sample data set according to a determined index system, carrying out joint authentication on the key index values under a plurality of working conditions by utilizing a multiple hypothesis test method, mapping an authentication result into a model grade, and determining an evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model grade.

Optionally, the model accuracy related evaluation index is a calculation system for calculating an average absolute error, a maximum error and an interpretation Fang Chafen of the electric energy meter informationized evaluation calibration model.

Optionally, the key indicator system determining unit constructs a key indicator system for evaluating the informationized evaluation calibration model of the electric energy meter according to the evaluation indicator related to the accuracy of the model and the evaluation indicator related to the service requirement based on an analytic hierarchy process, and the key indicator system comprises:

Optionally, the key index system determining unit is further configured to: and carrying out consistency check on the comparison and judgment matrix, and calculating to obtain normalized relative importance vectors of each evaluation index system to the upper evaluation index system according to the comparison and judgment matrix after the consistency check is passed.

Optionally, the evaluation unit obtains an original data set of the informationized evaluation calibration model of the electric energy meter, and preprocessing the original data set to obtain a verification sample data set, including:

Optionally, the key factor is determined based on a pearson correlation analysis method and a maximum information coefficient method. Optionally, the evaluating unit performs joint verification on the value of the key indicator, including:

the single hypothesis test adopts Bayesian factor test;

Optionally, the evaluation unit maps the verification result to a model level, and determining the evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model level includes:

Based on the verification result, establishing a score mapping logic of an informationized evaluation calibration model of the electric energy meter;

the score mapping logic includes: box-cox transformation logic, logistic regression scoring mapping logic, or integrated tree model scoring mapping logic;

determining the credibility score of the informationized evaluation calibration model of the electric energy meter based on the score mapping logic;

and grading the credibility score based on a chi-square box grading method or a business-based grading method to obtain a model grade, and determining an evaluation result according to the model grade.

In yet another aspect, the present invention also provides a computing device comprising: one or more processors;

a processor for executing one or more programs;

the method as described above is implemented when the one or more programs are executed by the one or more processors.

In yet another aspect, the present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed, implements a method as described above.

Compared with the prior art, the invention has the beneficial effects that:

the invention provides an evaluation method for an informationized evaluation calibration model of an electric energy meter, which comprises the following steps: aiming at an electric energy meter informatization evaluation calibration model, determining an evaluation index related to the model accuracy of the electric energy meter informatization evaluation calibration model and an evaluation index related to service requirements; based on an analytic hierarchy process, constructing a key index system for evaluating the electric energy meter informatization evaluation calibration model according to the evaluation index related to the model accuracy and the evaluation index related to the service demand; the method comprises the steps of obtaining an original data set of an informatization evaluation calibration model of the electric energy meter, preprocessing the original data set to obtain a verification sample data set, calculating key index values under different working conditions in the verification sample data set according to a determined index system, carrying out joint verification on the key index values under a plurality of working conditions by utilizing a multiple hypothesis test method, mapping a verification result into a model grade, and determining an evaluation result of the performance of the informatization evaluation calibration model of the electric energy meter according to the model grade.

Drawings

FIG. 1 is a flow chart of the method of the present invention;

FIG. 2 is a schematic diagram of the method of the present invention for constructing key indicators;

FIG. 3 is a hierarchical structure diagram of each evaluation index of the method of the present invention;

FIG. 4 is a schematic diagram of a method verification flow of the present invention;

FIG. 5 is a schematic diagram of a conventional multiple hypothesis testing algorithm for the method of the present invention;

fig. 6 is a block diagram of the system of the present invention.

Detailed Description

The exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, however, the present invention may be embodied in many different forms and is not limited to the examples described herein, which are provided to fully and completely disclose the present invention and fully convey the scope of the invention to those skilled in the art. The terminology used in the exemplary embodiments illustrated in the accompanying drawings is not intended to be limiting of the invention. In the drawings, like elements/components are referred to by like reference numerals.

Unless otherwise indicated, terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. In addition, it will be understood that terms defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense.

Example 1:

the invention provides an evaluation method for an informationized evaluation calibration model of an electric energy meter, which is shown in fig. 1 and comprises the following steps:

step 1, aiming at an electric energy meter informatization evaluation calibration model, determining an evaluation index related to the model accuracy of the electric energy meter informatization evaluation calibration model and an evaluation index related to service requirements;

step 2, solving related evaluation indexes, and constructing a key index system for evaluating the informatization evaluation calibration model of the electric energy meter;

and step 3, acquiring an original data set of the informationized evaluation calibration model of the electric energy meter, preprocessing the original data set to obtain a verification sample data set, calculating key index values under different working conditions in the verification sample data set according to a determined index system, performing joint verification on the key index values under a plurality of working conditions by using a multiple hypothesis test method, mapping a verification result into a model grade, and determining an evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model grade. The evaluation index related to the model accuracy is a calculation system for calculating the average absolute error, the maximum error and the interpretation Fang Chafen of the informationized evaluation calibration model of the electric energy meter.

The evaluation index related to the business requirement is a calculation system for calculating the detection rate, the false detection rate and the area under the working characteristic curve of the test subject of the electric energy meter informationized evaluation calibration model.

Based on an analytic hierarchy process, constructing a key index system for evaluating the informationized evaluation calibration model of the electric energy meter according to the evaluation index related to the model accuracy and the evaluation index related to the service demand, wherein the key index system comprises the following components:

Before calculating and obtaining the normalized relative importance vector of each evaluation index system to the upper evaluation index system according to the comparison and judgment matrix, the method further comprises:

The method for obtaining the original data set of the informationized evaluation calibration model of the electric energy meter comprises the steps of:

The key factors are determined based on a Pearson correlation analysis method and a maximum information coefficient method.

Wherein, the joint verification of the values of the key indexes comprises:

The single hypothesis test adopts Bayesian factor test;

The method for determining the evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model grade comprises the following steps of:

and grading the credibility score based on a chi-square box grading method or a business-based grading method to obtain a model grade, and determining an evaluation result according to the model grade. The invention is further described in connection with specific examples as follows:

the implementation of the specific case comprises the following steps:

the informatization evaluation calibration model of the electric energy meter can solve the current situation that the electric equipment evaluation verification is mainly based on the traditional physical transmission calibration method and cannot meet the informatization accurate calibration requirement for a long time. As an innovative technology, the informationized evaluation calibration model of the electric energy meter needs to integrate a plurality of performance indexes through scientific and effective technical means to verify the accuracy of an output result.

The electric energy meter informatization evaluation calibration model is used for analyzing and constructing evaluation key indexes, the consistency of different indexes and actual service requirements is required to be comprehensively analyzed, and a plurality of evaluation indexes are determined from the two aspects of accuracy and service requirements of the model according to an operable and quantifiable principle. The research content combines a plurality of evaluation indexes through an analytic hierarchy process to obtain key indexes for evaluation of an informationized evaluation calibration model of the electric energy meter, lays a foundation for the effectiveness verification of the evaluation calibration model to be developed later, and the flow is shown in a figure 2 and comprises the following steps:

(1) Evaluation indexes related to model accuracy;

in the electric energy meter informatization evaluation calibration model evaluation verification platform, the load data generator can simulate electric energy meter data with different error values, and the accuracy of the electric energy meter informatization evaluation calibration model can be directly evaluated by comparing the electric energy meter error value output by the electric energy meter informatization evaluation calibration model with the electric energy meter error value set during simulation.

1) Average absolute error;

the Mean Absolute Error (MAE) is a performance measurement index commonly used for regression tasks in machine learning, and can reflect the deviation degree of a model output value and a true value, and the calculation mode is shown in a formula (1):

Wherein n represents the number of electric energy meters in the simulation platform area, y _i Represents the error value set by the ith electric energy meter in simulation,and the error value of the informatization evaluation calibration model of the electric energy meter to the output of the ith electric energy meter is represented. The smaller the value of the average absolute error is, the closer the error value of the electric energy meter output by the evaluation calibration model is to the actual error value of the electric energy meter, namely the better the performance of the evaluation calibration model is.

2) Maximum error;

the Maximum Error (ME) is also one of the performance metrics of the regression task, and is different from the average absolute error in that the average performance of the calibration model on all simulated electric energy meters is evaluated, and the maximum error reflects the worst performance of the calibration model on all simulated electric energy meters. The calculation mode is shown in the formula (2):

wherein y is _i Represents the error value set by the ith electric energy meter in simulation,and the error value of the informatization evaluation calibration model of the electric energy meter to the output of the ith electric energy meter is represented. The smaller the maximum error is, the closer the error value of the electric energy meter output by the evaluation calibration model is to the actual error value of the electric energy meter, namely the better the performance of the evaluation calibration model is.

3) Interpretation side difference;

interpretation Fang Chafen (EVS) can be used to measure how stable a model is to the size of a target value, as shown in equation (3):

Wherein y represents the error value set by all the electric energy meters during simulation,the error values of the informatization evaluation calibration model of the electric energy meter for all the electric energy meters in the simulation platform area are represented, and Var represents variance. The closer the value of the difference of the interpretation party is to 1, the smaller the influence of the magnitude of the error value of the electric energy meter on the performance of the evaluation calibration model is, namely the more stable the performance of the evaluation calibration model is.

(2) Evaluation indexes related to business requirements;

in actual business, the informatization evaluation calibration model of the electric energy meter needs to judge whether the electric energy meter is a misalignment electric energy meter or not through the calculated error value. In the electric energy meter informatization evaluation calibration model evaluation verification platform, the load data generator can simulate electric energy meter data with normal and different degrees of misalignment, and the accuracy of the electric energy meter informatization evaluation calibration model in the face of actual service demands can be evaluated by comparing whether the misalignment electric energy meter judged by the electric energy meter informatization evaluation calibration model is a simulated misalignment electric energy meter.

1) A confusion matrix;

because the number of the electric energy meters which normally work in the transformer area is generally far more than that of the electric energy meters which are not aligned, the evaluation calibration model can obtain extremely high accuracy even if the electric energy meters are considered to be the electric energy meters which normally work, and therefore indexes such as accuracy and the like which are commonly used in classification problems are not directly applicable to the informationized evaluation calibration model for the evaluation and verification electric energy meters. Judging whether the electric energy meter is a misalignment meter or not is regarded as a classification problem, and the confusion matrix can intuitively reflect the performance of the evaluation calibration model on actual service requirements. As shown in table 1, the greater the number of positive and negative checks and the smaller the number of false checks, the better the performance of the evaluation calibration model.

TABLE 1

2) Detecting rate;

the detection rate (TPR) is the ratio of the number of the misalignment electric energy meters correctly judged by the electric energy meter informatization evaluation calibration model to the number of the misalignment electric energy meters set in all simulation, and the calculation mode of combining the confusion matrix is shown in a formula (4):

TP represents the number of the misalignment electric energy meters correctly judged by the informatization evaluation calibration model of the electric energy meters, FN represents the number of the misalignment electric energy meters which cannot be judged by the evaluation calibration model, and the higher the detection rate is, the more the misalignment electric energy meters which can be correctly detected by the evaluation calibration model are indicated.

3) False detection rate;

the false detection rate (FPR) is the proportion of the number of the normal electric energy meters which are wrongly judged to be misaligned by the electric energy meter informatization evaluation calibration model to the number of the normal electric energy meters set in all simulation, and the calculation method of the combination confusion matrix is shown in a formula (5):

wherein FP represents the number of normal electric energy meters which are incorrectly judged as misalignment by the informatization evaluation calibration model of the electric energy meters, and TN represents the number of normal electric energy meters which are correctly judged by the evaluation calibration model. The lower the false detection rate is, the fewer the number of the electric energy meters which are judged as the misalignment by the evaluation calibration model to be normal electric energy meters is.

4) Area under the working characteristic curve of the subject;

And traversing different thresholds to convert the error value of the electric energy meter output by the electric energy meter informatization evaluation calibration model into whether the electric energy meter is misaligned or not, calculating the detection rate and the false detection rate under the thresholds, and drawing a subject working characteristic curve (ROC) of the evaluation calibration model. The area under the curve (AUC) of the curve and the coordinate axis is calculated, and the expected generalization performance of the evaluation calibration model, which is not influenced by the threshold value, can be reflected. The closer the value of the area under the curve is to 1, the better the performance of the evaluation calibration model on the actual business requirements.

(3) Synthesizing a plurality of indexes into key indexes;

the method can respectively determine a plurality of evaluation indexes such as average absolute error, interpretation difference, detection rate, false detection rate, area under a subject working characteristic curve and the like from two aspects of model accuracy and business requirement of the electric energy meter informatization evaluation calibration model. The evaluation indexes respectively reflect the performance of the evaluation calibration model from different angles, but the evaluation calibration model is generally difficult to obtain higher scores on all indexes at the same time, and the importance degree of different indexes on actual demands is different, so that weights are scientifically calculated according to the importance degrees of the indexes and synthesized into a comprehensive key index, which is more beneficial to the follow-up evaluation verification of the performance of the informationized evaluation calibration model of the electric energy meter.

Because it is difficult to directly give weights to a plurality of evaluation indexes such as average absolute error, interpretation difference, detection rate, false detection rate, area under a subject working characteristic curve, and the like, a multi-index and multi-layer evaluation index structure needs to be established first, and a qualitative and quantitative organic combination method is adopted, so that the complex evaluation problem is made clear. The analytic hierarchy process can decompose complex key performance indexes into combinations of evaluation indexes, groups the evaluation indexes into a hierarchical structure, determines the relative importance of the indexes in the hierarchy by a pairwise comparison method, and establishes a judgment matrix to calculate comprehensive key indexes.

1) Establishing a hierarchical level structure of an evaluation index;

the model accuracy and business requirement of the electric energy meter informatization evaluation calibration model are used as first-level indexes, and a plurality of specific evaluation indexes such as average absolute error, interpretation difference, detection rate, false detection rate, area under a subject working characteristic curve and the like are used as second-level indexes, so that a hierarchical structure as shown in fig. 3 can be established:

2) Constructing a pairwise comparison judgment matrix;

when calculating a certain high-grade index of the informatization evaluation calibration model of the electric energy meter, the low-grade evaluation indexes under the electric energy meter are required to be compared in pairs, and the grade is rated according to the importance degree of the low-grade evaluation indexes. The matrix formed by the comparison results of two pairs is called a judgment matrix, and the judgment matrix has the following properties:

Wherein a is _ij A is the importance comparison result of index i and index j _ji The importance of index j and index i is compared. Judging element a in matrix _ij The scale method of (2) is shown in table 2 below:

TABLE 2

Index i is compared with index j	Quantized value
		Equally important	1
Slightly important	3
		Is of great importance	5
Is of great importance	7
		Extremely important	9
Intermediate value of two adjacent judgments	2,4,6,8

The average absolute error, the maximum error and the interpretation difference obtained according to expert experience are compared in pairs to judge matrixes as shown in table 3:

TABLE 3 Table 3

The detection rate, false detection rate and area under the working characteristic curve of the test subject obtained according to expert experience are compared in pairs, and the judgment matrix is shown in table 4:

TABLE 4 Table 4

The judgment matrix of the model accuracy and the business requirement obtained according to expert experience is shown in table 5:

TABLE 5

3) Calculating normalized relative importance vectors of each evaluation index to the upper evaluation index;

after the comparison judgment matrix is constructed pairwise, a method root method can be used for solving a normalized relative importance vector W of each evaluation index to the upper evaluation index _i ⁰ . The method of root method is shown in formulas (7) and (8), wherein n is the dimension of the judgment matrix A.

4) Consistency test;

before the key index performance of the model calculated according to the judgment matrix, consistency test is required to be carried out on the judgment matrix so as to ensure the rationality of the judgment matrix. Consistency test first, a consistency index CI of a judgment matrix A is calculated according to formulas (9) and (10):

Then, the test coefficient CR corresponding to CI needs to be calculated according to formula (11):

wherein RI can be obtained by looking up a table:

TABLE 6

n	1	2	3	4	5	6	7	8	9	10
											RI	0	0	0.52	0.89	1.12	1.26	1.36	1.41	1.46	1.49

If CR <0.1, the judgment matrix is considered to pass the consistency check, otherwise, the judgment matrix needs to be properly corrected until the consistency requirement is met.

5) Calculating key index performance of the model;

after the normalized relative importance vector is obtained through calculation and the consistency test is passed, the hierarchical analysis method takes the normalized relative importance vector as a weight and calculates key indexes of the model layer by layer from low to high according to all secondary performance indexes of the electric energy meter informatization evaluation calibration model. The key performance indexes obtained through the analytic hierarchy process can comprehensively consider a plurality of different performance indexes from the angles of model accuracy, business requirements and the like of the electric energy meter informatization evaluation calibration model, and lays a foundation for the effectiveness evaluation verification of the subsequent development evaluation calibration model.

Research of an evaluation verification method aiming at an informationized evaluation calibration model of the electric energy meter;

the electric equipment evaluation and verification is mainly based on a traditional physical transmission calibration method for a long time, and is different from the traditional physical transmission calibration method, the electric energy meter error data is calculated and output through an evaluation calibration model, the electric energy meter error data is essentially an informatization calibration method, and at present, a scientific and effective technical means for evaluating the accuracy of each performance index and output result of the verification model is not available, and the existing method cannot adapt to the current situation of the informatization accurate calibration requirement, so that the research of the electric energy meter informatization evaluation calibration model evaluation and verification method is necessary.

Based on the informatization evaluation calibration model of the electric energy meter, obtaining corresponding model output values by simulating various input quantities and working conditions of the evaluation calibration model to form an original data set; the relevance of the input quantity and working condition parameters of the evaluation calibration model and the output result of the model is checked by using analysis methods such as a pearson correlation coefficient, a maximum information coefficient and the like, the input quantity and the working condition parameters are screened, key factors influencing the output accuracy of the evaluation calibration model are determined, the range of the key factors is determined by combining priori knowledge, and corresponding sample data are selected to form a reference data sample set for subsequent comprehensive verification of the evaluation calibration model; for all electric meters under a single working condition, calculating indexes of each level according to the result output by the evaluation calibration model and the reference value of the simulation model, and verifying the credibility of the model by adopting technologies such as Bayesian factor test, t-hypothesis test and the like and combining the importance degree of the working condition; then, the reliability of the output of the evaluation calibration model under different working conditions is verified in a combined mode by using a multiple hypothesis test theory; the mapping logic of determining the model reliability score and the discrete model evaluation grade by adopting box-cox transformation, logistic regression, tree model grading mapping logic and the like is researched, the method of determining threshold segmentation by combining a whole evaluation index system based on the optimal grade classification of technology, the threshold classification strategy based on business grade classification and the like is researched, the evaluation grade and the whole evaluation grade of the model under different working conditions are analyzed, and the evaluation verification of the model evaluation calibration result is realized. The workflow diagram of the main study is shown in fig. 4.

(1) Evaluating the calibration model to verify data preprocessing and determining key factors influencing output accuracy;

according to the model verification requirement, different input values and working condition parameters of the evaluation calibration model are changed to simulate different actual conditions, and the different input values and working condition parameters and corresponding model output results form an original data set. In order to solve the problems that the scale difference of each variable in an original data set is overlarge, redundancy exists or variables are irrelevant, the reusability of data is improved, a good data basis is provided for later model verification, the data are subjected to normalized description by adopting a data preprocessing method such as normalization, key factors influencing an evaluation calibration model are determined through research of feature selection methods such as pearson correlation coefficients, maximum Information Coefficients (MIC) and the like, and the range of the key factors is determined by combining priori knowledge. And screening sample data in the original data set by taking the key factors and the range thereof as the basis to form a verification sample data set, and removing factors with insignificant influence on output so as to reduce the experimental space, fully verify the model and reduce the running cost of verification experiments and a reference system.

1) Normalizing;

Because the dimensions of different features are different, the value ranges are greatly different, and if the dimensions of the different features are not unified, the model verification effect can be possibly affected. Therefore, in the study, a min-max normalization method is adopted for all the characteristics, and the value range of all the variables is limited in the [0,1] interval. Normalization is shown in formula (12):

wherein x is ^* For sample dataNormalized value, x _max For the maximum value of the sample data, x _min Is the sample data minimum.

2) Pearson correlation coefficient;

the pearson correlation coefficient (Pearson Correlation) is one way to measure the linear correlation between two variables X and Y. It evolved from a similar but slightly different idea, by cals pearson, from franciss Gao Erdu in the 80 s of the 19 th century. The pearson correlation coefficient between two variables is defined as the quotient of the covariance and standard deviation between the two variables, as shown in equation (13):

where the above equation defines the overall correlation coefficient, the greek lower case ρ is often used as a representative symbol. Sigma (sigma) _X ，σ _Y The standard deviation of X and Y is denoted cov (X, Y) which represents the covariance of the total X and Y, which is an indicator of how correlated two random variables are, if one variable is followed by the other variable, either becoming larger or smaller at the same time, the covariance of the two variables is positive, and vice versa. Although covariance can reflect the correlation degree of two random variables, the magnitude of the covariance value cannot well measure the correlation degree of the two random variables, so a pearson correlation coefficient is introduced, which is divided by the standard deviation of the two random variables on the basis of covariance, and the correlation degree of the two random variables is better measured. As shown in equation (14), the pearson correlation coefficient can be obtained by estimating the covariance and standard deviation of the samples, which is often indicated by the english lowercase r:

Wherein n represents the number of samples, X _i Representing the i-th sample. The output range of the pearson correlation coefficient is-1 to +1, and when the linear relation of two variables is enhanced, the correlation coefficient tends to be 1 or-1; when one isWhen the variable is increased and the other variable is also increased, the positive correlation is shown between the variable and the variable, and the correlation coefficient is larger than 0; if one variable increases, the other decreases, indicating a negative correlation between them, the correlation coefficient is less than 0; if the correlation coefficient is equal to 0, it indicates that there is no linear correlation between them.

3) Maximum information coefficient;

the pearson correlation coefficient method can measure the linear relation between variables well, but complex nonlinear relation can exist between the variables, so the maximum information coefficient method (The Maximal Information Coefficient) can be further adopted for analysis. The maximum information coefficient method is a correlation algorithm of the functional relation and the statistical relation among evaluation variables, which does not need to make any assumption on data distribution, and is proposed by Reshef in Science, and has wider applicability. The MIC algorithm process mainly comprises the following steps: if there is a certain association relationship between the two variables, after grid division of a certain specific scale is adopted in the scatter diagram of the two-variable joint sample, the mutual information value (Mutual Information) of the two variables can be calculated according to the marginal probability density function and the joint probability density function in the grid, and the association between the two variables can be detected as a normalized result. The specific steps for analysis using the maximum information coefficient are as follows:

Step 1: given a finite ordered pair dataset d= { (x) _i ,y _i ) If the X and Y axes are divided into X and Y grids, respectively, obtaining an X Y grid G, and the variable values in D fall into the grid of G to obtain a corresponding probability distribution d| _G Where x and y are positive integers, if the grid division number is fixed, different mutual information values are obtained by changing the grid division positions, and the maximum mutual information value is shown in formula (15):

I ^* (D,x,y)＝maxI(D| _G ) (15)

step 2: in order to facilitate comparison between different dimensions, the above formula is normalized so that its value is within the interval [0,1]:

step 3: given an ordered pair data set D of sample size n, the MIC formula defining the two variables X and Y in the set is as follows:

wherein xy.ltoreq.B (n) { B (n) =n ^a N is the size of data, in order to limit the grid size to divide the area, obtain the correlation value, the value of constant a can be set according to experience or scale. Setting a proper B (n) is very important, the value of the B (n) is closely related to the universality of the MIC algorithm with the maximum information coefficient, and if the value is too small, the universality of the algorithm is reduced so that the algorithm can only detect a simple association relation; if the value is too large, false correlation can be caused under the condition of limited samples.

In step 1, x and y mutual information I (D|in the data set D is calculated _G ) Based on the nature of the mutual information, the MIC algorithm can be deduced to have the following properties:

(1) For a grid division G of x y, since 0.ltoreq.I (D.| _G )≤log _min { x, y }, each element in the matrix has a value interval of [0,1 ]]The correlation coefficient value MIC is the result of normalization of all maximum mutual information, and therefore, this value is also [0,1 ]]Between them.

(2) Since mutual information has symmetry, I (X; Y) =i (Y; X), the feature matrix M (D) is a symmetric matrix, i.e., MIC (X, Y) =mic (Y, X).

(3) Due to data distribution D| _G The values of (2) are related to the ordering distribution of the variable values, when the data of the two variables X and Y are transformed but the ordering position is unchanged, the characteristic matrix is unchanged, and then the correlation value MIC is unchanged. (MIC algorithm has invariance under order transformation).

For the calculation process of the MIC algorithm, the difficulty is how to obtain the maximum mutual information in all grid divisions of two variables, i.e., I (D, x, y), and it takes more time to find the mutual information value between the two variables, resulting in low calculation efficiency, so in the algorithm designed by Reshef et al, a polynomial approximate solution process is designed, and although an accurate value cannot be obtained, a better approximate solution can be obtained.

(2) A joint verification method of multi-angle user load characteristics based on multiple hypothesis test;

and according to the established index systems of all levels, carrying out similarity analysis on the preprocessed model output result data and simulation system reference data with the digital space consistent with the physics, and calculating the result credibility score of the evaluation calibration model on the single ammeter of the platform area. The comprehensive study comprises statistical consistency quantitative verification methods such as t hypothesis test and Bayesian factor test, and the like, an evaluation verification method which is suitable for evaluating the calibration model is selected, and the reliability of the result of the evaluation calibration model on a plurality of electric meters in a platform area is comprehensively verified.

The effectiveness of the electric energy meter informatization evaluation calibration model is influenced by the actual working condition, the reliability of the model result under different input conditions can be different, and the reliability of the evaluation calibration model result under each input condition is verified. The probability of first class errors caused by multiple independent hypothesis testing is increased along with the increase of the number of testing times, multiple hypothesis testing verification is conducted on the reliability of the output of the evaluation calibration model under different working conditions by using the multiple hypothesis testing theory, the error rate of multiple hypothesis testing is reduced, and the joint verification of the evaluation calibration model results under complex and various platform scenes and multi-angle user load characteristics is achieved.

1) The single independent evaluation calibration model evaluation verification method;

single bulk sample t-test: the Student's t test is mainly used for normal distribution with smaller sample content and unknown total standard deviation sigma. the t-test is to use the t-distribution theory to infer the probability of occurrence of a difference, and thus compare whether the difference between two averages is significant. The single population t-test is to examine whether the difference of one sample average from one known population average is significant. When the population distribution is a normal distribution, such as the population standard deviation is unknown and the sample volume is less than 30, then the dispersion statistic of the sample mean and the population mean is in a t distribution. First, calculateSample mean; secondly, based on experience or previous findings, a hypothesis is put on the mean of the population, i.e. μ=μ ₀ (μ ₀ Is the overall average to be tested); then, the sample mean value calculated by analysis is derived from the mean value of mu ₀ If the probability is small, then the average of the population is considered to be other than mu ₀ . Single sample t-test procedure:

1. the original assumption and the alternative assumption are proposed: original assumption H ₀ It is considered that there is no significant difference between the overall mean and the test value, and the original assumption H ₀ ：μ＝μ ₀ Alternative hypothesis H ₁ ：μ≠μ ₀ ；

2. Determining test statistics: the test statistic is t statistic:

Wherein the method comprises the steps ofS is the sample standard deviation and n is the sample number. The statistic t assumes at zero: μ=μ ₀ Obeying t distribution with the degree of freedom of n under the condition of true;

3. calculating an observed value and a p-value of the test statistic (p-value refers to the probability of making a first type of error);

4. determining a significance level α and making a decision: typically, the most commonly used alpha value is 0.05, and 0.001,0.005,0.0001 may be used in combination with the specific case. If the p value is less than or equal to the significance level alpha, rejecting the original hypothesis, namely considering that a significant difference exists between the overall mean and the test value; if the p-value is greater than the significance level α, then the original assumption is accepted that there is no significant difference between the overall mean and the test value.

The single-population t-test is mainly used for testing whether a significant difference exists between the population mean value of a certain variable and a certain specified value. If the single sample test is a large sample, the single sample test is generally referred to as U test in statistics, and mainly adopts U statistics subject to normal distribution as test statistics; if small and the samples follow a normal distribution, a single sample t-test is performed using the t-statistic. In statistics, the density functions of the t-distribution and the normal distribution are quite close in the case of large samples according to the central limit theorem of probability theory. In practical analysis operations, it is possible to perform a single sample mean test with a t-test, whether large or small. For the t-test, the robustness is good, and the significance of this feature is that the t-test can be used for analysis if the sample distribution deviates from the normal distribution by no means particularly severe. Thus, the single-population t-test method may be one of the evaluation verification methods adapted to evaluate the calibration model.

Bayesian factor test: the method and rule of treatment of hypothesis testing problems by the scholars are different from those of classical statistical school, and different rules of detection often result in different test results. For the evaluation verification of evaluating the calibration model, two types of test methods are comprehensively analyzed, and a verification method suitable for evaluating the calibration model is selected.

Although the classical statistical-based hypothesis testing method is a widely used statistical inference method at present, its drawbacks are apparent. For a fixed level test it is necessary to give a significance level α in advance, and thus to determine the reject domain of the original hypothesis, but there should be no specific criterion for how much α should be given, and sometimes opposite test conclusions will be drawn from different significance levels. The p value calculated by the p value test is the probability that the test statistic takes a value under the test sample when the original assumption is true, and is a true significant level. Although the use of value test avoids the influence on test results due to the selection of different values, the use of p value for test judgment still has some problems, which are specifically expressed in: (1) probability that the p value is not true for the original hypothesis. The p value is the probability of obtaining the observed sample when the original assumption is true, and is the probability of the data, and is not an effective estimated value of the true probability of the original assumption. (2) when the sample volume is large, the p-value is not very effective. When the sample size is large enough, almost any one of the original hypotheses will correspond to a very small p-value, and any of the original hypotheses Will be rejected. There are studies found that: one at 10 ^-10 P-value rejection H of ₀ When n is sufficiently large, H is ₀ The posterior probability of (2) gradually approaches 1, and this surprising result is called the "Lindley paradox". Thus, the p-value test fails almost as the sample capacity increases.

The bayesian school of inspection method is straightforward relative to the classical statistical school of hypothesis inspection method. It directly calculates the original assumption H after posterior distribution is obtained ₀ And alternative hypothesis H ₁ Posterior probability alpha of (a) ₀ And alpha ₁ And calculates a posterior probability ratio to compare the magnitudes of the two posterior probabilities:

when (when)When receiving H ₀ ；

When (when)When receiving H ₁ ；

When (when)And further sampling or further acquiring prior information for judgment.

Under the prior-verification distribution, the above idea can be expressed as a decision function:

in view of the sometimes difficulty in directly computing posterior probabilities, the posterior probabilities can be calculated by Bayesian factorsTo calculate the posterior probability ratio, namely, the B of the Bayesian factor can be conveniently calculated by the known information ^π (x) The value is then multiplied by the prior probability ratio by a Bayesian factor to directly obtain the posterior probability ratio.

Advantages of bayesian hypothesis testing over traditional statistical hypothesis testing:

(1) The method is relatively simple. The hypothesis test of the Bayesian school is directly judged according to the size of the posterior probability, and the difficulty of the hypothesis test of the classical statistical school, namely the sampling distribution of the statistic is determined by selecting the test statistic, is avoided, so that the hypothesis test method of the Bayesian school is relatively simple.

(2) The sufficiency of the a priori information utilization. The assumption test of classical statistical school only uses the information of the sample, while the Bayesian school uses both the sample information and the prior information of the parameters in the assumption test, and synthesizes the information into posterior distribution and deduces according to the posterior distribution, so the Bayesian method is more sufficient in information utilization, and the judging process is more in accordance with the actual thinking mode of people.

2) Multiple hypothesis testing theory;

and under the complex and various platform area scenes and multi-angle user load characteristics, multiple times of verification on the evaluation and calibration model results are required. In hypothesis testing, the theory of the single hypothesis testing problem is to minimize the probability of making a second type of error when the probability of making the first type of error is controlled within an acceptable range. In the case of simultaneous hypothesis testing of multiple subjects, it is possible for each individual hypothesis test to make a first type of error, which, if no control measures are taken, increases with increasing number of tests m. For example, when two independent tests are performed, the probability of making a class of errors as a whole is 1- (1-alpha) ² The method comprises the steps of carrying out a first treatment on the surface of the The probability of gross error in one class when tested m times is 1- (1-alpha) ^m Taking the significance level α=0.05, when the number of tests is 100, the probability of an overall class-one error approaches 1, at which time the test result itself becomes very unreliable.

TABLE 7

Multiple hypothesis test verification is carried out on the reliability of the output of the evaluation calibration model under different working conditions by using the multiple hypothesis test theory, so that the error rate of multiple hypothesis tests is reduced, and the joint verification of the evaluation calibration model results under complex and various platform scenes and multi-angle user load characteristics is realized. Multiple hypothesis testing is performed by considering a number of singlets as a family of tests, and simultaneously testing each hypothesis in the family. In multiple hypothesis testing, the thought of single test can be used to consider several reasonable and effective methods to measure the first class of errors, and the measurement standard is controlled in a reasonable range by a certain testing method, so that the testing efficacy is as large as possible.

Currently common error metrics are mainly overall error rate (FWER), error discovery rate (FDR), and positive error discovery rate (pFDR). The choice of error metrics also differs when solving different problems. When the value of m in the multiple tests is smaller, the control of the overall error rate has substantial significance; the value of m is larger, only when the p value is small, a differential expression gene can be obtained, and the control of FWER is too conservative at the moment, so that FDR can be selectively controlled. A common multiple hypothesis testing algorithm is illustrated in fig. 5:

Controlling the overall error rate: the multiple verification process of controlling overall errors is a traditional control process, and there are three main types: a single step process, a step-down process, and a step-up process. The single step process means that the reject domain of each hypothesis is the same in the multiple verification process, and the original p values do not need to be ordered; for the latter two control procedures, let m hypothesis test H ₁ ,…,H _m The corresponding original p value is p ₁ ,…,p _m The sequence of the two is p after sorting from small to large ₍₁₎ ,p ₍₂₎ ,…,p _(m) . The step-up procedure refers to proceeding from the smallest p value to the largest p value based on such ordering; while the step-down procedure proceeds from a maximum p-value to a minimum p-value. Bonferroin algorithm,The Homl algorithm and the Hochberg algorithm are classical algorithms employing these three control procedures, respectively, by using which the overall error rate of evaluating the calibration model reliability test multiple times can be controlled.

Controlling the error discovery rate: in multiple hypothesis testing, the error metric FDR has been widely used, and the testing method is based on the p value. Benjamini and Hochberg have first proposed the concept of FDR and provided the BH algorithm flow. The BH algorithm uses the p value to obtain a demarcation point for rejecting or accepting an assumption under the condition that the significance level α is selected in advance. In order to facilitate application of Benjaminii and Hochberg, an equivalent method of BH algorithm is provided on the basis of the Benjaminii and Hochberg, p value is adjusted, and the adjusted p value is compared with significance level. Benjanimi and Yekutieli modify the algorithm process while promoting the BH algorithm conclusion to obtain BY algorithm. When the joint distribution of test statistics is arbitrary, the BY algorithm can still control the error discovery rate.

Control positive error rate: the basic ideas of the multiple inspection methods for controlling the overall error rate and the error discovery rate are to judge the original hypothesis according to the p value sequence, namely, given the significance level alpha, find the reject domain and control the error rate below alpha. The checking algorithm proposed by store (2003) is to determine the reject domain empirically, then calculate the estimated value of pFDR and define the q value (p value corrected by FDR method), and finally determine the check rejecting the original hypothesis through q value.

(3) Threshold value division and model performance grade assessment method based on prior information and data driving combination;

according to the established complete evaluation calibration model verification index system, the credibility index of the model result under different input conditions is obtained, and the score of the model result is usually mapped into a numerical value between 0 and 1000 by a specific method manually and then divided into discrete levels. Although the continuous scores can directly compare the difference of the reliability of the output results of the evaluation calibration model under different input conditions, in actual use, different strategies need to be adopted for the reliability of different models, development cost is very high when different model use strategies are distributed to each reliability score, and the stability of the whole strategy system is very poor, so that the reliability of the models needs to be divided into several evaluation grades. After the performance grade evaluation of the model of the simulation standard platform area is completed, the classification-based method is adopted to classify the performance grade of the complex working conditions possibly occurring in the real environment. Typically discretizing the rating scale into two steps, determining the score mapping logic and threshold rating.

1) Mapping the score;

the most central requirement of the mapping of the score is to map the model output to an index that is easier for some business person to understand. Common score mapping logic is: box-cox transforms, scoring mapping for logistic regression, and scoring mapping logic for integrated tree models, etc.

box-cox transform: the simplest mapping logic is to multiply the score of 0-1 by 1000, resulting in a score of 0-1000. However, in order to make the distribution more consistent, the score is usually adjusted toward the normal distribution pattern using a cox-box transform.

Scoring mapping of logistic regression: the method is a grading mapping method of scaling ratio which is commonly used for conversion of a grading card of a user in financial business.

Wherein score is the output after the score card mapping, P _{Positive direction} Probability of credible model result, P _{Negative pole} Is the probability that the model results are not trusted. The Base Score (Base Score) 650 and the step size (Point of Double Odds, PDO) 50 in the above formula need to be adjusted according to the service requirements. Still another mapping method is to directly make the following way without considering the true meaning of the score conversion:

score＝650+50×log ₂ (pred-lag) (21)

where pred is the reliability of the model output, lag is typically set to a value where the desired model reliability is equal to the underlying time-sharing correspondence. In addition, the step size can be changed into a dynamic step size which varies with pred, so that distribution after scoring mapping is denser.

Score mapping logic for an integrated tree model: the score mapping logic of XGBoost/LightGBM is the probability that the higher the original score is, the smaller the mapping score is, and the corresponding model is not trusted, and the score mapping formula is as follows:

2) A threshold ranking algorithm;

after determining the mapping logic for the validation model to output the confidence score, a suitable threshold ranking algorithm is selected to rank the mapped score. Score classification itself is a binning problem. There are a great number of box-dividing methods in the field of wind control, and all the methods have excellent performances. The credibility level of the evaluation calibration model can be divided by referring to the box division technology in the variable processing process. The confidence score of the model output result is a continuous variable. Reasonable hierarchical logic can be obtained by using an algorithm for discretizing continuous variables, and the dividing thresholds corresponding to different levels are obtained by combining prior knowledge of indexes of the lower levels of the whole evaluation index system.

Optimal grading based on technology: chi-square binning uses chi-square testing to determine an optimal binning threshold. If two adjacent bins have similar tag distributions, the two bins are merged. Low chi-square values indicate that they have a similar class distribution. Chi-square test is the analysis of the frequency of the classification data. Its application is mainly represented in two aspects: fitting goodness test and independence test. The fitting goodness is to test a classification variable, namely, according to the overall distribution condition, calculating the expected frequency of each class in the classification variable, comparing with the distributed observation frequency, and judging whether the expected frequency and the observation frequency have significant differences, thereby achieving the purpose of analyzing the classification variable. An independence test is a calculation between two characteristic variables that can be used to analyze whether two classification variables are independent or have an association.

Chi-square binning is to use an independence test to pick the threshold value of the partitioning node. The chi-square binning process can be split into initialization and merging. (1) initializing: the initial discretization is constructed by sorting according to the continuous variable value size, i.e. each individual value is treated as a bin. The purpose of this is to gradually merge from each individual. (2) combining: traversing the two combined chi-square values, combining the two combinations with the smallest chi-square values, and repeating until the limit of the number of the sub-boxes is met.

Service-based ranking: the method based on business classification refers to the fact that the sample number of each class is basically consistent with the previous version model after the current classification class is expected to be stored. It is desirable to ensure that the score threshold is unchanged during model iteration, so that the linkage strategy of the corresponding threshold does not need to be modified. At this time, the base fraction and the step length are required to be adjusted so that the duty ratio of the final classification result of the model in each interval is the same. The piecewise functions may be borrowed and the combined score mapping logic may be employed to make adjustments that do not affect the ordering ability of the final model.

Example 2:

the invention also provides an evaluation system 200 for the informationized evaluation calibration model of the electric energy meter, as shown in fig. 6, comprising:

An index system determining unit 201, configured to determine, for an electric energy meter informatization evaluation calibration model, an evaluation index related to a model accuracy of the electric energy meter informatization evaluation calibration model and an evaluation index related to a business requirement;

the key index system determining unit 202 is configured to construct a key index system for evaluating the informatization evaluation calibration model of the electric energy meter according to the evaluation index related to the accuracy of the model and the evaluation index related to the service requirement based on an analytic hierarchy process;

and the evaluation unit 203 is configured to obtain an original data set of the informationized evaluation calibration model of the electric energy meter, pre-process the original data set to obtain a verification sample data set, calculate key index values under different working conditions in the verification sample data set according to a determined index system, perform joint verification on the key index values under multiple working conditions by using a multiple hypothesis test method, map the verification result into a model level, and determine an evaluation result of the performance of the informationized evaluation calibration model of the electric energy meter according to the model level.

The evaluation index related to the model accuracy is a calculation system for calculating the average absolute error, the maximum error and the interpretation Fang Chafen of the informationized evaluation calibration model of the electric energy meter.

The key index system determining unit 202 constructs a key index system for evaluating the informationized evaluation calibration model of the electric energy meter according to the evaluation index related to the accuracy of the model and the evaluation index related to the service requirement based on an analytic hierarchy process, and the key index system comprises:

Wherein, the key index system determining unit 202 is further configured to: and carrying out consistency check on the comparison and judgment matrix, and calculating to obtain normalized relative importance vectors of each evaluation index system to the upper evaluation index system according to the comparison and judgment matrix after the consistency check is passed.

The evaluation unit 203 obtains an original data set of the informationized evaluation calibration model of the electric energy meter, and performs preprocessing on the original data set to obtain a verification sample data set, which includes:

Wherein the evaluation unit 203 performs joint verification on the values of the key indexes, including:

the single hypothesis test adopts Bayesian factor test;

Wherein the evaluation unit 203 maps the verification result to a model level, and determining the evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model level includes:

and grading the credibility score based on a chi-square box grading method or a business-based grading method to obtain a model grade, and determining an evaluation result according to the model grade. According to the technical scheme, according to the characteristics of the electric meter informatization evaluation calibration model in terms of model accuracy and business requirements, a model evaluation key index analysis and construction method based on analytic hierarchy process is provided, a plurality of evaluation indexes of average absolute error, interpretation difference and detection rate, false detection rate and area under a test subject working characteristic curve are determined, and according to different importance degrees of different indexes on actual requirements, an analytic hierarchy process is used for constructing an analytic hierarchy process, so that the problem that a plurality of single indexes are difficult to effectively synthesize key indexes is solved. In consideration of the fact that the effectiveness of the informatization evaluation calibration model of the electric energy meter is affected by actual working conditions, the reliability of model results under different input conditions may be different, the multi-working-condition combined verification method of the informatization evaluation calibration model of the intelligent electric energy meter based on multiple hypothesis test is provided, multiple hypothesis test verification is carried out on the reliability of the output of the evaluation calibration model under different working conditions by using multiple hypothesis test theory, the error rate of multiple hypothesis test is reduced, and the combined verification of the evaluation calibration model results comprehensively considering multiple indexes and multiple working conditions under complex and various platform scenes is realized.

Example 3:

based on the same inventive concept, the invention also provides a computer device comprising a processor and a memory for storing a computer program comprising program instructions, the processor for executing the program instructions stored by the computer storage medium. The processor may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application SpecificIntegrated Circuit, ASIC), off-the-shelf Programmable gate arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computational core and control core of the terminal adapted to implement one or more instructions, in particular adapted to load and execute one or more instructions within a computer storage medium to implement the corresponding method flow or corresponding functions to implement the steps of the method in the embodiments described above.

Example 4:

based on the same inventive concept, the present invention also provides a storage medium, in particular, a computer readable storage medium (Memory), which is a Memory device in a computer device, for storing programs and data. It is understood that the computer readable storage medium herein may include both built-in storage media in a computer device and extended storage media supported by the computer device. The computer-readable storage medium provides a storage space storing an operating system of the terminal. Also stored in the memory space are one or more instructions, which may be one or more computer programs (including program code), adapted to be loaded and executed by the processor. The computer readable storage medium herein may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory. One or more instructions stored in a computer-readable storage medium may be loaded and executed by a processor to implement the steps of the methods in the above-described embodiments.

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The scheme in the embodiment of the invention can be realized by adopting various computer languages, such as object-oriented programming language Java, an transliteration script language JavaScript and the like.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. An evaluation method for an informationized evaluation calibration model of an electric energy meter, which is characterized by comprising the following steps:

the method comprises the steps of obtaining an original data set of an informationized evaluation calibration model of the electric energy meter, preprocessing the original data set to obtain a verification sample data set, calculating key index values under different working conditions in the verification sample data set according to a determined index system, carrying out joint verification on the key index values under a plurality of working conditions by utilizing a multiple hypothesis test method, mapping a verification result into a model grade, and determining an evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model grade.

2. The method of claim 1, wherein the model accuracy-related evaluation index is a computing system for calculating an average absolute error, a maximum error, and interpretation Fang Chafen of the power meter informationized evaluation calibration model.

3. The method of claim 1, wherein the business requirement related evaluation index is a computing system for computing a detection rate, a false detection rate and an area under a subject's working characteristic curve of an informationized evaluation calibration model of the electric energy meter.

4. The method according to claim 1, wherein the establishing a key index system for evaluating the informationized evaluation calibration model of the electric energy meter based on the analytic hierarchy process according to the evaluation index related to the model accuracy and the evaluation index related to the service demand includes:

5. The method of claim 4, wherein before calculating the normalized relative importance vector of each evaluation index system to the upper layer evaluation index system according to the comparison and judgment matrix, the method further comprises:

6. The method of claim 1, wherein the obtaining the raw dataset of the informative evaluation calibration model of the electric energy meter, and the preprocessing the raw dataset to obtain the verification sample dataset, comprises:

7. The method of claim 6, wherein the key factors are determined based on a pearson correlation analysis method and a maximum information coefficient method.

8. The method of claim 1, wherein the joint verification of the key indicator value comprises:

the single hypothesis test adopts Bayesian factor test;

9. The method of claim 1, wherein mapping the validation result to a model level, and determining the evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model level comprises:

10. An evaluation system for an informationized evaluation calibration model of an electric energy meter, the system comprising:

11. The system of claim 10, wherein the model accuracy related evaluation index is a computational system for calculating an average absolute error, a maximum error, and interpretation Fang Chafen of the power meter informative evaluation calibration model.

12. The system of claim 10, wherein the business requirement related evaluation index is a computing system for computing a detection rate, a false detection rate and an area under a subject's working characteristic curve of an informationized evaluation calibration model of the electric energy meter.

13. The system according to claim 10, wherein the key index system determining unit constructs a key index system for evaluation of the informationized evaluation calibration model of the electric energy meter based on an evaluation index related to the accuracy of the model and an evaluation index related to the business requirement based on a hierarchical analysis method, comprising:

14. The system of claim 13, wherein the key metric system determination unit is further configured to: and carrying out consistency check on the comparison and judgment matrix, and calculating to obtain normalized relative importance vectors of each evaluation index system to the upper evaluation index system according to the comparison and judgment matrix after the consistency check is passed.

15. The system of claim 10, wherein the evaluation unit obtains an original data set of the electric energy meter informative evaluation calibration model, and preprocessing the original data set to obtain a verification sample data set comprises:

16. The system of claim 15, wherein the key factors are determined based on pearson correlation analysis and maximum information coefficient methods.

17. The system of claim 10, wherein the evaluation unit performs joint verification on the values of the key indicators, comprising:

the single hypothesis test adopts Bayesian factor test;

18. The system of claim 10, wherein the evaluation unit maps the verification result to a model level, and determining the evaluation result of the informationized evaluation calibration model performance of the electric energy meter according to the model level comprises:

19. A computer device, comprising:

one or more processors;

a processor for executing one or more programs;

the method of any of claims 1-9 is implemented when the one or more programs are executed by the one or more processors.

20. A computer readable storage medium, characterized in that a computer program is stored thereon, which computer program, when executed, implements the method according to any of claims 1-9.