CN118035110A - Informationized model evaluation method and system based on big data - Google Patents
Informationized model evaluation method and system based on big data Download PDFInfo
- Publication number
- CN118035110A CN118035110A CN202410247405.9A CN202410247405A CN118035110A CN 118035110 A CN118035110 A CN 118035110A CN 202410247405 A CN202410247405 A CN 202410247405A CN 118035110 A CN118035110 A CN 118035110A
- Authority
- CN
- China
- Prior art keywords
- model
- evaluation
- informationized
- data
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000011156 evaluation Methods 0.000 title claims abstract description 261
- 238000004458 analytical method Methods 0.000 claims abstract description 32
- 238000000034 method Methods 0.000 claims abstract description 12
- 238000012360 testing method Methods 0.000 claims description 32
- 238000012795 verification Methods 0.000 claims description 24
- 238000012544 monitoring process Methods 0.000 claims description 22
- 238000013210 evaluation model Methods 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 4
- 238000011157 data evaluation Methods 0.000 abstract description 2
- 238000005457 optimization Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 13
- 238000000605 extraction Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 3
- 241000533950 Leucojum Species 0.000 description 2
- 230000001174 ascending effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000008676 import Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/368—Test management for test version control, e.g. updating test cases to a new software version
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses an informationized model evaluation method and system based on big data, and relates to the technical field of model evaluation, wherein the method comprises the following steps: importing a model to be evaluated into an informationized model evaluation system, and setting an evaluation configuration item; the informationized model evaluation system generates an evaluation set according to the imported model and the evaluation configuration item; the informationized model evaluation system analyzes the imported model once by using an evaluation set to generate an evaluation index set; and carrying out secondary analysis on the imported model, and generating a model evaluation report based on the generated evaluation index set and the analysis result. Based on the data of each application scene in each field, data assurance is provided for different informationized models, data evaluation in the system is unified, indexes are generated, fairness of model evaluation is guaranteed, and meanwhile optimization references are provided.
Description
Technical Field
The invention relates to the technical field of model evaluation, in particular to an informationized model evaluation method and system based on big data.
Background
Along with the wide application of big data technology, informatization models become the mainstream of data analysis processing, for example, the current GPT and intelligent assistants of various applications are all informatization models, they can be fast to give a reply according to the requirement of the user, also have informatization models for prediction and decision, how to evaluate these models, and the technical field of evaluating the practicality and accuracy of the models is still relatively vacant, and no unified evaluation system is available for evaluating these different types of models.
Disclosure of Invention
The invention provides an informationized model evaluation method based on big data, which comprises the following steps:
Step1, importing a model to be evaluated into an informationized model evaluation system, and setting an evaluation configuration item;
step2, an informationized model evaluation system generates an evaluation set according to the imported model and an evaluation configuration item;
Step3, the informationized model evaluation system uses an evaluation set to analyze the imported model once to generate an evaluation index set;
Step4, performing secondary analysis on the imported model, and generating a model evaluation report based on the generated evaluation index set and the analysis result.
The informatization model evaluation method based on big data, as described above, wherein the configuration items in the evaluation configuration item page comprise application fields, purposes, data scenes, base models and test cases; the configuration items are selected and filled in by single selection, multiple selection and text boxes, and after configuration is completed, a configuration file is formed and shares an identification ID with the imported model.
The informationized model evaluation method based on big data, as described above, wherein an evaluation set is generated according to an imported model and an evaluation configuration item, specifically comprises the following sub-steps:
inquiring the corresponding evaluation configuration item content according to the identification ID of the imported model;
Inquiring corresponding test data according to the classification identification of the content of the evaluation configuration item;
And constructing a data scene of the evaluation model based on the test data, and arranging and outputting the data scene as an evaluation set.
The informationized model evaluation method based on big data, as described above, wherein the imported model is analyzed once by using the evaluation set to generate the evaluation index set, specifically comprises the following sub-steps:
According to the content of the input set of the test case, extracting corresponding data from the evaluation set, inputting the data into the model, and storing an output result as a verification set;
extracting corresponding data from the evaluation set according to the content of the test case output set and storing the data as a comparison set;
and determining an evaluation standard of the model according to the model application, taking data of the evaluation standard, the verification set and the comparison set as parameters of an evaluation index value calculation function, and outputting an evaluation index set.
The informationized model evaluation method based on big data, as described above, wherein the imported model is subjected to secondary analysis, and a model evaluation report is generated based on the generated evaluation index set and the analysis result, specifically comprising the following sub-steps:
analyzing various model parameters according to the characteristics of a base model adopted by the model to be tested;
The relevance of each model parameter and each evaluation index in the evaluation index set is analyzed;
and generating a model evaluation report based on the relevance of the model parameters and the evaluation indexes.
The informationized model evaluation method based on big data, as described above, wherein each model parameter is resolved according to the characteristics of a base model adopted by a model to be tested, specifically comprises the following sub-steps:
intercepting model data items around basic model parameters in an extending mode to serve as monitoring items;
Along with the input of the input set, the interception range of the monitoring item is gradually reduced until the value of the monitoring item is not changed any more, and the parameters except the basic model parameters in the monitoring item are extracted to be used as additional model parameters of the evaluation model.
The informationized model evaluation method based on big data, as described above, wherein the correlation between each model parameter and each evaluation index in the evaluation index set is analyzed, specifically comprises the following sub-steps:
in the input process of the input set, each model parameter is finely adjusted;
and comparing the changes of each evaluation index set before and after the model parameter adjustment, and analyzing the relevance of the model parameter and the evaluation index.
The invention also provides an informationized model evaluation system based on big data, which comprises the following steps: the system comprises a model importing module, an evaluation set generating module, a model analyzing module and an evaluation result displaying module;
the model importing module is used for importing a model to be evaluated into the informationized model evaluation system and setting an evaluation configuration item;
The evaluation set generation module is used for generating an evaluation set according to the imported model and the evaluation configuration item;
the model analysis module is used for analyzing the imported model by using the evaluation set to obtain various model parameters;
and the evaluation result display module is used for generating a model evaluation report based on the relevance of the model parameters and the evaluation indexes.
The informationized model evaluation system based on big data comprises a model analysis module, a data analysis module and a data analysis module, wherein the model analysis module comprises a primary analysis sub-module and a secondary analysis sub-module;
The primary analysis sub-module is used for generating an evaluation index set;
and the secondary analysis sub-module is used for analyzing the relevance between the model parameters and the evaluation indexes.
The beneficial effects achieved by the invention are as follows: based on the data of each application scene in each field, data assurance is provided for different informationized models, data evaluation in the system is unified, indexes are generated, fairness of model evaluation is guaranteed, and meanwhile optimization references are provided.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present invention, and other drawings may be obtained according to these drawings for a person having ordinary skill in the art.
Fig. 1 is a flowchart of an informationized model evaluation method based on big data according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1
As shown in fig. 1, a first embodiment of the present invention provides an informationized model evaluation method based on big data, including:
Step S10: importing a model to be evaluated into an informationized model evaluation system, and setting an evaluation configuration item;
The homepage of the informatization model evaluation system is provided with a model import frame which supports informatization models developed based on a plurality of languages such as Python, C, C #, C++, and the like, and after importing is completed, the informatization model evaluation system jumps to an evaluation configuration item page which is used for configuring details of the informatization models, wherein configuration items in the evaluation configuration item page comprise: application field, purpose, data scene, base model and test case; the configuration items are selected and filled in by single selection, multiple selection and text boxes, after configuration is completed, the configuration files and the imported models share one identification ID, and the identification ID can be an ID generated by a snowflake algorithm or an ID generated by a self-elevating sequence, so long as the unique identification can be used.
Step S20: the informationized model evaluation system generates an evaluation set according to the imported model and the evaluation configuration item;
the informationized model has no quality, only proper effects can be exerted in proper data scenes, a large amount of test data are maintained in the informationized model evaluation system to build the data scenes of the evaluated model, the data comprise data of each field and each data scene, the data are marked by different classification identifiers, the classification identifiers are consistent with the contents of each configuration item in an evaluation configuration item page, and the accuracy of the generated evaluation set is ensured, and the method comprises the following steps:
1. inquiring the corresponding evaluation configuration item content according to the identification ID of the imported model;
2. inquiring corresponding test data according to the classification identification of the content of the evaluation configuration item;
3. Constructing a data scene of an evaluation model based on the test data, and sorting and outputting the data scene as an evaluation set;
Taking a wind power prediction model as an example, extracting data types of an input set and an output set according to test cases in an evaluation configuration item, extracting corresponding data from test data according to the data types to form an evaluation set, and inquiring related data for wind power prediction according to the application field and the content of a data scene configuration item, wherein the related data comprises scene data of the weather data, wind power equipment data, wind power data and other models in normal application, and the scene data is newly added into the evaluation set.
Step S30: the informationized model evaluation system analyzes the imported model once by using an evaluation set to generate an evaluation index set;
The evaluation standards of the different models are different, the informationized model evaluation system analyzes the imported model, and a group of evaluation index sets are generated based on the evaluation standards of the model and data feedback of the evaluation sets and used for describing the model, in particular:
1. According to the content of the input set of the test case, extracting corresponding data from the evaluation set, inputting the data into the model, and storing an output result as a verification set;
The input set content extraction function is: t 1(P)＝{xi fi＝f0∧ai＝a0∧si＝s0 }, wherein P is an evaluation set, x i is the ith item of data in the evaluation set P, f i is a field code of the ith item of data in the evaluation set P, f 0 is a field code in the test case input set, a i is a field code of the ith item of data in the evaluation set P, a 0 is a field code in the test case input set, s i is a scene code of the ith item of data in the evaluation set P, and s 0 is a scene code in the test case input set;
The extracted content is input into the evaluation model to be tested as a new input set, and the output result is stored as a verification set and is expressed as Y p.
2. Extracting corresponding data from the evaluation set according to the content of the test case output set and storing the data as a comparison set;
The output set content extraction function is: t 2(P)＝{xi gi＝g0+t0\c∧fi＝fo0 }, wherein x i is the ith item of data in the evaluation set P, g i is the group code of the ith item of data in the evaluation set P, g 0 is the group code of the data items in the input set, T 0 is the timestamp of the data items in the input set, c is the prediction period of the evaluation model to be tested, and is integer division, when the evaluation model to be tested is the prediction model, the expression of T 0 \c exists, f i is the field code of the ith item of data in the evaluation set P, and fo 0 is the field code in the output set of the test case.
After the extraction is finished, storing the extract as a comparison set, and marking the comparison set as Y r;
3. determining an evaluation standard of the model according to the model application, taking data of the evaluation standard, the verification set and the comparison set as parameters of an evaluation index value calculation function, and outputting an evaluation index set;
The evaluation standard refers to the hardness performance of the model according to specific application, including real-time performance, accuracy and associative ductility, wherein the performances are described by a set of evaluation indexes, and meanwhile, the performance has a standard, the model which cannot reach the standard cannot be regarded as a qualified model naturally, and the specific analysis method is to use an evaluation index calculation function to obtain a series of index values of the evaluation indexes, and the index values are used for representing the performance of the model.
The evaluation index value calculation function is expressed as:
Wherein B is an evaluation standard set, B 1～b3 is a data item in the B set, B 0 is a standard item to be reached by an evaluation model, e 1～e5 is different evaluation indexes, a default value is 0, yp k is a kth data item in a verification set Y p, yr k is a kth data item in a comparison set Y r, k takes a value of 1-m, m is the total number of the data items in the verification set, lambda is a standard normal distribution coefficient, rt is response time of the model, and delta is a standard association sensitivity coefficient; the evaluation index value calculation function is of a double-layer verification structure, when the result of the first-layer verification b 1＝b0 is true, the second-layer verification is entered, and when the second-layer verification expression is When the result is true, a corresponding result (e 1, 1) is returned, when the second-level expressionWhen the result is true, then a corresponding result (e 2, 1) is returned, and so on.
After the calculation is completed, an evaluation index set d= { e j,vj},ej represents an evaluation index with a subscript of j, and v j represents an index value corresponding to the evaluation index with the subscript of j.
Step S40: performing secondary analysis on the imported model, and generating a model evaluation report based on the generated evaluation index set and analysis results, specifically:
1. Analyzing various model parameters according to the characteristics of a base model adopted by the model to be tested;
The characteristics of different base models are different, model parameters to be set in training are also different, the basic model parameters originally contained in various base models are maintained in an informationized model evaluation system, a plurality of informationized models are additionally added with some parameters to adapt to actual application scenes besides the original model parameters of the base models, the basic model parameters can be inquired in the system according to the content of the base model configuration items in an evaluation configuration page, and other model parameters are required to be resolved, and the resolving method comprises the following steps:
① Intercepting model data items around basic model parameters in an extending mode to serve as monitoring items;
For example, the model to be tested is a linear model, y=ax1+bx2+cx3+d, and a and d are basic model parameters, then a is taken as a base point and extends backwards until the next basic model parameter is encountered, then the next basic model parameter is taken as a d base point and extends backwards for interception until the last item of data, and the monitoring items after connection symbol interception are removed are ax1, bx2 and cx3;
② Along with the input of the input set, the interception range of the monitoring item is slowly reduced until the value of the monitoring item is not changed any more, and the parameters except the basic model parameters in the monitoring item are extracted to be used as additional model parameters of an evaluation model;
Along with the change of the input sets x 1-x 3, the values of the monitoring items ax1, bx2 and cx3 are also changed, the basic model parameters are removed, and the first monitoring item only remains x1, and the x1 item is always changed, so that the value is not the model parameter; the second term bx2 is firstly contracted by one term backwards, and only x2 and x2 are always changed, and no data item is left behind x2, so that the first term is taken for verification, b is unchanged along with the change of the input set, and b is a model parameter.
2. The relevance of each model parameter and each evaluation index in the evaluation index set is analyzed, and the analysis method comprises the following steps of;
① In the input process of the input set, each model parameter is finely adjusted;
Fine tuning refers to iterating, ascending or descending the model parameters, with each iteration step preferably being a bit smaller.
② Comparing the changes of each evaluation index set before and after the model parameter adjustment, and analyzing the relevance of the model parameter and the evaluation index;
the same group of input sets are repeatedly input along with iteration of model parameters, each evaluation index set is repeatedly calculated according to model output, only one model parameter is adjusted at a time, and then the association degree of the adjusted model parameter and each evaluation index is calculated respectively according to a gray association degree calculation formula.
3. Generating a model evaluation report based on the relevance of the model parameters and the evaluation indexes;
The evaluation report firstly displays each evaluation index, and the model parameter with the highest association degree is displayed in a prompting mode under the unqualified evaluation index for reference of the participators.
Example two
The second embodiment of the invention provides an informationized model evaluation system based on big data, which comprises: the system comprises a model importing module, an evaluation set generating module, a model analyzing module and an evaluation result displaying module;
(1) The model importing module is used for importing a model to be evaluated into the informationized model evaluation system and setting an evaluation configuration item;
The homepage of the informatization model evaluation system is provided with a model import frame which supports informatization models developed based on a plurality of languages such as Python, C, C #, C++, and the like, and after importing is completed, the informatization model evaluation system jumps to an evaluation configuration item page which is used for configuring details of the informatization models, wherein configuration items in the evaluation configuration item page comprise: application field, purpose, data scene, base model and test case; the configuration items are selected and filled in by single selection, multiple selection and text boxes, after configuration is completed, the configuration files and the imported models share one identification ID, and the identification ID can be an ID generated by a snowflake algorithm or an ID generated by a self-elevating sequence, so long as the unique identification can be used.
(2) The evaluation set generation module is used for generating an evaluation set according to the imported model and the evaluation configuration item;
the informationized model has no quality, only proper effects can be exerted in proper data scenes, a large amount of test data are maintained in the informationized model evaluation system to build the data scenes of the evaluated model, the data comprise data of each field and each data scene, the data are marked by different classification identifiers, the classification identifiers are consistent with the contents of each configuration item in an evaluation configuration item page, and the accuracy of the generated evaluation set is ensured, and the method comprises the following steps:
1. inquiring the corresponding evaluation configuration item content according to the identification ID of the imported model;
2. inquiring corresponding test data according to the classification identification of the content of the evaluation configuration item;
3. Constructing a data scene of an evaluation model based on the test data, and sorting and outputting the data scene as an evaluation set;
(3) The model analysis module is used for analyzing the imported model by using the evaluation set to obtain various model parameters, and comprises the following steps: a primary parsing sub-module and a secondary parsing sub-module;
The primary analysis sub-module is used for generating an evaluation index set;
The evaluation standards of the different models are different, the informationized model evaluation system analyzes the imported model, and a group of evaluation index sets are generated based on the evaluation standards of the model and data feedback of the evaluation sets and used for describing the model, in particular:
1. According to the content of the input set of the test case, extracting corresponding data from the evaluation set, inputting the data into the model, and storing an output result as a verification set;
The input set content extraction function is: t 1(P)＝{xi fi＝f0∧ai＝a0∧si＝s0 }, wherein P is an evaluation set, x i is the ith item of data in the evaluation set P, f i is a field code of the ith item of data in the evaluation set P, f 0 is a field code in the test case input set, a i is a field code of the ith item of data in the evaluation set P, a 0 is a field code in the test case input set, s i is a scene code of the ith item of data in the evaluation set P, and s 0 is a scene code in the test case input set;
The extracted content is input into the evaluation model to be tested as a new input set, and the output result is stored as a verification set and is expressed as Y p.
2. Extracting corresponding data from the evaluation set according to the content of the test case output set and storing the data as a comparison set;
The output set content extraction function is: t 2(P)＝{xi gi＝g0+t0\c∧fi＝fo0 }, wherein x i is the ith item of data in the evaluation set P, g i is the group code of the ith item of data in the evaluation set P, g 0 is the group code of the data items in the input set, T 0 is the timestamp of the data items in the input set, c is the prediction period of the evaluation model to be tested, and is integer division, when the evaluation model to be tested is the prediction model, the expression of T 0 \c exists, f i is the field code of the ith item of data in the evaluation set P, and fo 0 is the field code in the output set of the test case.
After the extraction is finished, storing the extract as a comparison set, and marking the comparison set as Y r;
3. determining an evaluation standard of the model according to the model application, taking data of the evaluation standard, the verification set and the comparison set as parameters of an evaluation index value calculation function, and outputting an evaluation index set;
The evaluation standard refers to the hardness performance of the model according to specific application, including real-time performance, accuracy and associative ductility, wherein the performances are described by a set of evaluation indexes, and meanwhile, the performance has a standard, the model which cannot reach the standard cannot be regarded as a qualified model naturally, and the specific analysis method is to use an evaluation index calculation function to obtain a series of index values of the evaluation indexes, and the index values are used for representing the performance of the model.
The evaluation index value calculation function is expressed as:
Wherein B is an evaluation standard set, B 1～b3 is a data item in the B set, e 1～e5 is different evaluation indexes, a default value is 0, yp k is a kth data item in a verification set Y p, yr k is a kth data item in a comparison set Y r, k takes a value of 1-m, m is the total number of the data items in the verification set, lambda is a standard normal distribution coefficient, rt is response time of a model, delta is a standard association sensitivity coefficient; the evaluation index value calculation function is of a double-layer verification structure, when the result of the first-layer verification b 1＝b0 is true, the second-layer verification is entered, and when the second-layer verification expression is When the result is true, a corresponding result (e 1, 1) is returned, when the second layer expression/>When the result is true, then a corresponding result (e 2, 1) is returned, and so on.
After the calculation is completed, an evaluation index set d= { e j,vj},ej represents an evaluation index with a subscript of j, and v j represents an index value corresponding to the evaluation index with the subscript of j.
The secondary analysis sub-module is used for analyzing the relevance between the model parameters and the evaluation indexes, and specifically:
1. Analyzing various model parameters according to the characteristics of a base model adopted by the model to be tested;
The characteristics of different base models are different, model parameters to be set in training are also different, the basic model parameters originally contained in various base models are maintained in an informationized model evaluation system, a plurality of informationized models are additionally added with some parameters to adapt to actual application scenes besides the original model parameters of the base models, the basic model parameters can be inquired in the system according to the content of the base model configuration items in an evaluation configuration page, and other model parameters are required to be resolved, and the resolving method comprises the following steps:
① Intercepting model data items around basic model parameters in an extending mode to serve as monitoring items;
For example, the model to be tested is a linear model, y=ax1+bx2+cx3+d, and a and d are basic model parameters, then a is taken as a base point and extends backwards until the next basic model parameter is encountered, then the next basic model parameter is taken as a d base point and extends backwards for interception until the last item of data, and the monitoring items after connection symbol interception are removed are ax1, bx2 and cx3;
② Along with the input of the input set, the interception range of the monitoring item is slowly reduced until the value of the monitoring item is not changed any more, and the parameters except the basic model parameters in the monitoring item are extracted to be used as additional model parameters of an evaluation model;
Along with the change of the input sets x 1-x 3, the values of the monitoring items ax1, bx2 and cx3 are also changed, the basic model parameters are removed, and the first monitoring item only remains x1, and the x1 item is always changed, so that the value is not the model parameter; the second term bx2 is firstly contracted by one term backwards, and only x2 and x2 are always changed, and no data item is left behind x2, so that the first term is taken for verification, b is unchanged along with the change of the input set, and b is a model parameter.
2. The relevance of each model parameter and each evaluation index in the evaluation index set is analyzed, and the analysis method comprises the following steps of;
① In the input process of the input set, each model parameter is finely adjusted;
Fine tuning refers to iterating, ascending or descending the model parameters, with each iteration step preferably being a bit smaller.
② Comparing the changes of each evaluation index set before and after the model parameter adjustment, and analyzing the relevance of the model parameter and the evaluation index;
repeatedly inputting the same input set along with iteration of the model parameters, repeatedly calculating each evaluation index set according to model output, only adjusting one model parameter at a time, and then respectively calculating the association degree of the adjusted model parameter and each evaluation index according to a gray association degree calculation formula;
(4) The evaluation result display module is used for generating a model evaluation report based on the relevance of the model parameters and the evaluation indexes;
The evaluation report firstly displays each evaluation index, and the model parameter with the highest association degree is displayed in a prompting mode under the unqualified evaluation index for reference of the participators.
The foregoing embodiments have been provided for the purpose of illustrating the general principles of the present invention in further detail, and are not to be construed as limiting the scope of the invention, but are merely intended to cover any modifications, equivalents, improvements, etc. based on the teachings of the invention.
Claims (9)
1. An informationized model evaluation method based on big data comprises the following steps:
Step1, importing a model to be evaluated into an informationized model evaluation system, and setting an evaluation configuration item;
step2, an informationized model evaluation system generates an evaluation set according to the imported model and an evaluation configuration item;
Step3, the informationized model evaluation system uses an evaluation set to analyze the imported model once to generate an evaluation index set;
Step4, performing secondary analysis on the imported model, and generating a model evaluation report based on the generated evaluation index set and the analysis result.
2. The informationized model evaluation method based on big data according to claim 1, wherein the configuration items in the evaluation configuration item page comprise application fields, purposes, data scenes, base models and test cases; the configuration items are selected and filled in by single selection, multiple selection and text boxes, and after configuration is completed, a configuration file is formed and shares an identification ID with the imported model.
3. The informationized model evaluation method based on big data according to claim 1, wherein the evaluation set is generated according to the imported model and the evaluation configuration item, specifically comprising the following sub-steps:
inquiring the corresponding evaluation configuration item content according to the identification ID of the imported model;
Inquiring corresponding test data according to the classification identification of the content of the evaluation configuration item;
And constructing a data scene of the evaluation model based on the test data, and arranging and outputting the data scene as an evaluation set.
4. The informationized model evaluation method based on big data according to claim 1, wherein the method comprises the following sub-steps:
According to the content of the input set of the test case, extracting corresponding data from the evaluation set, inputting the data into the model, and storing an output result as a verification set;
extracting corresponding data from the evaluation set according to the content of the test case output set and storing the data as a comparison set;
and determining an evaluation standard of the model according to the model application, taking data of the evaluation standard, the verification set and the comparison set as parameters of an evaluation index value calculation function, and outputting an evaluation index set.
5. The informationized model evaluation method based on big data according to claim 1, wherein the imported model is subjected to secondary analysis, and a model evaluation report is generated based on the generated evaluation index set and the analysis result, and specifically comprises the following sub-steps:
analyzing various model parameters according to the characteristics of a base model adopted by the model to be tested;
The relevance of each model parameter and each evaluation index in the evaluation index set is analyzed;
and generating a model evaluation report based on the relevance of the model parameters and the evaluation indexes.
6. The method for evaluating an informationized model based on big data according to claim 5, wherein each model parameter is resolved according to the characteristics of a base model adopted by the model to be tested, and specifically comprises the following sub-steps:
intercepting model data items around basic model parameters in an extending mode to serve as monitoring items;
Along with the input of the input set, the interception range of the monitoring item is continuously reduced until the value of the monitoring item is not changed any more, and the parameters except the basic model parameters in the monitoring item are extracted to be used as additional model parameters of the evaluation model.
7. The informationized model evaluation method based on big data according to claim 5, wherein the correlation between each model parameter and each evaluation index in the evaluation index set is analyzed, and the method specifically comprises the following sub-steps:
in the input process of the input set, each model parameter is finely adjusted;
and comparing the changes of each evaluation index set before and after the model parameter adjustment, and analyzing the relevance of the model parameter and the evaluation index.
8. An informationized model evaluation system based on big data, comprising: the system comprises a model importing module, an evaluation set generating module, a model analyzing module and an evaluation result displaying module;
the model importing module is used for importing a model to be evaluated into the informationized model evaluation system and setting an evaluation configuration item;
The evaluation set generation module is used for generating an evaluation set according to the imported model and the evaluation configuration item;
the model analysis module is used for analyzing the imported model by using the evaluation set to obtain various model parameters;
and the evaluation result display module is used for generating a model evaluation report based on the relevance of the model parameters and the evaluation indexes.
9. The informationized model evaluation system based on big data according to claim 8, wherein the model analysis module comprises a primary analysis sub-module and a secondary analysis sub-module;
The primary analysis sub-module is used for generating an evaluation index set;
and the secondary analysis sub-module is used for analyzing the relevance between the model parameters and the evaluation indexes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410247405.9A CN118035110B (en) | 2024-03-05 | 2024-03-05 | Informationized model evaluation method and system based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410247405.9A CN118035110B (en) | 2024-03-05 | 2024-03-05 | Informationized model evaluation method and system based on big data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118035110A true CN118035110A (en) | 2024-05-14 |
| CN118035110B CN118035110B (en) | 2024-08-06 |
Family
ID=90996695
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410247405.9A Active CN118035110B (en) | 2024-03-05 | 2024-03-05 | Informationized model evaluation method and system based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118035110B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102855586A (en) * | 2012-09-07 | 2013-01-02 | 四川壹玖壹玖企业管理连锁有限公司 | Electronic commerce order management platform based on B/S framework |
| US20200312301A1 (en) * | 2019-03-28 | 2020-10-01 | Microsoft Technology Licensing, Llc | Modular Language Model Adaptation |
| CN112416755A (en) * | 2020-11-02 | 2021-02-26 | 中关村科学城城市大脑股份有限公司 | Artificial intelligence model evaluation method and device, electronic equipment and storage medium |
| CN114492764A (en) * | 2022-02-21 | 2022-05-13 | 深圳市商汤科技有限公司 | Artificial intelligence model testing method and device, electronic equipment and storage medium |
| CN116483697A (en) * | 2023-03-29 | 2023-07-25 | 中国电力科学研究院有限公司 | Method and system for evaluating artificial intelligence model |
| CN117195505A (en) * | 2023-08-18 | 2023-12-08 | 中国电力科学研究院有限公司 | Evaluation method and system for informatization evaluation calibration model of electric energy meter |
| CN117272011A (en) * | 2023-09-28 | 2023-12-22 | 北京百度网讯科技有限公司 | Model evaluation method, device, electronic equipment and storage medium |
-
2024
- 2024-03-05 CN CN202410247405.9A patent/CN118035110B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102855586A (en) * | 2012-09-07 | 2013-01-02 | 四川壹玖壹玖企业管理连锁有限公司 | Electronic commerce order management platform based on B/S framework |
| US20200312301A1 (en) * | 2019-03-28 | 2020-10-01 | Microsoft Technology Licensing, Llc | Modular Language Model Adaptation |
| CN112416755A (en) * | 2020-11-02 | 2021-02-26 | 中关村科学城城市大脑股份有限公司 | Artificial intelligence model evaluation method and device, electronic equipment and storage medium |
| CN114492764A (en) * | 2022-02-21 | 2022-05-13 | 深圳市商汤科技有限公司 | Artificial intelligence model testing method and device, electronic equipment and storage medium |
| CN116483697A (en) * | 2023-03-29 | 2023-07-25 | 中国电力科学研究院有限公司 | Method and system for evaluating artificial intelligence model |
| CN117195505A (en) * | 2023-08-18 | 2023-12-08 | 中国电力科学研究院有限公司 | Evaluation method and system for informatization evaluation calibration model of electric energy meter |
| CN117272011A (en) * | 2023-09-28 | 2023-12-22 | 北京百度网讯科技有限公司 | Model evaluation method, device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 陈金华1;乔春霖2;周雄俊1;沈莉1;胡欣煜1: "面向大数据的基础教育信息化2.0"县·省"级动态综合测评模型建构与应用", 《电化教育研究》, vol. 44, no. 6, 30 June 2023 (2023-06-30), pages 66 - 73 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118035110B (en) | 2024-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109271975B (en) | Power quality disturbance identification method based on big data multi-feature extraction collaborative classification | |
| CN106845717B (en) | Energy efficiency evaluation method based on multi-model fusion strategy | |
| for Higgs et al. | Search for the standard model Higgs boson at LEP | |
| CN106649272A (en) | Named entity recognizing method based on mixed model | |
| CN106598950A (en) | Method for recognizing named entity based on mixing stacking model | |
| CN109063147A (en) | Online course forum content recommendation method and system based on text similarity | |
| CN104536881A (en) | Public testing error report priority sorting method based on natural language analysis | |
| CN107291840A (en) | A kind of user property forecast model construction method and device | |
| CN110310012B (en) | Data analysis method, device, equipment and computer readable storage medium | |
| Chakrabarty | A regression approach to distribution and trend analysis of quarterly foreign tourist arrivals in India | |
| CN105868534A (en) | Multi-objective optimization sampling based hydrologic model uncertainty analysis method | |
| CN112036737B (en) | Method and device for calculating regional electric quantity deviation | |
| CN113407729A (en) | Judicial-oriented personalized case recommendation method and system | |
| CN106651199A (en) | Steam pipe network scheduling rule system based on decision-making tree method | |
| Gensler et al. | An analog ensemble-based similarity search technique for solar power forecasting | |
| CN115345297A (en) | Platform area sample generation method and system based on generation countermeasure network | |
| CN108694475B (en) | Short-time-scale photovoltaic cell power generation capacity prediction method based on hybrid model | |
| CN117312578B (en) | Construction method and system of non-genetic carrier spectrum | |
| CN118035110B (en) | Informationized model evaluation method and system based on big data | |
| CN105279699A (en) | Recommendation method combining multi-class untrust relation based on collaborative filtering | |
| CN117494760A (en) | Semantic tag-rich data augmentation method based on ultra-large-scale language model | |
| CN117036781A (en) | Image classification method based on tree comprehensive diversity depth forests | |
| CN116484300A (en) | Transformer rated capacity detection method | |
| CN115526393A (en) | Construction cost prediction method based on transformer project key influence factor screening | |
| CN105183807A (en) | emotion reason event identifying method and system based on structure syntax |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |