CN111950738A - Machine learning model optimization effect evaluation method and device, terminal and storage medium - Google Patents
Machine learning model optimization effect evaluation method and device, terminal and storage medium Download PDFInfo
- Publication number
- CN111950738A CN111950738A CN202010797368.0A CN202010797368A CN111950738A CN 111950738 A CN111950738 A CN 111950738A CN 202010797368 A CN202010797368 A CN 202010797368A CN 111950738 A CN111950738 A CN 111950738A
- Authority
- CN
- China
- Prior art keywords
- machine learning
- evaluation
- learning model
- optimization
- sample data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06398—Performance of employee with respect to a job function
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Human Resources & Organizations (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Development Economics (AREA)
- Data Mining & Analysis (AREA)
- Strategic Management (AREA)
- Economics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Accounting & Taxation (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Marketing (AREA)
- Educational Administration (AREA)
- General Engineering & Computer Science (AREA)
- Entrepreneurship & Innovation (AREA)
- General Business, Economics & Management (AREA)
- Finance (AREA)
- Computing Systems (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Game Theory and Decision Science (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Technology Law (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the technical field of artificial intelligence, and provides a method, a device, a terminal and a storage medium for evaluating the optimization effect of a machine learning model, wherein the method comprises the following steps: segmenting a historical sample data set into a plurality of test sample data sets; predicting the plurality of test sample data sets by using a plurality of machine learning models to obtain a plurality of predicted values; constructing an evaluation function based on the service index and the technical index, and calculating a plurality of evaluation scores according to the plurality of predicted values and the evaluation function; constructing an assessment matrix based on the plurality of assessment scores; and evaluating the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix. The method and the device can evaluate the machine learning model subjected to repeated iterative optimization by combining the service index and the technical index.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a method, a device, a terminal and a storage medium for evaluating the optimization effect of a machine learning model.
Background
With the rapid development of machine learning, more and more business scenes are predicted by using a machine learning model. For example, the retention rate of the user is predicted by using a machine learning model in an insurance business scene.
In the prior art, a machine learning model is subjected to multiple iterative optimization and a latest optimization model is selected for prediction. However, in the process of implementing the present invention, the inventor finds that the prediction accuracy of the latest optimization model is certainly higher than that of the historical optimization model, and although the machine learning model can predict a certain sample, it may take a long time to determine whether the prediction result of the machine learning model on the sample is accurate.
Therefore, it is necessary to provide a technical solution for evaluating a machine learning model optimized by multiple iterations.
Disclosure of Invention
In view of the above, it is necessary to provide a method, an apparatus, a terminal, and a storage medium for evaluating an optimization effect of a machine learning model, in which a machine learning model optimized by multiple iterations is evaluated by combining a business index and a technical index for the first time.
The first aspect of the present invention provides a method for evaluating an optimization effect of a machine learning model, the method including:
segmenting a historical sample data set into a plurality of test sample data sets;
predicting the plurality of test sample data sets by using a plurality of machine learning models to obtain a plurality of predicted values;
constructing an evaluation function based on the service index and the technical index, and calculating a plurality of evaluation scores according to the plurality of predicted values and the evaluation function;
constructing an assessment matrix based on the plurality of assessment scores;
and evaluating the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix.
Preferably, the segmenting the history sample data set into a plurality of test sample data sets includes:
acquiring the online time of each machine learning model;
sequencing the online time, and determining the sequenced online time as a segmentation time node;
and segmenting the historical sample data set into a plurality of test sample data sets according to the segmentation time node.
Preferably, the calculating a plurality of assessment scores from the plurality of predictor values and the assessment function comprises:
obtaining a plurality of first predicted values in a plurality of predicted values corresponding to each machine learning model;
acquiring a plurality of actual values corresponding to the plurality of first predicted values;
calculating to obtain a service index according to the plurality of predicted values and the plurality of actual values;
calculating a performance index according to the plurality of predicted values;
and inputting the service index and the performance index into the evaluation function to obtain the evaluation score of each machine learning model.
Preferably, the evaluation function is: score is (a + b)/(a/AUC + b/RR), where AUC is a technical index, RR is a business index, a is a weight coefficient of the technical index, b is a weight coefficient of the business index, and a + b is 1.
Preferably, the evaluating the optimization effect value of each machine learning model according to the preset optimization effect evaluation model and the evaluation matrix includes:
for each machine learning model, obtaining evaluation scores of all other machine learning models in a column where the online time of the machine learning model is located from the evaluation matrix;
determining a maximum evaluation score of the evaluation scores of all other machine learning models;
and calculating the difference value between the evaluation score of the machine learning model and the maximum evaluation score as the optimization effect value of the machine learning model.
Preferably, after the optimization effect value of each machine learning model is evaluated according to the preset optimization effect evaluation model and the evaluation matrix, the method further includes:
selecting a machine learning model corresponding to the maximum optimization effect value as a target machine learning model;
and predicting at least one target user by using the target machine learning model and obtaining at least one prediction result.
Preferably, the method further comprises:
comparing the at least one prediction result with a plurality of preset ranges;
and writing the information of the target user corresponding to the prediction result in each preset range into the data queue corresponding to each preset range.
A second aspect of the present invention provides a machine learning model optimization effect evaluation apparatus, including:
the sample segmentation module is used for segmenting the historical sample data set into a plurality of test sample data sets;
the model prediction module is used for predicting the test sample data sets by using a plurality of machine learning models to obtain a plurality of predicted values;
the score calculation module is used for constructing an evaluation function based on the service index and the technical index and calculating a plurality of evaluation scores according to the plurality of predicted values and the evaluation function;
a matrix construction module for constructing an evaluation matrix based on the plurality of evaluation scores;
and the optimization evaluation module is used for evaluating the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix.
A third aspect of the present invention is a terminal, including:
a memory for storing a computer program;
a processor for implementing the method for evaluating the optimization effect of the machine learning model when executing the computer program.
A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the machine learning model optimization effect evaluation method.
In summary, the method, the device, the terminal and the storage medium for evaluating the optimization effect of the machine learning model according to the present invention construct an evaluation function based on the service index and the technical index, calculate a plurality of predicted values output by a plurality of machine learning models using the evaluation function to obtain an evaluation score, construct an evaluation matrix based on the evaluation score, and finally evaluate the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix, thereby achieving the effect of evaluating the machine learning models subjected to multiple iterative optimization. Based on the evaluation effect value, the machine learning model with the highest prediction accuracy can be conveniently selected from the machine learning models subjected to multiple iterative optimization, so that the machine learning model with the highest prediction accuracy is used for prediction, and the prediction accuracy can be improved.
Drawings
Fig. 1 is a flowchart of a method for evaluating an optimization effect of a machine learning model according to an embodiment of the present invention.
Fig. 2 is a structural diagram of a device for evaluating optimization effects of a machine learning model according to a second embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a terminal according to a third embodiment of the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a detailed description of the present invention will be given below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Fig. 1 is a flowchart of a method for evaluating an optimization effect of a machine learning model according to an embodiment of the present invention. The machine learning model optimization effect evaluation method is executed by a terminal, can be applied to intelligent government affairs, and promotes the construction of intelligent cities. The method for evaluating the optimization effect of the machine learning model specifically comprises the following steps, and the sequence of the steps in the flowchart can be changed and some steps can be omitted according to different requirements.
S11, the historical sample data set is segmented into a plurality of test sample data sets.
The terminal performs iterative optimization on the machine learning model periodically or aperiodically, for example, during the period from 9 months in 2019 to 2 months in 2020, the machine learning model is subjected to iterative optimization once every month, and the optimization scheme is different every month.
In the agent retention scene, because the effect improvement brought by the optimized machine learning model for predicting the current month of the user cannot be observed immediately, and the effect improvement can be observed only after the user is retained for several months, the upper limit of the prediction effect of the machine learning model optimized in each iteration is different and unknown, and the optimization performance of the machine learning model cannot be evaluated by a certain fixed reference line. The optimization performance of the machine learning model refers to the prediction accuracy of the machine learning model on the plurality of historical sample data sets.
In an optional embodiment, the segmenting the historical sample data set into a plurality of test sample data sets includes:
acquiring the online time of each machine learning model;
sequencing the online time, and determining the sequenced online time as a segmentation time node;
and segmenting the historical sample data set into a plurality of test sample data sets according to the segmentation time node.
The online time of the machine learning model is the time when the machine learning model first performs online prediction.
In order to evaluate the optimization performance of a plurality of iteratively optimized machine learning models, a terminal acquires a plurality of historical sample data sets and evaluates the optimization performance of the plurality of machine learning models based on the plurality of historical sample data sets. And each sample data in the historical sample data set corresponds to one acquisition time. The terminal sorts the sample data in the historical sample data set according to the acquisition time, and divides the historical sample data set by taking a plurality of online times of a plurality of machine learning models as division time nodes, so that a plurality of test sample data sets are obtained.
In this optional embodiment, sample data in the historical sample data set is segmented according to the online time of the machine learning model, so that each machine learning model corresponds to one test sample data set, and the test sample data set corresponding to each machine learning model is the historical sample data before the machine learning model is online and after the previous machine learning model is online, so that the machine learning model predicts the corresponding test sample data set to conform to an actual scene, and thus, the effectiveness of the evaluation on the optimization performance of the machine learning model is ensured.
And S12, predicting the test sample data sets by using a plurality of machine learning models to obtain a plurality of predicted values.
And the terminal predicts each test sample data set by using each machine learning model and acquires a plurality of predicted values of each test sample data set output by each machine learning model.
The machine learning model can only predict the test sample data set before the online time when the machine learning model is online, and in order to ensure the symmetry of data, the predicted value of the test sample data set which cannot be predicted is marked as 0.
For example, assuming that there are 3 machine learning models, each machine learning model corresponds to 1 test sample data set, and each test sample data set includes 4 test samples, each machine learning model tests each test sample data set and outputs 4 predicted values, that is, each machine learning model outputs 12 predicted values in total. Wherein the predicted value is any value between 0 and 1. Wherein, 4 predicted values output when the 1 st machine learning model predicts the 2 nd test sample data set and 4 predicted values output when the 3 rd test sample data set are recorded as 0, and 4 predicted values output when the 2 nd machine learning model predicts the 3 rd test sample data set are recorded as 0.
S13, constructing an evaluation function based on the service index and the technical index, and calculating a plurality of evaluation scores according to the plurality of predicted values and the evaluation function.
In the agent retention scene, the service index refers to retention rate, the technical index refers to Area (Area Under cut, AUC) surrounded by coordinate axes Under a Receiver Operating Characteristic Curve, and the retention rate and the AUC may have contradictions, so that an evaluation function needs to be constructed by the comprehensive predicted value and the AUC based on the service index and the technical index. Regarding the calculation process of AUC, the present invention is not described in detail herein for the prior art.
In an alternative embodiment, the evaluation function is: score ═ (a + b)/(a/AUC + b/RR).
Wherein AUC is a technical index, RR is a service index, a is a weight coefficient of the technical index, b is a weight coefficient of the service index, and a + b is 1. The terminal may initialize a-b-0.5.
In an optional embodiment, the terminal may set the sizes of a and b according to the actual demand, for example, when the actual demand is a business index in a predicted business scenario, b >0.5> a is set, and when the actual demand is only for evaluating the optimization performance of each machine learning model and performing intelligent recommendation, a >0.5> b is set.
The evaluation function combines the performance index of the machine learning model and the service index obtained by predicting the machine learning model applied in the actual service scene, so that the optimization performance of the machine learning model applied in the actual service scene can be effectively evaluated through the evaluation function.
In an optional embodiment, the calculating a plurality of assessment scores from the plurality of predictor values and the assessment function comprises:
obtaining a plurality of first predicted values in a plurality of predicted values corresponding to each machine learning model;
acquiring a plurality of actual values corresponding to the plurality of first predicted values;
calculating to obtain a service index according to the plurality of predicted values and the plurality of actual values;
calculating a performance index according to the plurality of predicted values;
and inputting the service index and the performance index into the evaluation function to obtain the evaluation score of each machine learning model.
Since a threshold is usually manually set during classification to convert the corresponding probability into a category, the threshold greatly affects the calculation of the accuracy of the machine learning model. Machine learning models are actually poor in performance, but are counterproductive in terms of accuracy. In addition, in the classification model, the prediction results are all expressed in the form of probability, and the AUC can well describe the overall performance of the model, so that the prediction result is an evaluation index for measuring the advantages and disadvantages of the two classification models and represents the probability that a positive case is arranged in front of a negative case. Therefore, the terminal adopts AUC as a performance index of the machine learning model.
S14, constructing an evaluation matrix based on the plurality of evaluation scores.
And setting an initialization matrix by the terminal, taking a machine learning model as a row key and a test sample data set as a column key, and writing predicted values corresponding to the machine learning model and the test sample data set into positions corresponding to the initialization matrix to obtain an evaluation matrix.
Illustratively, assume that the evaluation matrix is as follows:
and the horizontal axis of the evaluation matrix is the version of the machine learning model iterated every month, and the vertical axis of the evaluation matrix is the prediction sample data set of the evaluation month. Each column represents the change of the predicted effect from top to bottom if the machine learning model stops iterative optimization for a certain month. For example, if the 1908 model is online after 19 years and 9 months, the predicted effect of the 1908 model will decrease from 11.5% to 5.2%. Each row shows, from left to right, the change in the prediction effect after using the iteratively optimized machine learning model for a prediction sample data set of a month. For example, in 9 months of 19 years, the effect of the 1909 model on new line is improved from 10.5% to 10.8% compared with the 1908 model on the previous month, and the model is positively benefited by representing 1909 to carry out iterative optimization. The first column 1908 model represents the prediction of the test sample data set from 8 months-20 years 2 months of 19 years with the machine learning model on-line at 8 months of 19 years, and the optimization of the machine learning model on-line at 8 months of 19 years is evaluated with an evaluation function. It can be seen that the predicted value of the machine learning model on the line 8 months of 19 dropped from 11.5% to 5.2%. The same can be generalized to the 1909 model to the 2002 model. The current month model is the optimized effect value of the main diagonal line of the matrix.
In an optional embodiment, after said constructing an assessment matrix based on said plurality of assessment scores, said method further comprises:
and fitting an evaluation trend curve according to the evaluation matrix.
And fitting an evaluation trend curve for each machine learning model by using the segmentation time node as a horizontal axis and the plurality of evaluation scores as a vertical axis and adopting a least square function.
In the optional embodiment, the trend curve is estimated through fitting, and the change trend of the optimization performance of each machine learning model can be displayed more intuitively.
And S15, evaluating the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix.
The terminal can preset an optimization effect evaluation model for evaluating the optimization effect value of each machine learning model.
In an optional embodiment, the evaluating the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix includes:
for each machine learning model, obtaining evaluation scores of all other machine learning models in a column where the online time of the machine learning model is located from the evaluation matrix;
determining a maximum evaluation score of the evaluation scores of all other machine learning models;
and calculating the difference value between the evaluation score of the machine learning model and the maximum evaluation score as the optimization effect value of the machine learning model.
The optimization effect value is defined as the value of any major diagonal minus the maximum value of the left data. In the evaluation matrix, the (2002 model, 2002) data cell represents the machine learning model after 20 years and 2 months of optimization, and the predicted effect is obtained by predicting the test sample data set of 20 years and 2 months of optimization. The optimization effect is (2002 model, 2002) -max [ (1908 model, 2002), (1909 model, 2002), (1910 model, 2002), (1911 model, 2002), (1912 model, 2002), (2001 model, 2002) ]. The optimization effect is 0.25% by calculation, which shows that the optimization effect of the 2002 model is better.
In an optional embodiment, after the evaluating the optimization effect value of each machine learning model according to the preset optimization effect evaluation model and the evaluation matrix, the method further includes:
selecting a machine learning model corresponding to the maximum optimization effect value as a target machine learning model;
and predicting at least one target user by using the target machine learning model and obtaining at least one prediction result.
And comparing the optimization effect values corresponding to the machine learning model to determine the maximum optimization effect value. The larger the optimization effect value is, the better the performance of the corresponding machine learning model applied to the real business scene is. The smaller the optimization effect value is, the poorer the performance of the corresponding machine learning model applied to the real business scene is.
By selecting the machine learning model corresponding to the maximum optimization effect value as the target machine learning model and then predicting the target user on line by using the target machine learning model subsequently, the prediction result with high accuracy can be obtained, and the confidence coefficient of the prediction result is high.
In an optional embodiment, the method further comprises:
comparing the at least one prediction result with a plurality of preset ranges;
and writing the information of the target user corresponding to the prediction result in each preset range into the data queue corresponding to each preset range.
The terminal is pre-stored with a plurality of preset ranges, for example, a first preset range [ x1, x2], a second preset range [ x2, x3], a third preset range [ x3, x4], wherein x1< x2< x3< x 4.
And each preset range corresponds to one data queue and is used for storing the information of the user corresponding to the prediction result in the preset range. Writing the information of the target user corresponding to the prediction result in the first preset range into the first data queue, writing the information of the target user corresponding to the prediction result in the second preset range into the second data queue, and writing the information of the target user corresponding to the prediction result in the third preset range into the third data queue.
In this optional embodiment, the information of the target user is written into the corresponding data queue according to the prediction result, so that the manager can directly obtain the information from the data queue and take measures. For example, in an insurance officer retention scene, different prediction results are written into different data queues, so that a manager can conveniently and intuitively determine which insurance officers belong to insurance officers with longer retention time, the insurance officers with longer retention time are taken as key culture objects, and the insurance officers with shorter retention time are eliminated, so that the screening efficiency of the officers is improved, and the enterprise cost is saved.
The method and the device construct an evaluation function based on the service index and the technical index, calculate a plurality of predicted values output by a plurality of machine learning models by using the evaluation function to obtain an evaluation score, construct an evaluation matrix based on the evaluation score, and finally evaluate the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix, so that the effect of evaluating the machine learning models subjected to repeated iterative optimization is realized. Based on the evaluation effect value, the machine learning model with the highest prediction accuracy can be conveniently selected from the machine learning models subjected to multiple iterative optimization, so that the machine learning model with the highest prediction accuracy is used for prediction, and the prediction accuracy can be improved.
It is emphasized that the above evaluation function may be stored in a node of the blockchain in order to further ensure privacy and security of the above evaluation function.
Fig. 2 is a structural diagram of a device for evaluating optimization effects of a machine learning model according to a second embodiment of the present invention.
In some embodiments, the machine learning model optimization effectiveness evaluation apparatus 20 may include a plurality of functional modules composed of computer program segments. The computer programs of the respective program segments in the machine learning model optimization effectiveness evaluation device 20 can be stored in the memory of the terminal and executed by at least one processor to perform the functions of machine learning model optimization effectiveness evaluation (detailed in fig. 1).
In this embodiment, the machine learning model optimization effect evaluation device 20 may be divided into a plurality of functional modules according to the functions performed by the device. The functional module may include: the system comprises a sample segmentation module 201, a model prediction module 202, a score calculation module 203, a matrix construction module 204, a curve fitting module 205, an optimization evaluation module 206, a model determination module 207 and an information writing module 208. The module referred to herein is a series of computer program segments capable of being executed by at least one processor and capable of performing a fixed function and is stored in memory. In the present embodiment, the functions of the modules will be described in detail in the following embodiments.
The sample segmentation module 201 is configured to segment a history sample data set into a plurality of test sample data sets.
The terminal performs iterative optimization on the machine learning model periodically or aperiodically, for example, during the period from 9 months in 2019 to 2 months in 2020, the machine learning model is subjected to iterative optimization once every month, and the optimization scheme is different every month.
In the agent retention scene, because the effect improvement brought by the optimized machine learning model for predicting the current month of the user cannot be observed immediately, and the effect improvement can be observed only after the user is retained for several months, the upper limit of the prediction effect of the machine learning model optimized in each iteration is different and unknown, and the optimization performance of the machine learning model cannot be evaluated by a certain fixed reference line. The optimization performance of the machine learning model refers to the prediction accuracy of the machine learning model on the plurality of historical sample data sets.
In an optional embodiment, the sample segmentation module 201 segmenting the historical sample data set into a plurality of test sample data sets includes:
acquiring the online time of each machine learning model;
sequencing the online time, and determining the sequenced online time as a segmentation time node;
and segmenting the historical sample data set into a plurality of test sample data sets according to the segmentation time node.
The online time of the machine learning model is the time when the machine learning model first performs online prediction.
In order to evaluate the optimization performance of a plurality of iteratively optimized machine learning models, a terminal acquires a plurality of historical sample data sets and evaluates the optimization performance of the plurality of machine learning models based on the plurality of historical sample data sets. And each sample data in the historical sample data set corresponds to one acquisition time. The terminal sorts the sample data in the historical sample data set according to the acquisition time, and divides the historical sample data set by taking a plurality of online times of a plurality of machine learning models as division time nodes, so that a plurality of test sample data sets are obtained.
In this optional embodiment, sample data in the historical sample data set is segmented according to the online time of the machine learning model, so that each machine learning model corresponds to one test sample data set, and the test sample data set corresponding to each machine learning model is the historical sample data before the machine learning model is online and after the previous machine learning model is online, so that the machine learning model predicts the corresponding test sample data set to conform to an actual scene, and thus, the effectiveness of the evaluation on the optimization performance of the machine learning model is ensured.
The model prediction module 202 is configured to predict the multiple test sample data sets by using multiple machine learning models to obtain multiple predicted values.
And the terminal predicts each test sample data set by using each machine learning model and acquires a plurality of predicted values of each test sample data set output by each machine learning model.
The machine learning model can only predict the test sample data set before the online time when the machine learning model is online, and in order to ensure the symmetry of data, the predicted value of the test sample data set which cannot be predicted is marked as 0.
For example, assuming that there are 3 machine learning models, each machine learning model corresponds to 1 test sample data set, and each test sample data set includes 4 test samples, each machine learning model tests each test sample data set and outputs 4 predicted values, that is, each machine learning model outputs 12 predicted values in total. Wherein the predicted value is any value between 0 and 1. Wherein, 4 predicted values output when the 1 st machine learning model predicts the 2 nd test sample data set and 4 predicted values output when the 3 rd test sample data set are recorded as 0, and 4 predicted values output when the 2 nd machine learning model predicts the 3 rd test sample data set are recorded as 0.
The score calculating module 203 is configured to construct an evaluation function based on the service index and the technical index, and calculate a plurality of evaluation scores according to the plurality of predicted values and the evaluation function.
In the agent retention scene, the service index refers to retention rate, the technical index refers to Area (Area Under cut, AUC) surrounded by coordinate axes Under a Receiver Operating Characteristic Curve, and the retention rate and the AUC may have contradictions, so that an evaluation function needs to be constructed by the comprehensive predicted value and the AUC based on the service index and the technical index. Regarding the calculation process of AUC, the present invention is not described in detail herein for the prior art.
In an alternative embodiment, the evaluation function is: score ═ (a + b)/(a/AUC + b/RR).
Wherein AUC is a technical index, RR is a service index, a is a weight coefficient of the technical index, b is a weight coefficient of the service index, and a + b is 1. The terminal may initialize a-b-0.5.
In an optional embodiment, the terminal may set the sizes of a and b according to the actual demand, for example, when the actual demand is a business index in a predicted business scenario, b >0.5> a is set, and when the actual demand is only for evaluating the optimization performance of each machine learning model and performing intelligent recommendation, a >0.5> b is set.
The evaluation function combines the performance index of the machine learning model and the service index obtained by predicting the machine learning model applied in the actual service scene, so that the optimization performance of the machine learning model applied in the actual service scene can be effectively evaluated through the evaluation function.
In an alternative embodiment, the score calculation module 203 calculating a plurality of assessment scores based on the plurality of predictor values and the assessment function comprises:
obtaining a plurality of first predicted values in a plurality of predicted values corresponding to each machine learning model;
acquiring a plurality of actual values corresponding to the plurality of first predicted values;
calculating to obtain a service index according to the plurality of predicted values and the plurality of actual values;
calculating a performance index according to the plurality of predicted values;
and inputting the service index and the performance index into the evaluation function to obtain the evaluation score of each machine learning model.
Since a threshold is usually manually set during classification to convert the corresponding probability into a category, the threshold greatly affects the calculation of the accuracy of the machine learning model. Machine learning models are actually poor in performance, but are counterproductive in terms of accuracy. In addition, in the classification model, the prediction results are all expressed in the form of probability, and the AUC can well describe the overall performance of the model, so that the prediction result is an evaluation index for measuring the advantages and disadvantages of the two classification models and represents the probability that a positive case is arranged in front of a negative case. Therefore, the terminal adopts AUC as a performance index of the machine learning model.
The matrix construction module 204 is configured to construct an evaluation matrix based on the plurality of evaluation scores.
And setting an initialization matrix by the terminal, taking a machine learning model as a row key and a test sample data set as a column key, and writing predicted values corresponding to the machine learning model and the test sample data set into positions corresponding to the initialization matrix to obtain an evaluation matrix.
Illustratively, assume that the evaluation matrix is as follows:
and the horizontal axis of the evaluation matrix is the version of the machine learning model iterated every month, and the vertical axis of the evaluation matrix is the prediction sample data set of the evaluation month. Each column represents the change of the predicted effect from top to bottom if the machine learning model stops iterative optimization for a certain month. For example, if the 1908 model is online after 19 years and 9 months, the predicted effect of the 1908 model will decrease from 11.5% to 5.2%. Each row shows, from left to right, the change in the prediction effect after using the iteratively optimized machine learning model for a prediction sample data set of a month. For example, in 9 months of 19 years, the effect of the 1909 model on new line is improved from 10.5% to 10.8% compared with the 1908 model on the previous month, and the model is positively benefited by representing 1909 to carry out iterative optimization. The first column 1908 model represents the prediction of the test sample data set from 8 months-20 years 2 months of 19 years with the machine learning model on-line at 8 months of 19 years, and the optimization of the machine learning model on-line at 8 months of 19 years is evaluated with an evaluation function. It can be seen that the predicted value of the machine learning model on the line 8 months of 19 dropped from 11.5% to 5.2%. The same can be generalized to the 1909 model to the 2002 model. The current month model is the optimized effect value of the main diagonal line of the matrix.
And the curve fitting module 205 is configured to fit an evaluation trend curve according to the evaluation matrix.
And fitting an evaluation trend curve for each machine learning model by using the segmentation time node as a horizontal axis and the plurality of evaluation scores as a vertical axis and adopting a least square function.
In the optional embodiment, the trend curve is estimated through fitting, and the change trend of the optimization performance of each machine learning model can be displayed more intuitively.
The optimization evaluation module 206 is configured to evaluate an optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix.
The terminal can preset an optimization effect evaluation model for evaluating the optimization effect value of each machine learning model.
In an optional embodiment, the evaluating the optimization effect value of each machine learning model according to the preset optimization effect evaluation model and the evaluation matrix by the optimization evaluation module 206 includes:
for each machine learning model, obtaining evaluation scores of all other machine learning models in a column where the online time of the machine learning model is located from the evaluation matrix;
determining a maximum evaluation score of the evaluation scores of all other machine learning models;
and calculating the difference value between the evaluation score of the machine learning model and the maximum evaluation score as the optimization effect value of the machine learning model.
The optimization effect value is defined as the value of any major diagonal minus the maximum value of the left data. In the evaluation matrix, the (2002 model, 2002) data cell represents the machine learning model after 20 years and 2 months of optimization, and the predicted effect is obtained by predicting the test sample data set of 20 years and 2 months of optimization. The optimization effect is (2002 model, 2002) -max [ (1908 model, 2002), (1909 model, 2002), (1910 model, 2002), (1911 model, 2002), (1912 model, 2002), (2001 model, 2002) ]. The optimization effect is 0.25% by calculation, which shows that the optimization effect of the 2002 model is better.
The model determining module 207 is configured to select the machine learning model corresponding to the maximum optimization effect value as the target machine learning model after the optimization effect value of each machine learning model is estimated according to the preset optimization effect estimation model and the estimation matrix; and predicting at least one target user by using the target machine learning model and obtaining at least one prediction result.
And comparing the optimization effect values corresponding to the machine learning model to determine the maximum optimization effect value. The larger the optimization effect value is, the better the performance of the corresponding machine learning model applied to the real business scene is. The smaller the optimization effect value is, the poorer the performance of the corresponding machine learning model applied to the real business scene is.
By selecting the machine learning model corresponding to the maximum optimization effect value as the target machine learning model and then predicting the target user on line by using the target machine learning model subsequently, the prediction result with high accuracy can be obtained, and the confidence coefficient of the prediction result is high.
An information writing module 208, configured to compare the at least one prediction result with a plurality of preset ranges; and writing the information of the target user corresponding to the prediction result in each preset range into the data queue corresponding to each preset range.
The terminal is pre-stored with a plurality of preset ranges, for example, a first preset range [ x1, x2], a second preset range [ x2, x3], a third preset range [ x3, x4], wherein x1< x2< x3< x 4.
And each preset range corresponds to one data queue and is used for storing the information of the user corresponding to the prediction result in the preset range. Writing the information of the target user corresponding to the prediction result in the first preset range into the first data queue, writing the information of the target user corresponding to the prediction result in the second preset range into the second data queue, and writing the information of the target user corresponding to the prediction result in the third preset range into the third data queue.
In this optional embodiment, the information of the target user is written into the corresponding data queue according to the prediction result, so that the manager can directly obtain the information from the data queue and take measures. For example, in an insurance officer retention scene, different prediction results are written into different data queues, so that a manager can conveniently and intuitively determine which insurance officers belong to insurance officers with longer retention time, the insurance officers with longer retention time are taken as key culture objects, and the insurance officers with shorter retention time are eliminated, so that the screening efficiency of the officers is improved, and the enterprise cost is saved.
The method and the device construct an evaluation function based on the service index and the technical index, calculate a plurality of predicted values output by a plurality of machine learning models by using the evaluation function to obtain an evaluation score, construct an evaluation matrix based on the evaluation score, and finally evaluate the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix, so that the effect of evaluating the machine learning models subjected to repeated iterative optimization is realized. Based on the evaluation effect value, the machine learning model with the highest prediction accuracy can be conveniently selected from the machine learning models subjected to multiple iterative optimization, so that the machine learning model with the highest prediction accuracy is used for prediction, and the prediction accuracy can be improved.
It is emphasized that the above evaluation function may be stored in a node of the blockchain in order to further ensure privacy and security of the above evaluation function.
Fig. 3 is a schematic structural diagram of a terminal according to a third embodiment of the present invention. In the preferred embodiment of the present invention, the terminal 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.
It will be appreciated by those skilled in the art that the configuration of the terminal shown in fig. 3 is not limiting to the embodiments of the present invention, and may be a bus-type configuration or a star-type configuration, and the terminal 3 may include more or less hardware or software than those shown, or a different arrangement of components.
In some embodiments, the terminal 3 is a terminal capable of automatically performing numerical calculation and/or information processing according to preset or stored instructions, and the hardware includes but is not limited to a microprocessor, an application specific integrated circuit, a programmable gate array, a digital processor, an embedded device, and the like. The terminal 3 may further include a client device, which includes, but is not limited to, any electronic product capable of performing human-computer interaction with a client through a keyboard, a mouse, a remote controller, a touch panel, or a voice control device, for example, a personal computer, a tablet computer, a smart phone, a digital camera, and the like.
It should be noted that the terminal 3 is only an example, and other existing or future electronic products, such as those that can be adapted to the present invention, should also be included in the scope of the present invention, and are included herein by reference.
In some embodiments, the memory 31 has stored therein a computer program which, when executed by the at least one processor 32, implements all or part of the steps of the machine learning model optimization effectiveness assessment method as described. The Memory 31 includes a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an electronically Erasable rewritable Read-Only Memory (Electrically-Erasable Programmable Read-Only Memory (EEPROM)), an optical Read-Only disk (CD-ROM) or other optical disk Memory, a magnetic disk Memory, a tape Memory, or any other medium readable by a computer capable of carrying or storing data.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
In some embodiments, the at least one processor 32 is a Control Unit (Control Unit) of the terminal 3, connects various components of the entire terminal 3 by using various interfaces and lines, and executes various functions and processes data of the terminal 3 by running or executing programs or modules stored in the memory 31 and calling data stored in the memory 31. For example, the at least one processor 32, when executing the computer program stored in the memory, implements all or part of the steps of the machine learning model optimization effectiveness evaluation method described in the embodiments of the present invention; or realize all or part of the functions of the machine learning model optimization effect evaluation device. The at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips.
In some embodiments, the at least one communication bus 33 is arranged to enable connection communication between the memory 31 and the at least one processor 32 or the like.
Although not shown, the terminal 3 may further include a power supply (such as a battery) for supplying power to various components, and preferably, the power supply may be logically connected to the at least one processor 32 through a power management device, so as to implement functions of managing charging, discharging, and power consumption through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The terminal 3 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
The integrated unit implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to enable a terminal (which may be a personal computer, a terminal, or a network device) or a processor (processor) to execute parts of the methods according to the embodiments of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or that the singular does not exclude the plural. A plurality of units or means recited in the apparatus claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method for evaluating optimization effects of a machine learning model, the method comprising:
segmenting a historical sample data set into a plurality of test sample data sets;
predicting the plurality of test sample data sets by using a plurality of machine learning models to obtain a plurality of predicted values;
constructing an evaluation function based on the service index and the technical index, and calculating a plurality of evaluation scores according to the plurality of predicted values and the evaluation function;
constructing an assessment matrix based on the plurality of assessment scores;
and evaluating the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix.
2. The method of machine learning model optimization effectiveness evaluation according to claim 1, wherein said segmenting a history sample dataset into a plurality of test sample datasets comprises:
acquiring the online time of each machine learning model;
sequencing the online time, and determining the sequenced online time as a segmentation time node;
and segmenting the historical sample data set into a plurality of test sample data sets according to the segmentation time node.
3. The method of machine learning model optimization effectiveness evaluation according to claim 1, wherein the calculating a plurality of evaluation scores from the plurality of predicted values and the evaluation function comprises:
obtaining a plurality of first predicted values in a plurality of predicted values corresponding to each machine learning model;
acquiring a plurality of actual values corresponding to the plurality of first predicted values;
calculating to obtain a service index according to the plurality of predicted values and the plurality of actual values;
calculating a performance index according to the plurality of predicted values;
and inputting the service index and the performance index into the evaluation function to obtain the evaluation score of each machine learning model.
4. The method of evaluating the optimization effect of a machine learning model according to claim 1, wherein the evaluation function is: score is (a + b)/(a/AUC + b/RR), where AUC is a technical index, RR is a business index, a is a weight coefficient of the technical index, b is a weight coefficient of the business index, and a + b is 1.
5. The method for evaluating the optimization effect of machine learning models according to claim 1, wherein the evaluating the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix comprises:
for each machine learning model, obtaining evaluation scores of all other machine learning models in a column where the online time of the machine learning model is located from the evaluation matrix;
determining a maximum evaluation score of the evaluation scores of all other machine learning models;
and calculating the difference value between the evaluation score of the machine learning model and the maximum evaluation score as the optimization effect value of the machine learning model.
6. The method for evaluating the optimization effect of machine learning models according to claim 1, wherein after the evaluation of the optimization effect value of each machine learning model according to the preset optimization effect evaluation model and the evaluation matrix, the method further comprises:
selecting a machine learning model corresponding to the maximum optimization effect value as a target machine learning model;
and predicting at least one target user by using the target machine learning model and obtaining at least one prediction result.
7. The machine learning model optimization effectiveness evaluation method of claim 6, the method further comprising:
comparing the at least one prediction result with a plurality of preset ranges;
and writing the information of the target user corresponding to the prediction result in each preset range into the data queue corresponding to each preset range.
8. An apparatus for evaluating optimization effects of a machine learning model, the apparatus comprising:
the sample segmentation module is used for segmenting the historical sample data set into a plurality of test sample data sets;
the model prediction module is used for predicting the test sample data sets by using a plurality of machine learning models to obtain a plurality of predicted values;
the score calculation module is used for constructing an evaluation function based on the service index and the technical index and calculating a plurality of evaluation scores according to the plurality of predicted values and the evaluation function;
a matrix construction module for constructing an evaluation matrix based on the plurality of evaluation scores;
and the optimization evaluation module is used for evaluating the optimization effect value of each machine learning model according to a preset optimization effect evaluation model and the evaluation matrix.
9. A terminal, characterized in that the terminal comprises:
a memory for storing a computer program;
a processor for implementing the machine learning model optimization effectiveness evaluation method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the method for evaluating the optimization effect of a machine learning model according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010797368.0A CN111950738B (en) | 2020-08-10 | 2020-08-10 | Machine learning model optimization effect evaluation method, device, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010797368.0A CN111950738B (en) | 2020-08-10 | 2020-08-10 | Machine learning model optimization effect evaluation method, device, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111950738A true CN111950738A (en) | 2020-11-17 |
| CN111950738B CN111950738B (en) | 2023-09-15 |
Family
ID=73333180
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010797368.0A Active CN111950738B (en) | 2020-08-10 | 2020-08-10 | Machine learning model optimization effect evaluation method, device, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111950738B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112308466A (en) * | 2020-11-26 | 2021-02-02 | 东莞市盟大塑化科技有限公司 | Enterprise qualification auditing method and device, computer equipment and storage medium |
| CN112420197A (en) * | 2020-12-07 | 2021-02-26 | 武汉开普伟业科技有限公司 | Intelligent public health service management system and method |
| CN112685547A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Method and device for assessing dialect template, electronic equipment and storage medium |
| CN112990481A (en) * | 2021-03-12 | 2021-06-18 | 北京航空航天大学 | Automatic evaluation method for machine learning model based on block chain |
| CN113297169A (en) * | 2021-02-26 | 2021-08-24 | 阿里云计算有限公司 | Database instance processing method, system, device and storage medium |
| CN113516197A (en) * | 2021-07-27 | 2021-10-19 | 浙江大华技术股份有限公司 | Method and device for evaluating algorithm model, storage medium and electronic device |
| CN113742216A (en) * | 2021-07-26 | 2021-12-03 | 科大讯飞股份有限公司 | Method, device and storage medium for detecting efficiency of machine learning engine |
| CN114862256A (en) * | 2022-05-26 | 2022-08-05 | 华晨宝马汽车有限公司 | Factory workshop water consumption assessment method and system |
| CN114861569A (en) * | 2022-05-26 | 2022-08-05 | 中国航空工业集团公司沈阳空气动力研究所 | Pneumatic characteristic evaluation method, electronic device and storage medium |
| CN115860592A (en) * | 2023-03-03 | 2023-03-28 | 一夫科技股份有限公司 | Quality detection and evaluation method and system for gypsum building blocks |
| CN116528270A (en) * | 2023-06-27 | 2023-08-01 | 杭州电瓦特科技有限公司 | Base station energy saving potential evaluation method, device, equipment and storage medium |
| CN117725527A (en) * | 2023-12-27 | 2024-03-19 | 北京领雁科技股份有限公司 | Score model optimization method based on machine learning analysis rules |
| WO2024061252A1 (en) * | 2022-09-20 | 2024-03-28 | 维沃移动通信有限公司 | Information acquisition method and apparatus, network side device and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030212678A1 (en) * | 2002-05-10 | 2003-11-13 | Bloom Burton H. | Automated model building and evaluation for data mining system |
| CN108664605A (en) * | 2018-05-09 | 2018-10-16 | 北京三快在线科技有限公司 | A kind of model evaluation method and system |
| CN108985574A (en) * | 2018-06-23 | 2018-12-11 | 浙江工业大学 | A kind of polypropylene melt index flexible measurement method based on selective ensemble extreme learning machine |
| CN109376419A (en) * | 2018-10-16 | 2019-02-22 | 北京字节跳动网络技术有限公司 | A kind of method, apparatus of data modeling, electronic equipment and readable medium |
| CN109936582A (en) * | 2019-04-24 | 2019-06-25 | 第四范式（北京）技术有限公司 | Construct the method and device based on the PU malicious traffic stream detection model learnt |
| US20190197361A1 (en) * | 2017-12-27 | 2019-06-27 | Marlabs Innovations Private Limited | System and method for predicting and scoring a data model |
| CN110033117A (en) * | 2019-02-19 | 2019-07-19 | 阿里巴巴集团控股有限公司 | Model calibration method and device |
| CN110084374A (en) * | 2019-04-24 | 2019-08-02 | 第四范式（北京）技术有限公司 | Construct method, apparatus and prediction technique, device based on the PU model learnt |
| CN111382874A (en) * | 2018-12-28 | 2020-07-07 | 第四范式（北京）技术有限公司 | Method and device for realizing update iteration of online machine learning model |
-
2020
- 2020-08-10 CN CN202010797368.0A patent/CN111950738B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030212678A1 (en) * | 2002-05-10 | 2003-11-13 | Bloom Burton H. | Automated model building and evaluation for data mining system |
| US20190197361A1 (en) * | 2017-12-27 | 2019-06-27 | Marlabs Innovations Private Limited | System and method for predicting and scoring a data model |
| CN108664605A (en) * | 2018-05-09 | 2018-10-16 | 北京三快在线科技有限公司 | A kind of model evaluation method and system |
| CN108985574A (en) * | 2018-06-23 | 2018-12-11 | 浙江工业大学 | A kind of polypropylene melt index flexible measurement method based on selective ensemble extreme learning machine |
| CN109376419A (en) * | 2018-10-16 | 2019-02-22 | 北京字节跳动网络技术有限公司 | A kind of method, apparatus of data modeling, electronic equipment and readable medium |
| CN111382874A (en) * | 2018-12-28 | 2020-07-07 | 第四范式（北京）技术有限公司 | Method and device for realizing update iteration of online machine learning model |
| CN110033117A (en) * | 2019-02-19 | 2019-07-19 | 阿里巴巴集团控股有限公司 | Model calibration method and device |
| CN109936582A (en) * | 2019-04-24 | 2019-06-25 | 第四范式（北京）技术有限公司 | Construct the method and device based on the PU malicious traffic stream detection model learnt |
| CN110084374A (en) * | 2019-04-24 | 2019-08-02 | 第四范式（北京）技术有限公司 | Construct method, apparatus and prediction technique, device based on the PU model learnt |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112308466A (en) * | 2020-11-26 | 2021-02-02 | 东莞市盟大塑化科技有限公司 | Enterprise qualification auditing method and device, computer equipment and storage medium |
| CN112420197A (en) * | 2020-12-07 | 2021-02-26 | 武汉开普伟业科技有限公司 | Intelligent public health service management system and method |
| CN112685547A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Method and device for assessing dialect template, electronic equipment and storage medium |
| CN113297169A (en) * | 2021-02-26 | 2021-08-24 | 阿里云计算有限公司 | Database instance processing method, system, device and storage medium |
| CN112990481B (en) * | 2021-03-12 | 2022-10-28 | 北京航空航天大学 | Automatic evaluation method for machine learning model based on block chain |
| CN112990481A (en) * | 2021-03-12 | 2021-06-18 | 北京航空航天大学 | Automatic evaluation method for machine learning model based on block chain |
| CN113742216A (en) * | 2021-07-26 | 2021-12-03 | 科大讯飞股份有限公司 | Method, device and storage medium for detecting efficiency of machine learning engine |
| CN113742216B (en) * | 2021-07-26 | 2023-12-01 | 科大讯飞股份有限公司 | Method, device and storage medium for detecting efficiency of machine learning engine |
| CN113516197A (en) * | 2021-07-27 | 2021-10-19 | 浙江大华技术股份有限公司 | Method and device for evaluating algorithm model, storage medium and electronic device |
| CN114861569A (en) * | 2022-05-26 | 2022-08-05 | 中国航空工业集团公司沈阳空气动力研究所 | Pneumatic characteristic evaluation method, electronic device and storage medium |
| CN114862256A (en) * | 2022-05-26 | 2022-08-05 | 华晨宝马汽车有限公司 | Factory workshop water consumption assessment method and system |
| WO2024061252A1 (en) * | 2022-09-20 | 2024-03-28 | 维沃移动通信有限公司 | Information acquisition method and apparatus, network side device and storage medium |
| CN115860592A (en) * | 2023-03-03 | 2023-03-28 | 一夫科技股份有限公司 | Quality detection and evaluation method and system for gypsum building blocks |
| CN116528270A (en) * | 2023-06-27 | 2023-08-01 | 杭州电瓦特科技有限公司 | Base station energy saving potential evaluation method, device, equipment and storage medium |
| CN116528270B (en) * | 2023-06-27 | 2023-10-03 | 杭州电瓦特科技有限公司 | Base station energy saving potential evaluation method, device, equipment and storage medium |
| CN117725527A (en) * | 2023-12-27 | 2024-03-19 | 北京领雁科技股份有限公司 | Score model optimization method based on machine learning analysis rules |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111950738B (en) | 2023-09-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111950738A (en) | Machine learning model optimization effect evaluation method and device, terminal and storage medium | |
| CN112801718B (en) | User behavior prediction method, device, equipment and medium | |
| CN112445854A (en) | Multi-source business data real-time processing method and device, terminal and storage medium | |
| CN111694844B (en) | Enterprise operation data analysis method and device based on configuration algorithm and electronic equipment | |
| CN113592019A (en) | Fault detection method, device, equipment and medium based on multi-model fusion | |
| CN112700131B (en) | AB test method and device based on artificial intelligence, computer equipment and medium | |
| CN112396547B (en) | Course recommendation method, device, equipment and medium based on unsupervised learning | |
| CN112199417B (en) | Data processing method, device, terminal and storage medium based on artificial intelligence | |
| CN111768096A (en) | Rating method and device based on algorithm model, electronic equipment and storage medium | |
| CN112328646B (en) | Multitask course recommendation method and device, computer equipment and storage medium | |
| CN112862546B (en) | User loss prediction method and device, computer equipment and storage medium | |
| CN113435998A (en) | Loan overdue prediction method and device, electronic equipment and storage medium | |
| CN113516417A (en) | Service evaluation method and device based on intelligent modeling, electronic equipment and medium | |
| CN111860522A (en) | Identity card picture processing method and device, terminal and storage medium | |
| CN111984898A (en) | Label pushing method and device based on big data, electronic equipment and storage medium | |
| CN114219023A (en) | Data clustering method and device, electronic equipment and readable storage medium | |
| CN112598135A (en) | Model training processing method and device, computer equipment and medium | |
| CN112102011A (en) | User grade prediction method, device, terminal and medium based on artificial intelligence | |
| CN112818028B (en) | Data index screening method and device, computer equipment and storage medium | |
| CN115757075A (en) | Task abnormity detection method and device, computer equipment and storage medium | |
| CN111475541A (en) | Data decision method and device, electronic equipment and storage medium | |
| CN112365051A (en) | Agent retention prediction method and device, computer equipment and storage medium | |
| CN111539401A (en) | Lane line detection method, device, terminal and storage medium based on artificial intelligence | |
| CN115271821A (en) | Dot distribution processing method, dot distribution processing device, computer equipment and storage medium | |
| CN112330080B (en) | Factor screening method, device, equipment and medium based on connectivity graph |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |