CN110705598A - Intelligent model management method and device, computer equipment and storage medium - Google Patents
Intelligent model management method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110705598A CN110705598A CN201910841829.7A CN201910841829A CN110705598A CN 110705598 A CN110705598 A CN 110705598A CN 201910841829 A CN201910841829 A CN 201910841829A CN 110705598 A CN110705598 A CN 110705598A
- Authority
- CN
- China
- Prior art keywords
- model
- new version
- sample data
- data set
- initial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007726 management method Methods 0.000 title claims abstract description 36
- 238000012549 training Methods 0.000 claims abstract description 78
- 238000012360 testing method Methods 0.000 claims abstract description 44
- 238000000034 method Methods 0.000 claims abstract description 14
- 230000006870 function Effects 0.000 claims description 39
- 238000001514 detection method Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 7
- 238000000638 solvent extraction Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 abstract description 2
- 230000008569 process Effects 0.000 description 4
- 238000007477 logistic regression Methods 0.000 description 3
- 238000003066 decision tree Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000012417 linear regression Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Medical Informatics (AREA)
- Stored Programmes (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides an intelligent model management method, an intelligent model management device, computer equipment and a storage medium, wherein the method comprises the following steps: obtaining a current online model, and recording as a first initial model; detecting whether the accuracy of the first initial model reaches a preset accuracy threshold, if not, acquiring a current latest sample data set from a data source end, and recording the current latest sample data set as a first sample data set; dividing the first sample dataset into a first training set and a first test set; retraining the first initial model according to the first training set to generate a first new version model; and testing whether the first new version model meets the preset requirement or not according to the first test set, and if so, updating the first initial model into the first new version model. The method and the system can realize automatic monitoring and iteration of the model, reduce manual intervention operation after the model is established, and reduce the labor cost of model training and management personnel.
Description
Technical Field
The invention relates to the field of data modeling, in particular to an intelligent model management method, an intelligent model management device, computer equipment and a storage medium.
Background
The model is an important concept in machine learning, and in brief, refers to mapping from a feature space to an output space, and generally consists of assumed functions and parameters of the model. Common models in the industry include LR (Logistic Regression), GBDT (global Boosting Decision Tree, which is an iterative Decision Tree algorithm), SVM (Support Vector Machine), DNN (Deep Neural Network), and the like.
Many enterprises have a large number of models that need to be managed and provided with online services in a unified manner so that technicians can call related models to realize corresponding functions. The model involves model control and iteration in the process of providing service, and prior art needs manual intervention, not only wastes time and energy, wastes the human cost of model training and managers, and easily leads to making mistakes because of human error moreover.
Disclosure of Invention
In view of the above deficiencies of the prior art, the present invention provides an intelligent model management method, apparatus, computer device and storage medium, so as to implement automatic monitoring and iteration of a model, reduce manual intervention operations after the model creation is completed, and reduce labor costs of model training and management personnel.
In order to achieve the above object, the present invention provides an intelligent model management method, comprising the steps of:
obtaining a current online model, and recording as a first initial model;
detecting whether the accuracy of the first initial model reaches a preset accuracy threshold, if not, acquiring a current latest sample data set from a data source end, and recording the current latest sample data set as a first sample data set;
dividing the first sample dataset into a first training set and a first test set;
retraining the first initial model according to the first training set to generate a first new version model;
and testing whether the first new version model meets the preset requirement or not according to the first test set, and if so, updating the first initial model into the first new version model.
Further, the intelligent model management method further comprises the step of detecting whether the sample data set of the data source end is updated every a preset period, and if yes, executing the following steps:
obtaining a current online model, and recording as a second initial model;
acquiring a current latest sample data set from the data source end, and recording the current latest sample data set as a second sample data set;
dividing the second sample data set into a second training set and a second test set;
retraining the second initial model according to the second training set to generate a second new version model;
and testing whether the second new version model is superior to the second initial model or not according to the second test set, and if so, updating the second initial model to the second new version model.
Further, the step of retraining the first initial model according to the first training set to generate a first new version model is as follows:
and training the first initial model by using the first training set until the loss function value of the first initial model meets a preset requirement, and taking the first initial model with the loss function value meeting the preset requirement as the first new version model.
Further, the step of retraining the second initial model according to the second training set to generate a second new version model is as follows:
and training the second initial model by using the second training set until the loss function value of the second initial model meets the preset requirement, and taking the second initial model with the loss function value meeting the preset requirement as the second new version model.
Further, the sample data set of the data source end is marked with a time stamp;
the step of collecting the current latest sample data set from the data source end comprises: and acquiring a sample data set of which the marked time stamp is matched with the current time stamp from the data source end.
Further, the step of obtaining the sample data set with the timestamp matched with the current timestamp from the data source end includes:
calculating the similarity between a current timestamp and timestamps marked by all sample data sets at a data source end, and determining the timestamp with the highest similarity with the current timestamp as a target timestamp matched with the current timestamp;
and acquiring the sample data set marked as the target timestamp from the data source end.
Further, the intelligent model management method further comprises the following steps:
after the first initial model is updated to the first new version model, sending notification information that the first initial model is updated to the first new version model to a model administrator terminal and/or a model user terminal; and/or
And after the second initial model is updated to the second new version model, sending notification information that the second initial model is updated to the second new version model to a model administrator terminal and/or a model user terminal.
In order to achieve the above object, the present invention also provides an intelligent model management apparatus, including:
the first model obtaining module is used for obtaining a current online model and marking as a first initial model;
the first accuracy detection module is used for detecting whether the accuracy of the first initial model reaches a preset accuracy threshold value;
the first sample acquisition module is used for acquiring the current latest sample data set from the data source end and recording the current latest sample data set as a first sample data set when the detection result of the accuracy detection module is negative;
a first sample partitioning module for partitioning the first sample data set into a first training set and a first test set;
the first model training module is used for retraining the first initial model according to the first training set to generate a first new version model;
the first model testing module is used for testing whether the first new version model meets the preset requirement or not according to the first testing set;
and the first model updating module is used for updating the first initial model into the first new version model when the first new version model reaches a preset requirement.
Further, the intelligent model management apparatus further includes:
the sample updating detection module is used for detecting whether the sample data set of the data source end is updated or not at intervals of a preset period;
the second model obtaining module is used for obtaining a current online model and marking the current online model as a second initial model when the detection result of the sample updating detection module is positive;
the second sample acquisition module is used for acquiring the current latest sample data set from the data source end and recording the latest sample data set as a second sample data set;
a second sample dividing module, configured to divide the second sample data set into a second training set and a second test set;
the second model training module is used for retraining the second initial model according to the second training set to generate a second new version model;
the second model testing module is used for testing whether the second new version model is superior to the second initial model according to the second testing set;
and the second model updating module is used for updating the second initial model into the second new version model when the second new version model is superior to the second initial model.
Further, the first model training module is specifically configured to:
and training a first initial model by using the first training set until the loss function value of the first initial model meets a preset requirement, and taking the first initial model with the loss function value meeting the preset requirement as the first new version model.
Further, the second model training module is specifically configured to:
and training a second initial model by using the second training set until the loss function value of the second initial model meets the preset requirement, and taking the second initial model with the loss function value meeting the preset requirement as the second new version model.
Further, the sample data set of the data source end is marked with a time stamp;
and the first sample acquisition module and the second sample acquisition module acquire a sample data set with the marked timestamp matched with the current timestamp from the data source end as a current latest sample data set.
Further, the first sample acquisition module and the second sample acquisition module respectively comprise:
the target timestamp determining unit is used for calculating the similarity between the current timestamp and the timestamp marked by each sample data set at the data source end, and determining the timestamp with the highest similarity with the current timestamp as the target timestamp matched with the current timestamp;
and the latest sample data set acquisition unit is used for acquiring the sample data set marked as the target timestamp from the data source end.
Further, the intelligent model management apparatus further includes:
the first model updating notification module is used for sending notification information that the first initial model is updated to the first new version model to the model administrator terminal and/or the model user terminal after the first initial model is updated to the first new version model; and/or
And the second model updating notification module is used for sending notification information that the second initial model is updated to the second new version model to the model administrator terminal and/or the model user terminal after the second initial model is updated to the second new version model.
In order to achieve the above object, the present invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the aforementioned method when executing the computer program.
In order to achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the aforementioned method.
By adopting the technical scheme, the invention has the following effects:
1. the method automatically detects the accuracy of the current online model in real time, automatically acquires the latest sample data set to retrain the current online model to generate a first new version model if the accuracy of the current online model does not reach a preset accuracy threshold, and automatically updates the current online model to the first new version model if the generated first new version model can meet the preset requirement, thereby realizing the automatic iteration function of the model.
2. The method and the device regularly detect whether the sample data set in the data source end is updated or not, retrain the current online model by using the new sample data set to generate the second new version model, and automatically update the current online model to the second new version model if the generated second new version model is superior to the current online model, so that the regular training and updating of the model are realized, and the model updating efficiency is improved.
Drawings
FIG. 1 is a flow chart of one embodiment of an intelligent model management method of the present invention;
FIG. 2 is a block diagram of an embodiment of an intelligent model management apparatus according to the present invention;
fig. 3 is a hardware architecture diagram of the computer apparatus of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
Referring to fig. 1, a flowchart of an embodiment of an intelligent model management method of the present invention is shown. As shown in fig. 1, the intelligent model management method of the present invention includes the following steps:
and step S11, acquiring the current online model, and marking as a first initial model. The current online model refers to a model currently located at a model calling address, such as a deep convolutional neural network model, a logistic regression model, a linear regression model, a GBDT, a binary classification model, and the like. When the current online model is obtained, the model calling address is firstly searched, and then the model located at the model calling address at present is directly read to obtain the model.
In step S12, it is detected whether the accuracy of the first initial model reaches a predetermined accuracy threshold (e.g. 95%), and if not, step S13 is executed. Specifically, firstly, M predicted values obtained by predicting the first initial model and M measured values corresponding to the M predicted values one to one are collected, then the number N of the predicted values matched with the corresponding measured values is obtained, N/M is used as the accuracy of the first initial model, and finally whether the accuracy of the first initial model reaches a predetermined accuracy threshold value is judged, wherein the predetermined accuracy threshold value can be adjusted according to actual needs. Preferably, the method can show the change trend of the accuracy rate of the historical online model in a graphical mode.
Step S13, collecting the latest sample data set from the preset data source end, and recording as the first sample data set. In this embodiment, the data source end may be an HDFS (Hadoop distributed file system) end or a specified storage path, and the like, the sample data set of the data source end is marked with a time stamp, and acquiring the latest sample data set from the data source end means: and acquiring a sample data set of which the marked time stamp is matched with the current time stamp from the data source end. Specifically, firstly, calculating the similarity between a current timestamp and timestamps marked by each sample data set at a data source end, and determining the timestamp with the highest similarity with the current timestamp as a target timestamp matched with the current timestamp; and then acquiring the sample data set marked with the target timestamp from the data source end as a latest sample data set, and marking as a first sample data set.
Step S14, the first sample data set is divided into a first training set of a first proportion (e.g., 60%) and a first testing set of a second proportion (e.g., 40%).
Step S15, retraining the first initial model according to the first training set, and generating a first new version model. Specifically, the first initial model is retrained by using the first training set until the loss function value of the first initial model meets the preset requirement, and the first initial model with the loss function value meeting the preset requirement is used as a first new version model. Specifically, a loss function value of the first initial model during the K-th iterative training is calculated, then the gradient descent algorithm is used for performing the K + 1-th iterative training according to the loss function value during the K-th iterative training until the loss function value of the first initial model meets a preset requirement, and the first initial model when the loss function value meets the preset requirement is used as a first new-version model.
For example, assume that the first initial model is a linear regression model, whose expression is as follows:
the loss function of the first initial model is calculated as follows:
when the loss function value does not meet the preset requirement, the loss function is derived to obtain:
then, a Batch Gradient Descent (BGD) algorithm is adopted, the parameter weight is trained along the opposite direction of the derivative, and theta is updated according to the following formula (4):
then the theta obtained by training isj' As a new theta substitution formula (2), calculating the loss function value of the model again, if the obtained loss function value meets the preset requirement, finishing the training, and enabling the loss function value to meet the model with the preset requirementAs a first new version model. Wherein in the formulae (1) to (4), θjIs the weight, x, of the jth parameter in the first initial modeljIs the jth parameter in the first initial model, n is the total number of parameters in the first initial model, xiIs the input value, y, of the ith sample in the first training setiIs the output value of the ith sample in the first training set, and m is the number of samples in the first training set.
As can be seen from equation (4), the batch gradient descent algorithm requires all samples for each iteration, and if the number of samples m is large, the training speed is affected. Therefore, when the number of samples in the first training set is too large, the parameter weights are preferably trained by using a random gradient descent (SDG) algorithm, i.e. θ is updated according to the following equation (5):
wherein the meaning of the parameters in formula (5) is identical to the meaning of the same parameters in formula (4). As can be seen from equation (5), the stochastic gradient descent is iteratively updated once by each sample, if the sample size is large, only part of the samples may be used to already iterate θ to the optimal solution, whereas the batch gradient descent algorithm described above requires hundreds of thousands of training samples once, one iteration is not optimal, and if 10 iterations are performed, the training samples need to be traversed 10 times. The disadvantage of the stochastic gradient descent algorithm is that the SGD is more noisy than the BGD, so that the SGD is not going towards the global optimization every iteration. The accuracy is degraded although the training speed is fast.
Step S16, according to the first test set, it is tested whether the first new version model meets the preset requirement, i.e. whether each predetermined performance index of the first new version model reaches the preset index threshold, if yes, step S17 is executed. Specifically, a series of indicators of the first new version of model are first evaluated according to the first test set, for example, assuming that the first new model is a binary model, the following four indicators are usually evaluated: accuracy (Accuracy), Precision (Precision), Recall (Recall), and F1_ score (F1 score); and then, judging whether each index meets the preset requirement.
The evaluation principles of Accuracy, Precision, Recall and F1_ score are as follows: typically, with the sample of interest being a positive sample and the other samples being negative samples, the model predicts either correctly or incorrectly from the test data set. The total number of occurrences of the four cases is respectively recorded as: TP-the number of times positive samples are predicted as positive samples, FN-the number of times positive samples are predicted as negative samples, FP-the number of times negative samples are predicted as positive samples, TN-the number of times negative samples are predicted as negative samples, then: accuracy ═ TP + TN)/(TP + FP + TN + FN), representing the ratio of the number of samples correctly classified by the model to the total number of samples for a given test dataset; precision, TP/(TP + FP), representing the ratio of the number of positive samples correctly classified by the model to the total number of positive samples identified; recall is TP/(TP + FN), representing the ratio of the number of positive samples correctly classified by the model to the total number of positive samples in the test dataset; f1_ score is 2/[ (1/Precision) + (1/Recall) ], representing Precision and Recall weighted harmonic averages used to synthesize an index reflecting the model as a whole.
And after a series of indexes of the first initial model are obtained through evaluation, whether each index of the model can reach a preset index threshold value is judged. The preset index threshold value may be a fixed value set by a model administrator according to actual requirements, or may be set as an optimal index corresponding to a historical online model for adjustment.
Step S17, if each performance index of the first new-version model reaches a preset index threshold, automatically updating the current online model (i.e. the first initial model) to the first new-version model, that is, automatically taking the first initial model that does not meet the preset requirement off-line, and deploying the online first new-version model, thereby implementing the automatic iteration and intelligent deployment functions of the model. And then, sending notification information of updating the current online model into the first new version model to the model manager terminal and/or the model user terminal so as to notify the model manager and the model user that the current online model is updated into the first new version model.
In the invention, the intelligent model management method further comprises the steps of detecting whether the sample data set at the data source end is updated every a preset period (such as 1-6 weeks), and if the sample data set is updated, executing the following steps:
and step S21, acquiring the current online model, and marking as a second initial model.
And step S22, collecting the latest sample data set from the data source end and recording as a second sample data set. As described in step S13, the sample data set at the data source end is labeled with a time stamp, and this step is also implemented by acquiring the sample data set from the data source end, where the time stamp of the sample data set is matched with the current time stamp. Specifically, firstly, calculating the similarity between a current timestamp and timestamps marked by each sample data set at a data source end, and determining the timestamp with the highest similarity with the current timestamp as a target timestamp matched with the current timestamp; and then acquiring the sample data set marked with the target timestamp from the data source end as a latest sample data set, and marking as a second sample data set.
Step S23, the second sample data set is divided into a third proportion (e.g., 70%) of a second training set and a fourth proportion of a second validation set (e.g., 30%).
And step S24, retraining the second initial model according to the second training set, and generating a second new version model. Specifically, the second initial model is retrained by using the second training set until the loss function value of the second initial model meets the preset requirement, and the second initial model with the loss function value meeting the preset requirement is used as the first new version model. During training, firstly inputting a second training set into a second initial model, calculating a loss function value of the second initial model during the K-th iterative training, updating a parameter weight of the second initial model according to the loss function value obtained during the K-th iterative training by using a gradient descent algorithm, and then performing the K + 1-th iterative training until the loss function value of the second initial model meets a preset requirement, and taking the second initial model when the loss function value meets the preset requirement as a second new version model.
And step S25, testing whether the second new version model is superior to the second initial model according to the second test set, if so, executing step S26, otherwise, ending the process. Specifically, after the second new version model is trained, first, predetermined indexes of the second new version model, such as Accuracy (Accuracy), Precision (Precision), Recall (Recall), and F1_ score (F1 score), are evaluated according to the second test set, and then, whether the predetermined indexes of the second new version model are higher than corresponding indexes of the second initial model is judged, and if yes, it is determined that the second new version model is better than the second initial model, step S26 is performed.
And step S26, updating the second initial model into a second new version model, namely automatically taking the second initial model off line and deploying the second new version model with better online effect, thereby realizing the regular training and updating of the model and improving the model updating efficiency. And then, sending notification information of updating the current online model into the second new version model to a model manager terminal and/or a model user terminal so as to notify the model manager and/or the model user that the current online model is updated into the second new version model.
It should be noted that, for the sake of simplicity, the present embodiment is described as a series of acts, but those skilled in the art should understand that the present invention is not limited by the described order of acts, because some steps can be performed in other orders or simultaneously according to the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.
Example two
The present embodiment provides an intelligent model management apparatus 10, as shown in fig. 2, the apparatus 10 includes:
a first model obtaining module 111, configured to obtain a current online model, which is denoted as a first initial model;
a first accuracy detection module 112, configured to detect whether an accuracy of the first initial model reaches a predetermined accuracy threshold;
the first sample acquisition module 113 is configured to acquire a current latest sample data set from the data source end and record the current latest sample data set as a first sample data set when the detection result of the accuracy detection module is negative;
a first sample partitioning module 114 for partitioning the first sample data set into a first training set and a first test set;
a first model training module 115, configured to retrain the first initial model according to the first training set to generate a first new version model;
the first model testing module 116 is configured to test whether the first new version model meets a preset requirement according to the first test set;
the first model updating module 117 is configured to update the first initial model to the first new version model when the first new version model meets a preset requirement.
In this embodiment, the intelligent model management apparatus 10 further includes:
the sample update detection module 121 is configured to detect whether a sample data set at the data source end is updated every predetermined period;
a second model obtaining module 122, configured to obtain, when a detection result of the sample update detection module is yes, a current online model, which is recorded as a second initial model;
the second sample acquisition module 123 is configured to acquire a current latest sample data set from the data source end, and record the current latest sample data set as a second sample data set;
a second sample dividing module 124, configured to divide the second sample data set into a second training set and a second test set;
a second model training module 125, configured to retrain the second initial model according to the second training set, and generate a second new version model;
a second model testing module 126, configured to test whether the second new version model is better than the second initial model according to the second test set;
and the second model updating module 127 is used for updating the second initial model into the second new version model when the second new version model is better than the second initial model.
In this embodiment, the first model training module 115 is specifically configured to:
and training the first initial model by using the first training set until the loss function value of the first initial model meets the preset requirement, and taking the first initial model when the loss function value meets the preset requirement as a first new version model.
In this embodiment, the second model training module 125 is specifically configured to:
and training the second initial model by using the second training set until the loss function value of the second initial model meets the preset requirement, and taking the second initial model when the loss function value meets the preset requirement as a second new version model.
In this embodiment, the sample data set at the data source end is marked with a timestamp;
the first sample acquisition module 113 and the second sample acquisition module 123 acquire, from the data source, a sample data set whose timestamp matches the current timestamp as a current latest sample data set.
In this embodiment, the first sample acquiring module 113 and the second sample acquiring module 123 respectively include:
the target timestamp determining unit is used for calculating the similarity between the current timestamp and the timestamp marked by each sample data set at the data source end, and determining the timestamp with the highest similarity with the current timestamp as the target timestamp matched with the current timestamp;
and the latest sample data set acquisition unit is used for acquiring the sample data set marked as the target timestamp from the data source end.
In this embodiment, the intelligent model management apparatus 10 further includes:
the first model updating notification module is used for sending notification information that the first initial model is updated to the first new version model to the model administrator terminal and/or the model user terminal after the first initial model is updated to the first new version model; and/or
And the second model updating notification module is used for sending notification information that the second initial model is updated to the second new version model to the model administrator terminal and/or the model user terminal after the second initial model is updated to the second new version model.
It should also be understood by those skilled in the art that the embodiments described in the specification are preferred embodiments and that the modules referred to are not necessarily essential to the invention.
EXAMPLE III
The present invention also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers) capable of executing programs, and the like. The computer device 20 of the present embodiment includes at least, but is not limited to: a memory 21, a processor 22, which may be communicatively coupled to each other via a system bus, as shown in FIG. 3. It is noted that fig. 3 only shows the computer device 20 with components 21-22, but it is to be understood that not all shown components are required to be implemented, and that more or fewer components may be implemented instead.
In the present embodiment, the memory 21 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the storage 21 may be an internal storage unit of the computer device 20, such as a hard disk or a memory of the computer device 20. In other embodiments, the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device 20. Of course, the memory 21 may also include both internal and external storage devices of the computer device 20. In this embodiment, the memory 21 is generally used for storing an operating system and various application software installed in the computer device 20, such as the program codes of the intelligent model management apparatus 10 of the second embodiment. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, to execute the intelligent model management apparatus 10, so as to implement the intelligent model management method according to the first embodiment.
Example four
The present invention also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of this embodiment is used for storing the intelligent model management apparatus 10, and when being executed by a processor, the intelligent model management method of the first embodiment is implemented.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. An intelligent model management method is characterized by comprising the following steps:
obtaining a current online model, and recording as a first initial model;
detecting whether the accuracy of the first initial model reaches a preset accuracy threshold, if not, acquiring a current latest sample data set from a data source end, and recording the current latest sample data set as a first sample data set;
dividing the first sample dataset into a first training set and a first test set;
retraining the first initial model according to the first training set to generate a first new version model;
and testing whether the first new version model meets the preset requirement or not according to the first test set, and if so, updating the first initial model into the first new version model.
2. The intelligent model management method according to claim 1, further comprising detecting whether the sample data set at the data source end is updated every predetermined period, and if so, performing the following steps:
obtaining a current online model, and recording as a second initial model;
acquiring a current latest sample data set from the data source end, and recording the current latest sample data set as a second sample data set;
dividing the second sample data set into a second training set and a second test set;
retraining the second initial model according to the second training set to generate a second new version model;
and testing whether the second new version model is superior to the second initial model or not according to the second test set, and if so, updating the second initial model to the second new version model.
3. The intelligent model management method of claim 1, wherein the retraining the first initial model according to the first training set to generate a first new version of the model comprises:
and training the first initial model by using the first training set until the loss function value of the first initial model meets a preset requirement, and taking the first initial model with the loss function value meeting the preset requirement as the first new version model.
4. The intelligent model management method of claim 2, wherein the retraining of the second initial model according to the second training set and the generating of the second new version model are as follows:
and training the second initial model by using the second training set until the loss function value of the second initial model meets the preset requirement, and taking the second initial model with the loss function value meeting the preset requirement as the second new version model.
5. The intelligent model management method according to claim 1 or 2, wherein the sample data set at the data source end is labeled with a time stamp;
the step of collecting the current latest sample data set from the data source end comprises: and acquiring a sample data set of which the marked time stamp is matched with the current time stamp from the data source end.
6. The intelligent model management method of claim 5, wherein the step of obtaining from the data source end a sample data set with a timestamp matching a current timestamp comprises:
calculating the similarity between a current timestamp and timestamps marked by all sample data sets at a data source end, and determining the timestamp with the highest similarity with the current timestamp as a target timestamp matched with the current timestamp;
and acquiring the sample data set marked as the target timestamp from the data source end.
7. The intelligent model management method of claim 2, further comprising:
after the first initial model is updated to the first new version model, sending notification information that the first initial model is updated to the first new version model to a model administrator terminal and/or a model user terminal; and/or
And after the second initial model is updated to the second new version model, sending notification information that the second initial model is updated to the second new version model to a model administrator terminal and/or a model user terminal.
8. An intelligent model management apparatus, comprising:
the first model obtaining module is used for obtaining a current online model and marking as a first initial model;
the first accuracy detection module is used for detecting whether the accuracy of the first initial model reaches a preset accuracy threshold value;
the first sample acquisition module is used for acquiring the current latest sample data set from the data source end and recording the current latest sample data set as a first sample data set when the detection result of the accuracy detection module is negative;
a first sample partitioning module for partitioning the first sample data set into a first training set and a first test set;
the first model training module is used for retraining the first initial model according to the first training set to generate a first new version model;
the first model testing module is used for testing whether the first new version model meets the preset requirement or not according to the first testing set;
and the first model updating module is used for updating the first initial model into the first new version model when the first new version model reaches a preset requirement.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented by the processor when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910841829.7A CN110705598B (en) | 2019-09-06 | 2019-09-06 | Intelligent model management method, intelligent model management device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910841829.7A CN110705598B (en) | 2019-09-06 | 2019-09-06 | Intelligent model management method, intelligent model management device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110705598A true CN110705598A (en) | 2020-01-17 |
| CN110705598B CN110705598B (en) | 2024-05-28 |
Family
ID=69194358
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910841829.7A Active CN110705598B (en) | 2019-09-06 | 2019-09-06 | Intelligent model management method, intelligent model management device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110705598B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111588384A (en) * | 2020-05-27 | 2020-08-28 | 京东方科技集团股份有限公司 | Method, device and equipment for obtaining blood sugar detection result |
| CN111951012A (en) * | 2020-07-31 | 2020-11-17 | 中国工商银行股份有限公司 | Risk message identification method and system |
| CN112215357A (en) * | 2020-09-29 | 2021-01-12 | 三一专用汽车有限责任公司 | Model optimization method, device, equipment and computer readable storage medium |
| CN112508715A (en) * | 2020-11-30 | 2021-03-16 | 泰康保险集团股份有限公司 | Method and device for online deployment of insurance dual-core data model, electronic equipment and medium |
| CN112561332A (en) * | 2020-12-16 | 2021-03-26 | 北京百度网讯科技有限公司 | Model management method, model management apparatus, electronic device, storage medium, and program product |
| CN112767022A (en) * | 2021-01-13 | 2021-05-07 | 平安普惠企业管理有限公司 | Mobile application function evolution trend prediction method and device and computer equipment |
| CN113449773A (en) * | 2021-05-26 | 2021-09-28 | 北京有竹居网络技术有限公司 | Model updating method and device, storage medium and electronic equipment |
| CN113827233A (en) * | 2020-06-24 | 2021-12-24 | 京东方科技集团股份有限公司 | User characteristic value detection method and device, storage medium and electronic equipment |
| CN117314269A (en) * | 2023-09-14 | 2023-12-29 | 博瀚智能(深圳)有限公司 | Quality inspection model management method, system, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120284212A1 (en) * | 2011-05-04 | 2012-11-08 | Google Inc. | Predictive Analytical Modeling Accuracy Assessment |
| CN107316083A (en) * | 2017-07-04 | 2017-11-03 | 北京百度网讯科技有限公司 | Method and apparatus for updating deep learning model |
| CN109389030A (en) * | 2018-08-23 | 2019-02-26 | 平安科技（深圳）有限公司 | Facial feature points detection method, apparatus, computer equipment and storage medium |
| WO2019056499A1 (en) * | 2017-09-20 | 2019-03-28 | 平安科技（深圳）有限公司 | Prediction model training method, data monitoring method, apparatuses, device and medium |
-
2019
- 2019-09-06 CN CN201910841829.7A patent/CN110705598B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120284212A1 (en) * | 2011-05-04 | 2012-11-08 | Google Inc. | Predictive Analytical Modeling Accuracy Assessment |
| CN107316083A (en) * | 2017-07-04 | 2017-11-03 | 北京百度网讯科技有限公司 | Method and apparatus for updating deep learning model |
| WO2019056499A1 (en) * | 2017-09-20 | 2019-03-28 | 平安科技（深圳）有限公司 | Prediction model training method, data monitoring method, apparatuses, device and medium |
| CN109389030A (en) * | 2018-08-23 | 2019-02-26 | 平安科技（深圳）有限公司 | Facial feature points detection method, apparatus, computer equipment and storage medium |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111588384A (en) * | 2020-05-27 | 2020-08-28 | 京东方科技集团股份有限公司 | Method, device and equipment for obtaining blood sugar detection result |
| CN111588384B (en) * | 2020-05-27 | 2023-08-22 | 京东方科技集团股份有限公司 | Method, device and equipment for obtaining blood glucose detection result |
| CN113827233A (en) * | 2020-06-24 | 2021-12-24 | 京东方科技集团股份有限公司 | User characteristic value detection method and device, storage medium and electronic equipment |
| CN111951012A (en) * | 2020-07-31 | 2020-11-17 | 中国工商银行股份有限公司 | Risk message identification method and system |
| CN112215357A (en) * | 2020-09-29 | 2021-01-12 | 三一专用汽车有限责任公司 | Model optimization method, device, equipment and computer readable storage medium |
| CN112508715A (en) * | 2020-11-30 | 2021-03-16 | 泰康保险集团股份有限公司 | Method and device for online deployment of insurance dual-core data model, electronic equipment and medium |
| CN112561332A (en) * | 2020-12-16 | 2021-03-26 | 北京百度网讯科技有限公司 | Model management method, model management apparatus, electronic device, storage medium, and program product |
| CN112561332B (en) * | 2020-12-16 | 2023-07-25 | 北京百度网讯科技有限公司 | Model management method, device, electronic equipment, storage medium and program product |
| CN112767022A (en) * | 2021-01-13 | 2021-05-07 | 平安普惠企业管理有限公司 | Mobile application function evolution trend prediction method and device and computer equipment |
| CN112767022B (en) * | 2021-01-13 | 2024-02-27 | 湖南天添汇见企业管理咨询服务有限责任公司 | Mobile application function evolution trend prediction method and device and computer equipment |
| CN113449773A (en) * | 2021-05-26 | 2021-09-28 | 北京有竹居网络技术有限公司 | Model updating method and device, storage medium and electronic equipment |
| CN117314269A (en) * | 2023-09-14 | 2023-12-29 | 博瀚智能(深圳)有限公司 | Quality inspection model management method, system, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110705598B (en) | 2024-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110705598B (en) | Intelligent model management method, intelligent model management device, computer equipment and storage medium | |
| US11514347B2 (en) | Identifying and remediating system anomalies through machine learning algorithms | |
| US9129228B1 (en) | Robust and fast model fitting by adaptive sampling | |
| CN107533504A (en) | Anomaly analysis for software distribution | |
| US11157380B2 (en) | Device temperature impact management using machine learning techniques | |
| US10839308B2 (en) | Categorizing log records at run-time | |
| US9104753B2 (en) | Identifying an incident-addressing step | |
| CN113626241B (en) | Abnormality processing method, device, equipment and storage medium for application program | |
| CN109740760B (en) | Text quality inspection automatic training method, electronic device and computer equipment | |
| CN109871891B (en) | Object identification method and device and storage medium | |
| CN113505537A (en) | Building energy consumption detection method and device, computer equipment and storage medium | |
| US11343146B1 (en) | Automatically determining configuration-based issue resolutions across multiple devices using machine learning models | |
| CN113886237A (en) | Analysis report generation method and device, electronic equipment and storage medium | |
| CN117193088B (en) | Industrial equipment monitoring method and device and server | |
| CN116010187A (en) | Log detection method and related device | |
| US20210149793A1 (en) | Weighted code coverage | |
| CN114139636B (en) | Abnormal operation processing method and device | |
| US20220083320A1 (en) | Maintenance of computing devices | |
| US20230259117A1 (en) | Asset health identification from multi-modality data analysis | |
| US20230161661A1 (en) | Utilizing topology-centric monitoring to model a system and correlate low level system anomalies and high level system impacts | |
| US11188064B1 (en) | Process flow abnormality detection system and method | |
| US11586964B2 (en) | Device component management using deep learning techniques | |
| CN114610590A (en) | Method, device and equipment for determining operation time length and storage medium | |
| CN115757002A (en) | Energy consumption determination method, device and equipment and computer readable storage medium | |
| CN108897818B (en) | Method and device for determining aging state of data processing process and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |