CN114925939A - East-west calculation heat data prediction method and device and electronic equipment - Google Patents
East-west calculation heat data prediction method and device and electronic equipment Download PDFInfo
- Publication number
- CN114925939A CN114925939A CN202210845186.5A CN202210845186A CN114925939A CN 114925939 A CN114925939 A CN 114925939A CN 202210845186 A CN202210845186 A CN 202210845186A CN 114925939 A CN114925939 A CN 114925939A
- Authority
- CN
- China
- Prior art keywords
- cold data
- prediction model
- data
- cold
- east
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000004364 calculation method Methods 0.000 title description 16
- 238000012549 training Methods 0.000 claims abstract description 152
- 238000013524 data verification Methods 0.000 claims description 61
- 230000006870 function Effects 0.000 claims description 32
- 238000001914 filtration Methods 0.000 claims description 27
- 238000011156 evaluation Methods 0.000 claims description 20
- 238000004422 calculation algorithm Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 13
- 238000013502 data validation Methods 0.000 claims description 11
- 238000010801 machine learning Methods 0.000 claims description 9
- 238000010200 validation analysis Methods 0.000 claims description 2
- 238000013277 forecasting method Methods 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000005457 optimization Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000012795 verification Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 150000001875 compounds Chemical class 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000001174 ascending effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000002349 favourable effect Effects 0.000 description 2
- 238000003058 natural language processing Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G06Q50/40—
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Abstract
The invention provides a method, a device and electronic equipment for predicting east-west arithmetic thermal data, which relate to the technical field of east-west arithmetic and big data and are applied to east-west arithmetic center, wherein the method comprises the following steps: receiving a first business prediction model sent by a western computing center; the first business prediction model is obtained by training based on the filtered cold data sample; inputting the thermal data to be predicted into a first business prediction model for prediction, and obtaining a prediction result corresponding to the thermal data to be predicted. According to the method, the east computing power center receives the business prediction model sent by the west computing power center, and the business prediction model is obtained based on the cold data training with large deviation in the filtered cold data samples, so that the good applicability of the business prediction model on the hot data is realized.
Description
Technical Field
The invention relates to the technical field of east-west arithmetic and big data, in particular to a method and a device for predicting east-west arithmetic heat data and electronic equipment.
Background
With the formal start of the nationwide integrated computing power network system, the digital transformation cost and energy consumption are urgently needed to be reduced by means of 'east, west and west computation'. Under the scene of 'east-west calculation', the east calculation center classifies mass data generated by the Internet of things, artificial intelligence and the like into cold data and hot data according to the use frequency, time and the like, and transmits the cold data to the west data center in an incremental manner for storage. In the requirements of artificial intelligence and data mining, firstly, analyzing and mining accumulated historical sample cold data in a western power computing center to generate a model training result, then deploying the model to an east power computing center, and supporting the online application of the model in the east power computing center.
However, in many scenarios, the distribution of cold data samples is significantly different from the distribution of hot data samples, which results in a great degradation in the effect of applying the model obtained by training cold data in western power center to hot data in east.
Currently, solutions to the above problems include: 1. horizontal federal learning: a federal learning platform is constructed between the east computational power center and the west computational power center, and cold and hot data are jointly participated in model construction through a transverse federal learning mode; 2. pre-training-fine tuning scheme: and a neural network is constructed by utilizing a large number of cold data samples in the western force computing center, the model is sent to the east force computing center, and the east force computing center finely adjusts the neural network according to the thermal data.
However, the above solutions all have certain limitations, in the solution 1, the latest hot data sample characteristics of the eastern force computing center are considered in the horizontal federal study, but the western force computing center has "confrontation samples", which results in the overall effect of the model being reduced; in the scheme 2, on one hand, most of the pre-training-fine tuning architectures are applicable to neural network models and are not applicable to other relevant machine learning model scenes, and on the other hand, the problem of model applicability still exists under the condition that the sample distribution deviation is large.
Disclosure of Invention
The invention provides a method and a device for predicting east-west calculation heat data and electronic equipment, which are used for solving the defect of poor applicability of a model trained by a Chinese-western calculation force center by cold data in prediction of the east-west calculation force center on heat data in the prior art.
In a first aspect, the present invention provides a method for predicting eastern and western arithmetic thermal data, which is applied to eastern department arithmetic center, and includes:
receiving a first business prediction model sent by a western computing center; the first business prediction model is obtained by training based on the filtered cold data sample;
inputting the thermal data to be predicted into a first business prediction model for prediction, and obtaining a prediction result corresponding to the thermal data to be predicted.
Optionally, the receiving a first traffic prediction model sent by the western power center includes:
receiving a second service prediction model and a first AUC score sent by a western computing center; the second business prediction model is obtained by training a first cold data training set filtered based on a first preset threshold; the first AUC score is obtained by verifying the second service prediction model by using the sampled first cold data verification set;
predicting the thermal data samples in the thermal data application set based on the second service prediction model to obtain a second AUC score;
constructing an objective function based on the first AUC score and the second AUC score;
under the condition that the value of the objective function is minimum, receiving a first business prediction model which is sent by a western computing force center and obtained by training a second cold data training set and a second cold data verification set; the second cold data training set and the second cold data validation set are obtained by filtering the first cold data training set and the first cold data validation set based on a second preset threshold corresponding to the minimum value of the objective function.
Optionally, before receiving the second traffic prediction model and the first AUC score sent by the western computing center, the method further includes:
carrying out sample similarity training on the thermal data samples based on an LGB algorithm to obtain a sample similarity model;
sending the sample similarity model to a western force center; the sample similarity model is used for carrying out similarity evaluation on the cold data samples in the first cold data training set and the first cold data verification set to obtain a similarity score corresponding to each cold data sample.
In a second aspect, the present invention further provides a method for predicting eastern and western arithmetic thermal data, which is applied to western arithmetic center, and includes:
sending a first business prediction model to an east force computing center; the first business prediction model is obtained by training based on the filtered cold data samples.
Optionally, before the sending the first business prediction model to the east computing center, the method further includes:
receiving a sample similarity model sent by the east force computing center;
performing similarity evaluation on cold data samples in the first cold data training set and the first cold data verification set based on the sample similarity model to obtain a similarity score corresponding to each cold data sample;
and respectively filtering the cold data samples in the first cold data training set and the first cold data verification set based on a first preset threshold and the similarity score to obtain a third cold data training set and a third cold data verification set.
Optionally, after the obtaining the third cold data training set and the third cold data verification set, the method further includes:
training cold data samples in the third cold data training set by using a machine learning algorithm to obtain a second service prediction model;
verifying the second service prediction model by using the sampled first cold data verification set to obtain a first AUC score of the second service prediction model;
and sending the second service prediction model and the first AUC score to a western computing power center.
Optionally, after the sending the second traffic prediction model and the first AUC score to the western force center, the method further includes:
adjusting the first preset threshold value by adopting grid search to obtain a second preset threshold value; the second preset threshold corresponds to the minimum value of the target function; the objective function is constructed based on the first and second AUC scores; the second AUC score is obtained by predicting the second service prediction model by using the thermal data application set;
respectively filtering cold data samples in the first cold data training set and the first cold data verification set based on a second preset threshold value to obtain a second cold data training set and a second cold data verification set;
and training the second cold data training set and the second cold data verification set to obtain the first business prediction model.
In a third aspect, the present invention further provides an eastern and western arithmetic thermal data prediction apparatus, including:
the first receiving module is used for receiving a first business prediction model sent by a western computing center; the first business prediction model is obtained based on the filtered cold data sample training;
the first obtaining module is used for inputting the thermal data to be predicted into a first business prediction model for prediction, and obtaining a prediction result corresponding to the thermal data to be predicted.
In a fourth aspect, the present invention further provides an eastern and western arithmetic thermal data prediction apparatus, including:
the first sending module is used for sending the first business prediction model to the east computing center; the first business prediction model is obtained by training based on the filtered cold data samples.
In a fifth aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the east west arithmetic thermal data prediction method of the first or second aspect as described above when executing the computer program.
In a sixth aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the east-west arithmetic thermal data prediction method of the first or second aspect as described in any of the above.
In a seventh aspect, the present invention also provides a computer program product comprising a computer program which, when executed by a processor, implements the east-west computation thermal data prediction method of any of the first or second aspects as described above.
According to the east-west calculation heat data prediction method, the east-west calculation heat data prediction device and the electronic equipment, the east-west calculation force center receives the business prediction model sent by the west calculation force center, and the business prediction model is obtained by training cold data with large deviation in cold data samples based on filtering, so that the good applicability of the business prediction model on heat data is realized.
Drawings
In order to more clearly illustrate the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart of a method for predicting eastern and western arithmetic thermal data according to an embodiment of the present invention;
FIG. 2 is a second flowchart of a method for predicting eastern and western arithmetic thermal data according to an embodiment of the present invention;
FIG. 3 is a third flowchart illustrating a method for predicting eastern and western arithmetic thermal data according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of an east-west arithmetic thermal data prediction apparatus according to an embodiment of the present invention;
FIG. 5 is a second schematic diagram of a east-west arithmetic thermal data prediction apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art based on the embodiments of the present invention without inventive step, are within the scope of the present invention.
Fig. 1 is a schematic flow chart of a method for predicting eastern and western arithmetic and thermal data according to an embodiment of the present invention, and as shown in fig. 1, the present invention provides a method for predicting eastern and western arithmetic and thermal data, which is applied to an eastern arithmetic and power center, and includes:
Specifically, the eastern force computing center receives a first business prediction model sent by the western force computing center. The first business prediction model is obtained by the western computing power center through conducting business prediction training on the filtered cold data samples by using a machine learning algorithm, and the first business prediction model is a final model output by the western computing power center.
The cold data sample is composed of a first cold data training set and a first cold data validation set. The filtered cold data sample is obtained by the western force center after cold data filtering is performed on the first cold data training set and the first cold data verification set based on the determined optimal preset threshold.
Step 102, inputting thermal data to be predicted into a first business prediction model for prediction, and obtaining a prediction result corresponding to the thermal data to be predicted.
Specifically, after the east computing center receives a first Service prediction model, the east computing center solidifies a received first Service prediction model file, deploys the model in a function as a Service (FaaS) form or a batch prediction form, performs Service prediction on thermal data to be predicted by using the first Service prediction model after deployment, obtains a prediction result corresponding to the thermal data to be predicted, and stores the prediction result in a database for real-time calling.
According to the east-west calculation heat data prediction method provided by the embodiment of the invention, the east-west calculation center receives the business prediction model sent by the west calculation center, and the business prediction model is obtained by training cold data with large deviation in cold data samples through filtering, so that the good applicability of the business prediction model on heat data is realized.
Fig. 2 is a second flow diagram of the east-west computation thermal data prediction method according to the embodiment of the present invention, and as shown in fig. 2, before the thermal data is predicted by using the first business prediction model, the east computation center performs data analysis and sample similarity model training, evaluates the initial model output by the west computation center, that is, the second business prediction model, and constructs the objective function based on the result of the model evaluation.
Optionally, before receiving the second traffic prediction model and the first AUC score sent by the western computing center, the method further includes:
carrying out sample similarity training on the thermal data samples based on an LGB algorithm to obtain a sample similarity model;
sending the sample similarity model to a western computing power center; the sample similarity model is used for carrying out similarity evaluation on the cold data samples in the first cold data training set and the first cold data verification set to obtain a similarity score corresponding to each cold data sample.
Specifically, the eastern force computing center analyzes the distribution of the thermal data samples, obtains the proportion of the positive sample thermal data and the negative sample thermal data in the thermal data samples, and records the proportion as r.
The east computing center performs baseline model training, namely sample similarity training, on small batches of thermal data samples by adopting an LGB algorithm, and the baseline model aims to measure the similarity between the samples and obtain a trained baseline model, namely a sample similarity model.
The complexity of the baseline model is low, and in the LGB algorithm, the low complexity is reflected in the small depth of the related tree and the small data of the leaf node.
The eastern force computing center sends the sample similarity model to the western force computing center. The sample similarity model is used for the western force center to carry out similarity evaluation on the cold data samples in the first cold data training set and the first cold data verification set, so that a similarity evaluation result of each cold data sample is obtained, and the similarity evaluation result is quantized, so that a similarity score corresponding to each cold data sample is obtained. The similarity score was noted as f (x), where x represents a sample of cold data.
According to the east-west calculation hot data prediction method provided by the embodiment of the invention, the sample similarity model is trained on the basis of the hot data sample, and then the sample similarity model is sent to the west calculation center, so that the west calculation center is favorable for evaluating the similarity of the cold data sample, and the west calculation center is further favorable for filtering the cold data sample.
Optionally, the receiving a first traffic prediction model sent by the western computing center includes:
receiving a second service prediction model and a first AUC score sent by a western computing center; the second business prediction model is obtained by training a first cold data training set filtered based on a first preset threshold; the first AUC score is obtained by verifying the second service prediction model by using the sampled first cold data verification set;
predicting the thermal data samples in the thermal data application set based on the second service prediction model to obtain a second AUC score;
constructing an objective function based on the first AUC score and the second AUC score;
under the condition that the value of the objective function is minimum, receiving a first business prediction model which is sent by a western computing force center and obtained by training a second cold data training set and a second cold data verification set; the second cold data training set and the second cold data validation set are obtained by filtering the first cold data training set and the first cold data validation set based on a second preset threshold corresponding to the minimum value of the objective function.
Specifically, the eastern force computing center receives the second service prediction model and the first AUC score sent by the western force computing center.
The second service prediction model is obtained through the following process: a preset threshold value a is set in the western force calculation center, the value range of a is 0-1, and the initial value of the preset threshold value is a first preset threshold value; performing cold data filtering on the first cold data training set based on a first preset threshold and the similarity score to obtain a third cold data training set; and training the service prediction model for the cold data samples in the third cold data training set by adopting a machine learning algorithm, and carrying out internal tuning on algorithm-related parameters to obtain a second service prediction model.
The first model evaluation index (AUC) score is obtained by the western force center verifying the second service prediction model by using the sampled cold data samples in the first cold data verification set, obtaining a verification result, and performing model evaluation on the second service prediction model based on the verification result. And the east force computing center predicts the thermal data samples in the thermal data application set by using the second business prediction model to obtain a prediction result, evaluates the second business prediction model based on the prediction result and obtains a second AUC score of the second business prediction model.
The eastern computational force center constructs an objective function based on the first AUC score and the second AUC score. The expression of the objective function is as follows:
in the formula (I), the compound is shown in the specification,
the representation of the objective function is shown as,
representing the AUC scores obtained when the business prediction model predicts the application set of thermal data,
represents the regularization weight coefficients and,
the value range of (A) is 0 to 0.1,
representing the AUC scores obtained when the sampled first cold data validation set validates the traffic prediction model,
for evaluating the performance of the business prediction model on thermal data,
the method is used for evaluating the stability condition of the business prediction model in the cold and hot data.
When the preset threshold value a is a first preset threshold value, the obtained service prediction model is a second service prediction model,
in order to score the second AUC,
scoring the first AUC, the objective function
The result is obtained by the operation of the first AUC score and the second AUC score. Thus, the objective function is known
As the preset threshold a changes.
And adjusting a preset threshold value a by the western computational force center, and taking the corresponding preset threshold value when the objective function value is minimum as an optimal preset threshold value, namely a second preset threshold value. And after the optimal preset threshold value, namely a second preset threshold value, filtering the cold data of the non-positive and negative samples by utilizing the second preset threshold value to the first cold data training set and the first cold data verification set respectively to obtain filtered cold data of the positive and negative samples, and acquiring a second cold data training set and a second cold data verification set. And splicing the second cold data training set and the second cold data verification set, and training a business prediction model based on the spliced cold data sample set to obtain a final business prediction model, namely the first business prediction model.
And the east force computing center receives the first business prediction model sent by the west force computing center.
According to the east-west calculation thermal data prediction method provided by the embodiment of the invention, the automatic optimization and stable promotion of the model are realized by the hyper-parameterization of the preset threshold and the regularization of the model effect difference.
Fig. 3 is a third schematic flow chart of a method for predicting eastern and western arithmetic thermal data according to an embodiment of the present invention, and as shown in fig. 3, the present invention provides a method for predicting eastern and western arithmetic thermal data, which is applied to a western arithmetic center, and includes:
Specifically, the cold data sample consists of a first training set of cold data and a first validation set of cold data. The western force computing center performs cold data filtering on the first cold data training set and the first cold data verification set based on the optimal preset threshold value to obtain the filtered first cold data training set and the filtered first cold data verification set, namely to obtain the filtered cold data sample.
And the western computing power center performs service prediction training on the filtered cold data sample by using a machine learning algorithm to obtain a first service prediction model. The first business prediction model is the final model of the western force center output.
And the western force computing center sends the obtained first business prediction model to the east force computing center.
According to the east-west calculation heat data prediction method provided by the embodiment of the invention, the west calculation center filters the cold data samples to filter out the cold data with larger sample deviation, and then obtains the business prediction model based on the filtered cold data samples, and sends the business prediction model to the east calculation center, so that the good applicability of the business prediction model on heat data is realized.
Optionally, before sending the first business prediction model to the eastern computational force center, the method further includes:
receiving a sample similarity model sent by the east computational force center;
performing similarity evaluation on cold data samples in the first cold data training set and the first cold data verification set based on the sample similarity model to obtain a similarity score corresponding to each cold data sample;
and respectively filtering the cold data samples in the first cold data training set and the first cold data verification set based on a first preset threshold and the similarity score to obtain a third cold data training set and a third cold data verification set.
Specifically, the western force computing center first divides the cold data sample set into a first cold data training set and a first cold data verification set according to a preset proportion. The preset ratio may be 8: 2.
The western force computing center receives the sample similarity model sent by the eastern force computing center, similarity evaluation is conducted on cold data samples in the first cold data training set and the first cold data verification set through the sample similarity model, a similarity evaluation result of each cold data sample is obtained, the similarity evaluation result is quantized, and therefore a similarity score corresponding to each cold data sample is obtained. The similarity score was noted as f (x), where x represents a cold data sample.
After obtaining the similarity score corresponding to one cold data sample, the western computing center sets a preset threshold a, wherein the value range of a is 0-1, and the initial value of the preset threshold is a first preset threshold.
And the western force computing center performs cold data sample filtering on the first cold data training set based on a first preset threshold and the similarity score to obtain a third cold data training set.
The specific process of the first cold data training set for cold data sample filtering is as follows:
in the first case, if the ratio r of the positive sample hot data and the negative sample hot data in the hot data sample is greater than 1, the expression of the positive sample cold data in the third cold data training set is as follows:
wherein the content of the first and second substances,
representing positive sample cold data in a third training set of cold data,
indicating that the cold data samples in the first cold data training set are scored according to similarity
And (5) performing descending order, and taking the first n cold data samples as the cold data of the positive samples in the third cold data training set.
Wherein the expression of n is as follows:
wherein n represents the number of the cold data of the positive sample in the third cold data training set, a represents a preset threshold, the value range of a is 0-1,
representing similarity scores in a first Cold data training set
The number of (2).
The expression for the negative sample cold data in the third cold data training set is as follows:
wherein, the first and the second end of the pipe are connected with each other,
negative sample cold data in the third cold data training set,
indicating that the cold data samples in the first cold data training set are scored according to similarity
And (5) performing ascending arrangement, and taking the first m cold data samples as negative sample cold data in a third cold data training set.
Wherein, the first and the second end of the pipe are connected with each other,
the expression of (a) is as follows:
in the formula (I), the compound is shown in the specification,
the number of the negative sample cold data in the third cold data training set is represented, a represents a preset threshold value, the value range of a is 0-1,
representing similarity scores in a first Cold data training set
R represents the ratio of the positive sample thermal data to the negative sample thermal data in the thermal data samples.
In the second case, if the ratio r of the positive sample hot data and the negative sample hot data in the hot data sample is smaller than 1, the expression of the positive sample cold data in the third cold data training set is as follows:
in the formula (I), the compound is shown in the specification,
representing positive sample cold data in a third training set of cold data,
indicating that the cold data samples in the first cold data training set are scored according to similarity
And (5) performing descending order arrangement, and taking the first n cold data samples as the positive sample cold data in the third cold data training set.
Wherein the expression of n is as follows:
in the formula, n represents the number of the positive sample cold data in the third cold data training set, a represents a preset threshold, the value range of a is 0-1,
representing similarity scores in a first Cold data training set
R represents the ratio of the positive sample thermal data to the negative sample thermal data in the thermal data samples.
The expression for negative sample cold data in the third training set of cold data is as follows:
wherein, the first and the second end of the pipe are connected with each other,
negative sample cold data in the third cold data training set,
indicating that the cold data samples in the first cold data training set are scored according to similarity
And (5) performing ascending arrangement, and taking the first m cold data samples as negative sample cold data in a third cold data training set.
Wherein the content of the first and second substances,
the expression of (a) is as follows:
And the western computational force center performs cold data sample filtering on the first cold data verification set based on a first preset threshold and the similarity score to obtain a third cold data verification set. The specific process of the first cold data validation set for filtering the cold data samples is the same as the specific process of the first cold data training set for filtering the cold data samples, and is not described herein again.
According to the east-west heat data prediction method provided by the embodiment of the invention, the west force computing center performs similarity evaluation on cold data samples by using the sample similarity model sent by the east force computing center, the cold data samples are screened based on the similarity score and the preset threshold, and the data volume pressure of the western force computing center model training is further reduced due to the reduction of the sample data volume.
Optionally, after the acquiring the third cold data training set and the third cold data verification set, the method further includes:
training cold data samples in the third cold data training set by using a machine learning algorithm to obtain a second service prediction model;
verifying the second service prediction model by using the sampled first cold data verification set to obtain a first AUC score of the second service prediction model;
and sending the second service prediction model and the first AUC score to a western force center.
Specifically, a western computing power center sampling machine learning algorithm, such as an LGB algorithm, an XGB algorithm, or a Natural Language Processing (NLP) algorithm, trains a traffic prediction model for the cold data samples in the obtained third cold data training set, and performs internal tuning on algorithm-related parameters to obtain a second traffic prediction model.
The western force center samples the first cold data verification set in an undersampling or oversampling manner, so that the proportion of the positive sample cold data and the negative sample cold data in the sampled first cold data verification set is the same as the proportion of the positive sample hot data and the negative sample hot data in the hot data sample, that is, the proportion of the positive sample cold data and the negative sample cold data in the sampled first cold data verification set is also r.
The western force center verifies the second business prediction model by using the cold data samples in the sampled first cold data verification set to obtain a verification result, evaluates the second business prediction model based on the verification result, and obtains a first AUC score of the second business prediction model.
And the western computational force center sends the second business prediction model and the first AUC score to the eastern computational force center.
According to the east-west heat calculation data prediction method provided by the embodiment of the invention, the west force calculation center sends the model trained by the filtered cold data sample and the model evaluation result to the east force calculation center, so that the east force calculation center is facilitated to construct the target function.
Optionally, after the sending the second traffic prediction model and the first AUC score to the western force center, the method further includes:
adjusting the first preset threshold value by adopting grid search to obtain a second preset threshold value; the second preset threshold corresponds to the minimum value of the target function; the objective function is constructed based on the first and second AUC scores; the second AUC score is obtained by predicting the second service prediction model by the thermal data application set;
respectively filtering cold data samples in the first cold data training set and the first cold data verification set based on a second preset threshold value to obtain a second cold data training set and a second cold data verification set;
and training the second cold data training set and the second cold data verification set to obtain the first business prediction model.
Specifically, the western force center adjusts the preset threshold a in a grid search manner, and the upper limit of the preset threshold a is set to
The lower limit is
Step length of
And sequentially repeating the processes of cold data filtering, business prediction model training and model evaluation to obtain different objective function values corresponding to different preset thresholds. And taking the corresponding preset threshold value when the objective function value is minimum as a second preset threshold value, wherein the second preset threshold value is an optimal preset threshold value.
And after the optimal preset threshold value, namely a second preset threshold value, filtering the cold data of the non-positive and negative samples by utilizing the second preset threshold value to the first cold data training set and the first cold data verification set respectively to obtain filtered cold data of the positive and negative samples, and acquiring a second cold data training set and a second cold data verification set. And splicing the second cold data training set and the second cold data verification set, and training a business prediction model based on the spliced cold data sample set to obtain a final business prediction model, namely the first business prediction model.
The western force center sends the first business prediction model to the eastern force center.
According to the east-west calculation thermal data prediction method provided by the embodiment of the invention, the automatic optimization and stable promotion of the model are realized by carrying out hyper-parameterization on the preset threshold value and regularizing the difference of the model effect; compared with the transmission of sample data, the transmission of the model parameters has smaller additional burden on a network, and the safety problem existing in the transmission of the sample data is avoided.
According to the east-west computation thermal data prediction method provided by the embodiment of the invention, the automatic optimization and stability promotion of the model are realized through the reconstruction objective function and the hyper-parameter optimization. The concrete expression is as follows:
1. compared with the traditional scheme of cold data training and hot data application, the AUC score of the model on hot data is improved by 1%;
2. in the western force computing center, the number of samples participating in model training is relatively reduced by 30% by filtering cold data samples, and the average training time of the model training is shortened to 70% of the original training time;
3. the stability of the model is improved by constructing an objective function, and compared with the traditional scheme, the AUC score difference of the model in an east computational force center and a west computational force center is reduced from 0.05 to 0.02;
4. in the invention, the filtering of cold data samples is realized by transmitting sample similarity model parameters, and compared with sample alignment schemes such as Maximum Mean Difference (MMD) and the like, the data volume transmitted through a network is reduced from GB level to KB level, so that the network load is effectively reduced;
5. by means of parameterization of the preset threshold, collaborative model iterative optimization of the east computational force center and the west computational force center is achieved, and compared with single model training, AUC scores of the model on thermal data are improved by 0.3%.
The present invention provides an eastern and western calculated thermal data prediction apparatus, which can be referred to in correspondence with the eastern and western calculated thermal data prediction method described above.
Fig. 4 is a schematic structural diagram of an eastern and western thermal data prediction apparatus according to an embodiment of the present invention, and as shown in fig. 4, the present invention further provides an eastern and western thermal data prediction apparatus including: a first receiving module 401 and a first obtaining module 402, wherein:
the first receiving module 401 is configured to receive a first service prediction model sent by a western computing center; the first business prediction model is obtained based on the filtered cold data sample training;
the first obtaining module 402 is configured to input the thermal data to be predicted into a first service prediction model for prediction, and obtain a prediction result corresponding to the thermal data to be predicted.
Optionally, the first receiving module 401 includes:
the first receiving submodule is used for receiving a second service prediction model and a first AUC score sent by the western computing center; the second business prediction model is obtained by training a first cold data training set filtered based on a first preset threshold; the first AUC score is obtained by verifying the second service prediction model by using the sampled first cold data verification set;
the first obtaining submodule is used for predicting the thermal data samples in the thermal data application set based on the second business prediction model to obtain a second AUC score;
a construction submodule for constructing a target function based on the first AUC score and the second AUC score;
the second receiving submodule receives a first business prediction model which is sent by a western computing center and obtained by training a second cold data training set and a second cold data verification set under the condition that the value of the target function is minimum; the second cold data training set and the second cold data validation set are obtained by filtering the first cold data training set and the first cold data validation set based on a second preset threshold corresponding to the minimum value of the objective function.
Optionally, the apparatus further comprises:
the second acquisition module is used for carrying out sample similarity training on the thermal data samples based on an LGB algorithm to acquire a sample similarity model;
the second sending module is used for sending the sample similarity model to a western force computing center; the sample similarity model is used for carrying out similarity evaluation on the cold data samples in the first cold data training set and the first cold data verification set to obtain a similarity score corresponding to each cold data sample.
Specifically, the device for predicting eastern and western arithmetic thermal data provided in the embodiment of the present application can implement all the method steps implemented by the embodiment in which the main execution body of the method is the eastern arithmetic center, and can achieve the same technical effects, and detailed descriptions of the same parts and beneficial effects as those of the method embodiment in the embodiment are omitted here.
Fig. 5 is a second structural schematic diagram of an eastern and western thermal data prediction apparatus according to an embodiment of the present invention, and as shown in fig. 5, the present invention further provides an eastern and western thermal data prediction apparatus, including: a first transmitting module 501;
a first sending module 501, configured to send a first business prediction model to the east force computing center; the first business prediction model is obtained by training based on the filtered cold data samples.
Optionally, the apparatus further comprises:
the second receiving module is used for receiving the sample similarity model sent by the east computing power center;
the third obtaining module is used for carrying out similarity evaluation on the cold data samples in the first cold data training set and the first cold data verification set based on the sample similarity model to obtain a similarity score corresponding to each cold data sample;
and the fourth acquisition module is used for respectively filtering the cold data samples in the first cold data training set and the first cold data verification set based on a first preset threshold and the similarity score, and acquiring a third cold data training set and a third cold data verification set.
Optionally, the apparatus further comprises:
a fifth obtaining module, configured to train the cold data samples in the third cold data training set by using a machine learning algorithm, and obtain a second service prediction model;
the sixth obtaining module is configured to verify the second service prediction model by using the sampled first cold data verification set, and obtain a first AUC score of the second service prediction model;
and the third sending module is used for sending the second service prediction model and the first AUC score to a western force center.
Optionally, the apparatus further comprises:
a seventh obtaining module, configured to adjust the first preset threshold by using grid search, and obtain a second preset threshold; the second preset threshold corresponds to the minimum value of the target function; the objective function is constructed based on the first and second AUC scores; the second AUC score is obtained by predicting the second service prediction model by using the thermal data application set;
an eighth obtaining module, configured to filter cold data samples in the first cold data training set and the first cold data verification set based on a second preset threshold, respectively, so as to obtain a second cold data training set and a second cold data verification set;
and the ninth obtaining module is used for training the second cold data training set and the second cold data verification set to obtain the first business prediction model.
Specifically, the east-west computation thermal data prediction apparatus provided in the embodiment of the present application can implement all the method steps implemented by the embodiment in which the execution subject of the method is the west computation center, and can achieve the same technical effects, and details of the same parts and beneficial effects as those of the method embodiment in this embodiment are not repeated here.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 6, the electronic device may include: a processor (processor) 610, a communication Interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication Interface 620 and the memory 630 communicate with each other via the communication bus 640. Processor 610 may call logic instructions in memory 630 to perform a east west arithmetic thermal data prediction method comprising: receiving a first service prediction model sent by a western computing center; the first business prediction model is obtained by training based on the filtered cold data sample; inputting the thermal data to be predicted into a first business prediction model for prediction, and obtaining a prediction result corresponding to the thermal data to be predicted.
Or, further comprising: sending a first business prediction model to an east force computing center; the first business prediction model is trained based on the filtered cold data samples.
In addition, the logic instructions in the memory 630 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the eastern and western thermal data prediction method provided by the above methods, the method comprising: receiving a first service prediction model sent by a western computing center; the first business prediction model is obtained by training based on the filtered cold data sample; inputting the thermal data to be predicted into a first service prediction model for prediction, and obtaining a prediction result corresponding to the thermal data to be predicted.
Or, further comprising: sending a first business prediction model to an east force computing center; the first business prediction model is obtained by training based on the filtered cold data samples.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method for predicting eastern western thermal data provided by the above methods, the method comprising: receiving a first business prediction model sent by a western computing center; the first business prediction model is obtained based on the filtered cold data sample training; inputting the thermal data to be predicted into a first business prediction model for prediction, and obtaining a prediction result corresponding to the thermal data to be predicted.
Or, further comprising: sending a first business prediction model to an east force computing center; the first business prediction model is trained based on the filtered cold data samples.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment may be implemented by software plus a necessary general hardware platform, and may also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
The terms "first," "second," and the like in the embodiments of the present application are used for distinguishing between similar elements and not for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application are capable of operation in other sequences than those illustrated or otherwise described herein, and that the terms "first" and "second" used herein generally refer to a class and do not limit the number of objects, for example, a first object can be one or more.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for predicting east-west computation thermal data is applied to an east computation power center and comprises the following steps:
receiving a first service prediction model sent by a western computing center; the first business prediction model is obtained based on the filtered cold data sample training;
inputting the thermal data to be predicted into a first business prediction model for prediction, and obtaining a prediction result corresponding to the thermal data to be predicted.
2. The method of claim 1, wherein receiving the first traffic prediction model sent by the western force center comprises:
receiving a second service prediction model and a first AUC score sent by a western computing center; the second business prediction model is obtained by training a first cold data training set filtered based on a first preset threshold; the first AUC score is obtained by verifying the second service prediction model by using the sampled first cold data verification set;
predicting the thermal data samples in the thermal data application set based on the second service prediction model to obtain a second AUC score;
constructing an objective function based on the first AUC score and the second AUC score;
under the condition that the value of the objective function is minimum, receiving a first business prediction model which is sent by a western computing force center and obtained by training a second cold data training set and a second cold data verification set; the second cold data training set and the second cold data validation set are obtained by filtering the first cold data training set and the first cold data validation set based on a second preset threshold corresponding to the minimum value of the objective function.
3. The method for forecasting eastern West computation thermal data according to claim 2, wherein before receiving the second traffic forecasting model and the first AUC score sent by the western computation force center, the method further comprises:
performing sample similarity training on the thermal data sample based on an LGB algorithm to obtain a sample similarity model;
sending the sample similarity model to a western force center; the sample similarity model is used for carrying out similarity evaluation on the cold data samples in the first cold data training set and the first cold data verification set to obtain a similarity score corresponding to each cold data sample.
4. The east-west arithmetic thermal data prediction method is applied to a west arithmetic center and comprises the following steps:
sending a first business prediction model to an east force computing center; the first business prediction model is trained based on the filtered cold data samples.
5. The eastern West forecasting method according to claim 4, wherein before sending the first business forecasting model to the eastern force center, the method further comprises:
receiving a sample similarity model sent by the east force computing center;
performing similarity evaluation on cold data samples in the first cold data training set and the first cold data verification set based on the sample similarity model to obtain a similarity score corresponding to each cold data sample;
and respectively filtering cold data samples in the first cold data training set and the first cold data verification set based on a first preset threshold and the similarity score to obtain a third cold data training set and a third cold data verification set.
6. The east west arithmetic thermal data prediction method of claim 5 wherein after obtaining a third training set of cold data and a third validation set of cold data, further comprising:
training cold data samples in the third cold data training set by using a machine learning algorithm to obtain a second business prediction model;
verifying the second service prediction model by using the sampled first cold data verification set to obtain a first AUC score of the second service prediction model;
and sending the second service prediction model and the first AUC score to a western computing power center.
7. The eastern west arithmetic thermal data prediction method of claim 6 further comprising, after sending the second business prediction model and the first AUC score to a western force center:
adjusting the first preset threshold value by adopting grid search to obtain a second preset threshold value; the second preset threshold corresponds to the minimum value of the target function; the objective function is constructed based on the first and second AUC scores; the second AUC score is obtained by predicting the second service prediction model by using the thermal data application set;
respectively filtering cold data samples in the first cold data training set and the first cold data verification set based on a second preset threshold value to obtain a second cold data training set and a second cold data verification set;
and training the second cold data training set and the second cold data verification set to obtain the first business prediction model.
8. An eastern west arithmetic thermal data prediction apparatus comprising:
the first receiving module is used for receiving a first business prediction model sent by a western computing center; the first business prediction model is obtained by training based on the filtered cold data sample;
the first obtaining module is used for inputting the thermal data to be predicted into a first business prediction model for prediction, and obtaining a prediction result corresponding to the thermal data to be predicted.
9. An eastern west arithmetic thermal data prediction apparatus comprising:
the first sending module is used for sending the first business prediction model to the east computing center; the first business prediction model is trained based on the filtered cold data samples.
10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the east-west computed thermal data prediction method of any one of claims 1-3 or implements the east-west computed thermal data prediction method of any one of claims 4-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210845186.5A CN114925939B (en) | 2022-07-19 | 2022-07-19 | East-west calculation heat data prediction method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210845186.5A CN114925939B (en) | 2022-07-19 | 2022-07-19 | East-west calculation heat data prediction method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114925939A true CN114925939A (en) | 2022-08-19 |
| CN114925939B CN114925939B (en) | 2022-10-21 |
Family
ID=82816216
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210845186.5A Active CN114925939B (en) | 2022-07-19 | 2022-07-19 | East-west calculation heat data prediction method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114925939B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190258904A1 (en) * | 2018-02-18 | 2019-08-22 | Sas Institute Inc. | Analytic system for machine learning prediction model selection |
| CN112580268A (en) * | 2021-02-25 | 2021-03-30 | 上海冰鉴信息科技有限公司 | Method and device for selecting machine learning model based on business processing |
| CN113918884A (en) * | 2020-07-09 | 2022-01-11 | 顺丰科技有限公司 | Traffic prediction model construction method and traffic prediction method |
| CN114445143A (en) * | 2022-01-29 | 2022-05-06 | 中国农业银行股份有限公司 | Service data prediction method, device, equipment and medium |
| CN114462465A (en) * | 2021-09-30 | 2022-05-10 | 马上消费金融股份有限公司 | Training method of classification model, quality inspection prediction method and corresponding device |
-
2022
- 2022-07-19 CN CN202210845186.5A patent/CN114925939B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190258904A1 (en) * | 2018-02-18 | 2019-08-22 | Sas Institute Inc. | Analytic system for machine learning prediction model selection |
| CN113918884A (en) * | 2020-07-09 | 2022-01-11 | 顺丰科技有限公司 | Traffic prediction model construction method and traffic prediction method |
| CN112580268A (en) * | 2021-02-25 | 2021-03-30 | 上海冰鉴信息科技有限公司 | Method and device for selecting machine learning model based on business processing |
| CN114462465A (en) * | 2021-09-30 | 2022-05-10 | 马上消费金融股份有限公司 | Training method of classification model, quality inspection prediction method and corresponding device |
| CN114445143A (en) * | 2022-01-29 | 2022-05-06 | 中国农业银行股份有限公司 | Service data prediction method, device, equipment and medium |
Non-Patent Citations (1)
| Title |
|---|
| 易春燕: "基于机器学习的混合存储冷热数据分析及优化", 《中国优秀硕士学位论文全文数据库（信息科技辑）》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114925939B (en) | 2022-10-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111177792B (en) | Method and device for determining target business model based on privacy protection | |
| CN109117380B (en) | Software quality evaluation method, device, equipment and readable storage medium | |
| CN112289034A (en) | Deep neural network robust traffic prediction method based on multi-mode space-time data | |
| CN111967971B (en) | Bank customer data processing method and device | |
| CN112561568A (en) | Target customer prediction method, device and storage medium | |
| CN114925939B (en) | East-west calculation heat data prediction method and device and electronic equipment | |
| CN116363452B (en) | Task model training method and device | |
| CN111400964A (en) | Fault occurrence time prediction method and device | |
| CN116010832A (en) | Federal clustering method, federal clustering device, central server, federal clustering system and electronic equipment | |
| CN111507499A (en) | Construction method, test method, device and system of model for prediction | |
| CN114492847B (en) | Efficient personalized federal learning system and method | |
| CN116108155A (en) | Man-machine conversation system, data processing method thereof, electronic equipment and storage medium | |
| CN110490244A (en) | A kind of data processing method and device | |
| CN114549147A (en) | Carbon emission treatment method, carbon emission treatment device, electronic equipment and storage medium | |
| CN112132219A (en) | General deployment scheme of deep learning detection model based on mobile terminal | |
| CN112199980A (en) | Overhead line robot obstacle identification method | |
| CN111160662A (en) | Risk prediction method, electronic equipment and storage medium | |
| CN116527411B (en) | Data security intelligent protection model construction method and device and collaboration platform | |
| CN115496200B (en) | Neural network quantization model training method, device and equipment | |
| CN116957072A (en) | Method and device for generating rule based on monotone sub-box and electronic equipment | |
| CN116885707A (en) | Demand side resource adjustable potential collaborative prediction method based on VMD and GA-BP | |
| CN117556212A (en) | Stream big data analysis method, device, system and storage medium | |
| CN116451587A (en) | Method for simulating operation curve by deep neural network | |
| CN116343760A (en) | Speech recognition method, system and computer equipment based on federal learning | |
| CN116385222A (en) | Teaching quality evaluation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |