CN110188919A - A kind of load forecasting method based on shot and long term memory network - Google Patents
A kind of load forecasting method based on shot and long term memory network Download PDFInfo
- Publication number
- CN110188919A CN110188919A CN201910325295.2A CN201910325295A CN110188919A CN 110188919 A CN110188919 A CN 110188919A CN 201910325295 A CN201910325295 A CN 201910325295A CN 110188919 A CN110188919 A CN 110188919A
- Authority
- CN
- China
- Prior art keywords
- data
- load
- term memory
- feature
- memory network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007787 long-term memory Effects 0.000 title claims abstract description 24
- 238000013277 forecasting method Methods 0.000 title abstract 2
- 238000012549 training Methods 0.000 claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 20
- 230000006835 compression Effects 0.000 claims abstract description 7
- 238000007906 compression Methods 0.000 claims abstract description 7
- 230000006403 short-term memory Effects 0.000 claims description 21
- 230000015654 memory Effects 0.000 claims description 17
- 238000003066 decision tree Methods 0.000 claims description 7
- 238000000513 principal component analysis Methods 0.000 claims description 5
- 238000003064 k means clustering Methods 0.000 claims description 4
- 238000011478 gradient descent method Methods 0.000 claims description 2
- 230000005611 electricity Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 239000011159 matrix material Substances 0.000 description 5
- 230000008569 process Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000008034 disappearance Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000010248 power generation Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000012106 screening analysis Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Economics (AREA)
- General Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- Strategic Management (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Tourism & Hospitality (AREA)
- Development Economics (AREA)
- Biophysics (AREA)
- Water Supply & Treatment (AREA)
- Public Health (AREA)
- Game Theory and Decision Science (AREA)
- Entrepreneurship & Innovation (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Primary Health Care (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention belongs to Techniques for Prediction of Electric Loads fields, disclose a kind of load forecasting method based on shot and long term memory network, comprising: acquisition target area forms raw data set in the power load data and corresponding weather characteristics data of certain time period；Missing values processing is carried out to raw data set using Spark cluster；Feature selecting is carried out to raw data set；Feature Compression is carried out to raw data set；Establish the prediction model based on shot and long term memory network；Distributed training is carried out to prediction model using Spark cluster；According to the weather characteristics data of previous time point and power load data, Distributed Predictive is carried out using prediction model, obtains the predicted load of current point in time.The present invention can solve is difficult under big data scene the problem of fast and efficiently carrying out electro-load forecast in the prior art, effectively can quickly be extracted, be handled and operation to large data sets.
Description
Technical Field
The invention relates to the technical field of power load prediction, in particular to a load prediction method based on a long-term and short-term memory network.
Background
The load prediction problem is about predicting the power load required by a power enterprise at a specific future time, and is one of the core contents in power grid planning. The power enterprise forecasts the change situation and the development trend of the power load within a period of time in the future according to the historical data analysis of the load and the judgment of the future development trend. An accurate load forecast is crucial to the short-term scheduling arrangement and the long-term system planning of the power enterprise, and is the basis for making power supply planning, development planning, capital financial planning and the like.
By the end of 2017, the new energy grid-connected capacity in the national grid dispatching range reaches 2.80 hundred million kilowatts, wherein 14539 ten thousand kilowatts of wind power and 12083 ten thousand kilowatts of solar power generation are the first in the world; the accumulated installation of the intelligent electric meter exceeds 4 hundred million, and the automatic acquisition and full coverage of the electricity utilization information are basically realized. The data is growing day by day, and load prediction enters big data age, and load prediction and big data technology are amalgamated and produced to the practice that is imperative, have important strategic meaning to the development of electric power industry.
To address the challenges presented by massive amounts of data, "big data" has spawned a large number of distributed parallel computing and storage technologies based on computer technology. Based on the MapReduce programming framework and the Google File System developed by Google, the Hadoop project of Apache starts the era of enterprise-level big data processing, and the advent of various distributed computing and storage platforms surrounding Hadoop research and development comes with the Hadoop. With the demand for real-time computing becoming higher, the related stream processing engines represented by Spark Streaming and Flink have started a new wave of big data development.
In order to solve the problem of rapid and efficient power load prediction in a big data scene, a whole set of load prediction data processing and modeling scheme suitable for a big data processing scene needs to be provided.
Disclosure of Invention
The load prediction method based on the long-term and short-term memory network solves the problem that the power load prediction is difficult to rapidly and efficiently perform in a big data scene in the prior art.
The embodiment of the application provides a load prediction method based on a long-term and short-term memory network, which comprises the following steps:
s1, collecting power load data and corresponding weather characteristic data of a target area in a certain time period to form an original data set;
step S2, missing value processing is carried out on the original data set by utilizing a Spark cluster;
step S3, selecting the characteristics of the original data set;
step S4, performing feature compression on the original data set;
step S5, establishing a prediction model based on the long-term and short-term memory network;
step S6, performing distributed training on the prediction model by using the Spark cluster;
and S7, according to the weather characteristic data and the power load data of the previous time point, performing distributed prediction by using the prediction model to obtain a load prediction value of the current time point.
Preferably, the load prediction method based on the long-short term memory network further includes:
step S8: and storing the power load data and the weather characteristic data which are acquired in real time in an Hbase cluster, and displaying a load predicted value and a load real-time acquisition value.
Preferably, in step S1, the formed raw data set is stored into the Hbase cluster.
Preferably, in step S2, after K-means clustering is performed on the missing value by using the Spark cluster, a similar mean interpolation strategy is adopted for processing.
Preferably, in step S3, the pearson correlation between the electrical load data and the weather feature data is calculated, and feature importance ranking is performed by training a gradient boosting decision tree.
Preferably, feature variables with significance higher than 0.05 are removed, all variables are subjected to relevancy ranking, and the top 30% of features are taken as a first feature set; training by using a gradient lifting decision tree to obtain feature variable importance ranking, and taking the first 30% of features as a second feature set; and taking the intersection feature of the first feature set and the second feature set as the screened feature.
Preferably, in step S4, feature compression is performed on the weather feature data by using principal component analysis.
Preferably, in step S5, the parameter is updated by using a truncated propagation along the time direction.
Preferably, in step S6, based on the Spark cluster, a distributed training of data parallel is performed by using an asynchronous random gradient descent method.
Preferably, in step S7, the power load data and the weather characteristic data at the previous time point are acquired, the parallelism is set according to the data size and the cluster hardware information, and the load size is predicted by using the Spark cluster.
One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:
in the embodiment of the application, a historical electric load data set and a historical weather characteristic data set are obtained, and data preprocessing, characteristic selection and characteristic compression are carried out; then, an asynchronous random gradient descent distributed training is carried out by using a cut-off type long-short term memory network which reversely propagates along time, a prediction model of the electricity utilization load is established, and electricity utilization amount prediction at a certain moment is carried out. The invention can effectively and rapidly extract, process and operate the big data set.
Drawings
In order to more clearly illustrate the technical solution in the present embodiment, the drawings needed to be used in the description of the embodiment will be briefly introduced below, and it is obvious that the drawings in the following description are one embodiment of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
Fig. 1 is a data processing platform of a load prediction method based on a long-term and short-term memory network according to an embodiment of the present invention;
fig. 2 is a topological diagram of a long-short term memory network model in a load prediction method based on a long-short term memory network according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the truncated backward propagation along time in the load prediction method based on the long-short term memory network according to the embodiment of the present invention;
fig. 4 is a topological diagram of Spark common mode training in the load prediction method based on the long-term and short-term memory network according to the embodiment of the present invention;
fig. 5 is a topological diagram of Spark grid pattern training in the load prediction method based on the long and short term memory network according to the embodiment of the present invention.
Detailed Description
In order to better understand the technical solution, the technical solution will be described in detail with reference to the drawings and specific embodiments.
The load prediction method based on the long-term and short-term memory network provided by the embodiment mainly comprises the following steps:
step S1: historical electricity load data of a certain area and related weather data of the same area and the same time period are collected to form an original data set, and the original data set is stored in an Hbase cluster.
Historical data (namely an original data set) is stored in the Hbase, and due to the characteristic that the Hbase level is well expanded, the requirement for large data volume can be met, the operation requirements for data persistence and reading are met, and meanwhile, interactive query operation can be performed by using Hive.
Step S2: and carrying out missing value processing operation on the data set by utilizing the Spark cluster.
By providing related information for the drive nodes of the Spark cluster, and using the computing nodes in the Spark cluster to load and persist the big data into the distributed memory of the cluster host in parallel, the data is abstracted into an elastic distributed data set in the Spark cluster and is computed based on the memory, so that the time overhead of data preprocessing is reduced. The data missing value is subjected to parallel similar mean value interpolation processing based on K-mean value clustering, and is closer to a true value compared with the common mean value interpolation.
Step S3: feature selection of the data is performed.
The method comprises the steps of calculating the Pearson correlation degree of each weather characteristic data and the power load data, training by using a gradient lifting decision tree to obtain characteristic variable importance ranking, and combining the two to perform characteristic screening, so that original data characteristics are reserved to the maximum extent, and training data volume is reduced.
Step S4: feature compression of the data is performed.
And the principal component is used for analyzing the compressed data, so that the input data volume of the algorithm model is reduced and the calculation speed of the model is improved while the characteristics of the original data are kept to the maximum extent.
Step S5: and establishing a model based on the long-term and short-term memory network.
The load prediction modeling is carried out on the compressed data by using the long-term and short-term memory network, so that the problem of gradient disappearance can be prevented. The method can improve the frequency of parameter updating, so that the neural network can be shaped more quickly with the same operational capability.
Step S6: the Spark cluster is used for distributed training of the model.
Based on Spark clusters, the asynchronous random gradient descent data parallelism training method is used, and the Spark cluster mode training network can greatly reduce the data volume of parameter updating between nodes, so that the model training speed is improved.
Step S7: and according to the power load data and the weather characteristic data of the previous time point, performing distributed prediction through the established long-term and short-term memory network model to obtain a load prediction value of the current time point.
Reading the power load data and the weather characteristic data at the previous moment, setting reasonable parallelism, predicting the load capacity by using a Spark cluster, and improving the prediction speed.
Step S8: and storing the power load data and the weather data acquired in real time in an Hbase cluster, and displaying the load predicted value and the load real-time acquired value (namely the real load value) on a Web end through a graphical interface.
And storing the latest power load data and the weather characteristic data which are acquired in real time into the Hbase cluster. The real-time real load value (i.e. the real-time load collection value) and the predicted load value are displayed together so as to observe errors and trends.
The present invention is described in further detail below.
Fig. 1 shows a block diagram of a data processing platform provided in the present invention, which includes: providing a distributed file system HDFS in Hadoop stored at the bottom layer; a calculation framework MapReduce for providing data operation interfaces for Hbase and Hive; cluster computing framework Spark; a message queue MQ that accepts real-time data. The Hbase is a distributed database, and the Hive is used for providing SQL type data operation for related workers.
The method synthesizes a real scene, considers factors such as weather factors, time factors, area relevance and the like, and predicts the electricity utilization load amount of a certain future area at a certain moment by mathematical modeling under the condition of ensuring a certain fault tolerance and precision.
The load prediction method based on the long-term and short-term memory network provided by the embodiment mainly comprises the following steps:
step S1: and collecting historical electricity load data of a certain area and related weather data of the same area and the same time period in the Hbase cluster.
The original electric quantity data set and the weather data set are respectively stored in Hbase in advance, the Hbase can store massive data sets, and data columns of the data sets can be dynamically increased according to requirements as a column storage type NoSQL type database to meet the requirement of load characteristic data with multiple dimensions.
Step S2: and carrying out missing value processing operation on the data set by utilizing the Spark cluster.
The meteorological characteristic missing values are subjected to K-means clustering processing to obtain a plurality of different data clusters, mean interpolation is carried out in the same data cluster, compared with common mean interpolation, the interpolated mean value is closer to the true value, and therefore the accuracy of the model is improved.
K-means is an iterative algorithm, and it is assumed that we want to cluster data into K groups, the method is that first, K random points are selected, called cluster centers; for each data in the data set, the data is characterized by a characteristic vector 2-norm according to the distance from each central point. It is associated with the closest central point and all points associated with the same central point are grouped into one category. The average value of each group is calculated and the center point associated with the group is moved to the position of the average value. The K-means minimization problem is to minimize the sum of the distances between all data points and their associated cluster center points, and the cost function of K-means is as follows:
wherein,representative and feature data vector x(i)The nearest cluster center point, the algorithm optimization goal is to find c which minimizes the cost function(1),…c(m)And u1,…,uk。
The algorithm flow is as follows:
(1) randomly creating K points as an initial center;
(2) calculating the distance of each feature vector in the meteorological feature data set relative to each center;
(3) assigning the feature vector to its closest center;
(4) for the newly obtained groups, calculating the vector center of each group;
and (5) repeating the processes (2) to (4) until the algorithm converges.
The K-means algorithm cannot determine the number of clusters, and too much, although it makes the cost function smaller, will cause an overfitting of the data. The selection of a suitable cluster number is important to the accuracy of the mean interpolation. And calculating error values of the meteorological feature data under different types of quantity, and selecting the type quantity which enables the error reduction rate to be maximum as a classification basis of the type difference value. And respectively carrying out mean interpolation on the divided weather characteristic categories so as to finish the missing value processing work.
And step S3, selecting the characteristics of the data.
And 4, calculating the correlation degree of the Pearson, training a model by using a gradient lifting decision algorithm, and combining the two to select characteristics, so that the method has higher characteristic fault tolerance.
And (3) calculating the correlation degree and significance of the pearson, and calculating the correlation degree of the load quantity and each weather characteristic by using the following formula:
wherein,andare each XiSample and YiAverage of samples. And removing the characteristic variables with the significance higher than 0.05, carrying out relevancy sorting on all the variables, and taking the top 30% of the characteristics as a characteristic set A.
And training by using a gradient lifting decision tree to obtain feature variable importance ranking, and taking the first 30% of features as a feature set B. And taking the intersection feature of the feature set A and the feature set B as the feature after screening.
And step S4, performing characteristic compression of the data.
Using principal component analysis dimensionality reduction, a d x k dimensional transformation matrix W is constructed, such that an eigenvector x can be mapped into a new k dimensional eigen-space, the dimensions of which are smaller than the original d dimensional eigen-space:
the principal component analysis algorithm flow is as follows:
(1) carrying out standardization processing on the original d-dimensional data set;
(2) constructing a sample covariance matrix;
(3) calculating an eigenvalue of the covariance matrix and a corresponding eigenvector;
(4) selecting eigenvectors corresponding to the first k largest eigenvalues, where k is the dimension of the new eigenspace (k < d);
(5) constructing a mapping matrix W through the previous feature vector;
(6) the d-dimensional input data set is converted to a new k-dimensional feature subspace by means of a mapping matrix W.
For the compressed time series data set in the Spark cluster, the following 8: and 2, splitting the data into a training characteristic data set and a test data set. Wherein the feature data set is used to train the model and the test data set is used to evaluate the model.
And step S5, establishing a circulating network model based on the long-term and short-term memory network.
Because the load per hour of 24h in the future is predicted, the time step of 24 of the long-short term memory network, that is, the output of the sequence of continuous 24h loads, is selected as a sample, and the specific structure is shown in fig. 2. The training characteristics are compressed weather characteristic data, and the training labels are load values of the next time point. By adopting a long-short term memory network structure, even if the time step is longer, the model training is not influenced by 'gradient disappearance'.
The long and short term memory network has high requirement on the setting calculation capacity when processing longer sequences, and a cut-off type rapid setting network is used for backward propagation along time. Taking the truncated backward propagation along time with length 4 in fig. 3 as an example, each backward propagation of the parameter update only passes through 4 time steps, so that the complexity of the parameter update in the network can be reduced. When inputting longer sequence data, the truncated time-reversal propagation algorithm can be adopted to increase the updating frequency of parameters, so that the neural network can be shaped more quickly with the same operational capability.
Step S6, carrying out distributed training on the model by using Spark clusters.
A distributed training scheme based on data-parallel asynchronous random gradient descent (asynchronous random gradient descent) is used. Defining the loss function as L, the gradient vector of the loss function for n parameters is:
the parameter vector W is, after i +1 th iteration of SGD at the learning rate a:
wherein, WiAs a result of the ith iteration of the parameter vector,and training the gradient vector of the loss function obtained after the node data copy is trained for the jth computing node, wherein n is the number of the computing nodes.
Updating values of parameters in asynchronous random gradient descentIs applied to the vector of parameters when the calculation is completed, and does not need to strictly follow the period of updating the parameters. Asynchronous random gradient descent can achieve higher throughput in distributed systems: rather than waiting for the parameter averaging step to complete, the worker node may take more time to perform useful calculations. Second, the worker node merges information from other worker nodes faster than when using synchronous updates.
Performing distributed model training by using asynchronous random gradient descent based on Spark computing cluster, and using a common mode when the number of nodes in the cluster is less than 32, as shown in fig. 4; when the cluster size is large, the network mode is used, see fig. 5. The two training modes update the parameters between the nodes for coding and compressing, thereby reducing the communication traffic between the nodes and effectively improving the model training speed.
In the normal mode, quantized coded updates are passed by the worker node to the master node, which then propagates the updates to the remaining nodes. This may ensure that the master node always holds the latest version of the model. Meanwhile, reliable communication of the nodes can be guaranteed by means of a fault-tolerant mechanism of the Spark cluster, and if the Spark Master node fails, a new Master node can be elected by the cluster, so that the single-point problem is avoided.
The grid pattern is a multi-way tree, and the root node of the grid pattern is Spark Master. By default, each node may have a maximum of eight nodes, and the node tree within the Spark cluster may have a maximum of five levels. In network mode, each node relays the encoding updates to all nodes connected to it, and each node aggregates the updates received from all other nodes connected to it. In mesh mode, the Master node is no longer a bottleneck for performance because it receives less traffic directly.
And step S7, according to the power load data and the weather characteristic data of the previous time point, carrying out distributed prediction through the established long-term and short-term memory network model to obtain the load prediction quantity of the current time point.
And reading the power load data and the weather characteristic data at the last moment from the Hbase cluster, using a Spark cluster to predict the load amount in parallel, and setting reasonable data partition number and partition size according to the cluster condition to improve the prediction speed to the maximum extent.
And step S8, storing the real-time collected power load data and weather data in the Hbase cluster, and displaying the data on a Web end through a graphical interface.
And storing the load quantity and the weather characteristic data acquired in real time into Hbase for use in a later training model. And writing the prediction result into a message queue, receiving the load prediction value and the real value in the message queue in real time by using a network programming technology, and displaying in a line graph mode so as to observe errors and trends.
The load prediction method based on the long-term and short-term memory network provided by the embodiment of the invention at least comprises the following technical effects:
based on Hadoop, Spark and other big data processing technologies, K-means clustering is used for carrying out similar mean interpolation, and Pearson relevance and gradient boosting decision tree feature screening and principal component analysis are combined to carry out data processing operation of reducing dimension and compressing feature data, so that original data features are reserved to the greatest extent, the data volume of model training is greatly reduced, and the model training speed is accelerated. The distributed load modeling training is carried out on the data by using the long-term and short-term memory network, and compared with the traditional single machine training, the training process is greatly accelerated. By storing the training model on the distributed file system and calculating the cluster by means of Spark, the distributed parallel load prediction is completed, and the prediction time cost is reduced. The method can efficiently and accurately complete the electric load prediction in a big data scene. The method and the system predict the electricity utilization load of a certain area corresponding to a certain moment in the future under the condition of ensuring a certain fault tolerance rate and precision, and provide certain reference significance for relevant departments for dispatching the electricity resources.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention has been described in detail with reference to examples, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, which should be covered by the claims of the present invention.
Claims (10)
1. A load prediction method based on a long-term and short-term memory network is characterized by comprising the following steps:
s1, collecting power load data and corresponding weather characteristic data of a target area in a certain time period to form an original data set;
step S2, missing value processing is carried out on the original data set by utilizing a Spark cluster;
step S3, selecting the characteristics of the original data set;
step S4, performing feature compression on the original data set;
step S5, establishing a prediction model based on the long-term and short-term memory network;
step S6, performing distributed training on the prediction model by using the Spark cluster;
and S7, according to the weather characteristic data and the power load data of the previous time point, performing distributed prediction by using the prediction model to obtain a load prediction value of the current time point.
2. The long-short term memory network-based load prediction method of claim 1, further comprising:
step S8: and storing the power load data and the weather characteristic data which are acquired in real time in an Hbase cluster, and displaying a load predicted value and a load real-time acquisition value.
3. The method for load prediction based on long-short term memory network as claimed in claim 1, wherein in step S1, the formed raw data set is stored into Hbase cluster.
4. The load prediction method based on the long-short term memory network as claimed in claim 1, wherein in step S2, after K-means clustering is performed on the missing values by using the Spark cluster, a homogeneous mean interpolation strategy is adopted for processing.
5. The load prediction method based on the long-short term memory network as claimed in claim 1, wherein in step S3, the pearson correlation degree between the electrical load data and the weather feature data is calculated, and the feature importance ranking is performed by training a gradient boosting decision tree.
6. The load prediction method based on the long-short term memory network as claimed in claim 5, characterized in that feature variables with significance higher than 0.05 are removed, all variables are subjected to relevancy ranking, and the top 30% of features are taken as a first feature set; training by using a gradient lifting decision tree to obtain feature variable importance ranking, and taking the first 30% of features as a second feature set; and taking the intersection feature of the first feature set and the second feature set as the screened feature.
7. The load prediction method based on the long-short term memory network as claimed in claim 1, wherein in step S4, the weather feature data is feature compressed by principal component analysis.
8. The method for load prediction based on long-short term memory network as claimed in claim 1, wherein in step S5, the parameters are updated by using the truncated propagation along the time direction.
9. The load prediction method based on the long-short term memory network as claimed in claim 1, wherein in step S6, based on the Spark cluster, a distributed training of data parallel is performed by using an asynchronous random gradient descent method.
10. The load prediction method based on the long and short term memory network as claimed in claim 1, wherein in step S7, the power load data and the weather characteristic data of the previous time point are obtained, and the load amount is predicted by using the Spark cluster according to the parallelism between the data amount and the cluster hardware information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910325295.2A CN110188919A (en) | 2019-04-22 | 2019-04-22 | A kind of load forecasting method based on shot and long term memory network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910325295.2A CN110188919A (en) | 2019-04-22 | 2019-04-22 | A kind of load forecasting method based on shot and long term memory network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110188919A true CN110188919A (en) | 2019-08-30 |
Family
ID=67714857
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910325295.2A Pending CN110188919A (en) | 2019-04-22 | 2019-04-22 | A kind of load forecasting method based on shot and long term memory network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110188919A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110879802A (en) * | 2019-10-28 | 2020-03-13 | 同济大学 | Log pattern extraction and matching method |
| CN110991689A (en) * | 2019-10-17 | 2020-04-10 | 国网河南省电力公司鹤壁供电公司 | Distributed photovoltaic power generation system short-term prediction method based on LSTM-Morlet model |
| CN111091243A (en) * | 2019-12-13 | 2020-05-01 | 南京工程学院 | PCA-GM-based power load prediction method, system, computer-readable storage medium, and computing device |
| CN111178587A (en) * | 2019-12-06 | 2020-05-19 | 广东工业大学 | Spark framework-based short-term power load rapid prediction method |
| CN111311001A (en) * | 2020-02-17 | 2020-06-19 | 合肥工业大学 | Bi-LSTM network short-term load prediction method based on DBSCAN algorithm and feature selection |
| CN111680074A (en) * | 2019-12-31 | 2020-09-18 | 国网浙江省电力有限公司 | Clustering algorithm-based electric power collection load leakage point feature mining method |
| CN113450141A (en) * | 2021-06-09 | 2021-09-28 | 重庆锦禹云能源科技有限公司 | Intelligent prediction method and device based on electricity selling quantity characteristics of large-power customer groups |
| CN113988393A (en) * | 2021-10-21 | 2022-01-28 | 青岛联众芯云科技有限公司 | Energy internet load prediction method |
| CN114881343A (en) * | 2022-05-18 | 2022-08-09 | 清华大学 | Short-term load prediction method and device of power system based on feature selection |
| CN114970938A (en) * | 2022-03-11 | 2022-08-30 | 武汉大学 | Self-adaptive residential load prediction method considering user privacy protection |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108985965A (en) * | 2018-06-22 | 2018-12-11 | 华北电力大学 | A kind of photovoltaic power interval prediction method of combination neural network and parameter Estimation |
| CN109032671A (en) * | 2018-06-25 | 2018-12-18 | 电子科技大学 | A kind of distributed deep learning method and system based on data parallel strategy |
| CN109117864A (en) * | 2018-07-13 | 2019-01-01 | 华南理工大学 | Coronary heart disease risk prediction technique, model and system based on heterogeneous characteristic fusion |
| CN109325624A (en) * | 2018-09-28 | 2019-02-12 | 国网福建省电力有限公司 | A kind of monthly electric power demand forecasting method based on deep learning |
| CN109508811A (en) * | 2018-09-30 | 2019-03-22 | 中冶华天工程技术有限公司 | Parameter prediction method is discharged based on principal component analysis and the sewage treatment of shot and long term memory network |
| CN109543203A (en) * | 2017-09-22 | 2019-03-29 | 山东建筑大学 | A kind of Building Cooling load forecasting method based on random forest |
-
2019
- 2019-04-22 CN CN201910325295.2A patent/CN110188919A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109543203A (en) * | 2017-09-22 | 2019-03-29 | 山东建筑大学 | A kind of Building Cooling load forecasting method based on random forest |
| CN108985965A (en) * | 2018-06-22 | 2018-12-11 | 华北电力大学 | A kind of photovoltaic power interval prediction method of combination neural network and parameter Estimation |
| CN109032671A (en) * | 2018-06-25 | 2018-12-18 | 电子科技大学 | A kind of distributed deep learning method and system based on data parallel strategy |
| CN109117864A (en) * | 2018-07-13 | 2019-01-01 | 华南理工大学 | Coronary heart disease risk prediction technique, model and system based on heterogeneous characteristic fusion |
| CN109325624A (en) * | 2018-09-28 | 2019-02-12 | 国网福建省电力有限公司 | A kind of monthly electric power demand forecasting method based on deep learning |
| CN109508811A (en) * | 2018-09-30 | 2019-03-22 | 中冶华天工程技术有限公司 | Parameter prediction method is discharged based on principal component analysis and the sewage treatment of shot and long term memory network |
Non-Patent Citations (1)
| Title |
|---|
| 刘琪琛等: ""基于Spark平台和并行随机森林回归算法的短期电力负荷预测"", 《电力建设》 * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110991689B (en) * | 2019-10-17 | 2022-10-04 | 国网河南省电力公司鹤壁供电公司 | Distributed photovoltaic power generation system short-term prediction method based on LSTM-Morlet model |
| CN110991689A (en) * | 2019-10-17 | 2020-04-10 | 国网河南省电力公司鹤壁供电公司 | Distributed photovoltaic power generation system short-term prediction method based on LSTM-Morlet model |
| CN110879802A (en) * | 2019-10-28 | 2020-03-13 | 同济大学 | Log pattern extraction and matching method |
| CN111178587A (en) * | 2019-12-06 | 2020-05-19 | 广东工业大学 | Spark framework-based short-term power load rapid prediction method |
| CN111178587B (en) * | 2019-12-06 | 2022-11-22 | 广东工业大学 | Spark framework-based short-term power load rapid prediction method |
| CN111091243A (en) * | 2019-12-13 | 2020-05-01 | 南京工程学院 | PCA-GM-based power load prediction method, system, computer-readable storage medium, and computing device |
| CN111680074A (en) * | 2019-12-31 | 2020-09-18 | 国网浙江省电力有限公司 | Clustering algorithm-based electric power collection load leakage point feature mining method |
| CN111680074B (en) * | 2019-12-31 | 2023-07-04 | 国网浙江省电力有限公司 | Clustering algorithm-based power acquisition load leakage point feature mining method |
| CN111311001A (en) * | 2020-02-17 | 2020-06-19 | 合肥工业大学 | Bi-LSTM network short-term load prediction method based on DBSCAN algorithm and feature selection |
| CN111311001B (en) * | 2020-02-17 | 2021-11-19 | 合肥工业大学 | Bi-LSTM network short-term load prediction method based on DBSCAN algorithm and feature selection |
| CN113450141A (en) * | 2021-06-09 | 2021-09-28 | 重庆锦禹云能源科技有限公司 | Intelligent prediction method and device based on electricity selling quantity characteristics of large-power customer groups |
| CN113450141B (en) * | 2021-06-09 | 2023-09-01 | 重庆锦禹云能源科技有限公司 | Intelligent prediction method and device based on electricity sales quantity characteristics of large power customer group |
| CN113988393A (en) * | 2021-10-21 | 2022-01-28 | 青岛联众芯云科技有限公司 | Energy internet load prediction method |
| CN114970938A (en) * | 2022-03-11 | 2022-08-30 | 武汉大学 | Self-adaptive residential load prediction method considering user privacy protection |
| CN114970938B (en) * | 2022-03-11 | 2024-05-07 | 武汉大学 | Self-adaptive house load prediction method considering user privacy protection |
| CN114881343A (en) * | 2022-05-18 | 2022-08-09 | 清华大学 | Short-term load prediction method and device of power system based on feature selection |
| CN114881343B (en) * | 2022-05-18 | 2023-11-14 | 清华大学 | Short-term load prediction method and device for power system based on feature selection |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110188919A (en) | A kind of load forecasting method based on shot and long term memory network | |
| Xuan et al. | Multi-model fusion short-term load forecasting based on random forest feature selection and hybrid neural network | |
| Zhang et al. | Short-term load forecasting based on big data technologies | |
| CN110969290B (en) | Runoff probability prediction method and system based on deep learning | |
| CN110503256B (en) | Short-term load prediction method and system based on big data technology | |
| CN112561156A (en) | Short-term power load prediction method based on user load mode classification | |
| CN108921358B (en) | Prediction method, prediction system and related device of power load characteristics | |
| CN107833153B (en) | Power grid load missing data completion method based on k-means clustering | |
| CN109190672A (en) | Operation of Electric Systems operating condition unsupervised clustering and device | |
| CN106980906B (en) | Spark-based Ftrl voltage prediction method | |
| CN106779219A (en) | A kind of electricity demand forecasting method and system | |
| CN114792156A (en) | Photovoltaic output power prediction method and system based on curve characteristic index clustering | |
| CN112308298B (en) | Multi-scenario performance index prediction method and system for semiconductor production line | |
| CN115204698A (en) | Real-time analysis method for power supply stability of low-voltage transformer area | |
| CN117458440A (en) | Method and system for predicting generated power load based on association feature fusion | |
| CN116720132A (en) | Power service identification system, method, device, medium and product | |
| CN113886454B (en) | LSTM-RBF-based cloud resource prediction method | |
| CN117578434B (en) | Power distribution network flexibility evaluation method and device considering flexible resource adjustability | |
| CN117318033B (en) | Power grid data management method and system combining data twinning | |
| CN117034780B (en) | Multi-scale sub-season precipitation prediction method based on deep learning | |
| CN108694475A (en) | Short-term time scale photovoltaic cell capable of generating power amount prediction technique based on mixed model | |
| CN117200204A (en) | Multi-element load prediction method, device and storage medium | |
| CN114781699B (en) | Reservoir water level prediction and early warning method based on improved particle swarm Conv1D-Attention optimization model | |
| CN116404637A (en) | Short-term load prediction method and device for electric power system | |
| CN115118015A (en) | Platform district power supply stability monitoring system based on fuse terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190830 |