CN113377755B - Integrity detection and missing repair method for electric power spot data - Google Patents
Integrity detection and missing repair method for electric power spot data Download PDFInfo
- Publication number
- CN113377755B CN113377755B CN202110699778.6A CN202110699778A CN113377755B CN 113377755 B CN113377755 B CN 113377755B CN 202110699778 A CN202110699778 A CN 202110699778A CN 113377755 B CN113377755 B CN 113377755B
- Authority
- CN
- China
- Prior art keywords
- integrity
- data
- vertex
- attribute
- integrity detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/06—Electricity, gas or water supply
Abstract
A method for detecting integrity and repairing missing of electric power spot data belongs to the technical field of data integrity detection and missing repair. The invention aims to solve the problem that the existing method can not evaluate the integrity of data generated in edge equipment, so that the quality of data repair can not be ensured, and further wrong power system decision is caused. The invention provides a transaction data integrity evaluation model based on a Bayesian network by considering the dependency of power transaction time series data. On the basis of the Bayesian network, the data integrity of the transaction data is defined according to different granularities, and an evaluation criterion of the transaction data integrity is designed. And then the Bi-GRU + Attention model of the invention is adopted or the data is effectively repaired directly according to the relationship between the vertexes and the probability table. The method can be applied to integrity detection and deletion repair of the electric power spot data.
Description
Technical Field
The invention belongs to the technical field of data integrity detection and deletion repair, and particularly relates to a method for detecting the integrity and repairing the deletion of electric power spot data.
Background
In recent years, various devices have been deployed to record data generated in energy internet transactions. Such transactional data contains a variety of potential insights into the industry and academia that have become one of the key bases for many tasks, such as data mining, data fusion, and data analysis.
Some data is of low quality in some data quality dimensions due to equipment failures or update delays. For example, certain values in the transaction data are missing, inconsistent or erroneous and do not reflect the true details of the transaction. This low quality data hinders knowledge discovery of various algorithms for different targets. In the data life cycle, data forgery, loss and tampering can affect the data quality, and then affect the data analysis result, eventually leading to wrong decisions. Therefore, in order to improve the data quality, we need to know the quality of the data first, which makes the data integrity evaluation and data repair based on the integrity evaluation result to be a problem to be solved urgently.
In the multiple transaction steps of the electric power spot market, complex dependency relationship exists among data. In complex transaction analysis, missing steps can be vital details that may lead to erroneous decisions. For example, power transaction data includes data describing wind power generation, photovoltaic power generation, electrical loads and electricity prices, etc., which come from various edge devices in a time series. Some of the values in such a time series may be lost in transmission, such as electricity prices, making it difficult to recover these values without knowing all relevant information for the transaction. Such missing values prevent algorithms from discovering causal relationships between prices and other factors, which makes it difficult to predict price trends in future transactions with existing algorithms. Therefore, in the edge calculation, the quality of the electric power transaction data should be guaranteed in order to perform better transaction analysis. The existing methods improve data quality by modifying classical dependency and probability methods, but the methods cannot evaluate the integrity of data generated in edge equipment, so that the quality of data repair cannot be guaranteed, and further wrong power system decision is caused.
Disclosure of Invention
The invention aims to solve the problems that the integrity of data generated in edge equipment cannot be evaluated by adopting the existing method, so that the quality of data repair cannot be ensured, and further wrong power system decision is caused, and provides an integrity detection and missing repair method for power spot data.
The technical scheme adopted by the invention for solving the technical problems is as follows:
based on one aspect of the invention, the method for detecting the integrity and repairing the missing of the electric power spot data specifically comprises the following steps:
constructing a directed graph G = { V, E }, where V is a vertex and E is an edge connecting the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical value integrity detection, weight integrity detection, path integrity detection, transaction integrity detection and data integrity detection;
the transaction integrity detection is as follows: detecting whether all paths meet path integrity; if all paths meet the path integrity, missing repair is not needed, otherwise, data integrity detection is continued;
the data integrity detection is as follows: calculating the percentage of the total number of paths meeting the path integrity in the total number of paths;
and restoring the electric power spot data according to the data integrity detection result and by combining the block chain and the Bi-GRU + Attention model.
Based on another aspect of the present invention, a method for integrity detection and loss repair of power spot data specifically includes the following steps:
constructing a directed graph G = { V, E, F }, wherein V is a vertex, E is an edge connecting the vertices, and F represents the relationship between the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical integrity detection, weight integrity detection, path integrity detection and transaction integrity detection;
if the transaction integrity detection result is that the transaction is complete, the electric power spot data does not need to be repaired, otherwise, the electric power spot data is repaired by adopting the following repairing mode:
according to the relation F between the vertexes, data restoration is carried out on the vertexes which do not meet the numerical integrity;
and repairing the weight of the vertex according to the probability table.
The invention has the beneficial effects that: the invention provides an integrity detection and deletion repair method for electric power spot data, and provides a transaction data integrity evaluation model based on a Bayesian network by considering the dependency of electric power transaction time sequence data. On the basis of the Bayesian network, the data integrity of the transaction data is defined according to different granularities, and an evaluation criterion of the transaction data integrity is designed. And then the Bi-GRU + Attention model of the combined block chain is adopted or the data is effectively repaired directly according to the relation between the vertexes and the probability table.
Experimental verification shows that the Bayesian network-based data integrity evaluation model provided by the invention can fully reflect the integrity of power transaction time sequence data, and adopts a Bi-GRU + Attention network or carries out data restoration according to the relation between vertexes and a probability table, so that the restoration precision and efficiency of data integrity are improved, the quality of data restoration is ensured, and a correct power system decision is conveniently made.
Drawings
FIG. 1 is a Bayesian network diagram of a transactional data model;
in the figure, bidding Day is a Bidding Day, running Day is a Running Day;
FIG. 2 is a path diagram on a transaction data integrity diagram;
FIG. 3 is a diagram of a Bi-GRU + Attention model structure;
FIG. 4 is a diagram of a GRU structure;
FIG. 5 is a structural view of a Bi-GRU;
FIG. 6 is a schematic illustration of data integrity for power spot data on different dates;
FIG. 7 is a graph comparing repair data integrity of GRUs with power data of the present invention;
in the figure, target represents original data, bi-GRU + Attention represents data repaired by a Bi-GRU + Attention model adopting the method of the invention, and GRU represents data repaired by a GRU model;
FIG. 8 is a graphical illustration of repair values for different dates;
FIG. 9 is a graph comparing the efficiency of the consensus mechanism;
FIG. 10 is a graph of the impact of data quality on the efficiency of the consensus mechanism.
Detailed Description
The first specific implementation way is as follows: the method for detecting integrity and repairing missing of electric power spot data specifically comprises the following steps:
constructing a directed graph G = { V, E }, wherein V is a vertex and E is an edge connecting the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical value integrity detection, weight integrity detection, path integrity detection, transaction integrity detection and data integrity detection;
the transaction integrity detection is as follows: detecting whether all paths meet the path integrity; if all paths meet the path integrity, missing repair is not needed, otherwise, data integrity detection is continued;
the data integrity detection is as follows: calculating the percentage of the total number of paths meeting the path integrity in the total number of paths;
and restoring the power spot data according to the data integrity detection result and by combining the block chain and the Bi-GRU + Attention model.
The block chain and attention mechanism based data evaluation and repair algorithm (CBBA) of the present embodiment is shown in table 1:
TABLE 1
In Table 1, the CBBA first calls the algorithm of Table 2 to evaluate the data integrity of the data set. Then, for the missing value, the Bi-GRU + Attention network is used to learn the power spot data to predict the missing value. For each repair, N rep Will be repaired and SAT and MSE are calculated by equation (10) and equation (14). If the SAT is too small, repair will stop and repair data is returned. Based on different granularities of data integrity, by formula
Calculate data integrity, | f trading (T j ) L represents the total number of complete paths and n represents the total number of all paths.
Table 2 data integrity assessment
Degree of satisfaction of repair
After the data with low integrity is repaired according to the repair rule, calculating the satisfaction SAT of data repair according to the integrity and repair time of the repaired data:
SAT＝αD repair -βT (1)
0＜α＜1 (2)
0＜β＜1 (3)
α+β＝1 (4)
where α and β are weight coefficients of integrity and repair time, respectively. Equation (1) indicates that the satisfaction SAT is represented by a difference between the data integrity multiplier and the repair time multiplier of the repaired data. Equations (2) - (4) indicate that the weight coefficients α, β are between 0 and 1, and the sum of α and β is 1. The higher the SAT value, the better the repair algorithm.
Repair time
The repair time consists of two parts; one is the time t at which we fetch data from the chain d The data quantity and the bandwidth size in the data transmission process are determined; second, time t for local repair l The algorithm running on the raw data set can yield:
t repair ＝t d +t l (6)
wherein the content of the first and second substances,
C i respectively representing the amount of data and the bandwidth in the ith data transmission link. Time of day
By volume of data
Sum bandwidth C i Is expressed as shown in equation (5). Total repair time t repair By
And
the joint decision is shown in equation (6).
To calculate the repair satisfaction, we apply equation (7) to the repair time t repair Normalization processing is performed, and the result is mapped to the (0,1) section.
Error repair
The error of data repair is calculated using the mean square error:
where X' represents the average of two values before and after the missing value, and Y represents the predicted data.
The second embodiment is as follows: the difference between this embodiment and the first embodiment is that the numerical integrity detection includes attribute integrity detection and tuple integrity detection;
the specific process of detecting the integrity of the attribute comprises the following steps: will be directed to graph GAs an attribute, all attributes including: predicted load L predicted by actual load on day of bidding day 1 Predicted photovoltaic power generation amount E predicted by actual photovoltaic power generation amount on day of bidding day 1 Predicted wind power generation amount E predicted by actual wind power generation amount on day of bidding day 2 Predicted electricity price P predicted by electricity price on day of bidding day 1 Bid winning power consumption E on bid date 3 Winning power generation amount E on bidding day 4 Actual power generation amount E of producer 5 Actual load L 2 A market competitive price P given by market clearing after all market participants submit bids 2 And a total price P provided by the producer 3 ；
If the attribute is missing in the electric power spot data, the attribute is indicated to be incomplete, otherwise, the attribute is complete;
when the attributes are incomplete, directly considering that the vertexes corresponding to the missing attributes do not meet the numerical integrity, and then carrying out tuple integrity detection on the attributes which are not missing;
when the attribute is complete, directly carrying out tuple integrity detection;
the specific process of tuple integrity detection is as follows: for any attribute, if the attribute has data at each moment in the electric power spot data, the attribute meets the integrity of the tuple, otherwise, the attribute does not meet the integrity of the tuple;
the vertex corresponding to the attribute which meets the integrity of the tuple is complete in numerical value, and the vertex corresponding to the attribute which does not meet the integrity of the tuple is not complete in numerical value;
the specific process of the weight integrity detection is as follows:
in the directed graph G, for a vertex, if the weight associated with the vertex is not missing and the sum of the weights associated with the vertex is equal to 1, the vertex is weighted completely, otherwise the vertex is weighted incompletely.
The specific process of path integrity detection is as follows:
for any path, if the path satisfies the path integrity, the path should satisfy the following conditions at the same time:
1) Each vertex in the path is weighted completely;
2) Each vertex in the path is numerically complete.
Other steps and parameters are the same as those in the first embodiment.
The third concrete implementation mode: this embodiment will be described with reference to fig. 3 and 5. The difference between this embodiment and the first or second embodiment is that the Bi-GRU + Attention model is composed of an input layer, a Bi-GRU layer, an Attention layer, a Dense layer, a softmax layer, and an output layer.
First input vector { x 1 ,x 2 ,......,x n Processing the sequence vector into a sequence vector mode, calculating an output sequence vector through a Bi-GRU layer, and extracting deep features of the input sequence vector through the Bi-GRU layer. Then, an Attention probability value that should be generated for each data vector is generated through the Attention layer. The final Attention value is obtained by the accumulated sum of the Attention probability value generated by the Attention layer and the product of the hidden layer state, and then the softmax function is used for normalization operation. As shown in equations (9) - (12).
1)Bi-GRU
Based on LSTM, GRU combines forgetting gate and input gate into an update gate, and mixes the cell state and the hidden state. As shown in figure 4 of the drawings,
z t ＝σ(W z ·[h t-1 ,x t ]) (9)
r t ＝σ(W r ·[h t-1 ,x t ]) (10)
in the formula (9), z t Indicates the output of the update gate at time t, W z Show moreWeight of new door, h t-1 Output, x, representing the hidden state at time t-1 t Representing data input at the time t, wherein sigma represents a sigmoid function; in the formula (10), r t Denotes the output of the reset gate, σ denotes the sigmoid function, W r Representing the weight of the reset gate; in the formula (11), the first and second groups,
an output representing a hidden state at time t, tanh representing a tanh function, W representing a weight of memory information, r t A specific gravity indicating the output of the reset gate at time t, i.e., the forgotten information; in the formula (12), h t Output representing hidden state at time t, h t-1 Output representing hidden state at time t-1, z t Indicating the output of the update gate at time t, i.e., the weight of the past information being retained.
First, the update gate z is calculated according to the formula (9) and the formula (10) t And a reset gate r t Then by equation (11) and resetting the gate r t Is calculated to obtain
Finally, the door z is updated t Act on at
(1-z t ) Act at h t-1 Adding to obtain the final state output h t . The bidirectional GRU processes the input sequence in forward and backward order, respectively, so that the output sequence at each time step contains past and future context information in the input sequence at the current time.
2) Mechanism of Attention
Combining the attention mechanism with the GRU makes the model more concerned about important information in the time series data. The attention mechanism is given by equation (13) -equation (15):
u t ＝tanh(W d h t +b l ) (13)
wherein h is t Is the output of the previous Bi-GRU neural network layer, W d Represents a weight coefficient, b l Denotes the offset coefficient u d Indicating the initialized attention matrix, u t Denotes the attention vector, a t The similarity coefficient at time t is represented, and a represents a context vector.
Other steps and parameters are the same as those of the seventh or eighth embodiment.
The fourth concrete implementation mode: the difference between this embodiment and the first to third embodiments is that, according to the data integrity detection result, the block chain and the Bi-GRU + Attention model are combined to repair the power spot data, and the specific process is as follows:
step S1, a power spot data owner initiates a transaction request, and after the identity of the data owner is verified by a block chain, data at each moment in the power spot data are respectively sent to corresponding nodes (the data in each node are different);
selecting a node corresponding to the data at the moment with the highest data integrity through a consensus mechanism, and taking the selected node as a leader node (the leader node has the right to write the block into the block chain);
s2, packaging time data with the highest data integrity into blocks by a leader node, broadcasting the blocks to other nodes for verification, if the verification is passed (if more than half of the nodes vote to pass, the verification is passed), adding the blocks to a block chain, and if the verification is not passed, repairing the time data corresponding to the leader node by using a Bi-GRU + Attention model deployed on the leader node, wherein the blocks are invalid;
and S3, reselecting a leader node from the time data which are not added to the block chain through a consensus mechanism, and repeating the process of the step S2 until all the time data are added to the block chain.
The proposed consensus mechanism based on data quality can reduce the block forming time and improve the efficiency of transaction verification.
The consensus mechanism functions to bring the ledgers of all nodes into agreement. The POW consensus mechanism is that all nodes solve a common problem, the node that first solves the problem will generate a block. All nodes solve the same problem, and extremely high computational cost is consumed. The PODQ consensus mechanism proposed by the inventor is that each node solves the problem of the node, and the difficulty of the problem is inconsistent. The PODQ consensus process is performed on the basis of data integrity evaluations. Each node agrees according to its own data integrity.
When the consensus process starts, based on the data integrity, then the integrity D completeness The highest node pushes the consensus process between the participating nodes. Selecting a data integrity D completeness Taking the selected node as a leader node according to the node corresponding to the highest time data; and the leader node packs the data at the time with the highest data integrity into blocks and broadcasts the blocks to other nodes for verification, if the verification is passed (when more than half of the nodes vote to pass, the verification is passed), the blocks are added to the block chain, and all other nodes are updated into corresponding block chains.
Other steps and parameters are the same as those in one of the first to third embodiments.
Fifth embodiment this embodiment will be described with reference to fig. 1. The method for detecting integrity and repairing missing of the electric power spot data in the embodiment specifically comprises the following steps:
constructing a directed graph G = { V, E, F } of the Bayesian network, wherein V is a vertex, E is an edge connecting the vertices, and F represents the relationship between the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical integrity detection, weight integrity detection, path integrity detection and transaction integrity detection;
if the transaction integrity detection result is that the transaction is complete, the electric power spot data does not need to be repaired, otherwise, the electric power spot data is repaired by adopting the following repairing mode:
according to the relation F between the vertex and the vertex, data restoration is carried out on the vertex which does not meet the numerical integrity;
and repairing the weight of the vertex according to a probability table given by an expert system.
The sixth specific implementation mode: the difference between this embodiment and the fifth embodiment is that the numerical integrity detection includes attribute integrity detection and tuple integrity detection;
the specific process of detecting the integrity of the attribute comprises the following steps: taking each vertex in the directed graph G as an attribute, wherein all the attributes comprise: predicted load L predicted by actual load on day of bidding day 1 Predicted photovoltaic power generation amount E predicted by actual photovoltaic power generation amount on day of bidding day 1 Predicted wind power generation amount E predicted by actual wind power generation amount on day of bidding day 2 Predicted electricity price P predicted by electricity price on day of bidding day 1 Bid amount E on bid day 3 Winning power generation amount E on bidding day 4 Actual power generation amount E of producer 5 Actual load L 2 A market competitive price P given by market clearing after all market participants submit bids 2 And a total price P provided by the producer 3 ；
If the attribute is missing in the electric power spot data, the attribute is indicated to be incomplete, otherwise, the attribute is complete;
when the attributes are incomplete, directly considering that the vertexes corresponding to the missing attributes do not meet the numerical integrity, and then carrying out tuple integrity detection on the attributes which are not missing;
when the attribute is complete, directly carrying out tuple integrity detection;
the specific process of tuple integrity detection is as follows: for any attribute, if the attribute has data at each moment in the power spot data, the attribute meets tuple integrity, otherwise, the attribute does not meet tuple integrity;
the vertex corresponding to the attribute satisfying the tuple integrity is complete in value, and the vertex corresponding to the attribute not satisfying the tuple integrity is not complete in value.
1) Attribute integrity is denoted A c This means whether the attribute values in the dataset are complete. For the attributes in table 3, such as (photovoltaic power generation, wind power generation, electrical load, electricity price), the absence of any one or more of them indicates that the attribute is incomplete.
2) Tuple integrity is represented as T c This means whether a tuple is complete in the dataset. For any one tuple in table 3, such as (01/01, 01 00,0.5371,0.5882,0.9371, 0.56), the absence of any one or more of them indicates that the tuple is missing.
TABLE 3
The other steps and parameters are the same as those in the fifth embodiment.
The seventh concrete implementation mode: this embodiment will be described with reference to fig. 2. The difference between this embodiment and one of the fifth to sixth embodiments is that the specific process of detecting the integrity of the weight is as follows:
in the directed graph G, for a vertex, if the weight associated with the vertex is not missing and the sum of the weights associated with the vertex is equal to 1, the vertex is weighted completely, otherwise the vertex is weighted incompletely.
As shown in equation (16), the vertex is not weight-complete:
p(P 2 |L 1 )+p(P 2 |E 1 )+p(P 2 |E 2 )+p(P 2 |P 1 )≠1 (16)
wherein, P (P) 2 |L 1 ) Represents a point L 1 Point of direction P 2 Probability of time, i.e. load L by prediction 1 Calculating a bid price P 2 Probability of time. For vertex P 2 In the path diagram, the vertex L is 1 、E 1 、E 2 And P 1 Are all directed to the vertexP 2 Therefore, with the vertex P 2 The associated weights refer to: point L 1 Point of direction P 2 Probability of time, point E 1 Point of direction P 2 Probability of time, point E 2 Point of orientation P 2 Probability of time, point P 1 Point of direction P 2 Probability of time, and vertex P 2 The associated weight sum refers to P (P) 2 |L 1 )、p(P 2 |E 1 )、p(P 2 |E 2 ) And P (P) 2 |P 1 ) The sum of (1).
The other steps and parameters are the same as those of the fifth or sixth embodiment.
The specific implementation mode is eight: the difference between this embodiment and one of the fifth to seventh embodiments is that the specific process of detecting the integrity of the path is as follows:
for any path, if the path satisfies the path integrity, the path should satisfy the following conditions at the same time:
1) Each vertex in the path is weighted completely;
2) Each vertex in the path is numerically complete.
Other steps and parameters are the same as those of one of the fifth to seventh embodiments.
The specific implementation method nine: the difference between this embodiment and one of the fifth to eighth embodiments is that the specific process of the transaction integrity detection is as follows: and detecting whether each path meets the path integrity, if so, the electric power spot data meets the transaction integrity, otherwise, the electric power spot data does not meet the transaction integrity.
Other steps and parameters are the same as those of the fifth to eighth embodiments.
The specific implementation mode is ten: this embodiment is different from one of the fifth to ninth embodiments in that the relationship F between the vertex and the vertex includes:
1)E 5 ＝E 3 ＝E 4 ＝L 1 ；
2) According to L 1 、E 1 、E 2 And P 1 Calculating P 2 (ii) a From L 1 、E 1 、E 2 、P 1 Obtaining the price P of the day 2 ；
3)
Wherein, P + Is a positive unbalanced adjustment price, P - Is a negative imbalance adjustment price.
The double priced electricity market refers to a situation where an imbalance is solved with two different prices. Typically, if a producer produces more energy than it sells, the excess energy is sold at a spot price that is less than the market time unit. Conversely, when a producer produces less energy than sales, it must purchase the energy it lacks at a price higher than the spot price per market time unit. In such a market, the price offered by the producer is P for a certain unit of market time 3 . A positive imbalance price is typically lower than or equal to the day-ahead market bid price, so any remaining energy compared to the bid is rewarded at a price lower than the spot price. Likewise, a negatively unbalanced price is typically higher than or equal to the day-ahead market bid price, and thus any missing energy will be penalized at a price higher than the day-ahead market bid price.
Other steps and parameters are the same as those in one of the fifth to ninth embodiments.
Experimental part
A. Experimental setup
In order to verify the performance of the proposed bayesian network based data integrity evaluation and repair. The public data sets are sorted, and time sequence data sets required by power transaction are constructed. The data set is composed of a prediction load, an actual power load, photovoltaic power generation capacity, wind power generation capacity, bid-winning power consumption, actual power generation capacity, quotation, a bidding price and actual pricing. Wherein the predicted load is predicted from the actual load of the previous day. Probability tables of a data integrity evaluation model based on a bayesian network are given by an expert system. The raw data we present are complete. In order to meet the requirements of our experiments, we perform random data missing processing based on Gaussian distribution on the data. We performed performance analysis of the proposed model using the processed missing data set.
We have realized the web service based on block chain in Ubuntu system, verify the efficiency of the data quality consensus mechanism (PODQ) put forward. All experiments were performed using Python3.7 on an IntelCorei7-9750HCPU2.6GHz, 16GB memory and Windows1064 bit PC.
B. Performance evaluation
1) Data integrity assessment
We calculated the completeness of the data set according to the proposed bayesian network based data integrity evaluation model, evaluation rules and equation (16). As shown in FIG. 6, the missing data accounts for 8.9% of the total data set, and the data with integrity of 2/7, 3/7, 5/7 and 6/7 account for 0.46%, 10.65%, 3.01% and 85.88% of the missing data, respectively, indicating that our data integrity model can capture the missing values in the power transaction data.
2) Effectiveness of data repair: as shown in fig. 7. We use a Bi-GRU + Attention based network to repair data. First, we fill the missing value with the average of the values of the missing data at the previous and next time instants. Time series data predictions were then made using Bi-GRU + Attention to determine the value of the deletion. When the data repair satisfaction is greater than r, we will stop the data repair. Meanwhile, when the error between the predicted value and the average value is satisfied, the data will be restored. We can see from fig. 8 that the integrity of the repaired data has improved significantly.
3) Efficiency of consensus mechanism: in order to verify the efficiency of the proposed data quality consensus mechanism and to evaluate the impact of data quality repair on the data quality consensus mechanism. After the integrity of the data is evaluated, the low integrity data is traced according to the traceable characteristic of the block chain. And then, uploading the repaired node data to the block chain again, forming a block through a consensus mechanism, and recording the transaction in the block chain. As shown in fig. 9, we compare the proposed data quality consensus mechanism with the pow consensus mechanism, where we set the difficulty of pow to the lowest. In the figure, the dotted line represents the block formation time using the pow consensus mechanism, and the solid line represents the block formation time using the data quality consensus mechanism. Since the curves have a consistent trend, the travel time of only the first 100 blocks is shown here for the sake of clarity of the experimental plot. We can clearly see that the proposed data quality consensus mechanism (PODQ) has a shorter chunk formation time and higher efficiency. In addition, fig. 10 is a comparison of block formation times of low quality repair data and repair data. The dotted line is the block forming time of the low-quality data, and the solid line is the block forming time after data recovery. In general, our proposed data quality consensus mechanism improves the formation time of the block and the higher the data quality the more efficient the transaction is achieved.
The above-described calculation examples of the present invention are merely to explain the calculation model and the calculation flow of the present invention in detail, and are not intended to limit the embodiments of the present invention. It will be apparent to those skilled in the art that other variations and modifications of the present invention can be made based on the above description, and it is not intended to be exhaustive or to limit the invention to the precise form disclosed, and all such modifications and variations are possible and contemplated as falling within the scope of the invention.
Claims (3)
1. The integrity detection and missing repair method of the electric power spot data is characterized by comprising the following steps:
constructing a directed graph G = { V, E }, wherein V is a vertex and E is an edge connecting the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical value integrity detection, weight integrity detection, path integrity detection, transaction integrity detection and data integrity detection;
the transaction integrity detection is as follows: detecting whether all paths meet path integrity; if all paths meet the path integrity, missing repair is not needed, otherwise, data integrity detection is continued;
the data integrity detection is as follows: calculating the percentage of the total number of paths meeting the path integrity in the total number of paths;
restoring the electric power spot data according to the data integrity detection result and by combining a block chain and a Bi-GRU + Attention model;
the numerical integrity check comprises an attribute integrity check and a tuple integrity check;
the specific process of detecting the integrity of the attribute comprises the following steps: taking each vertex in the directed graph G as an attribute, wherein all the attributes comprise: predicted load L predicted by actual load on day of bidding day 1 Predicted photovoltaic power generation amount E predicted by actual photovoltaic power generation amount on day of bidding day 1 Predicted wind power generation amount E predicted by actual wind power generation amount on day of bidding day 2 Predicted electricity price P predicted by electricity price on day of bidding day 1 Bid winning power consumption E on bid date 3 Winning power generation amount E on bidding day 4 Actual power generation amount E of producer 5 Actual load L 2 A market competitive price P given by market clearing after all market participants submit bids 2 And a total price P provided by the producer 3 ；
If the attribute is missing in the electric power spot data, the attribute is indicated to be incomplete, otherwise, the attribute is complete;
when the attributes are incomplete, directly considering that the vertexes corresponding to the missing attributes do not meet the numerical integrity, and then carrying out tuple integrity detection on the attributes which are not missing;
when the attribute is complete, directly carrying out tuple integrity detection;
the specific process of tuple integrity detection is as follows: for any attribute, if the attribute has data at each moment in the electric power spot data, the attribute meets the integrity of the tuple, otherwise, the attribute does not meet the integrity of the tuple;
the vertex corresponding to the attribute which meets the integrity of the tuple is complete in numerical value, and the vertex corresponding to the attribute which does not meet the integrity of the tuple is not complete in numerical value;
the specific process of the weight integrity detection is as follows:
in the directed graph G, for a vertex, if the weight associated with the vertex is not missing and the sum of the weights associated with the vertex is equal to 1, the vertex is completely weighted, otherwise the vertex is not completely weighted;
the specific process of the path integrity detection is as follows:
for any path, if the path satisfies the path integrity, the path should satisfy the following conditions at the same time:
1) Each vertex in the path is weighted completely;
2) Each vertex in the path is numerically complete;
the method is characterized in that the electric power spot data is repaired according to the data integrity detection result and by combining a block chain and a Bi-GRU + Attention model, and the specific process is as follows:
s1, a power spot data owner initiates a transaction request, and after the identity of the data owner is verified by a block chain, data at each moment in the power spot data are respectively sent to corresponding nodes;
selecting a node corresponding to the data at the moment with the highest data integrity through a consensus mechanism, and taking the selected node as a leader node;
s2, packaging the time data with the highest data integrity into a block by the leader node, broadcasting the block to other nodes for verification, adding the block to a block chain if the verification is passed, and restoring the time data corresponding to the leader node by using a Bi-GRU + Attention model deployed on the leader node if the verification is not passed;
and S3, reselecting a leader node from the time data which are not added to the block chain through a consensus mechanism, and repeating the process of the step S2 until all the time data are added to the block chain.
2. The method of claim 1, wherein the Bi-GRU + Attention model is composed of an input layer, a Bi-GRU layer, an Attention layer, a sense layer, a softmax layer, and an output layer.
3. The integrity detection and missing repair method of the electric power spot data is characterized by comprising the following steps:
constructing a directed graph G = { V, E, F }, wherein V is a vertex, E is an edge connecting the vertices, and F represents the relationship between the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical value integrity detection, weight integrity detection, path integrity detection and transaction integrity detection;
if the transaction integrity detection result is that the transaction is complete, the electric power spot data does not need to be repaired, otherwise, the electric power spot data is repaired by adopting the following repairing mode:
according to the relation F between the vertex and the vertex, data restoration is carried out on the vertex which does not meet the numerical integrity;
repairing the weight of the vertex according to the probability table;
the numerical integrity detection comprises attribute integrity detection and tuple integrity detection;
the specific process of detecting the integrity of the attribute comprises the following steps: taking each vertex in the directed graph G as an attribute, wherein all the attributes comprise: predicted load L predicted by actual load on day of bidding day 1 Predicted photovoltaic power generation amount E predicted by actual photovoltaic power generation amount on day of bidding day 1 Predicted wind power generation amount E predicted by actual wind power generation amount on day of bidding day 2 Predicted electricity price P predicted by electricity price on day of bidding day 1 Bid amount E on bid day 3 Winning power generation amount E on bidding day 4 Actual power generation amount E of producer 5 Actual load L 2 After all market participants submit bids, the market competitive price P is given by market clearing 2 And a total price P provided by the producer 3 ；
If the attribute is missing in the electric power spot data, the attribute is indicated to be incomplete, otherwise, the attribute is complete;
when the attributes are incomplete, directly considering that the vertexes corresponding to the missing attributes do not meet the numerical integrity, and then carrying out tuple integrity detection on the attributes which are not missing;
when the attribute is complete, directly carrying out tuple integrity detection;
the specific process of tuple integrity detection is as follows: for any attribute, if the attribute has data at each moment in the electric power spot data, the attribute meets the integrity of the tuple, otherwise, the attribute does not meet the integrity of the tuple;
the vertex corresponding to the attribute which meets the integrity of the tuple is complete in numerical value, and the vertex corresponding to the attribute which does not meet the integrity of the tuple is not complete in numerical value;
the specific process of the weight integrity detection is as follows:
in the directed graph G, for a vertex, if the weight associated with the vertex is not missing and the sum of the weights associated with the vertex is equal to 1, the vertex is completely weighted, otherwise the vertex is not completely weighted;
the specific process of path integrity detection is as follows:
for any path, if the path satisfies the path integrity, the path should satisfy the following conditions at the same time:
1) Each vertex in the path is weighted completely;
2) Each vertex in the path is numerically complete;
the transaction integrity detection comprises the following specific processes: detecting whether each path meets the path integrity, if so, the electric power spot data meets the transaction integrity, otherwise, the electric power spot data does not meet the transaction integrity;
the vertex-to-vertex relationship F includes:
1)E 5 ＝E 3 ＝E 4 ＝L 1 ；
2) According to L 1 、E 1 、E 2 And P 1 Calculating P 2 ；
3)
Wherein, P + Is a positive imbalance of the adjusted price, P - Is a negative imbalance adjustment price.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110699778.6A CN113377755B (en) | 2021-06-23 | 2021-06-23 | Integrity detection and missing repair method for electric power spot data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110699778.6A CN113377755B (en) | 2021-06-23 | 2021-06-23 | Integrity detection and missing repair method for electric power spot data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113377755A CN113377755A (en) | 2021-09-10 |
| CN113377755B true CN113377755B (en) | 2022-12-16 |
Family
ID=77578829
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110699778.6A Active CN113377755B (en) | 2021-06-23 | 2021-06-23 | Integrity detection and missing repair method for electric power spot data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113377755B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109445972A (en) * | 2018-09-21 | 2019-03-08 | 深圳供电局有限公司 | Data recovery method, device, equipment and storage medium |
| CN110599163A (en) * | 2019-08-20 | 2019-12-20 | 江苏大学 | Transaction record outsourcing method facing block chain transaction supervision |
| CN111160880A (en) * | 2019-12-10 | 2020-05-15 | 国网天津市电力公司 | Block chain-based industrial user demand response-oriented transaction settlement control method |
| CN111523785A (en) * | 2020-04-16 | 2020-08-11 | 三峡大学 | Power system dynamic security assessment method based on generation countermeasure network |
| CN111709672A (en) * | 2020-07-20 | 2020-09-25 | 国网黑龙江省电力有限公司 | Virtual power plant economic dispatching method based on scene and deep reinforcement learning |
| CN112001744A (en) * | 2020-07-28 | 2020-11-27 | 安徽电力交易中心有限公司 | Power generator auxiliary quotation system and method based on prospect theory in electric power spot market |
| CN112785429A (en) * | 2021-01-08 | 2021-05-11 | 西北工业大学 | Local area multi-microgrid power transaction pairing method based on block chain technology |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080005194A1 (en) * | 2006-05-05 | 2008-01-03 | Lockheed Martin Corporation | System and method for immutably cataloging and storing electronic assets in a large scale computer system |
| US10659473B2 (en) * | 2017-11-09 | 2020-05-19 | Nokia Solutions And Networks Oy | Method and apparatus for blockchain powered integrity protection system |
| CN111709779A (en) * | 2020-06-01 | 2020-09-25 | 广东电网有限责任公司 | Trading variety optimization system for electric power spot market |
| CN112381263B (en) * | 2020-09-23 | 2023-07-14 | 四川大学 | Block chain-based distributed data storage multi-microgrid pre-day robust electric energy transaction method |
| CN112434343B (en) * | 2020-11-25 | 2024-03-01 | 江西理工大学 | Virtual power plant safety scheduling and trading method based on dual block chain technology |
-
2021
- 2021-06-23 CN CN202110699778.6A patent/CN113377755B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109445972A (en) * | 2018-09-21 | 2019-03-08 | 深圳供电局有限公司 | Data recovery method, device, equipment and storage medium |
| CN110599163A (en) * | 2019-08-20 | 2019-12-20 | 江苏大学 | Transaction record outsourcing method facing block chain transaction supervision |
| CN111160880A (en) * | 2019-12-10 | 2020-05-15 | 国网天津市电力公司 | Block chain-based industrial user demand response-oriented transaction settlement control method |
| CN111523785A (en) * | 2020-04-16 | 2020-08-11 | 三峡大学 | Power system dynamic security assessment method based on generation countermeasure network |
| CN111709672A (en) * | 2020-07-20 | 2020-09-25 | 国网黑龙江省电力有限公司 | Virtual power plant economic dispatching method based on scene and deep reinforcement learning |
| CN112001744A (en) * | 2020-07-28 | 2020-11-27 | 安徽电力交易中心有限公司 | Power generator auxiliary quotation system and method based on prospect theory in electric power spot market |
| CN112785429A (en) * | 2021-01-08 | 2021-05-11 | 西北工业大学 | Local area multi-microgrid power transaction pairing method based on block chain technology |
Non-Patent Citations (3)
| Title |
|---|
| 基于CNN和LSTM联合预测并修正的电量缺失数据预测;郭蕴颖 等;《计算机系统应用》;20200815;第29卷(第08期);192-198 * |
| 基于规则的电力数据指标检查系统设计与实现;荀华 等;《江西电力》;20200425;第44卷(第04期);15-19 * |
| 省级电力现货市场技术支持系统技术架构设计;蔡宇 等;《电力系统自动化》;20210325;第45卷(第06期);176-185 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113377755A (en) | 2021-09-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR100766149B1 (en) | Methods and systems for efficiently sampling portfolios for optimal underwriting | |
| CN110298574B (en) | Convolutional neural network-based electricity user payment risk rating method | |
| JP2004506959A (en) | Evaluation value prediction model when information is missing | |
| JP2004500641A (en) | Method and system for automatically estimating credit score evaluation value | |
| JP2004500646A (en) | Competitive bidding revenue simulation method and system | |
| AU2585401A (en) | Rapid valuation of portfolios of assets such as financial instruments | |
| JP2004500642A (en) | Methods and systems for assessing cash flow recovery and risk | |
| Giahi et al. | Design optimization for resilience for risk-averse firms | |
| CN113868537B (en) | Recommendation method based on multi-behavior session graph fusion | |
| Kell et al. | Long-term electricity market agent based model validation using genetic algorithm based optimization | |
| CN113377755B (en) | Integrity detection and missing repair method for electric power spot data | |
| Majadi et al. | Real-time detection of shill bidding in online auctions: A literature review | |
| Camelia et al. | A Computational Grey Based Model for Companies Risk Forecasting. | |
| US20230252285A1 (en) | Spatio-temporal graph neural network for time series prediction | |
| Emami et al. | A conceptual analysis of equilibrium bidding strategy in a combined oligopoly and oligopsony wholesale electricity market | |
| Król et al. | Fuzzy system model to assist with real estate appraisals | |
| CN112581177B (en) | Marketing prediction method combining automatic feature engineering and residual neural network | |
| CN114004530A (en) | Enterprise power credit score modeling method and system based on sequencing support vector machine | |
| CN111915082B (en) | Prediction method, prediction device, storage medium and prediction equipment | |
| Kaneko et al. | Sensitivity analysis of factors relevant to extreme imbalance between procurement plans and actual demand: Case study of the Japanese electricity market | |
| Orłowski et al. | Conceptual fuzzy model of the Polish Internet mortgage market | |
| Seo et al. | A Unified Model for Bid Landscape Forecasting in the Mixed Auction Types of Real-Time Bidding | |
| US20240112150A1 (en) | Systems and methods to facilitate decision making for utility networks | |
| CN114154277B (en) | Gathering and transportation system layout optimization method, device and equipment | |
| Yasak et al. | A Forecasting with Multilayer Perceptron Algorithm the Occupancy Rate of Accommodation Establishments in Turkey |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |