CN113377755A - Integrity detection and missing repair method for electric power spot data - Google Patents
Integrity detection and missing repair method for electric power spot data Download PDFInfo
- Publication number
- CN113377755A CN113377755A CN202110699778.6A CN202110699778A CN113377755A CN 113377755 A CN113377755 A CN 113377755A CN 202110699778 A CN202110699778 A CN 202110699778A CN 113377755 A CN113377755 A CN 113377755A
- Authority
- CN
- China
- Prior art keywords
- integrity
- data
- integrity detection
- vertex
- attribute
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 104
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000008439 repair process Effects 0.000 title claims abstract description 53
- 238000010248 power generation Methods 0.000 claims description 31
- 230000007246 mechanism Effects 0.000 claims description 25
- 230000008569 process Effects 0.000 claims description 24
- 230000005611 electricity Effects 0.000 claims description 12
- 238000012795 verification Methods 0.000 claims description 12
- 230000002860 competitive effect Effects 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 abstract description 7
- 238000012217 deletion Methods 0.000 abstract description 5
- 230000037430 deletion Effects 0.000 abstract description 5
- 238000013210 evaluation model Methods 0.000 abstract description 4
- 235000019580 granularity Nutrition 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 6
- 230000015572 biosynthetic process Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- 241000288105 Grus Species 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000001364 causal effect Effects 0.000 description 1
- 238000011157 data evaluation Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 239000011541 reaction mixture Substances 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- Development Economics (AREA)
- Marketing (AREA)
- Tourism & Hospitality (AREA)
- Quality & Reliability (AREA)
- Entrepreneurship & Innovation (AREA)
- Game Theory and Decision Science (AREA)
- Databases & Information Systems (AREA)
- Educational Administration (AREA)
- Health & Medical Sciences (AREA)
- Operations Research (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
Abstract
A method for detecting integrity and repairing missing of electric power spot data belongs to the technical field of data integrity detection and missing repair. The invention aims to solve the problem that the existing method can not evaluate the integrity of data generated in edge equipment, so that the quality of data repair can not be ensured, and further wrong power system decision is caused. The invention provides a transaction data integrity evaluation model based on a Bayesian network by considering the dependency of power transaction time series data. On the basis of the Bayesian network, the data integrity of the transaction data is defined according to different granularities, and an evaluation criterion of the transaction data integrity is designed. And then the Bi-GRU + Attention model of the invention is adopted or the data is effectively repaired directly according to the relationship between the vertexes and the probability table. The method can be applied to integrity detection and deletion repair of the electric power spot data.
Description
Technical Field
The invention belongs to the technical field of data integrity detection and deletion repair, and particularly relates to a method for detecting the integrity and repairing the deletion of electric power spot data.
Background
In recent years, various devices have been deployed to record data generated in energy internet transactions. Such transactional data contains a variety of potential insights into the industry and academia that have become one of the key bases for many tasks, such as data mining, data fusion, and data analysis.
Some data is of low quality in some data quality dimensions due to equipment failures or update delays. For example, certain values in the transaction data are missing, inconsistent or erroneous and do not reflect the true details of the transaction. This low quality data hinders knowledge discovery of various algorithms for different targets. In the data life cycle, data forgery, loss and tampering can affect the data quality, and then affect the data analysis result, eventually leading to wrong decisions. Therefore, in order to improve the data quality, we need to know the quality of the data first, which makes the data integrity evaluation and data repair based on the integrity evaluation result to be a problem to be solved urgently.
In the multiple transaction steps of the electric power spot market, complex dependency relationship exists among data. In complex transaction analysis, missing steps can be vital details, which can lead to erroneous decisions. For example, power trading data includes data describing wind power generation, photovoltaic power generation, electrical loads and prices of electricity, etc., which come from various edge devices in a time series. Some values in such a time series may be lost in the transmission, such as electricity prices, which makes it difficult to recover these values without knowing all relevant information for the transaction. Such missing values prevent algorithms from discovering causal relationships between prices and other factors, which makes it difficult to predict price trends in future transactions with existing algorithms. Therefore, in the edge calculation, the quality of the electric power transaction data should be guaranteed in order to perform better transaction analysis. The existing methods improve data quality by modifying classical dependency and probability methods, but the methods cannot evaluate the integrity of data generated in edge equipment, so that the quality of data repair cannot be guaranteed, and further wrong power system decision is caused.
Disclosure of Invention
The invention aims to solve the problems that the integrity of data generated in edge equipment cannot be evaluated by adopting the existing method, so that the quality of data repair cannot be ensured, and further wrong power system decision is caused, and provides an integrity detection and missing repair method for power spot data.
The technical scheme adopted by the invention for solving the technical problems is as follows:
based on one aspect of the invention, the method for detecting the integrity and repairing the missing of the electric power spot data specifically comprises the following steps:
constructing a directed graph G ═ V, E }, wherein V is a vertex and E is an edge connecting the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical value integrity detection, weight integrity detection, path integrity detection, transaction integrity detection and data integrity detection;
the transaction integrity detection is as follows: detecting whether all paths meet path integrity; if all paths meet the path integrity, missing repair is not needed, otherwise, data integrity detection is continued;
the data integrity detection is as follows: calculating the percentage of the total number of paths meeting the path integrity in the total number of paths;
and restoring the electric power spot data according to the data integrity detection result and by combining the block chain and the Bi-GRU + Attention model.
Based on another aspect of the present invention, a method for integrity detection and loss repair of power spot data specifically includes the following steps:
constructing a directed graph G ═ V, E, F, wherein V is a vertex, E is an edge connecting the vertices, and F represents the relationship between the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical integrity detection, weight integrity detection, path integrity detection and transaction integrity detection;
if the transaction integrity detection result is that the transaction is complete, the electric power spot data does not need to be repaired, otherwise, the electric power spot data is repaired by adopting the following repairing mode:
according to the relation F between the vertex and the vertex, data restoration is carried out on the vertex which does not meet the numerical integrity;
and repairing the weight of the vertex according to the probability table.
The invention has the beneficial effects that: the invention provides an integrity detection and deletion repair method for electric power spot data, and provides a transaction data integrity evaluation model based on a Bayesian network by considering the dependency of electric power transaction time sequence data. On the basis of the Bayesian network, the data integrity of the transaction data is defined according to different granularities, and an evaluation criterion of the transaction data integrity is designed. And then the Bi-GRU + Attention model of the combined block chain is adopted or the data is effectively repaired directly according to the relation between the vertexes and the probability table.
Experimental verification shows that the Bayesian network-based data integrity evaluation model provided by the invention can fully reflect the integrity of power transaction time sequence data, and adopts a Bi-GRU + Attention network or carries out data restoration according to the relation between vertexes and a probability table, so that the restoration precision and efficiency of data integrity are improved, the quality of data restoration is ensured, and a correct power system decision is conveniently made.
Drawings
FIG. 1 is a Bayesian network diagram of a transactional data model;
in the figure, Bidding Day is a Bidding Day, Running Day is a Running Day;
FIG. 2 is a path diagram on a transaction data integrity diagram;
FIG. 3 is a diagram of a Bi-GRU + Attention model structure;
FIG. 4 is a diagram of a GRU structure;
FIG. 5 is a structural view of a Bi-GRU;
FIG. 6 is a schematic illustration of data integrity for power spot data on different dates;
FIG. 7 is a graph comparing repair data integrity of GRUs with power data of the present invention;
in the figure, target represents original data, Bi-GRU + Attention represents data repaired by a Bi-GRU + Attention model adopting the method of the invention, and GRU represents data repaired by a GRU model;
FIG. 8 is a graphical illustration of repair values for different dates;
FIG. 9 is a graph comparing the efficiency of the consensus mechanism;
FIG. 10 is a graph of the impact of data quality on the efficiency of the consensus mechanism.
Detailed Description
The first embodiment is as follows: the method for detecting integrity and repairing missing of the electric power spot data in the embodiment specifically comprises the following steps:
constructing a directed graph G ═ V, E }, wherein V is a vertex and E is an edge connecting the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical value integrity detection, weight integrity detection, path integrity detection, transaction integrity detection and data integrity detection;
the transaction integrity detection is as follows: detecting whether all paths meet path integrity; if all paths meet the path integrity, missing repair is not needed, otherwise, data integrity detection is continued;
the data integrity detection is as follows: calculating the percentage of the total number of paths meeting the path integrity in the total number of paths;
and restoring the electric power spot data according to the data integrity detection result and by combining the block chain and the Bi-GRU + Attention model.
The block chain and attention mechanism based data evaluation and repair algorithm (CBBA) of the present embodiment is shown in table 1:
TABLE 1
In table 1, the CBBA first invokes the algorithm of table 2 to evaluate the data integrity of the data set. Then, for the missing value, the Bi-GRU + Attention network is used to learn the power spot data to predict the missing value. For each repair, NrepWill be repaired and SAT and MSE are calculated by equation (10) and equation (14). If the SAT is too small, repair will stop and repair data is returned. Based on different granularities of data integrity, by formula
Calculate data integrity, | ftrading(Tj) L represents the total number of complete paths and n represents the total number of all paths.
Table 2 data integrity assessment
Degree of satisfaction of repair
After the data with low integrity is repaired according to the repair rule, calculating the satisfaction SAT of data repair according to the integrity and repair time of the repaired data:
SAT＝αDrepair-βT (1)
0＜α＜1 (2)
0＜β＜1 (3)
α+β＝1 (4)
where α and β are weight coefficients of integrity and repair time, respectively. Equation (1) indicates that the satisfaction SAT is represented by a difference between the data integrity multiplier and the repair time multiplier of the repaired data. Equations (2) to (4) represent that the weight coefficients α, β are between 0 and 1, and the sum of α and β is 1. The higher the SAT value, the better the repair algorithm.
Repair time
The repair time consists of two parts; one is the time t at which we fetch data from the chaindIn the course of data transmissionData volume and bandwidth size determination; second, time t for local repairlThe algorithm running on the raw data set can yield:
trepair＝td+tl (6)
wherein the content of the first and second substances,
Cirespectively representing the amount of data and the bandwidth in the ith data transmission link. Time of day
By data volume
Sum bandwidth CiIs expressed as shown in equation (5). Total repair time trepairBy
And
the joint decision is shown in equation (6).
To calculate the repair satisfaction, we apply equation (7) to the repair time trepairNormalization processing is performed to map the result to the (0, 1) section.
Error repair
The error of data repair is calculated using the mean square error:
wherein X' represents the average of two values before and after the missing value, and Y represents the predicted data.
The second embodiment is as follows: the difference between this embodiment and the first embodiment is that the numerical integrity detection includes attribute integrity detection and tuple integrity detection;
the specific process of detecting the integrity of the attribute comprises the following steps: taking each vertex in the directed graph G as an attribute, wherein all the attributes comprise: predicted load L predicted by actual load on day of bidding day1Predicted photovoltaic power generation amount E predicted by actual photovoltaic power generation amount on day of bidding day1Predicted wind power generation amount E predicted by actual wind power generation amount on day of bidding day2Predicted electricity price P predicted by electricity price on day of bidding day1Bid winning power consumption E on bid date3Winning power generation amount E on bidding day4Actual power generation amount E of producer5Actual load L2A market competitive price P given by market clearing after all market participants submit bids2And a total price P provided by the producer3；
If the attribute is missing in the electric power spot data, the attribute is indicated to be incomplete, otherwise, the attribute is complete;
when the attributes are incomplete, directly considering that the vertexes corresponding to the missing attributes do not meet the numerical integrity, and then carrying out tuple integrity detection on the attributes which are not missing;
when the attribute is complete, directly carrying out tuple integrity detection;
the specific process of tuple integrity detection is as follows: for any attribute, if the attribute has data at each moment in the electric power spot data, the attribute meets the integrity of the tuple, otherwise, the attribute does not meet the integrity of the tuple;
the vertex corresponding to the attribute which meets the integrity of the tuple is complete in numerical value, and the vertex corresponding to the attribute which does not meet the integrity of the tuple is not complete in numerical value;
the specific process of the weight integrity detection is as follows:
in the directed graph G, for a vertex, if the weight associated with the vertex is not missing and the sum of the weights associated with the vertex is equal to 1, the vertex is weighted completely, otherwise the vertex is weighted incompletely.
The specific process of path integrity detection is as follows:
for any path, if the path satisfies the path integrity, the path should satisfy the following conditions at the same time:
1) each vertex in the path is weighted completely;
2) each vertex in the path is numerically complete.
Other steps and parameters are the same as those in the first embodiment.
The third concrete implementation mode: this embodiment will be described with reference to fig. 3 and 5. The present embodiment is different from the first to the second embodiments in that the Bi-GRU + orientation model is composed of an input layer, a Bi-GRU layer, an orientation layer, a sense layer, a softmax layer, and an output layer.
First, input vector { x1,x2,......,xnProcessing the sequence vector into a sequence vector mode, calculating an output sequence vector through a Bi-GRU layer, and extracting deep features of the input sequence vector through the Bi-GRU layer. Then, an Attention probability value that should be generated for each data vector is generated through the Attention layer. The final Attention value is obtained by the accumulated sum of the Attention probability value generated by the Attention layer and the product of the hidden layer state, and then the softmax function is used for normalization operation. As shown in equations (9) - (12).
1)Bi-GRU
Based on LSTM, GRU combines forgetting gate and input gate into an update gate, and mixes the cell state and the hidden state. As shown in figure 4 of the drawings,
zt＝σ(Wz·[ht-1,xt]) (9)
rt＝σ(Wr·[ht-1,xt]) (10)
in formula (9), ztIndicating the output of the update gate at time t, WzRepresents the weight of the update gate, ht-1Output, x, representing the hidden state at time t-1tRepresenting data input at the time t, wherein sigma represents a sigmoid function; in the formula (10), rtDenotes the output of the reset gate, σ denotes the sigmoid function, WrRepresenting the weight of the reset gate; in the formula (11), the reaction mixture,
an output representing a hidden state at time t, tanh representing a tanh function, W representing a weight of memory information, rtThe specific gravity indicating the output of the reset gate at the time t, i.e. the forgotten information; in the formula (12), htOutput representing hidden state at time t, ht-1Output representing hidden state at time t-1, ztIndicating the output of the update gate at time t, i.e., the weight of the past information being retained.
First, the update gate z is calculated according to the formula (9) and the formula (10)tAnd a reset gate rtIs then passed through equation (11) and reset gate rtIs calculated to obtain
Finally, the door z is updatedtAct on at
(1-zt) Act at ht-1Adding to obtain the final state output ht. The bidirectional GRU processes the input sequence in forward and backward order, respectively, so that the output sequence at each time step contains past and future context information in the input sequence at the current time.
2) Mechanism of Attention
Combining the attention mechanism with the GRU makes the model focus more on important information in the time series data. The attention mechanism is given by equation (13) -equation (15):
ut＝tanh(Wdht+bl) (13)
wherein h istIs the output of the previous Bi-GRU neural network layer, WdRepresents a weight coefficient, blDenotes the offset coefficient udIndicating the initialized attention matrix, utDenotes the attention vector, atThe similarity coefficient at time t is represented, and a represents a context vector.
Other steps and parameters are the same as those of the seventh or eighth embodiment.
The fourth concrete implementation mode: the difference between this embodiment and the first to third embodiments is that, according to the data integrity detection result, the block chain and the Bi-GRU + Attention model are combined to repair the power spot data, and the specific process is as follows:
step S1, the electric power spot data owner initiates a transaction request, and after the identity of the data owner is verified by the blockchain, the data at each moment in the electric power spot data are respectively sent to corresponding nodes (the data in each node are different);
selecting a node corresponding to the data at the moment with the highest data integrity through a consensus mechanism, and taking the selected node as a leader node (the leader node has the right to write the block into the block chain);
step S2, the leader node packs the time data with the highest data integrity into blocks and broadcasts the blocks to other nodes for verification, if the verification passes (when more than half of the nodes vote to pass, the verification passes), the blocks are added to a block chain, if the verification does not pass, the blocks are invalid, and the time data corresponding to the leader node is repaired by using a Bi-GRU + Attention model deployed on the leader node;
and step S3, reselecting the leader node from the time data which is not added to the block chain through a consensus mechanism, and repeating the process of step S2 until all the time data are added to the block chain.
The proposed consensus mechanism based on data quality can reduce the block forming time and improve the efficiency of transaction verification.
The consensus mechanism functions to bring the ledgers of all nodes into agreement. The POW consensus mechanism is that all nodes solve a common problem, and the node that first solves the problem will generate a block. All nodes solve the same problem, and extremely high computational cost is consumed. The PODQ consensus mechanism proposed by the inventor is that each node solves the problem of the node, and the difficulty of the problem is inconsistent. The PODQ consensus process is performed on the basis of data integrity evaluations. Each node agrees according to its own data integrity.
When the consensus process starts, based on the data integrity, then the integrity DcompletenessThe highest node pushes the consensus process between the participating nodes. Selecting a data integrity DcompletenessTaking the selected node as a leader node according to the node corresponding to the highest time data; and the leader node packs the data at the time with the highest data integrity into blocks and broadcasts the blocks to other nodes for verification, if the verification is passed (when more than half of the nodes vote to pass, the verification is passed), the blocks are added to the block chain, and all other nodes are updated into corresponding block chains.
Other steps and parameters are the same as those in one of the first to third embodiments.
Fifth embodiment this embodiment will be described with reference to fig. 1. The method for detecting integrity and repairing missing of the electric power spot data in the embodiment specifically comprises the following steps:
constructing a directed graph G of the Bayesian network, wherein V is a vertex, E is an edge connecting the vertex, and F represents the relationship between the vertex and the vertex; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical integrity detection, weight integrity detection, path integrity detection and transaction integrity detection;
if the transaction integrity detection result is that the transaction is complete, the electric power spot data does not need to be repaired, otherwise, the electric power spot data is repaired by adopting the following repairing mode:
according to the relation F between the vertex and the vertex, data restoration is carried out on the vertex which does not meet the numerical integrity;
and repairing the weight of the vertex according to a probability table given by an expert system.
The sixth specific implementation mode: the difference between this embodiment and the fifth embodiment is that the numerical integrity detection includes attribute integrity detection and tuple integrity detection;
the specific process of detecting the integrity of the attribute comprises the following steps: taking each vertex in the directed graph G as an attribute, wherein all the attributes comprise: predicted load L predicted by actual load on day of bidding day1Predicted photovoltaic power generation amount E predicted by actual photovoltaic power generation amount on day of bidding day1Predicted wind power generation amount E predicted by actual wind power generation amount on day of bidding day2Predicted electricity price P predicted by electricity price on day of bidding day1Bid winning power consumption E on bid date3Winning power generation amount E on bidding day4Actual power generation amount E of producer5Actual load L2A market competitive price P given by market clearing after all market participants submit bids2And a total price P provided by the producer3；
If the attribute is missing in the electric power spot data, the attribute is indicated to be incomplete, otherwise, the attribute is complete;
when the attributes are incomplete, directly considering that the vertexes corresponding to the missing attributes do not meet the numerical integrity, and then carrying out tuple integrity detection on the attributes which are not missing;
when the attribute is complete, directly carrying out tuple integrity detection;
the specific process of tuple integrity detection is as follows: for any attribute, if the attribute has data at each moment in the electric power spot data, the attribute meets the integrity of the tuple, otherwise, the attribute does not meet the integrity of the tuple;
the vertex corresponding to the attribute satisfying the tuple integrity is complete in value, and the vertex corresponding to the attribute not satisfying the tuple integrity is not complete in value.
1) Attribute integrity is denoted AcThis means whether the attribute values in the dataset are complete. For the attributes in table 3, such as (photovoltaic power generation, wind power generation, electrical load, electricity price), the absence of any one or more of them indicates that the attribute is incomplete.
2) Tuple integrity is represented as TcThis means whether a tuple is complete in the dataset. For any tuple in Table 3, e.g., (01/01, 01:00:00, 0.5371, 0.5882, 0.9371, 0.56), the absence of any one or more of them indicates that the tuple is missing.
TABLE 3
The other steps and parameters are the same as those in the fifth embodiment.
The seventh embodiment: this embodiment will be described with reference to fig. 2. The difference between this embodiment and one of the fifth to sixth embodiments is that the specific process of detecting the integrity of the weight is as follows:
in the directed graph G, for a vertex, if the weight associated with the vertex is not missing and the sum of the weights associated with the vertex is equal to 1, the vertex is weighted completely, otherwise the vertex is weighted incompletely.
As shown in equation (16), the vertex is not weight-complete:
p(P2|L1)+p(P2|E1)+p(P2|E2)+p(P2|P1)≠1 (16)
wherein, P (P)2|L1) Represents a point L1Point of orientation P2Probability of time, i.e. load L by prediction1Calculating a bid price P2Probability of time. For vertex P2In the path diagram, the vertex L is1、E1、E2And P1All point to the vertex P2Therefore, with the vertex P2The associated weights refer to: point L1Point of orientation P2Probability of time, point E1Point of orientation P2Probability of time, point E2Point of orientation P2Probability of time, point P1Point of orientation P2Probability of time, and vertex P2The associated weight sum refers to P (P)2|L1)、p(P2|E1)、p(P2|E2) And P (P)2|P1) The sum of (1).
The other steps and parameters are the same as those of the fifth or sixth embodiment.
The specific implementation mode is eight: the difference between this embodiment and one of the fifth to seventh embodiments is that the specific process of detecting the integrity of the path is as follows:
for any path, if the path satisfies the path integrity, the path should satisfy the following conditions at the same time:
1) each vertex in the path is weighted completely;
2) each vertex in the path is numerically complete.
Other steps and parameters are the same as those of one of the fifth to seventh embodiments.
The specific implementation method nine: the difference between this embodiment and one of the fifth to eighth embodiments is that the specific process of the transaction integrity detection is as follows: and detecting whether each path meets the path integrity, if so, the electric power spot data meets the transaction integrity, otherwise, the electric power spot data does not meet the transaction integrity.
Other steps and parameters are the same as those of the fifth to eighth embodiments.
The detailed implementation mode is ten: this embodiment is different from one of the fifth to ninth embodiments in that the relationship F between the vertex and the vertex includes:
1)E5＝E3＝E4＝L1；
2) according to L1、E1、E2And P1Calculating P2(ii) a From L1、E1、E2、P1Obtaining the price P of the day2；
3)
Wherein, P+Is a positive imbalance of the adjusted price, P-Is a negative imbalance adjustment price.
The double priced electricity market refers to a situation where an imbalance is solved with two different prices. Typically, if a producer produces more energy than it sells, the excess energy is sold at a spot price that is less than the market time unit. Conversely, when a producer produces less energy than sales, it must purchase the energy it lacks at a price higher than the spot price per market time unit. In such a market, the price offered by the producer is P for a certain unit of market time3. A positive imbalance price is typically lower than or equal to the day-ahead market bid price, so any remaining energy compared to the bid is rewarded at a price lower than the spot price. Likewise, a negatively unbalanced price is typically higher than or equal to the day-ahead market bid price, and thus any missing energy will be penalized at a price higher than the day-ahead market bid price.
Other steps and parameters are the same as those in one of the fifth to ninth embodiments.
Experimental part
A. Experimental setup
In order to verify the performance of the proposed bayesian network based data integrity evaluation and repair. The public data sets are sorted, and time sequence data sets required by power transaction are constructed. The data set is composed of a prediction load, an actual power load, photovoltaic power generation capacity, wind power generation capacity, bid-winning power consumption, actual power generation capacity, quotation, a bidding price and actual pricing. Wherein the predicted load is predicted from the actual load of the previous day. Probability tables of a data integrity evaluation model based on a bayesian network are given by an expert system. The raw data we present is complete. In order to meet the experimental requirements, random data missing processing based on Gaussian distribution is carried out on the test sample. We performed performance analysis of the proposed model using the processed missing data set.
We have realized the web service based on block chain in Ubuntu system, verify the efficiency of the data quality consensus mechanism (PODQ) put forward. All experiments were performed using Python3.7 on an Intel Corei7-9750HCPU2.6GHz, 16GB memory and Windows 1064-bit PC.
B. Performance evaluation
1) Data integrity assessment
We calculated the completeness of the data set according to the proposed bayesian network based data integrity assessment model, assessment rules and equation (16). As shown in fig. 6, the missing data accounts for 8.9% of the total data set, and the data with integrity 2/7, 3/7, 5/7, and 6/7 account for 0.46%, 10.65%, 3.01%, and 85.88% of the missing data, respectively, indicating that our data integrity model can capture the missing values in the power transaction data.
2) Effectiveness of data repair: as shown in fig. 7. We use a Bi-GRU + Attention based network to repair data. First, we fill the missing value with the average of the values of the missing data at the previous and next time instants. Time series data predictions were then made using Bi-GRU + Attention to determine the value of the deletion. When the data repair satisfaction is greater than r, we will stop the data repair. Meanwhile, when the error between the predicted value and the average value is satisfied, the data will be restored. We can see from fig. 8 that the integrity of the repaired data has improved significantly.
3) Efficiency of consensus mechanism: in order to verify the efficiency of the proposed data quality consensus mechanism and to evaluate the impact of data quality repair on the data quality consensus mechanism. After the data integrity is evaluated, the low integrity data is traced according to the traceable nature of the blockchain. And then, uploading the repaired node data to the block chain again, forming a block through a consensus mechanism, and recording the transaction in the block chain. As shown in fig. 9, we compare the proposed data quality consensus mechanism with the pow consensus mechanism, where we set the difficulty of pow to the lowest. In the figure, the dotted line represents the block formation time using the pow consensus mechanism, and the solid line represents the block formation time using the data quality consensus mechanism. Since the curves have a consistent trend, the travel time of only the first 100 blocks is shown here for the sake of clarity of the experimental plot. We can clearly see that the proposed data quality consensus mechanism (PODQ) has a shorter chunk formation time and higher efficiency. In addition, fig. 10 is a comparison of block formation times of low quality repair data and repair data. The dotted line is the block forming time of the low-quality data, and the solid line is the block forming time after data recovery. In general, our proposed data quality consensus mechanism improves the formation time of the block and the higher the data quality the more efficient the transaction is achieved.
The above-described calculation examples of the present invention are merely to explain the calculation model and the calculation flow of the present invention in detail, and are not intended to limit the embodiments of the present invention. It will be apparent to those skilled in the art that other variations and modifications of the present invention can be made based on the above description, and it is not intended to be exhaustive or to limit the invention to the precise form disclosed, and all such modifications and variations are possible and contemplated as falling within the scope of the invention.
Claims (10)
1. The integrity detection and missing repair method of the electric power spot data is characterized by comprising the following steps:
constructing a directed graph G ═ V, E }, wherein V is a vertex and E is an edge connecting the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical value integrity detection, weight integrity detection, path integrity detection, transaction integrity detection and data integrity detection;
the transaction integrity detection is as follows: detecting whether all paths meet path integrity; if all paths meet the path integrity, missing repair is not needed, otherwise, data integrity detection is continued;
the data integrity detection is as follows: calculating the percentage of the total number of paths meeting the path integrity in the total number of paths;
and restoring the electric power spot data according to the data integrity detection result and by combining the block chain and the Bi-GRU + Attention model.
2. The method for integrity detection and loss repair of power spot data according to claim 1, wherein the numerical integrity detection comprises an attribute integrity detection and a tuple integrity detection;
the specific process of detecting the integrity of the attribute comprises the following steps: taking each vertex in the directed graph G as an attribute, wherein all the attributes comprise: predicted load L predicted by actual load on day of bidding day1Predicted photovoltaic power generation amount E predicted by actual photovoltaic power generation amount on day of bidding day1Predicted wind power generation amount E predicted by actual wind power generation amount on day of bidding day2Predicted electricity price P predicted by electricity price on day of bidding day1Bid winning power consumption E on bid date3Winning power generation amount E on bidding day4Actual power generation amount E of producer5Actual load L2A market competitive price P given by market clearing after all market participants submit bids2And a total price P provided by the producer3；
If the attribute is missing in the electric power spot data, the attribute is indicated to be incomplete, otherwise, the attribute is complete;
when the attributes are incomplete, directly considering that the vertexes corresponding to the missing attributes do not meet the numerical integrity, and then carrying out tuple integrity detection on the attributes which are not missing;
when the attribute is complete, directly carrying out tuple integrity detection;
the specific process of tuple integrity detection is as follows: for any attribute, if the attribute has data at each moment in the electric power spot data, the attribute meets the integrity of the tuple, otherwise, the attribute does not meet the integrity of the tuple;
the vertex corresponding to the attribute which meets the integrity of the tuple is complete in numerical value, and the vertex corresponding to the attribute which does not meet the integrity of the tuple is not complete in numerical value;
the specific process of the weight integrity detection is as follows:
in the directed graph G, for a vertex, if the weight associated with the vertex is not missing and the sum of the weights associated with the vertex is equal to 1, the vertex is weighted completely, otherwise the vertex is weighted incompletely.
The specific process of path integrity detection is as follows:
for any path, if the path satisfies the path integrity, the path should satisfy the following conditions at the same time:
1) each vertex in the path is weighted completely;
2) each vertex in the path is numerically complete.
3. The method of claim 2, wherein the Bi-GRU + Attention model is composed of an input layer, a Bi-GRU layer, an Attention layer, a sense layer, a softmax layer, and an output layer.
4. The method for integrity detection and loss repair of power spot data according to claim 3, wherein the method for repairing the power spot data by combining a block chain and a Bi-GRU + Attention model according to the data integrity detection result comprises the following specific processes:
step S1, the electric power spot data owner initiates a transaction request, and after the identity of the data owner is verified by the blockchain, the data at each moment in the electric power spot data are respectively sent to corresponding nodes;
selecting a node corresponding to the data at the moment with the highest data integrity through a consensus mechanism, and taking the selected node as a leader node;
step S2, the leader node packs the time data with the highest data integrity into blocks and broadcasts the blocks to other nodes for verification, if the verification is passed, the blocks are added to a block chain, if the verification is not passed, the blocks are invalid, and the time data corresponding to the leader node is repaired by using a Bi-GRU + Attention model deployed on the leader node;
and step S3, reselecting the leader node from the time data which is not added to the block chain through a consensus mechanism, and repeating the process of step S2 until all the time data are added to the block chain.
5. The integrity detection and missing repair method of the electric power spot data is characterized by comprising the following steps:
constructing a directed graph G ═ V, E, F, wherein V is a vertex, E is an edge connecting the vertices, and F represents the relationship between the vertices; carrying out integrity detection on the electric power spot data based on the constructed directed graph;
the integrity detection comprises numerical integrity detection, weight integrity detection, path integrity detection and transaction integrity detection;
if the transaction integrity detection result is that the transaction is complete, the electric power spot data does not need to be repaired, otherwise, the electric power spot data is repaired by adopting the following repairing mode:
according to the relation F between the vertex and the vertex, data restoration is carried out on the vertex which does not meet the numerical integrity;
and repairing the weight of the vertex according to the probability table.
6. The method for integrity detection and loss repair of power spot data according to claim 5, wherein the numerical integrity detection comprises an attribute integrity detection and a tuple integrity detection;
the specific process of detecting the integrity of the attribute comprises the following steps: taking each vertex in the directed graph G as an attribute, wherein all the attributes comprise: predicted load L predicted by actual load on day of bidding day1Predicted photovoltaic power generation amount E predicted by actual photovoltaic power generation amount on day of bidding day1Predicted wind power generation amount E predicted by actual wind power generation amount on day of bidding day2Predicted electricity price P predicted by electricity price on day of bidding day1Bid winning power consumption E on bid date3Winning power generation amount E on bidding day4Actual power generation amount E of producer5Actual load L2A market competitive price P given by market clearing after all market participants submit bids2And a total price P provided by the producer3；
If the attribute is missing in the electric power spot data, the attribute is indicated to be incomplete, otherwise, the attribute is complete;
when the attributes are incomplete, directly considering that the vertexes corresponding to the missing attributes do not meet the numerical integrity, and then carrying out tuple integrity detection on the attributes which are not missing;
when the attribute is complete, directly carrying out tuple integrity detection;
the specific process of tuple integrity detection is as follows: for any attribute, if the attribute has data at each moment in the electric power spot data, the attribute meets the integrity of the tuple, otherwise, the attribute does not meet the integrity of the tuple;
the vertex corresponding to the attribute satisfying the tuple integrity is complete in value, and the vertex corresponding to the attribute not satisfying the tuple integrity is not complete in value.
7. The method for integrity detection and loss repair of power spot data according to claim 6, wherein the specific process of weight integrity detection is as follows:
in the directed graph G, for a vertex, if the weight associated with the vertex is not missing and the sum of the weights associated with the vertex is equal to 1, the vertex is weighted completely, otherwise the vertex is weighted incompletely.
8. The method for integrity detection and loss repair of power spot data according to claim 7, wherein the path integrity detection specifically comprises:
for any path, if the path satisfies the path integrity, the path should satisfy the following conditions at the same time:
1) each vertex in the path is weighted completely;
2) each vertex in the path is numerically complete.
9. The method for integrity detection and loss repair of power spot data according to claim 8, wherein the transaction integrity detection comprises: and detecting whether each path meets the path integrity, if so, the electric power spot data meets the transaction integrity, otherwise, the electric power spot data does not meet the transaction integrity.
10. The method of claim 9, wherein the vertex-to-vertex relationship F comprises:
1)E5＝E3＝E4＝L1；
2) according to L1、E1、E2And P1Calculating P2；
3)
Wherein, P+Is a positive imbalance of the adjusted price, P-Is a negative imbalance adjustment price.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110699778.6A CN113377755B (en) | 2021-06-23 | 2021-06-23 | Integrity detection and missing repair method for electric power spot data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110699778.6A CN113377755B (en) | 2021-06-23 | 2021-06-23 | Integrity detection and missing repair method for electric power spot data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113377755A true CN113377755A (en) | 2021-09-10 |
| CN113377755B CN113377755B (en) | 2022-12-16 |
Family
ID=77578829
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110699778.6A Active CN113377755B (en) | 2021-06-23 | 2021-06-23 | Integrity detection and missing repair method for electric power spot data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113377755B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070283417A1 (en) * | 2006-05-05 | 2007-12-06 | Lockheed Martin Corporation | Systems and methods for establishing authenticity of electronic records in an archives system |
| CN109445972A (en) * | 2018-09-21 | 2019-03-08 | 深圳供电局有限公司 | Data recovery method, device, equipment and storage medium |
| US20190141054A1 (en) * | 2017-11-09 | 2019-05-09 | Nokia Solutions And Networks Oy | Method and apparatus for blockchain powered integrity protection system |
| CN110599163A (en) * | 2019-08-20 | 2019-12-20 | 江苏大学 | Transaction record outsourcing method facing block chain transaction supervision |
| CN111160880A (en) * | 2019-12-10 | 2020-05-15 | 国网天津市电力公司 | Block chain-based industrial user demand response-oriented transaction settlement control method |
| CN111523785A (en) * | 2020-04-16 | 2020-08-11 | 三峡大学 | Power system dynamic security assessment method based on generation countermeasure network |
| CN111709672A (en) * | 2020-07-20 | 2020-09-25 | 国网黑龙江省电力有限公司 | Virtual power plant economic dispatching method based on scene and deep reinforcement learning |
| CN111709779A (en) * | 2020-06-01 | 2020-09-25 | 广东电网有限责任公司 | Trading variety optimization system for electric power spot market |
| CN112001744A (en) * | 2020-07-28 | 2020-11-27 | 安徽电力交易中心有限公司 | Power generator auxiliary quotation system and method based on prospect theory in electric power spot market |
| CN112381263A (en) * | 2020-09-23 | 2021-02-19 | 四川大学 | Block chain distributed data storage based multi-microgrid day-ahead robust electric energy transaction method |
| CN112434343A (en) * | 2020-11-25 | 2021-03-02 | 江西理工大学 | Virtual power plant safety scheduling and transaction method based on dual block chain technology |
| CN112785429A (en) * | 2021-01-08 | 2021-05-11 | 西北工业大学 | Local area multi-microgrid power transaction pairing method based on block chain technology |
-
2021
- 2021-06-23 CN CN202110699778.6A patent/CN113377755B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070283417A1 (en) * | 2006-05-05 | 2007-12-06 | Lockheed Martin Corporation | Systems and methods for establishing authenticity of electronic records in an archives system |
| US20190141054A1 (en) * | 2017-11-09 | 2019-05-09 | Nokia Solutions And Networks Oy | Method and apparatus for blockchain powered integrity protection system |
| CN109445972A (en) * | 2018-09-21 | 2019-03-08 | 深圳供电局有限公司 | Data recovery method, device, equipment and storage medium |
| CN110599163A (en) * | 2019-08-20 | 2019-12-20 | 江苏大学 | Transaction record outsourcing method facing block chain transaction supervision |
| CN111160880A (en) * | 2019-12-10 | 2020-05-15 | 国网天津市电力公司 | Block chain-based industrial user demand response-oriented transaction settlement control method |
| CN111523785A (en) * | 2020-04-16 | 2020-08-11 | 三峡大学 | Power system dynamic security assessment method based on generation countermeasure network |
| CN111709779A (en) * | 2020-06-01 | 2020-09-25 | 广东电网有限责任公司 | Trading variety optimization system for electric power spot market |
| CN111709672A (en) * | 2020-07-20 | 2020-09-25 | 国网黑龙江省电力有限公司 | Virtual power plant economic dispatching method based on scene and deep reinforcement learning |
| CN112001744A (en) * | 2020-07-28 | 2020-11-27 | 安徽电力交易中心有限公司 | Power generator auxiliary quotation system and method based on prospect theory in electric power spot market |
| CN112381263A (en) * | 2020-09-23 | 2021-02-19 | 四川大学 | Block chain distributed data storage based multi-microgrid day-ahead robust electric energy transaction method |
| CN112434343A (en) * | 2020-11-25 | 2021-03-02 | 江西理工大学 | Virtual power plant safety scheduling and transaction method based on dual block chain technology |
| CN112785429A (en) * | 2021-01-08 | 2021-05-11 | 西北工业大学 | Local area multi-microgrid power transaction pairing method based on block chain technology |
Non-Patent Citations (3)
| Title |
|---|
| 荀华 等: "基于规则的电力数据指标检查系统设计与实现", 《江西电力》 * |
| 蔡宇 等: "省级电力现货市场技术支持系统技术架构设计", 《电力系统自动化》 * |
| 郭蕴颖 等: "基于CNN和LSTM联合预测并修正的电量缺失数据预测", 《计算机系统应用》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113377755B (en) | 2022-12-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR100766149B1 (en) | Methods and systems for efficiently sampling portfolios for optimal underwriting | |
| CN110298574B (en) | Convolutional neural network-based electricity user payment risk rating method | |
| JP2004506959A (en) | Evaluation value prediction model when information is missing | |
| JP2004500641A (en) | Method and system for automatically estimating credit score evaluation value | |
| JP2004500646A (en) | Competitive bidding revenue simulation method and system | |
| JP2003535387A (en) | Rapid evaluation of asset portfolios such as financial products | |
| JP2004500642A (en) | Methods and systems for assessing cash flow recovery and risk | |
| Boral et al. | An integrated approach for fuzzy failure mode and effect analysis using fuzzy AHP and fuzzy MARCOS | |
| JP2003526147A (en) | Cross-correlation tool to automatically calculate portfolio description statistics | |
| Giahi et al. | Design optimization for resilience for risk-averse firms | |
| McCluskey et al. | The mass appraisal of residential property in Northern Ireland | |
| Wu et al. | Gridtopo-GAN for distribution system topology identification | |
| Camelia et al. | A Computational Grey Based Model for Companies Risk Forecasting. | |
| CN113987808A (en) | Electricity user complaint early warning method of feature weighted Bayesian network | |
| Emami et al. | A conceptual analysis of equilibrium bidding strategy in a combined oligopoly and oligopsony wholesale electricity market | |
| CN113377755B (en) | Integrity detection and missing repair method for electric power spot data | |
| US20230252285A1 (en) | Spatio-temporal graph neural network for time series prediction | |
| Król et al. | Fuzzy system model to assist with real estate appraisals | |
| Kaneko et al. | Sensitivity analysis of factors relevant to extreme imbalance between procurement plans and actual demand: Case study of the Japanese electricity market | |
| CN114004530A (en) | Enterprise power credit score modeling method and system based on sequencing support vector machine | |
| Mateo et al. | Strategic bidding under uncertainty using genetic algorithms | |
| CN111915082B (en) | Prediction method, prediction device, storage medium and prediction equipment | |
| Huang et al. | Day-Ahead Probabilistic Load Forecasting: A Multi-Information Fusion and Noncrossing Quantiles Method | |
| Singh et al. | Using quality attributes and computational intelligence to generate and evaluate system architecture alternatives | |
| Seo et al. | A Unified Model for Bid Landscape Forecasting in the Mixed Auction Types of Real-Time Bidding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |