CN111767338A - Distributed data storage method and system for online super real-time simulation of power system - Google Patents
Distributed data storage method and system for online super real-time simulation of power system Download PDFInfo
- Publication number
- CN111767338A CN111767338A CN202010083967.6A CN202010083967A CN111767338A CN 111767338 A CN111767338 A CN 111767338A CN 202010083967 A CN202010083967 A CN 202010083967A CN 111767338 A CN111767338 A CN 111767338A
- Authority
- CN
- China
- Prior art keywords
- data
- data storage
- simulation
- storage
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 114
- 238000004088 simulation Methods 0.000 title claims abstract description 111
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000003860 storage Methods 0.000 claims abstract description 112
- 238000007726 management method Methods 0.000 claims abstract description 53
- 230000015654 memory Effects 0.000 claims description 27
- 230000006870 function Effects 0.000 claims description 9
- 230000002688 persistence Effects 0.000 claims description 8
- 230000005540 biological transmission Effects 0.000 claims description 7
- 238000005259 measurement Methods 0.000 claims description 7
- 230000002085 persistent effect Effects 0.000 claims description 7
- 238000012217 deletion Methods 0.000 claims description 6
- 230000037430 deletion Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 description 13
- 238000003379 elimination reaction Methods 0.000 description 11
- 230000008030 elimination Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000004140 cleaning Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 3
- 238000005265 energy consumption Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000031068 symbiosis, encompassing mutualism through parasitism Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a distributed data storage method and a system for online super real-time simulation of an electric power system, which comprises the following steps: constructing a distributed power system online super real-time simulation system comprising metadata nodes and data storage nodes; according to a storage resource scheduling request of a simulation task and an available storage space of each data storage node, distributing two mutually independent data storage nodes as an original storage node and a summary storage node by a metadata node; continuously and periodically acquiring state data of power grid operation from a power grid, sending the state data to an original data storage node and a data storage node, and operating the online super real-time simulation of the power system by each data storage node according to a simulation task and the state data to obtain a local super real-time simulation result; and summarizing the local simulation results of the data storage nodes to a summarized storage node to obtain a complete super real-time simulation result of the power grid. The invention can provide data storage management for simulation while running the power system simulation.
Description
Technical Field
The invention relates to the field of online super real-time simulation of an electric power system in system simulation, in particular to an online super real-time simulation method and system of the electric power system based on distributed data.
Background
The online super real-time simulation of the power system refers to the steps that power grid operation state data are continuously and periodically (with 10 seconds as a period) collected from an interconnected large power grid through online measurement equipment such as an SCADA (supervisory control and data acquisition), the power grid operation state data are transmitted back to a data center through a high-speed interconnected network, online measurement data are rapidly calculated and analyzed through a power system simulation software system, the stability and the safety of the current operation state of the interconnected large power grid are further judged, and the operation state in a short period of time (within a few seconds) in the future is accurately predicted. Measurement data of hundreds of megabytes are generated in each measurement period of a general regional backbone power grid, gigabytes of calculation result data are generated through calculation and analysis of a power system simulation software system, and the data have important value on analysis and management of an interconnected large power grid, so that efficient storage and management of the data are needed. At present, a disk array mode is mostly adopted in the aspect of online super real-time simulation big data storage of a power system, and the disk array is generally high in price and limited in expansibility; although the existing distributed storage system in the market has a small number of applications in the field, a large bottleneck exists in the aspect of storage I/O efficiency, and general commercial products are low in general cost.
In the face of massive and rapidly-increased power system simulation data, the scale and cost of data storage are one of the problems to be solved. Conventional distributed storage systems generally improve the storage reliability of data by means of data redundancy (such as multiple copies, erasure codes, and the like), which further results in reduction of storage space efficiency and improvement of storage cost. In the online super real-time simulation environment of the power system, in addition to having higher requirements for storage I/O performance, important consideration is also needed in terms of storage space efficiency. The simulation input data and the simulation output data comprise thousands of files, the content and the value of the data stored in each file are different, if the files store relatively static configuration information, some files store periodically generated online measurement data such as voltage, current and power, the data stored in some files need to be stored for a long time for subsequent analysis and mining, and the data stored in some files can be repeatedly generated through simulation calculation. Thus, historical data is generally required to be preserved for different periods of time, such as three months or half a year, or even a year, depending on the level of value. In addition, in order to enable the online super real-time simulation of the power system to be capable of continuously and stably operating for a long time, the system can have fault isolation and self-healing capabilities under the conditions that a single server fails, a single disk fails or the disk storage space is full, and the like, so that the system can be ensured to continuously operate online without being affected.
The existing distributed storage system in the market has less application in the field of power system simulation, on one hand, the cost is higher and the maintenance threshold is high; on the other hand, the I/O throughput capacity of the simulation system is difficult to meet the performance requirement of online super real-time simulation, and the simulation system is often a performance bottleneck of simulation calculation. This is because, in the power system simulation software system calculation process, a large number of intermediate results and final calculation results are generated, and these data are often output in the same time period, i.e. a high I/O load is generated in a short time, and it is generally difficult for the existing distributed storage system to cope with such application challenges. In addition, the independent distributed storage cluster occupies a large amount of machine room space, and the energy consumption of the machine room is greatly increased. Therefore, it is generally rarely used in actual production.
Disclosure of Invention
The invention aims to solve the problems of I/O performance and high cost in the field of power system simulation in the conventional distributed storage technology, and provides a method and a system for online super real-time simulation large data distributed storage and management of a power system based on a computing-storage coupling architecture.
Aiming at the defects of the prior art, the invention provides a distributed data storage method for online super real-time simulation of a power system, which comprises the following steps:
step 3, continuously and periodically acquiring state data of power grid operation from the power grid through online measurement equipment, sending the state data to the original data storage nodes and the data storage nodes, and operating online super real-time simulation of the power system by each data storage node according to the simulation task and the state data to obtain a local super real-time simulation result;
and 4, summarizing the local simulation results of the data storage nodes to the summarized storage nodes to obtain a complete super real-time simulation result of the power grid.
The distributed data storage method for the online super real-time simulation of the power system is characterized in that the data storage nodes adopt distributed message queues to realize real-time transmission of real-time data, and adopt a distributed storage system based on a centralized structure to finish persistent storage and access functions of non-real-time data.
The distributed data storage method for the online super real-time simulation of the power system is characterized in that data exchange is carried out among the data storage nodes by adopting a TCP (transmission control protocol).
The distributed data storage method for the online super real-time simulation of the power system is characterized in that the metadata node comprises a metadata management module used for eliminating management of data in a disk and eliminating management of data in a memory.
In the distributed data storage method for the online super real-time simulation of the power system, after receiving a signal of finishing data persistence, the metadata management module updates the use space of a corresponding disk in a database, and when the use space is larger than a preset value, the database table is inquired to find out the data with the earliest storage time; sending a signal to a data storage node where the earliest data is located; and after receiving the signal, the data storage node deletes the related data according to the instruction and sends a deletion success instruction to the metadata management module.
The invention also provides a distributed data storage system for online super real-time simulation of the power system, which comprises the following components:
the method comprises the following steps that 1, an online super real-time simulation system of the distributed power system is constructed, wherein the online super real-time simulation system comprises a metadata node and a plurality of data storage nodes;
the module 2 allocates two mutually independent data storage nodes as an original storage node and a summary storage node respectively according to the storage resource scheduling request of the simulation task and the available storage space of each data storage node;
the module 3 continuously and periodically collects state data of power grid operation from the power grid through the online measuring equipment, and sends the state data to the original data storage nodes and the data storage nodes, and each data storage node operates the online super real-time simulation of the power system according to the simulation task and the state data to obtain a local super real-time simulation result;
and the module 4 summarizes the local simulation results of the data storage nodes to the summarized storage nodes to obtain the complete super real-time simulation result of the power grid.
The distributed data storage system of the online super real-time simulation of the power system is characterized in that the data storage nodes adopt distributed message queues to realize real-time transmission of real-time data, and adopt a distributed storage system based on a centralized structure to finish persistent storage and access functions of non-real-time data.
The distributed data storage system for the online super real-time simulation of the power system adopts a TCP protocol for data exchange among the data storage nodes.
The distributed data storage system of the online super real-time simulation of the power system is characterized in that the metadata node comprises a metadata management module, and the metadata management module is used for eliminating management of data in a disk and eliminating management of data in a memory.
In the distributed data storage system for online super real-time simulation of the power system, after receiving a signal of finishing data persistence, the metadata management module updates the use space of a corresponding disk in a database, and when the use space is larger than a preset value, the database table is inquired to find out the data with the earliest storage time; sending a signal to a data storage node where the earliest data is located; and after receiving the signal, the data storage node deletes the related data according to the instruction and sends a deletion success instruction to the metadata management module.
According to the scheme, the invention has the advantages that:
firstly, the computing and storing capabilities of the server can be fully utilized, the storage management of mass data is provided for simulation while the simulation computing of the power system is operated, the machine room space is saved, and the energy consumption is reduced. Secondly, the memory cache and the locality principle can be utilized, the storage I/O load in the running process of the simulation program is greatly reduced, the storage I/O path is shortened, and the running efficiency of the simulation program is improved. Thirdly, a storage reliability strategy of file granularity can be customized according to the value grade of the data, the storage space is reasonably and effectively utilized, indexes such as the value grade, the copy quantity, the generation time and the like of the data are comprehensively considered, the data elimination priority is dynamically calculated, the data in the storage system can be automatically eliminated according to reasonable rules to provide available storage capacity, and the continuous online operation capacity of the power system simulation is guaranteed.
Drawings
FIG. 1 is a diagram of a storage system software architecture;
FIG. 2 is a block diagram of simulation data for a power system;
FIG. 3 is a distributed storage system deployment logic diagram;
FIG. 4 is a data storage operation workflow diagram;
FIG. 5 is a raw data storage workflow diagram;
FIG. 6 is a summary data storage workflow diagram;
FIG. 7 is a diagram of data storage state changes on a summarized data storage node;
FIG. 8 is a diagram of data storage state changes on non-summarized data storage nodes.
Detailed Description
In order to achieve the technical effects, the invention comprises the following three key points:
the key point 1 is a power system simulation calculation and storage service symbiosis and resource cooperative scheduling technology. The technical effects are as follows: the simulation calculation and data storage management can be simultaneously carried out on the same high-performance server cluster, and each server runs a simulation calculation program and a storage service program simultaneously. This can bring about two advantages: in the aspect of economy, the computing and storing capacity of the server can be fully utilized, the cluster scale of the server is reduced, and the space occupation and energy consumption of a machine room are reduced; in the aspect of performance, the locality of simulation data storage I/O can be improved, a data moving path is shortened, the I/O performance is improved, meanwhile, the processes which are in charge of storing the I/O input/output function in the simulation program are dispersedly dispatched to different computing nodes to run through the cooperative dispatching of computing and storage resources, the I/O load of each server is shared, and I/O hot spots caused by a small number of servers are avoided, so that the simulation I/O performance is not influenced;
a key point 2, a memory-based storage I/O acceleration technology; the technical effects are as follows: in the periodic operation process of the power system simulation program, disk I/O intensive operations of reading original input data, configuring information, exchanging intermediate result data, outputting result data and the like need to be periodically executed, which occupies a large proportion in the operation time of the simulation program, the method has a non-negligible influence on the performance of the simulation program, and in order to solve the problem, the scheme uses the memory file system as a transfer cache layer of input and output data, temporarily stores original input data, configuration information and the like in the memory file system, the simulation program is used for calling, intermediate result data and calculation result data generated in the running process of the simulation program are also written into the memory file system, data that needs to be persisted is written out asynchronously by the storage service to disks of local or other servers, thereby avoiding bottlenecks in the computation process of the emulator. In order to ensure the operation safety and performance of the simulation program, a memory occupation upper limit is set for a memory file system, a storage service is responsible for monitoring the space occupation condition of the memory file system and writing data out of a disk in time, and under the necessary condition (for example, when the memory file system is about to reach the memory occupation upper limit due to the large delay of the data written out of the disk), the data which can be discarded is calculated through a certain strategy, so that the memory space is released, and the continuous and stable operation of the online super real-time simulation program of the power system is ensured;
a key point 3, a fine granularity mixed reliability strategy taking a file as a unit and a data elimination dynamic priority mechanism; the technical effects are as follows: different reliability guarantee strategies can be designed according to the value grade of the simulation data of the power system, and the different reliability guarantee strategies coexist in the same data storage management system and can be set and adjusted as required. For example, for the simulated original input data, the value level is high, in order to ensure the reliability, the 3-copy mode can be adopted for storage, and for the simulated calculation result data, the requirement on the reliability of storage is low because the simulated calculation result data can be calculated according to the corresponding original input data, and the single-copy mode can be adopted for storage. For a specific power system simulation platform, because the storage space capacity of the server cluster is limited and the data capable of being stored is also limited, in order to always ensure that the newer simulation data can be timely and reliably stored, old data stored in the system must be eliminated. However, different data have different value levels, use requirements and use modes, and corresponding elimination priorities are different, for example, the value level of the simulation input data is higher, such data should be kept as much as possible when old data is eliminated to release more available storage space, data generated within 1 day should be kept as much as possible due to higher probability of being accessed, and the like. Therefore, the dynamic priority of data elimination is calculated by comprehensively considering indexes of three dimensions of value level, copy number and generation time of the data, and the data are eliminated according to the priority.
In order to make the aforementioned features and effects of the present invention more comprehensible, embodiments accompanied with figures are described in detail below.
The online super real-time power system simulation data distributed storage management system is mainly divided into three parts: a data storage layer, a communication service layer and an interface layer, as shown in fig. 1.
(1) Data storage layer
Since the data that needs to be stored is divided into two types, real-time data and non-real-time data, it will correspond to two different data storage services. For real-time messages, a message queue is adopted to realize real-time transmission of data, and the message queue can be a distributed message queue based on a plurality of storage node memories; for non-real-time messages, a distributed storage system based on a centralized structure is adopted to complete the functions of persistent storage and access to data.
(2) Communication service layer
In order to ensure the reliability of data, the TCP protocol is adopted for data exchange.
(3) Interface layer
In order to facilitate the reading of data by the human-computer interaction terminal, a series of data access interfaces are provided.
A power system simulation job will include a plurality of computation tasks, which are denoted as N computation tasks, the computation results will form N simple summaries, N detailed results, N final summaries, N difference files, and in addition, the N tasks share the same input data, as shown in fig. 2.
The distributed storage system deployment logic is shown in fig. 3, the system includes two parts, namely a metadata node and a storage node, the storage of metadata is based on a relational database, such as MySQL, in order to ensure the reliability of metadata storage management, an HA architecture can be adopted for metadata storage, and the storage node is also used as a computing node for simulation program operation, that is, the storage management system and the simulation computing system operate on the same physical computing node. The metadata node records metadata information such as a storage node where the data is located, a disk (or a storage path) of the storage node where the data is located, and the like, and is also responsible for scheduling a storage request, that is, the metadata node determines on which storage node the data is stored.
The data storage operation workflow of the distributed storage system is mainly divided into the following steps:
1) requesting storage resources
And the simulation calculation scheduling node sends a storage resource scheduling request to the metadata node, and requests the metadata node to allocate a storage space for the data of the simulation calculation. And the metadata node manages the storage resources by taking the disk as a unit, and selects the disk with the most available space according to the current state of the available disk in the storage cluster and the available storage space condition. When the metadata node allocates storage resources, two independent disks are allocated, one disk is used for storing a summarized data packet, and the other disk is used for storing an original data packet. After the selection is completed, the metadata management module sends < summary data disk ID, task ID > to the summary data storage node, and sends < original data disk ID, task ID > to the original data storage node, after the related nodes receive the notification, the roles of the related nodes are confirmed, the final persistent path of the data is confirmed according to the disk ID, and then the data summarization stage is started.
2) Data packet summarization
Because the copies of the original data are cached in the memory file system of each computing node, the summary is not needed. After the original data storage node receives the notification < the summarized data disk ID and the task ID >, a confirmation message is sent to a metadata management module of the main data node, the metadata management module sets the reading state of the task data in a data table to be a ramfs available state, and then the metadata management module synchronously updates the database table of the slave metadata node. The raw data storage workflow is shown in fig. 5.
The summary process for summarizing packets is as follows, as shown in fig. 6:
(1) after receiving a summarizing command sent by a metadata management module, the summarizing storage node sends a data packet collecting instruction to the storage node where the IO process is located;
(2) the summarizing node waits for data recovery, and the storage node where the IO process is located sends the data to the summarizing node;
(3) after the collection of the collection nodes is finished, sending a collection completion signal to a metadata management module on the main metadata node;
(4) the metadata management module updates the data to a ramfs available state while synchronizing the data to the slave metadata nodes.
3) And (5) data persistence.
After the summarized data storage nodes and the original data storage nodes are persisted to the designated disk directory, the storage nodes update the available state of the data packets (data can be accessed in the disks) to the main metadata node, and simultaneously update the used space of the corresponding disks.
In order to automatically maintain the availability of the storage space, the system provides a data elimination function, and relates to two aspects: elimination management of data in a disk and elimination management of data in a memory. In order to implement automatic and reasonable elimination of summarized data, the storage state of the summarized data is managed, as shown in fig. 7. Fig. 8 shows the state change of the relevant summarized data (mainly related to data elimination management of memory space) on the non-summarized data storage node.
The automatic elimination process of the data stored on the disk is as follows:
the detailed information of the number of disks, the disk space, the used capacity of the disks and the like of each node is recorded in the database. The cleaning of disk space will be initiated by the metadata management module. The conditions for triggering the metadata management module to initiate data cleaning include the following strategies:
(1) calculating the space utilization rate of each disk, and when the average value of the utilization rates of all the node disks exceeds a configured threshold value;
(2) calculating the overall disk utilization rate by taking all disks as a whole, and when the utilization rate exceeds a configured threshold value;
after receiving the signal of data persistence completion, the metadata management module updates the usage space of the corresponding disk in the database. If the trigger condition is found to be met, the following steps are sequentially executed:
(1) and querying a database table to find out the oldest N (optional) pieces of data in the whole situation.
(2) The metadata management module sends a signal to the node where the oldest data is located. This signal is < list of task data IDs to delete >.
(3) And after receiving the signal, the related node deletes the related data according to the instruction and sends a deletion success instruction to the metadata management module.
(4) The metadata management module updates the database table: the data is set to a clean state and the relative disk utilization is updated.
(5) Synchronizing data to the slave metadata node.
The automatic elimination process of the cached data in the memory is as follows:
considering the normal operation of the system, the proportion of the physical space occupied by the mount space of the memory file system is configured according to the configuration options in the configuration file. Meanwhile, a memory file system cleaning module cleans the relevant data, and the cleaning period is 8s (available).
The main functions of the memory file system cleaning module are as follows:
(1) and recording the state of all data in the memory file system. There are three states of the summary data: the summarized data is received, the summarized data is durably received, and the summarized data is available to a disk; the raw data has three states: raw data reception, in RAMFS, raw data persistence, and raw data disk availability. (a file is maintained separately for each data on the storage node, and the state of the corresponding data is stored in the file. when the file is modified, namely the data is modified/deleted, the file needs to be locked first to ensure that the modification operation is atomic.)
(2) The module will check the memory file system space to ensure that the usage (used/total) of the memory file system does not exceed 75% (this entry is available).
(3) After the threshold is reached, the oldest summarized data (the data already summarized is somewhat older than the output data not yet summarized), the oldest raw data and the oldest output data are deleted directly.
The following are system examples corresponding to the above method examples, and this embodiment can be implemented in cooperation with the above embodiments. The related technical details mentioned in the above embodiments are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related-art details mentioned in the present embodiment can also be applied to the above-described embodiments.
The invention also provides a distributed data storage system for online super real-time simulation of the power system, which comprises the following components:
the method comprises the following steps that 1, an online super real-time simulation system of the distributed power system is constructed, wherein the online super real-time simulation system comprises a metadata node and a plurality of data storage nodes;
the module 2 allocates two mutually independent data storage nodes as an original storage node and a summary storage node respectively according to the storage resource scheduling request of the simulation task and the available storage space of each data storage node;
the module 3 continuously and periodically collects state data of power grid operation from the power grid through the online measuring equipment, and sends the state data to the original data storage nodes and the data storage nodes, and each data storage node operates the online super real-time simulation of the power system according to the simulation task and the state data to obtain a local super real-time simulation result;
and the module 4 summarizes the local simulation results of the data storage nodes to the summarized storage nodes to obtain the complete super real-time simulation result of the power grid.
The distributed data storage system of the online super real-time simulation of the power system is characterized in that the data storage nodes adopt distributed message queues to realize real-time transmission of real-time data, and adopt a distributed storage system based on a centralized structure to finish persistent storage and access functions of non-real-time data.
The distributed data storage system for the online super real-time simulation of the power system adopts a TCP protocol for data exchange among the data storage nodes.
The distributed data storage system of the online super real-time simulation of the power system is characterized in that the metadata node comprises a metadata management module, and the metadata management module is used for eliminating management of data in a disk and eliminating management of data in a memory.
In the distributed data storage system for online super real-time simulation of the power system, after receiving a signal of finishing data persistence, the metadata management module updates the use space of a corresponding disk in a database, and when the use space is larger than a preset value, the database table is inquired to find out the data with the earliest storage time; sending a signal to a data storage node where the earliest data is located; and after receiving the signal, the data storage node deletes the related data according to the instruction and sends a deletion success instruction to the metadata management module.
Claims (10)
1. A distributed data storage method for online super real-time simulation of a power system is characterized by comprising the following steps:
step 1, constructing a distributed power system online super real-time simulation system comprising a metadata node and a plurality of data storage nodes;
step 2, according to the storage resource scheduling request of the simulation task and the available storage space of each data storage node, the metadata node distributes two mutually independent data storage nodes as an original storage node and a summary storage node respectively;
step 3, continuously and periodically acquiring state data of power grid operation from the power grid through online measurement equipment, sending the state data to the original data storage nodes and the data storage nodes, and operating online super real-time simulation of the power system by each data storage node according to the simulation task and the state data to obtain a local super real-time simulation result;
and 4, summarizing the local simulation results of the data storage nodes to the summarized storage nodes to obtain a complete super real-time simulation result of the power grid.
2. The distributed data storage method for the online super real-time simulation of the power system as claimed in claim 1, wherein the data storage node uses a distributed message queue to implement real-time transmission of real-time data, and uses a distributed storage system based on a centralized structure to implement persistent storage and access functions of non-real-time data.
3. The distributed data storage method for the online super real-time simulation of the power system as claimed in claim 1, wherein the data storage nodes exchange data with each other by using a TCP protocol.
4. The distributed data storage method for online super real-time simulation of electric power system of claim 1, wherein the metadata node comprises a metadata management module for obsolete management of data in disk and obsolete management of data in memory.
5. The online super real-time simulation distributed data storage method of the power system as claimed in claim 4, wherein the metadata management module updates the usage space of the corresponding disk in the database after receiving the signal of data persistence completion, and when the usage space is greater than a preset value, queries the database table to find out the data with the earliest storage time; sending a signal to a data storage node where the earliest data is located; and after receiving the signal, the data storage node deletes the related data according to the instruction and sends a deletion success instruction to the metadata management module.
6. A distributed data storage system for online super real-time simulation of an electric power system, comprising:
the method comprises the following steps that 1, an online super real-time simulation system of the distributed power system is constructed, wherein the online super real-time simulation system comprises a metadata node and a plurality of data storage nodes;
the module 2 allocates two mutually independent data storage nodes as an original storage node and a summary storage node respectively according to the storage resource scheduling request of the simulation task and the available storage space of each data storage node;
the module 3 continuously and periodically collects state data of power grid operation from the power grid through the online measuring equipment, and sends the state data to the original data storage nodes and the data storage nodes, and each data storage node operates the online super real-time simulation of the power system according to the simulation task and the state data to obtain a local super real-time simulation result;
and the module 4 summarizes the local simulation results of the data storage nodes to the summarized storage nodes to obtain the complete super real-time simulation result of the power grid.
7. The distributed data storage system for online super real-time simulation of electric power system of claim 6, wherein the data storage node uses distributed message queue to realize real-time transmission of real-time data, and uses distributed storage system based on centralized structure to realize persistent storage and access function of non-real-time data.
8. The distributed data storage system for online super real-time simulation of electric power system of claim 6, wherein the data storage nodes exchange data with each other using TCP protocol.
9. The distributed data storage system for online super real-time simulation of electric power system of claim 6, wherein the metadata node comprises a metadata management module for obsolete management of data in disk and obsolete management of data in memory.
10. The online super real-time simulation distributed data storage system of the power system as claimed in claim 9, wherein the metadata management module updates the usage space of the corresponding disk in the database after receiving the signal of data persistence completion, and when the usage space is greater than a preset value, queries the database table to find out the data with the earliest storage time; sending a signal to a data storage node where the earliest data is located; and after receiving the signal, the data storage node deletes the related data according to the instruction and sends a deletion success instruction to the metadata management module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010083967.6A CN111767338A (en) | 2020-02-10 | 2020-02-10 | Distributed data storage method and system for online super real-time simulation of power system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010083967.6A CN111767338A (en) | 2020-02-10 | 2020-02-10 | Distributed data storage method and system for online super real-time simulation of power system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111767338A true CN111767338A (en) | 2020-10-13 |
Family
ID=72719045
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010083967.6A Pending CN111767338A (en) | 2020-02-10 | 2020-02-10 | Distributed data storage method and system for online super real-time simulation of power system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111767338A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115933972A (en) * | 2022-11-24 | 2023-04-07 | 中国华能集团清洁能源技术研究院有限公司 | Distributed data storage method and system for multi-professional simulation platform |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013178528A1 (en) * | 2012-06-01 | 2013-12-05 | Thomson Licensing | Method of data storing and data synchronization in a distributed data storage system |
| CN104516967A (en) * | 2014-12-25 | 2015-04-15 | 国家电网公司 | Electric power system mass data management system and use method thereof |
| CN108256263A (en) * | 2018-02-07 | 2018-07-06 | 中国电力科学研究院有限公司 | A kind of electric system hybrid simulation concurrent computational system and its method for scheduling task |
| CN109815002A (en) * | 2017-11-21 | 2019-05-28 | 中国电力科学研究院有限公司 | A kind of distributed paralleling calculation platform and its method based on in-circuit emulation |
-
2020
- 2020-02-10 CN CN202010083967.6A patent/CN111767338A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013178528A1 (en) * | 2012-06-01 | 2013-12-05 | Thomson Licensing | Method of data storing and data synchronization in a distributed data storage system |
| CN104516967A (en) * | 2014-12-25 | 2015-04-15 | 国家电网公司 | Electric power system mass data management system and use method thereof |
| CN109815002A (en) * | 2017-11-21 | 2019-05-28 | 中国电力科学研究院有限公司 | A kind of distributed paralleling calculation platform and its method based on in-circuit emulation |
| CN108256263A (en) * | 2018-02-07 | 2018-07-06 | 中国电力科学研究院有限公司 | A kind of electric system hybrid simulation concurrent computational system and its method for scheduling task |
Non-Patent Citations (1)
| Title |
|---|
| ADNB34G: "大数据分布式存储的部署模式：分离式or超融合", 《HTTP://T.CSDN.CN/UBDWX》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115933972A (en) * | 2022-11-24 | 2023-04-07 | 中国华能集团清洁能源技术研究院有限公司 | Distributed data storage method and system for multi-professional simulation platform |
| CN115933972B (en) * | 2022-11-24 | 2024-05-31 | 中国华能集团清洁能源技术研究院有限公司 | Distributed data storage method and system of multi-specialty simulation platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102882983B (en) | Rapid data memory method for improving concurrent visiting performance in cloud memory system | |
| CN104965850B (en) | A kind of database high availability implementation method based on open source technology | |
| CN103139302A (en) | Real-time copy scheduling method considering load balancing | |
| CN111787055B (en) | Redis-based transaction mechanism and multi-data center oriented data distribution method and system | |
| CN113553346B (en) | Large-scale real-time data stream integrated processing, forwarding and storing method and system | |
| CN102833352A (en) | Distributed cache management system and method for implementing distributed cache management | |
| CN104202424B (en) | A kind of method using software architecture to expand buffer memory | |
| CN116777182B (en) | Task dispatch method for semiconductor wafer manufacturing | |
| CN107454171A (en) | Message service system and its implementation | |
| CN115587118A (en) | Task data dimension table association processing method and device and electronic equipment | |
| CN109241194A (en) | The load-balancing method and device of Database Systems based on High-Performance Computing Cluster distribution | |
| CN103530335A (en) | In-stockroom operation method and device of electric power measurement acquisition system | |
| CN110515938B (en) | Data aggregation storage method, equipment and storage medium based on KAFKA message bus | |
| CN107436904B (en) | Data acquisition method, data acquisition device, and computer-readable storage medium | |
| CN111767338A (en) | Distributed data storage method and system for online super real-time simulation of power system | |
| CN114610567A (en) | Container monitoring method, network device and storage medium | |
| CN117667447A (en) | Multi-log high concurrency message queue processing method and system | |
| CN111367880A (en) | Universal real-time data storage management system and implementation method thereof | |
| CN116775420A (en) | Information creation cloud platform resource display and early warning method and system based on Flink flow calculation | |
| CN116226067A (en) | Log management method, log management device, processor and log platform | |
| CN112445590A (en) | Computing resource access and scheduling system and method | |
| CN108023914B (en) | Memory data sharing system, and memory data writing and reading method | |
| CN112799835A (en) | Method and system for processing metadata of distributed database system | |
| CN101420439B (en) | Region matching method based on displacement residual error in distributed virtual environment | |
| CN116301663B (en) | Data storage method, device and host |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201013 |