CN108921728A - Distributed real-time database system based on power network dispatching system - Google Patents
Distributed real-time database system based on power network dispatching system Download PDFInfo
- Publication number
- CN108921728A CN108921728A CN201810722584.1A CN201810722584A CN108921728A CN 108921728 A CN108921728 A CN 108921728A CN 201810722584 A CN201810722584 A CN 201810722584A CN 108921728 A CN108921728 A CN 108921728A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- time database
- cluster
- power network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013500 data storage Methods 0.000 claims abstract description 39
- 238000007726 management method Methods 0.000 claims abstract description 38
- 238000003860 storage Methods 0.000 claims abstract description 32
- 230000002688 persistence Effects 0.000 claims abstract description 7
- 238000009826 distribution Methods 0.000 claims description 44
- 238000006062 fragmentation reaction Methods 0.000 claims description 18
- 238000013467 fragmentation Methods 0.000 claims description 17
- 230000008901 benefit Effects 0.000 claims description 6
- 230000003139 buffering effect Effects 0.000 claims description 5
- 239000004744 fabric Substances 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 5
- 239000000306 component Substances 0.000 description 40
- 238000012360 testing method Methods 0.000 description 28
- 238000000034 method Methods 0.000 description 15
- 101100202502 Caenorhabditis elegans scd-2 gene Proteins 0.000 description 12
- 238000012217 deletion Methods 0.000 description 10
- 230000037430 deletion Effects 0.000 description 10
- 238000012545 processing Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 7
- 238000007792 addition Methods 0.000 description 6
- 238000005192 partition Methods 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 239000012634 fragment Substances 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 239000000203 mixture Substances 0.000 description 4
- 241001269238 Data Species 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 239000008358 core component Substances 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000011056 performance test Methods 0.000 description 3
- 238000011282 treatment Methods 0.000 description 3
- 238000013523 data management Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 235000006508 Nelumbo nucifera Nutrition 0.000 description 1
- 240000002853 Nelumbo nucifera Species 0.000 description 1
- 235000006510 Nelumbo pentapetala Nutrition 0.000 description 1
- 101150042597 Scd2 gene Proteins 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Business, Economics & Management (AREA)
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Economics (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides the distributed real-time database systems based on power network dispatching system, including data storage component, data high-speed Buffer Unit, data service component and cluster management assembly；Data storage component for obtaining company-data, and is responsible for the persistence of company-data, and company-data includes model data and real time data；Data high-speed Buffer Unit, for obtaining real time data, and storage and management real time data；Data service component, for access to company-data and responsible company-data it is superfluous standby；Cluster management component monitors cluster stage condition, the dilatation capacity reducing of management node resource and cluster for managing cluster service priority.The present invention can support power grid mass data storage and data access, have the characteristics that it is high handle up, be High Availabitity, expansible, be more in line with the demand of dispatch automated system mass data storage and access.
Description
Technical field
The present invention relates to technical field of power systems, more particularly, to the distributed real-time database system based on power network dispatching system
System.
Background technique
As extra-high voltage alternating current-direct current serial-parallel power grid scale is fast-developing, power grid integrated feature is highlighted, and local fault influences
Globalization, the influence to power grid such as exceedingly odious weather increase, and need dispatch automated system to the data and external rings of the whole network
Border data carry out large-scale real-time storage and processing, realize the Situation Awareness to the whole network.Real-time data base be dispatching of power netwoks from
Dynamicization system core component part undertakes the data storage and management of system.
Real-time data base is the memory database based on sheet form storage, and data read-write efficiency is much higher than traditional relational
Database is data acquisition and monitoring SCADA (Supervisory Control And Data Acquisition) system
One of core component, for storing grid equipment model and real-time measurement data.
Real-time data base is always the core component of dispatch automated system, at home CC2000, OPEN3000, and
The real-time data base of D5000 system, being all based on share memory technology realizes storage to a certain moment Power System Steady-state data,
And local and remote access interface is provided for each application.It is accessed by the local interface of real-time database, is substantially
Access to this node shared drive, and accessed by the remote interface of real-time database, indeed through ICP/IP protocol to remote
The shared drive of Cheng Jiedian accesses.The real-time data base of D5000 platform is made of several tables, and each table is assigned one
Block shared drive, the number that the size of shared drive depends on the length of list item record and entirely records.
At present in automation system for the power network dispatching, real-time database is all single node centralised storage framework, and data storage is held
Amount and data throughput capabilities are limited by data scale, seriously affect the operation and extension of system, influence to adjust to a certain extent
The business of degree center production.
It is superfluous standby with data not have scalability mainly due to real-time data base uses single node centralised storage framework for this
Property, influence the stability of system operation and the continuity of business.Firstly, the real-time data base for lacking superfluous standby property limits data
There is single node failure in high availability.It is disposed in real time in existing automation system for the power network dispatching using standby machine mode
Database keeps real-time database in standby machine that identical data is written, to guarantee standby machine real-time database model data by service application
And the consistency of metric data.Although solving the problems, such as real-time database single node failure, such mode depends on system business application,
It forces service application in standby machine node while running identical services to synchronize real-time database, the synchronous process of data is cumbersome, intermediate
Link is more, and data synchronous safety is poor.Secondly, the real-time data base for lacking scalability constrains the distribution of SCADA application
Data processing can not promote the treatment effeciency of real time data by multi-host parallel mode, while data storage capacity is limited to save
Point disk and memory source, limit the expansion of data processing scale, and SCADA in future application processing can not be supported extensive real-time
The ability of data.
In conclusion lacking a kind of efficient real-time database system in the prior art.
Summary of the invention
It, can be in view of this, the purpose of the present invention is to provide the distributed real-time database system based on power network dispatching system
Power grid mass data storage and data access are supported, has the characteristics that height is handled up, is High Availabitity, expansible, is more in line with scheduling
The demand of automated system mass data storage and access.
In a first aspect, the embodiment of the invention provides the distributed real-time database systems based on power network dispatching system, including：Number
According to storage assembly, data high-speed Buffer Unit, data service component and cluster management assembly；
The data storage component is connected with the data high-speed Buffer Unit, for obtaining company-data, and is responsible for
The persistence of the company-data, wherein the company-data includes model data and real time data；
The data high-speed Buffer Unit, is connected with data service component, for obtaining the real time data, and stores
With the management real time data；
The data service component is connected with the data storage component, for accessing to the company-data,
And it is responsible for the superfluous standby of the company-data；
The cluster management component takes with the data storage component, the data high-speed Buffer Unit and the data
Business component is connected, and for managing cluster service priority, monitors cluster stage condition, the expansion of management node resource and cluster
Hold capacity reducing.
With reference to first aspect, the embodiment of the invention provides the first possible embodiments of first aspect, wherein institute
Stating company-data includes data fragmentation, metadata and operation log, wherein the data fragmentation is the distributed real-time database system
System splits table data, and the metadata includes Data distribution information data, index information data, clustered node Information Number
According to and server data, the operation log be the data fragmentation is increased, deleted and changed to operate.
With reference to first aspect, the embodiment of the invention provides second of possible embodiments of first aspect, wherein institute
Stating data storage component includes meta data file, data file and journal file, and the data high-speed Buffer Unit includes data
Fragment area, meta-data region and operation log area.
With reference to first aspect, the embodiment of the invention provides the third possible embodiments of first aspect, wherein institute
Stating data service component includes data service unit, Metadata Service unit, data subscription service unit, data synchronization service list
Member and data interface unit.
With reference to first aspect, the embodiment of the invention provides the 4th kind of possible embodiments of first aspect, wherein institute
Stating cluster management component includes cluster observation unit, service managing unit, node management unit and resilient expansion unit.
With reference to first aspect, the embodiment of the invention provides the 5th kind of possible embodiments of first aspect, wherein institute
The clustered deploy(ment) mode of distributed real-time database system is stated as independent deployment or is disposed in the application server.
With reference to first aspect, the embodiment of the invention provides the 6th kind of possible embodiments of first aspect, wherein institute
State distributed real-time database system distributed storage rule include random distribution rule or by area's distribution rule, wherein it is described with
Machine distribution rule be by after table keyword consistency Hash, random storage in clustered node, it is described by area's distribution rule be by
The model data presses physical region program and district, defines data allocation rule by customized mode.
With reference to first aspect, the embodiment of the invention provides the 7th kind of possible embodiments of first aspect, wherein also
Including distributed real-time data air interface.
With reference to first aspect, the embodiment of the invention provides the 8th kind of possible embodiments of first aspect, wherein institute
Stating distributed real-time data air interface includes man-machine interface access interface and application programming access interface.
Second aspect, the embodiment of the invention provides the distributed real-time database systems based on power network dispatching system, including such as
The distributed real-time database system based on power network dispatching system is gone up, further includes：
Application programming access interface is by using metadata location mechanism and counting parallel using divide and conquer as core
Frame is calculated to realize.
The present invention provides the distributed real-time database systems based on power network dispatching system, including data storage component, data
Speed buffering component, data service component and cluster management assembly；Data storage component for obtaining company-data, and is responsible for
The persistence of company-data, company-data include model data and real time data；Data high-speed Buffer Unit, for obtaining in real time
Data, and storage and management real time data；Data service component, for accessing to company-data and responsible cluster number
According to it is superfluous standby；Cluster management component monitors cluster stage condition for managing cluster service priority, management node resource with
And the dilatation capacity reducing of cluster.The present invention can support power grid mass data storage and data access, there is height to handle up, Gao Ke
With, expansible feature, it is more in line with the demand of dispatch automated system mass data storage and access.
Other features and advantages of the present invention will illustrate in the following description, also, partly become from specification
It obtains it is clear that understand through the implementation of the invention.The objectives and other advantages of the invention are in specification, claims
And specifically noted structure is achieved and obtained in attached drawing.
To enable the above objects, features and advantages of the present invention to be clearer and more comprehensible, preferred embodiment is cited below particularly, and cooperate
Appended attached drawing, is described in detail below.
Detailed description of the invention
It, below will be to specific in order to illustrate more clearly of the specific embodiment of the invention or technical solution in the prior art
Embodiment or attached drawing needed to be used in the description of the prior art be briefly described, it should be apparent that, it is described below
Attached drawing is some embodiments of the present invention, for those of ordinary skill in the art, before not making the creative labor
It puts, is also possible to obtain other drawings based on these drawings.
Fig. 1 is the distributed real-time database system schematic provided in an embodiment of the present invention based on power network dispatching system；
Fig. 2 is another distributed real-time database system schematic based on power network dispatching system provided in an embodiment of the present invention；
Fig. 3 is distribution SCADA clustered deploy(ment) schematic diagram provided in an embodiment of the present invention；
Fig. 4 is consistency hash algorithm schematic diagram provided in an embodiment of the present invention；
Fig. 5 is that user partition provided in an embodiment of the present invention definition indicates to be intended to；
Fig. 6 is that distributing real-time data bank provided in an embodiment of the present invention unifies access interface schematic diagram；
Fig. 7 is that distributing real-time data bank provided in an embodiment of the present invention uniformly accesses api interface schematic diagram；
Fig. 8 is that distributing real-time data bank provided in an embodiment of the present invention tests environment schematic.
Icon：
10- data storage component；20- data high-speed Buffer Unit；30- data service component；40- cluster management component.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with attached drawing to the present invention
Technical solution be clearly and completely described, it is clear that described embodiments are some of the embodiments of the present invention, rather than
Whole embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art are not making creative work premise
Under every other embodiment obtained, shall fall within the protection scope of the present invention.
Currently, real-time database is all single node centralised storage framework in automation system for the power network dispatching, data storage is held
Amount and data throughput capabilities are limited by data scale, seriously affect the operation and extension of system, influence to adjust to a certain extent
The business of degree center production.Based on this, the distributed real-time database system provided in an embodiment of the present invention based on power network dispatching system,
Power grid mass data storage and data access can be supported, have the characteristics that it is high handle up, be High Availabitity, expansible, be more in line with
The demand of dispatch automated system mass data storage and access.
Embodiment one：
Referring to Fig.1, a kind of distributed real-time database system based on power network dispatching system, including：Data storage component 10, number
According to speed buffering component 20, data service component 30 and cluster management assembly 40；
Data storage component 10 is connected with data high-speed Buffer Unit, for obtaining company-data, and is responsible for cluster number
According to persistence, wherein company-data includes model data and real time data；
Data high-speed Buffer Unit 20, is connected with data service component, for obtaining real time data, and storage and management
Real time data；
Data service component 30, is connected with data storage component, for accessing to company-data and responsible collection
Group data it is superfluous standby；
Cluster management component 40 is connected with data storage component, data high-speed Buffer Unit and data service component, uses
In management cluster service priority, cluster stage condition, the dilatation capacity reducing of management node resource and cluster are monitored.
Further, company-data includes data fragmentation, metadata and operation log, wherein data fragmentation is distribution
Real-time database system splits table data, and metadata includes Data distribution information data, index information data, clustered node letter
Data and server data are ceased, operation log is to be increased, deleted and changed operation to data fragmentation and formed.
Further, data storage component 10 includes meta data file, data file and journal file, data high-speed buffering
Component 20 includes data fragmentation area, meta-data region and operation log area.
Further, data service component 30 includes data service unit, Metadata Service unit, data subscription service list
Member, data synchronization service unit and data interface unit.
Further, cluster management component 40 includes cluster observation unit, service managing unit, node management unit and bullet
Property expanding element.
Further, the clustered deploy(ment) mode of distributed real-time database system is independently to dispose or be deployed in application server
In.
Further, the distributed storage rule of distributed real-time database system includes random distribution rule or is distributed rule by area
Then, wherein random distribution rule is by after table keyword consistency Hash, and random storage is distributed by area and is advised in clustered node
It is then by model data by physical region program and district, data allocation rule is defined by customized mode.
It further, further include distributed real-time data air interface.
Further, distributed real-time data air interface includes man-machine interface access interface and application programming access
Interface.
Embodiment two：
Existing real-time data base there are aiming at the problem that, distributing real-time data bank proposes a kind of based on distributed storage
The distributed real-time data storage scheme of technology has studied data store strategy, data redundancy, unified access, data synchronize
Key technology, the distributed real-time database for realizing High Availabitity, easily extending solve current real-time database memory capacity and are limited, and data gulp down
The technical problems such as scarce capacity are spat, SCADA application is effectively supported to realize distributed data processing ability, increases real time data and handles up
Amount improves real time data processing efficiency, better adapts to future scheduling automated system mass data storage and access.
Distributing real-time data bank uses decentralization, holosymmetric cluster topology, by distributed data management system and
The multiplexing technologies such as cluster operational management system and real-time database memory technology, real-time database lock technology, real-time database index technology
It organically combines, constitutes distributing real-time data bank.Distributed real-time database overall structure is as shown in Figure 2.
Distributed real-time database software is respectively data storage, data high-speed buffering, unified access service sum aggregate from bottom to top
The parts such as group's management form.By meta data file, data file, journal file composition data storage assembly, it is responsible for company-data
Persistence；By data fragmentation area, meta-data region, operation log district's groups at data high-speed caching component, it is real to be responsible for storage and management
When data；It is made of data service, Metadata Service, data subscription service, data synchronization service, data-interface (rtdbapi)
Data service component, responsible company-data access, company-data are superfluous standby；By cluster observation, service management, node administration, elasticity
Extension composition cluster management component, is responsible for cluster service priority management, clustered node Stateful Inspection, node resource management, collection
Group's dilatation and capacity reducing.Each part mentioned above assembly function is independent, and it is soft to organically combine composition high cohesion, the distributed real-time database of lower coupling
Part system.
The key technology of distributing real-time data bank provided in an embodiment of the present invention includes following aspects, first below
The storage of company-data is illustrated.
Distributed real-time database cluster is made of multiple holosymmetric back end.Back end by cluster management component,
Operation data serviced component and cluster-based storage data realize the management of clustered node.The data storage method of back end with it is general
The data storage method of logical real-time database is identical, is realized using shared drive mapped file mode, and there is data the read-write of great number to imitate
Rate, file layout have persistence.It is written in mapped file, is kept in real time by system kernel when internal storage data variation
The consistency of data in memory and file.
Company-data in back end is divided into data fragmentation, metadata, operation log three classes data.Distributed real-time database
Table data are split into N parts of data fragmentations, data fragmentation dispersion are stored in company-data node according to Distribution Algorithm, with number
Realize that data distribution formula stores according to the mode of fragment；Data distribution information data, index information data, clustered node Information Number
Metadata is referred to as according to, service information data etc., metadata is for inquiring and safeguarding cluster state information, location data position,
Make distributed real-time database unified logic, metadata is consistent by Metadata Service in each back end；Data fragmentation is carried out
Additions and deletions, which change operation, can generate operation log, and operation log is used for backed up in synchronization data, and operation log is pushed away by data subscription service
It send to backup node, backup node re-executes operation log by data synchronization service, realizes that data are superfluous standby.
Distributed real-time database clustered deploy(ment) is illustrated below.
Distributed real-time database cluster can be disposed independently, can also dispose in the application server, existing real-time database is generally taken
Deployment in the application server, applies and accesses data in local server, is not necessarily to network communication, improves data to the maximum extent and reads
Write efficiency.Distributed real-time database provides self-defining data distribution mode, supports in data fragmentation and stores custom content, works as distribution
When formula real-time database is disposed in the application server, so that data fragmentation is stored locally applied required data as much as possible, avoid or subtract
Few network visits frequency, improves reading and writing data effect.Fig. 3 show distributed real-time database clustered deploy(ment) showing in the application server
Example, dotted line frame server indicate to support cluster expansion.
Distributed storage strategy is illustrated below.
The distributed storage strategy of the embodiment of the present invention provides random distribution and is distributed two kinds of distributed storage rules by area.
Random distribution rule is by after table keyword consistency Hash, and random storage is in clustered node, its advantage is that data distribution is equal
It is even, it is suitble to storage visiting frequency moderate and has the table largely recorded.It is by model data by physical areas by area's distribution storage rule
Domain program and district, can customize the lower dress node of subregion, and the multidomain treat-ment of connected applications makes the data of application access in local
Processing improves and applies treatment effeciency.
(1) random distribution
Random distribution rule uses consistency hash algorithm random distribution mechanism, data are random, be evenly distributed on each number
According in node.Consistency Hash Distribution Algorithm constructs the Hash ring of 0-2^32, will be located in Hash ring after back end Hash.
When there is access, to key assignments Hash, cryptographic Hash is obtained, be assigned to corresponding to the maximum node cryptographic Hash less than key assignments cryptographic Hash
Back end (first back end encountered of walking clockwise).Schematic illustration is referring to fig. 4.
Consistency hash algorithm random distribution mechanism has good balance and monotonicity, lower dispersed and negative
Lotus.
Enhance balance, using MD5 algorithm as hashing algorithm, when MD5 algorithm is to back end Hash, even if data section
Point only has a difference, and cryptographic Hash can also generate huge difference, guarantee back end title similar to when cryptographic Hash be distributed as far as possible
Uniformly.When back end is less in cluster, a data node virtual can be gone out more by way of increasing dummy node
A dummy node, mapping relations are changed to by cryptographic Hash to dummy node by cryptographic Hash to back end again to back end.
Guarantee monotonicity.If increasing or deleting a server, impacted data are only that new demand servicing device arrives it
Number between a server (i.e. along inverse or first back end encountered of walking clockwise) before or after in annular space
According to other data are unaffected.
Reduce dispersibility and load.Identical hash algorithm is used to table key assignments Hash and back end Hash, is guaranteed
The cryptographic Hash that same key assignments calculates on different terminals is identical, keeps the identical data cryptographic Hash of different terminals identical, is mapped to same
Dispersibility is reduced on back end.Using the characteristic of MD5 algorithm (even if data only have little difference, the difference of Hash result
Also can be very big), it avoids a large amount of different data from being mapped to same back end, causes single back end load excessively high.
(2) customized distribution
Customized distribution rule is to define table according to specified partition, according to customer service requirement definition data allocation rule.
Partition definition table as shown in Figure 5,1 corresponding data node 1 of subregion, 2 corresponding data node 2 of subregion, 3 corresponding data section of subregion
Point 3,4 corresponding data node 4 of subregion.
System designing user data table data properties of distributions, data distribution details is defined into each user data table, such as
In user data table M shown in Fig. 5 and N, the zone attribute of the record is all had recorded in every record, and it is right that record institute is marked
The subregion answered.When by data distribution to each back end, partition definition table is matched according to data distribution attribute list actual value, is obtained
To after distribution rule, by data distribution into each back end.Every record of as shown in the figure user data table M and N, can be with
According to zone attribute 1,2,3,4, back end 1,2,3,4 corresponding to its subregion are found, are then stored in data corresponding
Back end.
Data distribution attribute and table major key establish mapping relations, index as global data.User visits according to table keyword
When asking data, by inquiry global data index, subregion where data is determined, the corresponding back end of inquiry subregion, access should
Back end obtains the value of record.For example, as it is known that first record major key of user data table M, inquiry global data indexes to obtain
Record partitioning is 1, is matched to back end 1 according to partition definition table, finally to data record is accessed on back end 1.
Data communication and network will be illustrated below.
Real-time data access interface provided in an embodiment of the present invention includes the api interface of man-machine interface access and application access
Access two ways.As shown in fig. 6, to realize access of the man-machine thin-client to backstage real time data, distributed real-time data
Library devises real time data middle layer service, including request response and subscription two kinds of access modes of publication, customizable browse mould
The real time data of type table.The api interface of application range such as provides increasing to real time data for application program, deletes, changes, looking at the operation letter
Number.Api interface uses metadata location mechanism and combines and realizes by the parallel computation frame of core of divide and conquer.Api interface is
Using shielding data positioning, concurrent access, the process flows such as failover, distributed storage can pellucidly be accessed and exist by making to apply
Data in cluster.
(1) api interface
Api interface (application programming interface) is the number for one group of unified logic that distributing real-time data bank externally provides
According to access interface, to realize that the transparence of real-time data base accesses.As shown in fig. 7, distributing real-time data bank utilizes first number
According to management company-data distributed intelligence, unified interface sets up part by quickly positioning, completes data using metadata query interface
Positioning work.According to the location information of acquisition, based on thought of dividing and ruling, this access is split as multiple parallel tasks, and will
Task distributes to different threads, and concurrently operation is distributed in the data of different nodes, and result is then carried out merger and is returned.
Api interface can record operation log using memory journaling techniques after accessing data, in case redundancy backup uses.
(2) man-machine interface accesses
Man-machine interface access is that distributing real-time data bank is one group of service that man-machine program provides, will by network communication
Data are sent to man-machine program.Distributing real-time data bank provides request response for man-machine interface and subscribes to two kinds of access moulds of publication
Formula.
It requests response modes to provide service with single request single response mode, is taken by real-time database middle layer random access
Api interface is called in business, provides transparence access for man-machine interface.
It subscribes to release model and provides service with the multiple response mode of single request, will be asked by real-time database picture agency service
The data asked are grouped according to distribution node, based on thought of dividing and ruling, distribute the request to corresponding node by parallel task component
Picture Push Service, and continue to push delta data.
Embodiment three：
The distributed real-time data realized using distributing real-time data bank key technology provided by Embodiment 2 of the present invention
Library module has been deployed in the dispatch automated system environment built in laboratory.Distributing real-time data bank module will be carried out
Performance test, usability testing, scalability are tested.It is as shown in Figure 8 to test environment.Server has 6 compositions altogether
Distributing real-time data bank cluster (scd1, scd2, scd3, scd4, scd5, scd6).It tests in environment herein, database service
Device and the public server of processing server, 6 servers are interconnected by gigabit networking interchanger, simulate main website scheduling controlling
The database purchase of system and data processing.Cluster server model be Huawei production RH5885 V3 (inside save as 32GB,
CPUxx core dominant frequency is 2.2GHz, and installation is domestic to lose in thought (SuSE) Linux OS).
1, performance compares test
Performance test tests distributing real-time data bank the function that combined data is handled together.Simultaneously with it is original
Regulator control system (D5000) is compared, and tests the raising of its data storage and access efficiency.By state's tune, North China, Central China, East China,
The full electric network model data in northeast, northwest is sequentially stored in 6 nodes of test environment according to the storage strategy of subregion fragment
In real-time database cluster, data distribution is as shown in table 1.Same full electric model is deployed in the single section of original regulator control system simultaneously
In point environment.Respectively by test case call database access distinct interface, more former regulator control system real-time database with it is existing
Distributed real-time database data storage and access data capability.Test result is as shown in table 2 and table 3.Wherein data sheet in table 2
Position is point/second.Unit is item/second in table 3.
1 test model of table and data distribution
2 distributing real-time data bank interface performance test of table
The test of 3 distributing real-time data bank collection group energy of table
Test result shows the interface access performance and original regulator control system and single node performance phase of distributed real-time database
When, but with the increase of clustered node number, write the trend that database efficiency promotes presentation at double.Distributed real-time database is simultaneously
Data-handling efficiency is not reduced due to the functions such as data distribution algorithms, data redundancy be spare, with outstanding data processing energy
Power.
2, usability testing
In the case of usability testing will simulate cluster arbitrary node catastrophic failure, the data access feelings of distributed real-time database
Condition.The modes such as server are closed or directly closed using network interface card, and simulation clustered node scd1 node is failure, other nodes
(scd2~scd6) state keeps normal.Test result is detailed in the following table 4.Test process is as follows：
1) measpoint table is read in scd2~scd6 node respectively, it is complete to read 441354 datas.
2) the record that measpoint table is distributed on scd1 node is deleted in scd2~scd6 arbitrary node, respectively
Measpoint table is read in scd2~scd6,441353 records is read, has lacked a record, illustrated that deletion record operates successfully.
3) it is inserted into the record for being distributed in scd1 node to measpoint table in scd2~scd6 arbitrary node, respectively
Measpoint table is read in scd2~scd6, reads 441354 records, more records illustrate that insertion records and operate successfully.
4) record that measpoint table is distributed in scd1 node is updated in scd2~scd6 arbitrary node, existed respectively
Scd2~scd6 node reads this record, it is found that this record is updated, and illustrates that update operates successfully.
4 distributing real-time data bank usability testing of table
Test result shows not influencing other node databases in cluster when some node failure or abnormal service
Additions and deletions change and look into operation, the data on malfunctioning node can normal use, the backup node switching host node of failure taken over therefore
Hinder the task of node, thus proves that distributed real-time database cluster has high availability.
3, scalability is tested
Scalability test will add back end in distributed type assemblies and delete back end, check company-data point
Cloth situation and data integrity.Newly-increased clustered node test result is shown in Table 5.Test process is as follows：
1) in tri- node cluster of scd1, scd2, scd3, addition nodal operation is executed, cluster is added in scd4 node.
2) data that measpoint is distributed locally are read in scd1, scd2, scd3 node respectively, discovery record number tails off,
In scd4 node reader measpoint table data, record number are equal to the summation of first three node reduction.
3) the full table data of measpoint table are read in four node arbitrary nodes, can completely reads whole table, totally 441354
Data.
4) measpoint mono- record is deleted in four node arbitrary nodes, reads the full table of measpoint table in arbitrary node
The data total number of records is all 441353 datas, and that record deleted is not wherein, illustrates deletion record success.
5) a record is added to measpoint table in four node arbitrary nodes, all inquires this note in arbitrary node
Record can inquire, and the full table total number of records is 451354, illustrate that addition records successfully.
6) one, measpoint table record is updated in four node arbitrary nodes, records and sends out in this table of any querying node
Now record content changes, and explanation is updated successfully.
5 distributed real-time data cluster scalability of table (increasing node) test
Deletion of node test result is shown in Table 6, and test process is as follows:
1) in tetra- node cluster of scd1, scd2, scd3, scd4, deletion of node operation is executed, scd4 knot removal collection
Group.
2) data that measpoint is distributed locally are read in scd1, scd2, scd3 node respectively, record number becomes more, increases
Summation be equal to scd4 node on data.
3) the full table data of measpoint table are read in three node arbitrary nodes, can completely reads whole table, totally 441354
Data.
4) measpoint mono- record is deleted in three node arbitrary nodes, reads the full table of measpoint table in arbitrary node
The data total number of records is all 441353 datas, and that record deleted is not wherein, illustrates deletion record success.
5) a record is added to measpoint table in three node arbitrary nodes, all inquires this note in arbitrary node
Record can inquire, and the full table total number of records is 451354, illustrate that addition records successfully.
6) one, measpoint table record is updated in three node arbitrary nodes, records and sends out in this table of any querying node
Now record content changes, and explanation is updated successfully.
The test of 6 distributed real-time data cluster scalability (deletion of node) of table
Above-mentioned newly-increased node and deletion of node test result show when increasing or deleting company-data node, distributed
The data of each node can be carried out the fragment again of data by real-time database according to certain rules, and keep data integrity, no
The additions and deletions for influencing the database of clustered node, which change, looks into operation, and the test demonstrated the scalabilities of real-time database.
The technical effect of the embodiment of the present invention is：Realize the storage of data distribution formula and unified access, high performance data
The functions such as redundancy backup have the characteristics such as data distribution, unified access, redundancy backup, high efficient and reliable and resilient expansion, preliminary to have
For the ability of mass data storage, data safety, resource expansion etc..Using this technological means, real-time data base is improved
Readwrite performance and efficiency.In addition, can reinforce monitoring granularity, Analysis Service and data real time execution shape in terms of cluster observation
State；Cluster management mode can be optimized in terms of human-computer interaction, abundant information is provided and operates succinct tools interfaces.
Distributed real-time database system based on power network dispatching system provided by the embodiment of the present invention, including store program
The computer readable storage medium of code, the instruction that said program code includes can be used for executing described in previous methods embodiment
Method, specific implementation can be found in embodiment of the method, details are not described herein.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description
It with the specific work process of device, can refer to corresponding processes in the foregoing method embodiment, details are not described herein.
Finally it should be noted that：Embodiment described above, only a specific embodiment of the invention, to illustrate the present invention
Technical solution, rather than its limitations, scope of protection of the present invention is not limited thereto, although with reference to the foregoing embodiments to this hair
It is bright to be described in detail, those skilled in the art should understand that：Anyone skilled in the art
In the technical scope disclosed by the present invention, it can still modify to technical solution documented by previous embodiment or can be light
It is readily conceivable that variation or equivalent replacement of some of the technical features；And these modifications, variation or replacement, do not make
The essence of corresponding technical solution is detached from the spirit and scope of technical solution of the embodiment of the present invention, should all cover in protection of the invention
Within the scope of.Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. a kind of distributed real-time database system based on power network dispatching system, which is characterized in that including：Data storage component, number
According to speed buffering component, data service component and cluster management assembly；
The data storage component is connected with the data high-speed Buffer Unit, for obtaining company-data, and is responsible for described
The persistence of company-data, wherein the company-data includes model data and real time data；
The data high-speed Buffer Unit, is connected with data service component, for obtaining the real time data, and stores and manages
Manage the real time data；
The data service component is connected with the data storage component, for accessing to the company-data, and
It is responsible for the superfluous standby of the company-data；
The cluster management component, with the data storage component, the data high-speed Buffer Unit and the data service group
Part is connected, and for managing cluster service priority, monitors cluster stage condition, the dilatation contracting of management node resource and cluster
Hold.
2. the distributed real-time database system according to claim 1 based on power network dispatching system, which is characterized in that the collection
Group's data include data fragmentation, metadata and operation log, wherein the data fragmentation will for the distributed real-time database system
Table data are split, the metadata include Data distribution information data, index information data, clustered node information data and
Server data, the operation log are that the data fragmentation is increased, deleted and changed to operate.
3. the distributed real-time database system according to claim 1 based on power network dispatching system, which is characterized in that the number
It include meta data file, data file and journal file according to storage assembly, the data high-speed Buffer Unit includes data fragmentation
Area, meta-data region and operation log area.
4. the distributed real-time database system according to claim 1 based on power network dispatching system, which is characterized in that the number
According to serviced component include data service unit, Metadata Service unit, data subscription service unit, data synchronization service unit and
Data interface unit.
5. the distributed real-time database system according to claim 1 based on power network dispatching system, which is characterized in that the collection
Group's management assembly includes cluster observation unit, service managing unit, node management unit and resilient expansion unit.
6. the distributed real-time database system according to claim 1 based on power network dispatching system, which is characterized in that described point
The clustered deploy(ment) mode of cloth real-time database system is independent deployment or disposes in the application server.
7. the distributed real-time database system according to claim 1 based on power network dispatching system, which is characterized in that described point
The distributed storage rule of cloth real-time database system includes random distribution rule or by area's distribution rule, wherein described random point
Cloth rule be by after table keyword consistency Hash, random storage in clustered node, it is described be by area's distribution rule will be described
Model data presses physical region program and district, defines data allocation rule by customized mode.
8. the distributed real-time database system according to claim 1 based on power network dispatching system, which is characterized in that further include
Distributed real-time data air interface.
9. the distributed real-time database system according to claim 8 based on power network dispatching system, which is characterized in that described point
Cloth real-time data space interface includes man-machine interface access interface and application programming access interface.
10. a kind of distributed real-time database system based on power network dispatching system, which is characterized in that including such as claim 1 to power
Benefit requires 9 described in any item distributed real-time database systems based on power network dispatching system, further includes：
Application programming access interface is by using metadata location mechanism and using divide and conquer as the parallel computation frame of core
Frame is realized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810722584.1A CN108921728B (en) | 2018-07-03 | 2018-07-03 | Distributed real-time library system based on power grid dispatching system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810722584.1A CN108921728B (en) | 2018-07-03 | 2018-07-03 | Distributed real-time library system based on power grid dispatching system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108921728A true CN108921728A (en) | 2018-11-30 |
| CN108921728B CN108921728B (en) | 2020-11-13 |
Family
ID=64424435
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810722584.1A Active CN108921728B (en) | 2018-07-03 | 2018-07-03 | Distributed real-time library system based on power grid dispatching system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108921728B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109933631A (en) * | 2019-03-20 | 2019-06-25 | 江苏瑞中数据股份有限公司 | Distributed parallel database system and data processing method based on Infiniband network |
| CN112162902A (en) * | 2020-09-24 | 2021-01-01 | 国网辽宁省电力有限公司辽阳供电公司 | Multi-active redundancy method for improving operation level of application module of regulation and control system |
| CN113495894A (en) * | 2020-04-01 | 2021-10-12 | 北京京东振世信息技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN114185488A (en) * | 2021-11-29 | 2022-03-15 | 广东财经大学 | Storage optimization method and system for big data cluster |
| CN115882479A (en) * | 2023-02-21 | 2023-03-31 | 国网江西省电力有限公司经济技术研究院 | Distributed energy storage system multi-objective optimization configuration method for toughness improvement |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102662795A (en) * | 2012-03-20 | 2012-09-12 | 浪潮电子信息产业股份有限公司 | Metadata fault-tolerant recovery method in distributed storage system |
| CN103152352A (en) * | 2013-03-15 | 2013-06-12 | 北京邮电大学 | Perfect information security and forensics monitoring method and system based on cloud computing environment |
| CN104123300A (en) * | 2013-04-26 | 2014-10-29 | 上海云人信息科技有限公司 | Data distributed storage system and method |
| CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL all-in-one machine system |
| US20150169624A1 (en) * | 2013-12-13 | 2015-06-18 | BloomReach Inc. | Distributed and fast data storage layer for large scale web data services |
| CN106210053A (en) * | 2016-07-12 | 2016-12-07 | 国家电网公司 | A kind of electrical power system wide-area information management method |
| CN106527981A (en) * | 2016-10-31 | 2017-03-22 | 华中科技大学 | Configuration-based data fragmentation method for adaptive distributed storage system |
| CN106682106A (en) * | 2016-12-05 | 2017-05-17 | 国网宁夏电力公司信息通信公司 | Distributed management system based on massive electric power real-time data |
| US9699205B2 (en) * | 2015-08-31 | 2017-07-04 | Splunk Inc. | Network security system |
| CN107037989A (en) * | 2017-05-17 | 2017-08-11 | 北京小米移动软件有限公司 | Data processing method and device in distributed computing system |
| CN107045422A (en) * | 2016-02-06 | 2017-08-15 | 华为技术有限公司 | Distributed storage method and equipment |
| CN107169144A (en) * | 2017-06-17 | 2017-09-15 | 南京中蓝数智信息技术有限公司 | Block chain distribution account book storage method and server |
-
2018
- 2018-07-03 CN CN201810722584.1A patent/CN108921728B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102662795A (en) * | 2012-03-20 | 2012-09-12 | 浪潮电子信息产业股份有限公司 | Metadata fault-tolerant recovery method in distributed storage system |
| CN103152352A (en) * | 2013-03-15 | 2013-06-12 | 北京邮电大学 | Perfect information security and forensics monitoring method and system based on cloud computing environment |
| CN104123300A (en) * | 2013-04-26 | 2014-10-29 | 上海云人信息科技有限公司 | Data distributed storage system and method |
| US20150169624A1 (en) * | 2013-12-13 | 2015-06-18 | BloomReach Inc. | Distributed and fast data storage layer for large scale web data services |
| CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL all-in-one machine system |
| US9699205B2 (en) * | 2015-08-31 | 2017-07-04 | Splunk Inc. | Network security system |
| CN107045422A (en) * | 2016-02-06 | 2017-08-15 | 华为技术有限公司 | Distributed storage method and equipment |
| CN106210053A (en) * | 2016-07-12 | 2016-12-07 | 国家电网公司 | A kind of electrical power system wide-area information management method |
| CN106527981A (en) * | 2016-10-31 | 2017-03-22 | 华中科技大学 | Configuration-based data fragmentation method for adaptive distributed storage system |
| CN106682106A (en) * | 2016-12-05 | 2017-05-17 | 国网宁夏电力公司信息通信公司 | Distributed management system based on massive electric power real-time data |
| CN107037989A (en) * | 2017-05-17 | 2017-08-11 | 北京小米移动软件有限公司 | Data processing method and device in distributed computing system |
| CN107169144A (en) * | 2017-06-17 | 2017-09-15 | 南京中蓝数智信息技术有限公司 | Block chain distribution account book storage method and server |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109933631A (en) * | 2019-03-20 | 2019-06-25 | 江苏瑞中数据股份有限公司 | Distributed parallel database system and data processing method based on Infiniband network |
| CN113495894A (en) * | 2020-04-01 | 2021-10-12 | 北京京东振世信息技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN112162902A (en) * | 2020-09-24 | 2021-01-01 | 国网辽宁省电力有限公司辽阳供电公司 | Multi-active redundancy method for improving operation level of application module of regulation and control system |
| CN114185488A (en) * | 2021-11-29 | 2022-03-15 | 广东财经大学 | Storage optimization method and system for big data cluster |
| CN115882479A (en) * | 2023-02-21 | 2023-03-31 | 国网江西省电力有限公司经济技术研究院 | Distributed energy storage system multi-objective optimization configuration method for toughness improvement |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108921728B (en) | 2020-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108921728A (en) | Distributed real-time database system based on power network dispatching system | |
| CN106506605B (en) | SaaS application construction method based on micro-service architecture | |
| CN108833503A (en) | A kind of Redis cluster method based on ZooKeeper | |
| CN103688259B (en) | For the method by compressing and file storage carries out automaticdata placement | |
| US9489443B1 (en) | Scheduling of splits and moves of database partitions | |
| CN108183961A (en) | A kind of distributed caching method based on Redis | |
| CN109218100A (en) | Distributed objects storage cluster and its request responding method, system and storage medium | |
| US10657154B1 (en) | Providing access to data within a migrating data partition | |
| CN104735110B (en) | Metadata management method and system | |
| AU2018290745A1 (en) | Systems and methods of database tenant migration | |
| CN108182213A (en) | A kind of data processing optimization device and method based on distributed system | |
| CN107315776A (en) | A kind of data management system based on cloud computing | |
| CN109299056B (en) | A kind of method of data synchronization and device based on distributed file system | |
| CN102033912A (en) | Distributed-type database access method and system | |
| JP5243991B2 (en) | Storage system, capacity management method, and management computer | |
| CN107734066A (en) | A kind of data center's total management system services administering method | |
| US10579597B1 (en) | Data-tiering service with multiple cold tier quality of service levels | |
| JP2007241486A (en) | Memory system | |
| CN111984191A (en) | Multi-client caching method and system supporting distributed storage | |
| CN107426319B (en) | Store controlled shared content item synchronous method, system and storage medium | |
| CN111857977B (en) | Elastic expansion method, device, server and storage medium | |
| CN103595799A (en) | Method for achieving distributed shared data bank | |
| WO2015088813A1 (en) | Elastic data metadata compression in a distributed data grid | |
| US7069270B1 (en) | Automated method and mechanism for converting a single instance application to a multiple instance application | |
| Tsai et al. | Scalable SaaS indexing algorithms with automated redundancy and recovery management. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |