CN106055590A - Power grid data processing method and system based on big data and graph database - Google Patents
Power grid data processing method and system based on big data and graph database Download PDFInfo
- Publication number
- CN106055590A CN106055590A CN201610343953.7A CN201610343953A CN106055590A CN 106055590 A CN106055590 A CN 106055590A CN 201610343953 A CN201610343953 A CN 201610343953A CN 106055590 A CN106055590 A CN 106055590A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- service request
- database
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 39
- 238000000034 method Methods 0.000 claims description 34
- 238000003860 storage Methods 0.000 claims description 32
- 230000008569 process Effects 0.000 claims description 13
- 230000003068 static effect Effects 0.000 claims description 12
- 230000005012 migration Effects 0.000 claims description 11
- 238000013508 migration Methods 0.000 claims description 11
- 230000035945 sensitivity Effects 0.000 claims description 7
- 230000002123 temporal effect Effects 0.000 claims description 5
- 238000010586 diagram Methods 0.000 claims description 4
- 239000000203 mixture Substances 0.000 claims description 2
- 230000008901 benefit Effects 0.000 abstract description 4
- 238000005457 optimization Methods 0.000 abstract description 2
- 238000004220 aggregation Methods 0.000 abstract 1
- 230000002776 aggregation Effects 0.000 abstract 1
- 230000006870 function Effects 0.000 description 13
- 230000009897 systematic effect Effects 0.000 description 7
- 238000005259 measurement Methods 0.000 description 6
- 230000004044 response Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 230000014509 gene expression Effects 0.000 description 4
- 230000001105 regulatory effect Effects 0.000 description 4
- 230000011664 signaling Effects 0.000 description 4
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 235000013399 edible fruits Nutrition 0.000 description 3
- 230000005611 electricity Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000013075 data extraction Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000002567 autonomic effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004040 coloring Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000005194 fractionation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Water Supply & Treatment (AREA)
- Tourism & Hospitality (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Primary Health Care (AREA)
- Marketing (AREA)
- Human Resources & Organizations (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention relates to the technical field of database application, and discloses a power grid data processing method and system based on big data and graph database. The power grid data processing method comprises the steps of receiving a data operation service request through a unified interface; performing data routing on the data operation service request; executing data operation by one or more of the graph database, a cache database and a memory file system according to the data routing strategy, and obtaining one or more corresponding execution results; performing data aggregation on the one or more execution results, and obtaining a final result corresponding to the data operation service request. and outputting the final result through the unified interface. According to the technical scheme, optimization modeling is carried out for the power grid through various data processing means according to data characteristics and performance requirements of the power grid, business requirements of various different data types in the real power grid are satisfied by full utilization of the advantages of the various data processing means, and therefore the whole performance of the system can be largely improved.
Description
Technical field
The present invention relates to database application technical field, particularly relate to a kind of based on big data and the power network of chart database
Network data processing method and system.
Background technology
In the long history of Database Technolgy Development, relevant database is always in occupation of leading position, but relationship type
Data base is the most unable to do what one wishes when the big request of data that reply magnanimity and height are concurrent, exposes the problem being much difficult to overcome.
On this basis, for the challenge solving large-scale data set and multiple data kind is brought, the biggest market demand
A difficult problem, the data base (NoSQL) of various non-relationals occurs rapidly and develops rapidly.
Currently, the large database of traditional industries, can not adaptive system extension mainly based on relevant database
And the needs that user's request increases, seriously hinder the development of industry.
Typically, as a example by power system of the prior art, when it uses existing relational data library management electric power
Network data also completes total data when processing, and is primarily present following problem:
The issued transaction of the most traditional relevant database typically can only support the data volume of millions, data volume is the biggest,
Performance is the poorest, cannot adapt to the development of large-scale power network the most completely；
2. being affected by traditional Relational DataBase data-storing total amount and performance, existing electric power networks is difficult to full dose and preserves
Operation data (operation data of general only preservation 30%), and storage limited time (about 2 years)；
The most traditional relevant database does not has to be described for the feature of electric power networks data, manage, classify and excellent
Changing, data processing form is limited, it is impossible to meet various business demand, and a lot of particular service response speeds are extremely slow, inefficiency.
Summary of the invention
For the defect of relevant database in prior art, present invention is primarily targeted at offer a kind of based on big number
According to and the electric power networks data handling system of chart database and method, strengthened the data of system by the data base of mixed type
Disposal ability.
For reaching above-mentioned purpose, on the one hand, the invention provides a kind of based on big data and the electric power networks of chart database
Data processing method, including step:
By unified interface data manipulation service request；
Described data manipulation service request is carried out data route；
The strategy routeing according to described data, by chart database, cache database and memory file system or
The described data manipulation of multiple execution, and obtain one or more execution results of correspondence；
The one or more is performed result and carries out convergence, it is thus achieved that described data manipulation service request is corresponding
Termination fruit；
Described final result is exported by described unified interface.
Preferably, described to described data manipulation service request carry out data route include: described data manipulation is serviced
Data content in request is carried out splitting and be routed to corresponding data processing means by data type.
Preferably, in described method: described chart database, described cache database and described memory file system are the most right
Static data, real time data and historical data in electric power networks performs described data manipulation and obtains the execution result of correspondence.
Preferably, described method also includes: be divided into cold and hot according to configurable temporal sensitivity by described historical data
Data, wherein, dsc data is saved in memory file system, and cold data are saved in cloud storage system.
Preferably, described method also includes: automatically judge the cold and hot degree of described historical data according to migration rules, described
Cold data are automatically migrated in described cloud storage system, and described dsc data is loaded onto described internal memory literary composition from cloud storage system automatically
In part system；Wherein, in described memory file system, described historical data is performed described data manipulation all the time.
On the other hand, the present invention provides the process of a kind of electric power networks data based on big data and chart database the most simultaneously
System, including service agent module, data routing module and data processing module, described service agent module includes receiver module
And output module, described data routing module includes that request splits module and result convergence module, wherein:
Described receiver module, for by unified interface data manipulation service request；
Described request routing module, for carrying out data route to described data manipulation service request；
Described data processing module, for the strategy that route according to described data, by chart database, cache database and interior
Deposit the described data manipulation of one or more execution in file system, and obtain one or more execution results of correspondence；
Described result convergence module, carries out convergence for the one or more is performed result, it is thus achieved that described number
According to the final result that operation service request is corresponding；
Described output module, for exporting described final result by described unified interface.
Preferably, described request routing module includes: split routing module, for by described data manipulation service request
In data content carry out splitting and be routed to corresponding data processing means by data type.
Preferably, described data processing module includes chart database module, data cached library module and memory file system
System module, wherein: described chart database module, described data cached library module and described memory file system module are for respectively
To the static data in electric power networks, real time data and historical data perform described data manipulation and obtain correspondence perform knot
Really.
Preferably, described data processing module also includes: historical data divides module, for according to the configurable time
Described historical data is divided into cold and hot data by sensitivity, and wherein, dsc data is saved in memory file system, and cold data preserve
In cloud storage system.
Preferably, described data processing module also includes: Data Migration module, for automatically judging according to migration rules
The cold and hot degree of described historical data, described cold data are automatically migrated in described cloud storage system, and described dsc data is deposited from cloud
Storage system is loaded onto in described memory file system automatically.
The technical scheme of the embodiment of the present invention provides at a kind of electric power networks data based on big data and chart database
Reason system and method, is optimized modeling by multiple data processing means to electric power networks, makes full use of multiple data and processes
The speciality of means, meets the business demand of various different types of data in real power network, thus it is overall to reach system
The significant increase of performance.
Accompanying drawing explanation
Fig. 1 is the stream of electric power networks data processing method based on big data and chart database in one embodiment of the invention
Journey schematic diagram；
Fig. 2 is the mould of electric power networks data handling system based on big data and chart database in another embodiment of the present invention
Block structure schematic diagram；
Fig. 3 is the complete of electric power networks data handling system based on big data and chart database in yet another embodiment of the invention
Illustrative view of functional configuration.
Detailed description of the invention
For making the goal of the invention of the present invention, feature, the advantage can be the most obvious and understandable, below in conjunction with the present invention
Accompanying drawing in embodiment, is clearly and completely described the technical scheme in the embodiment of the present invention, it is clear that described reality
Executing example is only a part of embodiment of the present invention, and not all embodiments.Based on the embodiment in the present invention, people in the art
The every other embodiment that member is obtained under not making creative work premise, broadly falls into the scope of protection of the invention.
Traditional relevant database only processes (increase, delete, change) aspect better performances at underlying transaction, and other such as expand
The many-sides such as exhibition ability, concurrency, storage total amount and multiformity are the most undesirable, seriously limit the development of power system.Existing
Although technology also occurs in that some non-relational database (NoSQL), but on the one hand by the electricity of ultra-large type relevant database
The cost and risk that Force system is completely migrating to non-relational database is the highest, and on the other hand non-relational database is at data
During reason, self there is also certain defect.Such as, key-value (key-value) although class data base simply, easily dispose, in data
Access aspect performance and efficiency are the highest, but its data Un-structured, the form of expression is limited, is difficult to the business that reply is complex
Demand；And although chart database can preferably express the topological structure of electric power networks, data access, the efficiency inquired about and travel through are also
Higher, but because being limited by graph structure, chart database many times needs whole figure calculates the letter that just can draw needs
Breath, therefore underlying transaction process performance is poor, and also its data structure poorly does distributed type colony scheme.
For the drawbacks described above of prior art, the invention provides a kind of brand-new based on big data and the electricity of chart database
The technical scheme of power network data processing, in an embodiment of the present invention, is entered chart database by multiple data processing means
Row is supported, thus significantly improves its open defect while making full use of the performance characteristics of chart database, utilizes multiple number
According to the combination of the means of process, respectively take the chief, divide and rule, fully meet the various different types of data in real power network
Business demand, with reach systematic entirety can significant increase.Specifically, as it is shown in figure 1, The embodiment provides
A kind of electric power networks data processing method based on big data and chart database, including step:
S1, by unified interface data manipulation service request；
In a preferred embodiment of the invention, provide system platform and outside mutual unique interface by service broker,
The service that this interface can be supported by include but not limited to certification, authenticate, control, data input, the function such as data output function, institute
State concrete operations type and the data content including data manipulation in data manipulation service request.
S2, carries out data route to described data manipulation service request；
In a preferred embodiment of the invention, the data content in described data manipulation service request is entered by data type
Row splits and is routed to corresponding data processing means.For example, the preferred embodiment of the present invention is considered as from performance
Multiple data acess method processes different types of data, and therefore, the outside data entering system platform need fractionation to deposit,
The data of external inquiry need to extract respectively synthesis；Additionally, include that deletion, message, notice etc. control class data and are typically also required to
Implement respectively to control.
S3, the strategy routeing according to described data, by one in chart database, cache database and memory file system
Or the multiple described data manipulation of execution, and obtain one or more execution results of correspondence；
In a preferred embodiment of the invention, by multiple data acess method achieve in real power network each
Plant the support of the business demand of different types of data；Preferably data acess method includes chart database, cache database and interior
That deposits in file system is one or more；It is highly preferred that described chart database, described cache database and described memory file system
One or more for distributed system in system.In further preferred embodiment of the present invention, described chart database, described slow
Deposit data storehouse and described memory file system perform institute to the static data in electric power networks, real time data and historical data respectively
State data manipulation and obtain the execution result of correspondence.
Wherein, according to graph theory, figure is a class abstract data structure more common in computer science, in structurally and semantically side
Face is more increasingly complex than linear list and tree, has more general expression ability, it is believed that diagram data is the set of point, line, tree.Structure
The most crucial of chart database is become to have two: point (Vertex), limit (edge)；Wherein limit has directivity, comprises unidirectional, double
To or undirected three kinds of states；Limit also can attach weight properties, represents significance level.And electric power networks is typically by power equipment and line
Road is constituted, and wherein power equipment is equivalent to the node of chart database, and circuit is equivalent to limit, and electric power flows to consumption end structure from generating end
Become the direction on limit.Use chart database can the topological structure of perfect expressions electric power networks, electric power networks number based on chart database
According to model in power system modeling, 3D solid displaying, power network topology analysis, electrical network coloring, the generation of feeder line group, state computation etc.
Aspect has greater advantage.Although chart database provides concordance support to affairs, but Core Superiority be data query with
Traversal, and additions and deletions change and wait transaction operation performance general, so common chart database is more suitable for storage static data (basic data).
And in general electrical network, such as remote measurement class dynamic real time data frequency acquisition is at a relatively high, even up to 10 Milliseconds, the highest
The data of frequency update, considerably beyond the UPS upper performance score of common chart database；Additionally, the historical data of remote measurement is mainly used in emulation
Calculate and planning application, need up to the several years or permanently store, and common chart database does not preserve node or side attribute
Historical information.Therefore, simple chart database is not particularly suited for processing electric power networks data.
Data characteristics and the demand of electric power networks is analyzed further, first, in electric power networks for these embodiments of the invention
Equipment or circuit all comprise many attribute (key-value to), divide according to temporal sensitivity, and these attributes can be divided into static state
Attribute (essential information) and dynamic attribute (secondary metamessage).Static attribute does not changes, as title, type, place,
GIS coordinate etc.；Dynamic attribute includes remote measurement, remote signalling, remote control, remote regulating information and various analysis and calculates intermediate data etc.；This
Outward, dynamic attribute is some information persistently changed, and the historical variations process of these information also it is frequently necessary in electric power networks
Inquire about, contrast and analyze, therefore electric power networks there is also the process demand to historical data.
Specifically, in electric power networks, important dynamic attribute generally includes:
Remote measurement (telemetry intelligence (TELINT)): the most remotely measure, refers generally to gather and transmit operational factor, including various electric parameters (lines
The values such as voltage on road, electric current, power) and load uncertainty etc.；
Remote signalling (remote signalling information): i.e. remote signal, refers generally to gather and transmit various protection and switching value information；
Remote control (remote information): the most remotely control, refers generally to accept and perform remote control command, mainly divide-shut brake,
I.e. some switching control devices are remotely controlled；
Remote regulating (remote regulating information): i.e. remote adjustment, refers generally to accept and perform remote adjustment order, to long-range controlled quentity controlled variable
Equipment carries out remote debugging, such as regulator generator output etc..
Wherein, most typical historical data is processed as the extraction of time profile data, and it is mainly for the historical data of remote measurement,
The relevant database of prior art is in the case of time span is relatively big, and extracted data efficiency is the lowest, it is often necessary to tens
More than minute；And chart database is owing to cannot effectively save historical data, this type of data manipulation also cannot be completed.
As can be seen here, perfection to support electric power networks, correlation data processing system at least needs to solve three class data and processes
Optimization problem: static data (basic data), real time data (remote measurement, remote signalling, remote control, remote regulating, the middle junction that calculates and analyze
Fruit etc.), historical data (time discontinuity surface extraction, relate generally to telemetry).
Chart database in the preferred embodiment of the present invention mainly preserves Back ground Information (the static letter on electric power networks node and limit
Breath) and topology information, because this kind of data renewal frequency is the lowest, the affairs performance of chart database be enough to support.Chart database is arrogated to oneself
Length storage has the network data structure of connecting relation, and is adapted for the process of big data, is relating to path computing and is changing simultaneously
There is greater advantage (from a minute level, a hour level, original data-handling capacity can be promoted to second level or Millisecond sound for aspect
Should), thus the structure of electric power networks and big data quantity can be carried out ideal statement and storage.The present invention's is further
In preferred embodiment, form distributed chart database by rationally splitting network boundary, make set expandability to be greatly promoted, can
Support the node of more than 1,000,000,000 grades, the relation of more than 10,000,000,000 grades easily.
The concurrent reading and writing performance of the cache database in the preferred embodiment of the present invention is optimal, main preserves dynamic data
New state, also preserves the business datum that business requirement of real-time is the highest, and cache database possesses affairs strong consistency.The present invention's
In further preferred embodiment, can promote further at the Real-time and Concurrent data of system by building distributed caching data base
Reason ability.By above-mentioned preferred implementation, the present invention can by original for power system data-handling capacity from a minute level, hour
Level, or even the business demand that can not complete, be all promoted to the response of second level even Millisecond, greatly improve systematic function.
Memory file system in the preferred embodiment of the present invention mainly preserves the historical information of dynamic data, and relatively
New data (historical data as produced in a year), the data pick-up of discontinuity surface when serving primarily in.Memory file system master
To improve system effectiveness by the way of shared drive, at concurrent reading and writing aspect of performance, memory file system is inferior to data cached
Storehouse and be better than chart database, can effectively support simulation calculation based on historical data.The further preferred embodiment of the present invention
In, the parallel processing capability of system can be promoted by building distributed memory file system further, and it is superfluous to reduce internal memory
Remaining, GC (Garbage Collection, the garbage reclamation) time etc..The application of memory file system, can be by respective time span
Bigger time discontinuity surface historical data extraction is improved to delete the response of second level.
Based on above-mentioned setting, the data content in described data manipulation service request is entered by data route according to data type
Row distribution, described chart database, described cache database and described memory file system are respectively to the static number in electric power networks
Perform described data manipulation according to, real time data and historical data and obtain the execution result of correspondence.
S4, performs result to the one or more and carries out convergence, it is thus achieved that described data manipulation service request correspondence
Final result；
Owing to different types of data is provided different execution result, such intermediate object program by different data processing means
And be unfavorable for that user understands and uses, therefore in a preferred embodiment of the invention, also the one or more performed result and enter
Row convergence, it is thus achieved that the final result that described data manipulation service request is corresponding.Data route now is main to performing knot
Fruit merges and/or logical operations, to ensure that final result expression form is unified, structure is consistent and the most reliable.
S5, exports described final result by described unified interface.
Finally, in a preferred embodiment of the invention, provide system platform mutual with outside again by service broker
Unique interface, now this interface is for exporting the final result that described data manipulation service request is corresponding.
In embodiments of the invention, by multiple data processing means, electric power networks is supported, makes full use of multiple
The speciality of data processing means, meets the business demand of various different types of data in real power network, thus reaches
The significant increase of systematic entirety energy.
Below by some preferred embodiments, technical scheme is carried out more detailed analytic explanation.
In a preferred embodiment of the invention, Back ground Information, topology information are stored in chart database, real time information
Using cache database, storage of history data P is in memory file system；Although in view of memory file system better performances, but
Relatively costly, and the data volume of the historical data of electric power networks also tends to (need to preserve several years even tens, decades relatively greatly
Data), therefore preferably further according to configurable temporal sensitivity, historical data is divided into cold and hot data.The most several
The data in year are relatively important and are divided into dsc data, and data several years ago are the most inessential and are divided into cold data, and business is frequent
The data accessed are relatively important and are divided into dsc data, and Operational Visit less data is the most inessential and is divided into cold data etc.
Deng.Dsc data total amount is less, is saved in better performances, relatively costly memory file system, and cold data total amount is more, preserves
In the cloud storage system of, poor-performing relatively low at cost, the preferred embodiment had both improve business operation performance, also reduced total
Body use cost.
Further, cloud storage system is preferably distributed file system, its can infinite expanding, possess data redundancy, from
The functions such as the migration of visibly moved mistake, fault, data high availability, support the data storage of more than P level.
In order to ensure effective management of cold and hot data, it is preferable that the present invention also carries out Autonomic Migration Framework, root to cold and hot data
Automatically judge that according to migration rules the cold and hot degree of historical data, cold data (historical data as the year before) are automatically migrated to cloud and deposit
In storage system, advantageously reduce cost；Dsc data is loaded onto in memory file system from cloud storage system automatically, to promote system
System performance.
Additionally, the preferred embodiment of the present invention further enhances systematic function and/or reliability also by some measures.
Such as, carry out safety and rights management, include but not limited to perform the authentication of external user (or service), purview certification,
Data safety certification etc.；Or provide data directory service, to realize quickly positioning target data.Further, due to
Each data processing means in the embodiment of the present invention the most all uses distributed structure/architecture, although the dispersion storage of each data processing means
Data quite independent, but remain a need for distributed transaction support, therefore preferably also provide for the distributed transaction coordination ability to ensure electricity
The integrity of network data.
By the above embodiment of the present invention, its technical scheme can effectively manage has complex topology structure in electric power networks
Mass data, Various types of data can be substantially improved simultaneously and process the performance of operation.
Further regard to Fig. 2, Fig. 2 illustrate provided in an alternative embodiment of the invention based on big data and figure number
According to the structural representation of the electric power networks data handling system in storehouse, for convenience of description, illustrate only and embodiment of the present invention phase
The part closed.The electric power networks data handling system 1 based on big data and chart database of Fig. 2 example mainly includes service broker
Module 11, data routing module 12 and data processing module 13, described service agent module 11 includes receiver module 111 and output
Module 112, described data routing module 12 includes asking routing module 121 and result convergence module 122, wherein, each function mould
Block describes in detail as follows:
Described receiver module 111, for by unified interface data manipulation service request；
Described request routing module 121, for carrying out data route to described data manipulation service request；
Described data processing module 13, for the strategy that route according to described data, by chart database, cache database and
The described data manipulation of one or more execution in memory file system, and obtain one or more execution results of correspondence；
Described result convergence module 122, carries out convergence for the one or more is performed result, it is thus achieved that described
The final result that data manipulation service request is corresponding；
Described output module 112, for exporting described final result by described unified interface.
Preferably, described request routing module includes: split routing module, for by described data manipulation service request
In data content carry out splitting and be routed to corresponding data processing means by data type.
Preferably, described data processing module includes chart database module, data cached library module and memory file system
System module, wherein: described chart database module, described data cached library module and described memory file system module are for respectively
To the static data in electric power networks, real time data and historical data perform described data manipulation and obtain correspondence perform knot
Really.
Preferably, described data processing module also includes: historical data divides module, for according to the configurable time
Described historical data is divided into cold and hot data by sensitivity, and wherein, dsc data is saved in memory file system, and cold data preserve
In cloud storage system.
Preferably, described data processing module also includes: Data Migration module, for automatically judging according to migration rules
The cold and hot degree of described historical data, described cold data are automatically migrated in described cloud storage system, and described dsc data is deposited from cloud
Storage system is loaded onto in described memory file system automatically.
Further, in the preferred embodiment of the present invention global function structure of above-mentioned data handling system as it is shown on figure 3, this is complete
Functional structure further enhancing the safety and reliability of system while ensureing systematic function.
It should be noted that electric power networks data based on big data and chart database described in various embodiments of the present invention
In processing system, the division of each functional module is merely illustrative of, in actual application can as required, such as corresponding hardware
The convenient consideration of the realization of configuration requirement or software, and above-mentioned functions distribution is completed by different functional modules, will
The internal structure of data handling system is divided into different functional modules, to complete all or part of function described above.
And, in actual application, the corresponding functional module in the present embodiment can be to be realized by corresponding hardware, it is also possible to by accordingly
Hardware perform corresponding software complete.Each embodiment that this specification provides all can apply foregoing description principle.
Additionally, data handling system described in various embodiments of the present invention is preferably integrated in one or more server apparatus
In, can be independent hardware, software and/or the firmware unit being loaded by server apparatus or calling further, it is also possible to be
The modular unit corresponding to software directly performed by server apparatus.
The embodiment of the present invention do not uses up details, refers to the corresponding description of earlier figures 1 method illustrated embodiment.
In several embodiments provided by the present invention, it should be understood that disclosed system and method, can be passed through it
Its mode realizes.Such as, system embodiment described above is only schematically, such as, and the division of described module, only
Being only a kind of logic function to divide, actual can have other dividing mode, the most multiple modules or assembly to tie when realizing
Close or be desirably integrated into another system, or some features can be ignored, or not performing.Another point, shown or discussed
Coupling each other or direct-coupling or communication connection can be the INDIRECT COUPLING by some interfaces, system or module or logical
Letter connects, and can be electrical, machinery or other form.
The described module illustrated as separating component can be or may not be physically separate, shows as module
The parts shown can be or may not be physical module, i.e. may be located at a place, or can also be distributed to multiple
On mixed-media network modules mixed-media.Some or all of module therein can be selected according to the actual needs to realize the mesh of the present embodiment scheme
's.
It addition, each functional module in each embodiment of the present invention can be integrated in a processing module, it is also possible to
It is that modules is individually physically present, it is also possible to two or more modules are integrated in a module.Above-mentioned integrated mould
Block both can realize to use the form of hardware, it would however also be possible to employ the form of software function module realizes.
If described integrated module realizes and as independent production marketing or use using the form of software function module
Time, can be stored in a computer read/write memory medium.Based on such understanding, technical scheme is substantially
The part that in other words prior art contributed or this technical scheme completely or partially can be with the form of software product
Embodying, this computer software product is stored in a storage medium, including some instructions with so that a computer
Equipment (can be personal computer, server, or the network equipment etc.) performs the complete of method described in each embodiment of the present invention
Portion or part steps.And aforesaid storage medium includes: USB flash disk, portable hard drive, read only memory (ROM, Read-Only
Memory), random access memory (RAM, Random Access Memory), magnetic disc or CD etc. are various can store journey
The medium of sequence code.
It should be noted that for aforesaid each method embodiment, in order to simplicity describes, therefore it is all expressed as a series of
Combination of actions, but those skilled in the art should know, the present invention is not limited by described sequence of movement because
According to the present invention, some step can use other order or carry out simultaneously.Secondly, those skilled in the art also should know
Knowing, it might not be all this that embodiment described in this description belongs to preferred embodiment, involved action and module
Bright necessary.
In the above-described embodiments, the description to each embodiment all emphasizes particularly on different fields, and does not has the portion described in detail in certain embodiment
Point, may refer to the associated description of other embodiments.
The technical scheme of the embodiment of the present invention provides at a kind of electric power networks data based on big data and chart database
Reason method and system, are supported electric power networks by multiple data processing means, make full use of multiple data processing means
Speciality, meet the business demand of various different types of data in real power network, thus reach systematic entirety energy
Significant increase.
Further, the technical scheme of the embodiment of the present invention has a following remarkable result:
1. achieve the high-performance of power industry full dose data, permanent storage scheme power industry available data storage
Rate less than 30%, the storage of history data P time at about several years, and the technical scheme of the embodiment of the present invention can support 1,000,000,000 grades with
On node (equipment), the limit (circuit) of more than 10,000,000,000 grades, more than P level historical data storage；
2. the technical scheme that the embodiment of the present invention is substantially improved achieving systematic entirety energy is former by power system
There is data-handling capacity from a minute level, hour level, or even the business demand that can not complete, be all promoted to second level or Millisecond
Response, the time discontinuity surface historical data extraction that respective time span is bigger is promoted to a minute level response；
3. achieve and process the technical scheme that the embodiment of the present invention is substantially improved on scope of data border by network number
Being promoted to 1,000,000,000 grades or more than 10,000,000,000 grades according to storage from millions, conceptual data amount of storage is promoted to more than P level within T level.
The above, only presently preferred embodiments of the present invention, be not intended to limit protection scope of the present invention, all
Any amendment, equivalent and the improvement etc. made within the spirit and principles in the present invention, should be included in the protection of the present invention
Within the scope of.
Claims (10)
1. an electric power networks data processing method based on big data and chart database, it is characterised in that described method includes
Step:
By unified interface data manipulation service request；
Described data manipulation service request is carried out data route；
The strategy routeing according to described data, one or more by chart database, cache database and memory file system
Perform described data manipulation, and obtain one or more execution results of correspondence；
The one or more is performed result and carries out convergence, it is thus achieved that the termination that described data manipulation service request is corresponding
Really；
Described final result is exported by described unified interface.
Method the most according to claim 1, it is characterised in that described to described data manipulation service request carry out data road
By including:
Carry out splitting and being routed at corresponding data by data type by the data content in described data manipulation service request
Reason means.
Method the most according to claim 1, it is characterised in that in described method:
Described chart database, described cache database and described memory file system respectively to the static data in electric power networks,
Real time data and historical data perform described data manipulation and obtain the execution result of correspondence.
Method the most according to claim 3, it is characterised in that described method also includes:
According to configurable temporal sensitivity, described historical data being divided into cold and hot data, wherein, dsc data is saved in internal memory
In file system, cold data are saved in cloud storage system.
Method the most according to claim 4, it is characterised in that described method also includes:
Automatically judge that according to migration rules the cold and hot degree of described historical data, described cold data are automatically migrated to the storage of described cloud
In system, described dsc data is loaded onto in described memory file system from cloud storage system automatically；Wherein, all the time in described
Deposit in file system and described historical data is performed described data manipulation.
6. an electric power networks data handling system based on big data and chart database, it is characterised in that described data process
System includes service agent module, data routing module and data processing module, and described service agent module includes receiver module
And output module, described data routing module includes asking routing module and result convergence module, wherein:
Described receiver module, for by unified interface data manipulation service request；
Described request routing module, for carrying out data route to described data manipulation service request；
Described data processing module, for the strategy routeing according to described data, by chart database, cache database and internal memory literary composition
The described data manipulation of one or more execution in part system, and obtain one or more execution results of correspondence；
Described result convergence module, carries out convergence for the one or more is performed result, it is thus achieved that described data are grasped
Make the final result that service request is corresponding；
Described output module, for exporting described final result by described unified interface.
Data handling system the most according to claim 6, it is characterised in that described request routing module includes:
Split routing module, for carrying out splitting and road by data type by the data content in described data manipulation service request
By to corresponding data processing means.
Data handling system the most according to claim 6, it is characterised in that described data processing module includes diagram data
Library module, data cached library module and memory file system module, wherein:
Described chart database module, described data cached library module and described memory file system module are for respectively to power network
Static data, real time data and historical data in network performs described data manipulation and obtains the execution result of correspondence.
Data handling system the most according to claim 8, it is characterised in that also include in described data processing module:
Historical data divides module, for described historical data being divided into cold and hot data according to configurable temporal sensitivity,
Wherein, dsc data is saved in memory file system, and cold data are saved in cloud storage system.
Data handling system the most according to claim 9, it is characterised in that also include in described data processing module:
Data Migration module, for automatically judging the cold and hot degree of described historical data according to migration rules, described cold data are certainly
Moving moves in described cloud storage system, and described dsc data is loaded onto described memory file system from cloud storage system automatically
In.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610343953.7A CN106055590A (en) | 2016-05-23 | 2016-05-23 | Power grid data processing method and system based on big data and graph database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610343953.7A CN106055590A (en) | 2016-05-23 | 2016-05-23 | Power grid data processing method and system based on big data and graph database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106055590A true CN106055590A (en) | 2016-10-26 |
Family
ID=57176569
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610343953.7A Pending CN106055590A (en) | 2016-05-23 | 2016-05-23 | Power grid data processing method and system based on big data and graph database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106055590A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106973036A (en) * | 2017-02-07 | 2017-07-21 | 杭州云象网络技术有限公司 | A kind of block chain method for secret protection based on asymmetric encryption |
| CN107169032A (en) * | 2017-04-17 | 2017-09-15 | 北京五八信息技术有限公司 | Database operation method and database server |
| CN107315856A (en) * | 2017-05-27 | 2017-11-03 | 全球能源互联网研究院 | The modeling method and device of a kind of power grid GIS based on chart database |
| CN108399206A (en) * | 2018-01-31 | 2018-08-14 | 佛山市聚成知识产权服务有限公司 | A kind of big data processing system |
| CN109947859A (en) * | 2017-08-25 | 2019-06-28 | 南方电网科学研究院有限责任公司 | Power distribution network drawing modeling method, system, storage medium and computer equipment |
| CN111562889A (en) * | 2020-05-14 | 2020-08-21 | 杭州海康威视系统技术有限公司 | Data processing method, device, system and storage medium |
| CN112100654A (en) * | 2020-09-02 | 2020-12-18 | 军事科学院系统工程研究院网络信息研究所 | Terminal-based equipment database resource management method |
| CN113672774A (en) * | 2021-07-29 | 2021-11-19 | 国电南瑞科技股份有限公司 | Distribution network equipment topology coloring method and device based on distribution cloud master station and graph database |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090326726A1 (en) * | 2008-06-25 | 2009-12-31 | Versify Solutions, Llc | Aggregator, monitor, and manager of distributed demand response |
| CN101826018A (en) * | 2010-05-20 | 2010-09-08 | 瑞斯康达科技发展股份有限公司 | Interactive development system and method for interface of network element management system |
| CN102982075A (en) * | 2012-10-30 | 2013-03-20 | 北京京东世纪贸易有限公司 | Heterogeneous data source access supporting system and method thereof |
| CN103605698A (en) * | 2013-11-06 | 2014-02-26 | 广东电子工业研究院有限公司 | Cloud database system used for distributed heterogeneous data resource integration |
| CN103778212A (en) * | 2014-01-16 | 2014-05-07 | 国网山东省电力公司青岛供电公司 | Data node-based parallel massive data processing method |
| CN103839138A (en) * | 2014-03-08 | 2014-06-04 | 成都文昊科技有限公司 | System for supporting interaction of multiple heterogeneous systems |
| CN104123437A (en) * | 2014-06-30 | 2014-10-29 | 华南理工大学 | Power transmission line crossover identification method and system based on graph database and spatial information |
| CN104516967A (en) * | 2014-12-25 | 2015-04-15 | 国家电网公司 | Electric power system mass data management system and use method thereof |
-
2016
- 2016-05-23 CN CN201610343953.7A patent/CN106055590A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090326726A1 (en) * | 2008-06-25 | 2009-12-31 | Versify Solutions, Llc | Aggregator, monitor, and manager of distributed demand response |
| CN101826018A (en) * | 2010-05-20 | 2010-09-08 | 瑞斯康达科技发展股份有限公司 | Interactive development system and method for interface of network element management system |
| CN102982075A (en) * | 2012-10-30 | 2013-03-20 | 北京京东世纪贸易有限公司 | Heterogeneous data source access supporting system and method thereof |
| CN103605698A (en) * | 2013-11-06 | 2014-02-26 | 广东电子工业研究院有限公司 | Cloud database system used for distributed heterogeneous data resource integration |
| CN103778212A (en) * | 2014-01-16 | 2014-05-07 | 国网山东省电力公司青岛供电公司 | Data node-based parallel massive data processing method |
| CN103839138A (en) * | 2014-03-08 | 2014-06-04 | 成都文昊科技有限公司 | System for supporting interaction of multiple heterogeneous systems |
| CN104123437A (en) * | 2014-06-30 | 2014-10-29 | 华南理工大学 | Power transmission line crossover identification method and system based on graph database and spatial information |
| CN104516967A (en) * | 2014-12-25 | 2015-04-15 | 国家电网公司 | Electric power system mass data management system and use method thereof |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106973036A (en) * | 2017-02-07 | 2017-07-21 | 杭州云象网络技术有限公司 | A kind of block chain method for secret protection based on asymmetric encryption |
| CN106973036B (en) * | 2017-02-07 | 2020-04-14 | 杭州云象网络技术有限公司 | Block chain privacy protection method based on asymmetric encryption |
| CN107169032A (en) * | 2017-04-17 | 2017-09-15 | 北京五八信息技术有限公司 | Database operation method and database server |
| CN107315856A (en) * | 2017-05-27 | 2017-11-03 | 全球能源互联网研究院 | The modeling method and device of a kind of power grid GIS based on chart database |
| CN109947859A (en) * | 2017-08-25 | 2019-06-28 | 南方电网科学研究院有限责任公司 | Power distribution network drawing modeling method, system, storage medium and computer equipment |
| CN109947859B (en) * | 2017-08-25 | 2023-12-05 | 南方电网科学研究院有限责任公司 | Power distribution network drawing modeling method, system, storage medium and computer equipment |
| CN108399206A (en) * | 2018-01-31 | 2018-08-14 | 佛山市聚成知识产权服务有限公司 | A kind of big data processing system |
| CN111562889A (en) * | 2020-05-14 | 2020-08-21 | 杭州海康威视系统技术有限公司 | Data processing method, device, system and storage medium |
| CN112100654A (en) * | 2020-09-02 | 2020-12-18 | 军事科学院系统工程研究院网络信息研究所 | Terminal-based equipment database resource management method |
| CN113672774A (en) * | 2021-07-29 | 2021-11-19 | 国电南瑞科技股份有限公司 | Distribution network equipment topology coloring method and device based on distribution cloud master station and graph database |
| CN113672774B (en) * | 2021-07-29 | 2023-09-29 | 国电南瑞科技股份有限公司 | Distribution network equipment topology coloring method and device based on distribution cloud master station and graph database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106055590A (en) | Power grid data processing method and system based on big data and graph database | |
| Zhao et al. | Geographical information system parallelization for spatial big data processing: a review | |
| Zheng et al. | Real-time big data processing framework: challenges and solutions | |
| CN102255962B (en) | Distributive storage method, device and system | |
| CN105139281A (en) | Method and system for processing big data of electric power marketing | |
| CN106611046A (en) | Big data technology-based space data storage processing middleware framework | |
| CN106708993A (en) | Spatial data storage processing middleware framework realization method based on big data technology | |
| CN105183834A (en) | Ontology library based transportation big data semantic application service method | |
| CN109918450B (en) | Distributed parallel database based on analysis type scene and storage method | |
| CN104504008B (en) | A kind of Data Migration algorithm based on nested SQL to HBase | |
| CN106453618A (en) | Remote sensing image processing service cloud platform system based on G-Cloud cloud computing | |
| CN102857560A (en) | Multi-service application orientated cloud storage data distribution method | |
| CN102158533B (en) | Distributed web service selection method based on QoS (Quality of Service) | |
| CN104572505A (en) | System and method for ensuring eventual consistency of mass data caches | |
| CN109947880A (en) | Three-dimensional spatial information serves urban-rural integration management isomeric data fusion method | |
| Jing et al. | An optimized method of HDFS for massive small files storage | |
| CN104219088A (en) | Hive-based network alarm information OLAP method | |
| CN102355502A (en) | Remote access method for remotely accessing storage system into desktop operation system | |
| CN115994197A (en) | GeoSOT grid data calculation method | |
| CN103365923A (en) | Method and device for assessing partition schemes of database | |
| Xie et al. | On massive spatial data retrieval based on spark | |
| CN104391947A (en) | Real-time processing method and system of mass GIS (geographic information system) data | |
| Dawei et al. | Research on the application of distributed key-value storage technology in computer database platform | |
| CN109687464B (en) | Power grid power flow analysis method and device, storage medium and processor | |
| CN104484763A (en) | Three-dimensional method and system for beautiful village and town construction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161026 |
|
| RJ01 | Rejection of invention patent application after publication |