CN106777141B - A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method - Google Patents
A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method Download PDFInfo
- Publication number
- CN106777141B CN106777141B CN201611177776.6A CN201611177776A CN106777141B CN 106777141 B CN106777141 B CN 106777141B CN 201611177776 A CN201611177776 A CN 201611177776A CN 106777141 B CN106777141 B CN 106777141B
- Authority
- CN
- China
- Prior art keywords
- data
- access
- time
- monitoring
- failure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/168—Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/06—Electricity, gas or water supply
Abstract
The invention discloses a kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage methods, comprising the following steps: establishes interface protocol according to each operation system data characteristics and establishes operation of power networks environment and device data model specification；Configure data access strategy, the configuration data verification rule of each operation system；Monitoring access in real time is using the information generated in operational process, for the fault message monitored, carries out data amended record to the data lost during failure；Establish real-time message queue, data broadcasting and the format specification for caching grid operation data, the online monitoring data of parsing, operation data, lightning data and meteorological data are broadcast to and stored in buffered message queue, Hadoop distributed storage file is written by the period；The data of access and monitoring data are visualized.Invention creates operation of power networks environment and device data model specifications, lay a good foundation for integrated power system systematic data modeling.
Description
Technical field
The present invention relates to the running environment of electric system and the access field of storage of facility information, and in particular to fusion power grid
The cross-platform data of running environment and facility information obtains and distributed storage method.
Background technique
Power transmission and transformation equipment state relevant information is valuator device state, diagnostic device failure, dynamic adjustment apparatus of load, comments
Estimate the important evidence that power grid risk influences equipment, it is accurate in real time to obtain the status related informations such as power transmission and transforming equipment, operation, environment
It is to analyse in depth equipment state and the accurate key foundation for controlling capacity of equipment level.
Currently, the data such as equipment involved in power transmission and transformation equipment state relevant information, operation, environment are mainly derived from energy
Management system (EMS), meteorological system, production management system (PMS), power transmission and transforming equipment on-line monitoring system, power grid spatial information
The different business systems such as service platform (GIS), lightning location system, robot used for intelligent substation patrol system.
Since each operation system is based on different platform, different application target, different agreement, the building of different data structure,
The characteristics of highly discrete, spatial and temporal distributions is presented, inconsistent, structuring that there are data-interface specifications between cross-platform data and
The problems such as unstructured data storage mode is not identical, weak related；Secondly, the data of each operation system are substantially all in accordance with relationship
Database or file mode storage are not easy to analyze and handle using Hadoop distributed technology Develop Data；Again, for
The real-time streams such as EMS, meteorology, on-line monitoring, the real-time and accuracy of access way are quickly to judge equipment state
Important link needs to establish a real-time data broadcast and Data Share System.
Summary of the invention
To solve the shortcomings of the prior art, the invention discloses fusion operation of power networks environment and facility informations across flat
Platform data acquisition and distributed storage method, the purpose of the present invention are obtaining data with realization from operation system automatically, are realizing
Uniform data access, real time monitoring calculating, distributed storage and the visual presentation of cross-platform information.
To achieve the above object, concrete scheme of the invention is as follows:
A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method, comprising the following steps:
Interface protocol is established according to each operation system data characteristics and establishes operation of power networks environment and device data model
Specification；
Configure each operation system data access strategy, configuration data verification rule, thus realize in real time access application and
History access application；
Access using log information, fault message and the warning information generated in the process of running and passes through in real time for monitoring,
For the fault message monitored, data amended record is carried out to the data lost during failure；
It establishes data broadcasting and caches the format specification of grid operation data, online monitoring data, operation number parsing
It is broadcast to and is stored in buffered message queue according to, lightning data and meteorological data, it is distributed by period write-in Hadoop
Storage file or HBase；
The data of access and monitoring data are visualized.
Further, the data model specification of above-mentioned foundation is the data attribute of each operation system, including energy pipe
Manage system model specification, lightning location system model specification and meteorological system model specification；
Lightning location system model specification: it formulates interface protocol and obtains lightning stroke time of origin, position, return stroke times；It establishes
Lightning systems model specification includes: thunder and lightning time of origin, accuracy coordinate, latitude coordinate, current strength and return stroke times；
Meteorological system model specification: it formulates interface protocol and obtains weather data, radar diagram data, cloud atlas data and day
Gas forecast data；Establish meteorological system monitoring information model, radar information model, cloud atlas information model and weather forecast information mould
Type；
Energy Management System model specification: it formulates interface protocol and obtains electric current, voltage, active power, reactive power data；
Establishing EMS system model specification includes measuring id, measuring the time, measure type and measuring value.
Further, the data access strategy for configuring each operation system, including the rule with each operation system interaction data
Then agreement, data source address, interface form, wherein the regular agreement of the operation system interaction data mainly includes real-time number
According to interaction protocol, historical data interaction protocol；
Real-time, interactive agreement mainly describes the explanation and system that system data sends required parameter to system data provider
The explanation for the data content attribute that data providing accordingly returns, description trigger mechanism are the frequency for using real-time data interaction protocol
Rate；Historical data interaction protocol and real-time data interaction protocol request and return identical, but non-periodically triggers；
Data source address mainly describes the address ip and port of system data provider's issuing service；
Interface form mainly describes the mode of the publication data of system data provider.
Further, the data check rule of configuration is used to describe for the pre- of the data that obtain from system data provider
Cleaning rule, including the cleaning based on time series, the cleaning based on clustering algorithm, the cleaning based on SVM, account cleaning rule
Then；
Wherein, access application in real time and history access are using integrated data access strategy and data check rule, according to number
Data access and the parsing of each operation system are realized according to model specification.
Further, the real-time access is the process for accessing application in real time using the log information generated in operational process
Information, record WebService starts access, a reading data action message is completed or completed in WebService access；
Fault message is the malfunction of access application in real time, including network failure, service stopping failure and storage failure；
Warning information is described as accessing the alarm status of application, including access delay or storage delay in real time.
Further, the data amended record describes after the fault recovery of real-time access application, and history run connects
Enter the data lost during application access failure.
Further, the format specification established data broadcasting and cache grid operation data, format specification are intended to root
Meet buffered message queue storage format, the data of differentiated service system and searchable inquiry according to data model norm-setting
The method that the traffic table and function decomposition into analytic function of structure go out each system operation data message.
Further, the data broadcasting is the real time data write-in buffered message queue that will be accessed, as real-time analysis
The data source of processing and application function；
The data of Hadoop distributed storage file storage regular persistence from caching message queue, as distribution
The data source of formula analysis processing.
Further, the data and monitoring number using access are accessed by showing interface in real time in the visual presentation
According to using measuring value in access data as the curve graph of time change is shown；Web service state is intuitively shown in monitoring data
With the showing interface of storage state.
Further, monitoring access application in real time further include: monitoring identification access application exists in the process of running
Network failure, service fault or storage failure, specifically:
It when totally service being requested mistake occur, disappears if mistake is not up to n times, is judged as that interruption stops servicing；
It waits and does not occur for overall request N-1 time wrong then releasing failure；
When totally service being requested mistake occur, if overall request reaches n times or more and mistake continuously occurs, judge at this time
Whether web services ip is communication, is judged as service stopping if communication；It does not communicate, is judged as that external network is obstructed；It waits
Service releases failure after having return；
Service monitoring request and the limitation of response duration are configured, if it exceeds the duration of setting, then be judged as request timed out, such as
The fruit N-1 times duration lower than setting then releases request timed out state；
When storage reports an error, then being judged as can not be written；Waiting releases failure after being successfully written；
Setting write-in duration limitation is judged as write latency if write-in is more than the duration of setting；It waits N-1 times and writing
Enter and then releases delaying state lower than duration limitation；
To the data of gaps and omissions in failure logging table, the data lost during application recovery failure are accessed by history.
Further, when realizing the data parsing of each operation system, the data of acquisition is parsed according to model specification, are wrapped
It includes: according to Energy Management System model specification, obtaining operation data analysis of object Property Name and carried out with the specification of caching
Match, operation data structured set object is packaged into after matching；According to lightning systems model specification, thunder is converted using XML component
Electric system data are XML document object, pack lightning data structured set object according to specification matching；According to meteorological system mould
Type specification is converted to the stream character string of BASE64 coding and is instantiated as object picture for picture file data.
Beneficial effects of the present invention:
1, invention creates operation of power networks environment and facility information data models, establish for integrated power system systematic data modeling
Basis is determined, has laid a good foundation for the data interaction inter-sectional with other of power specialty application, power industry.
2, the present invention describes the process for establishing data access application, mentions for the cross-platform data acquisition of other electric system
Technical basis is supplied, the application of data access monitoring means can offer reference to ensure the integrality of access data from now on.
3, real time data is cached using Distributed Message Queue Kafka, number after message number reaches setting value
According to distributed file system is written into, the condition that provides the foundation is handled for power grid big data distributed analysis.
Detailed description of the invention
The cross-platform data of fusion operation of power networks environment and facility information Fig. 1 of the invention obtains and distributed storage method
Flow chart.
Specific embodiment:
The present invention is described in detail with reference to the accompanying drawing:
As shown in Figure 1, the cross-platform data of fusion operation of power networks environment and facility information obtains and distributed storage method,
The following steps are included:
Step (1): interface protocol is established according to each operation system data characteristics and establishes data model specification；
Step (2): the access protocol according to each system in step (1) configures each system data access strategy, configuration data
Verification rule, realizes real-time data imputing system function and historical data access function；
Step (3): log information, fault message, the announcement generated in real-time access function operational process in monitoring step (2)
Alert information etc.；
Step (4): the fault message monitored in analytical procedure (3) retransmits request, loses during amended record failure
Data；For warning information, alarm cause is analyzed, excludes hidden danger.
Step (5): the format rule of broadcast and caching grid operation data are established according to the data model specification in step (1)
Model；
Step (6): according to the format specification in step (5), the online monitoring data accessed in step (2), number is run
It is broadcast in buffered message queue according to, lightning data and meteorological data etc., is written to Hadoop distributed storage file by the period
Or HBase；
Step (7): according to the data model specification in step (1), the data of the access of access application in real time in step (2)
It is visualized with the monitoring data in step (3).
Wherein in step (1), operation of power networks environment and device data model specification are established:
Data acquisition is shared using data-interface, data center, the safety under Network Isolation based on Enterprise Service Bus
The modes such as file transmission define relevant interface, the period, call the parameters such as frequency and object, automatically by configuring corresponding strategies
Data are extracted from operation system, solve platform database access, large data files across platform high speed concurrently reads, is cross-platform
Data security transmission with it is synchronous key issues of.Data are mainly derived from the relevant business application system of power grid, including power transmission and transformation
Equipment Condition Monitoring System, production management system PMS, Energy Management System EMS, power grid GIS GIS, weather information
System, lightning location system, intelligent robot cruising inspection system etc..
Data model specification describes the business datum attribute of system, and model specification includes Energy Management System model rule
Model, lightning location system model specification, meteorological system model specification etc..
To ensure to obtain the accuracy and consistency of data, the source system to grid operation data and facility information includes thunder
Electric positioning system, meteorological system, PMS and EMS etc. embark specification with reference to the data cases that itself is described, and as obtaining
The final explanation of the data taken, such as:
Lightning location system specification: formulating interface protocol method, obtains lightning stroke time of origin, position, return stroke times；It establishes
Lightning systems model specification includes: thunder and lightning time of origin, accuracy coordinate, latitude coordinate, current strength, return stroke times.
Meteorological system specification: formulating interface protocol includes: obtaining weather data method, radar map data method, cloud
Diagram data method, data of weather forecast method；Establish meteorological system monitoring information model (monitoring time, monitoring station, monitoring station institute
Belong to city, county where monitoring station, temperature, humidity, wind scale, wind speed, wind angle, extreme wind speed, very big angle, precipitation, energy
Degree of opinion, air pressure, issuing time, wind direction, very big wind wind scale, very big wind wind direction), radar information model (radar map date, latitude
Degree, longitude, radar map filestream data), cloud atlas information model (cloud atlas date, picture file flow data, centre coordinate), weather
(monitoring time, monitors affiliated city to forecast information model, and county where monitoring forecasts time span, the highest temperature, lowest temperature, weather
Situation, wind-force rank, wind direction code).
EMS Energy Management System: formulating interface protocol data capture method, obtains electric current, voltage, active power, idle
The data such as power；Establishing EMS system model specification includes measuring ID, measuring the time, measure type and measuring value.
In addition, the configuration system data access strategy in step (2) describes the rule with each system interaction data
Agreement, data source address, interface form etc., the regular agreement of system interaction mainly include real-time data interaction protocol, history number
According to interaction protocol etc..Real-time, interactive agreement mainly describe system data to system data provider send required parameter explanation and
The explanation for the data content attribute that system data provider accordingly returns, description trigger mechanism use real-time data interaction protocol
Frequency；Historical data interaction protocol and real-time data interaction protocol request and return identical, but non-periodically triggers；Data source
Location mainly describes the address ip and port of system data provider's issuing service；Interface form mainly describes system data provider
Publication data mode, such as webservice mode.
Access strategy describes to obtain the regular agreement of each system data, data source address, interface form etc., such as:
Lightning systems configuration data content: lightning stroke time of origin, position, return stroke times；Frequency: in real time；Source data address:
Lightning data service IP address and port numbers；Interface form: webservice.
Meteorological system configuration data content: weather data, radar map, cloud atlas, weather forecast information；Frequency: in real time；
Source data address: meteorological data service IP address and port numbers；Interface form: webservice.
EMS system configuration data content: electric current, voltage, active power, reactive power；Frequency: in real time；Source data address:
EMS data service IP address and port numbers；Interface form: webservice.
Configuration data verification rule: the data check rule configuration in the step (2) from each system data for providing
The preprocessing rule of the data just obtained, including the cleaning based on time series, the cleaning based on clustering algorithm, based on SVM's
Cleaning, equipment account cleaning etc., such as:
Cleaning rule of the EMS data configuration based on time series；
Meteorological data configures the cleaning based on clustering algorithm；
Lightning data configures the cleaning based on clustering algorithm.
In step (2), creation access data application: Integrated access strategy, data check rule, according in step (1)
Data model specification realizes operation system data access and parsing.
According to each operation system interface protocol, is generated using the wsimport tool of wsdl.jar and be based on webservice
Client, read operation system access strategy configuration, data check rule configuration parameter, such as: lightning location system frequency be 5
Minute；Energy Management System frequency is 1 minute and configures based on time series cleaning rule；Monitoring data frequency in meteorological data
For 10 minutes and configure based on clustering algorithm cleaning rule, radar map data frequency is 5 minutes, cloud atlas data frequency is 1 hour,
Data of weather forecast frequency 24 hours.
The data of acquisition are parsed according to model specification, such as: according to Energy Management System model specification, obtaining operation data pair
It is matched as parsing Property Name and with the specification of caching, operation data structured set object is packaged into after matching；According to
Lightning systems model specification converts lightning systems data using XML component as XML document object, packs thunder according to specification matching
Electric data structured collection object；BASE64 coding is converted to for picture file data according to meteorological system model specification
Stream character string is simultaneously instantiated as object picture.
Monitoring access is applied: network failure, service fault existing for monitoring identification access application in the process of running are deposited
Failure is stored up, method includes:
It when totally service being requested mistake occur, disappears if mistake is not up to n times, is judged as that interruption stops servicing；
It waits and does not occur for overall request N-1 time wrong then releasing failure.
When totally service being requested mistake occur, if overall request reaches n times or more and mistake continuously occurs, judge at this time
Whether web services ip is communication, is judged as service stopping if communication；It does not communicate, is judged as that external network is obstructed；It waits
Service releases failure after having return.
Service monitoring request and the limitation of response duration are configured, if it exceeds the duration of setting, then be judged as request timed out, such as
The fruit N-1 times duration lower than setting then releases request timed out state.
When storage reports an error, then being judged as can not be written；Waiting releases failure after being successfully written.
Setting write-in duration limitation is judged as write latency if write-in is more than the duration of setting；It waits N-1 times and writing
Enter and then releases delaying state lower than duration limitation.
The fault message of monitoring is applied by monitoring agent and is sent to being used for for monitoring server by socket socket
On socket server, and by monitoring server storage to failure logging table
To the data of gaps and omissions in failure logging table, the data lost during application recovery failure are accessed by history.
Log information in step (3) describes the procedural information of the real-time access application in step (2), record
The movements such as WebService starts access, WebService access is completed or completes a reading data.
Fault message describes the malfunction of the real-time access application in step (2), including network failure, service stop
Only failure, storage failure etc..
Warning information describes the alarm status of the real-time access application in step (2), including access delay or storage
Delay etc..
After data amended record in step (4) describes the fault recovery of the real-time access application in step (2), fortune
The data lost during history access application access failure in row step (2).
Format specification in step (5), it is intended to which buffered message is met according to the system model norm-setting in step (1)
The traffic table and function decomposition into analytic function of the data structure of queue storage format, differentiated service system and searchable inquiry go out each system fortune
The design method of row data-message.
Data broadcasting in step (6) is the real time data write-in buffered message queue that will be accessed, at real-time analysis
The data source of reason and application function.
The data of Hadoop distributed storage file storage regular persistence from caching message queue, as distribution point
Analyse the data source of processing.
Specifically, data broadcasting is shared and distributed storage: these three are big using Redis, kafka and Hadoop for this method
Data tool realizes the distribution of high-throughput for power grid environment and operation data broadcast, shared and distributed storage, kafka
News release is subscribed to, and redis realizes that data high-speed caching, Hadoop realize mass data distributed storage, pass through parallel mechanism
Unify Message Processing on line and offline, passes through cluster and consumption in real time is provided.Critical process storing data structure is focused on
It explains and is illustrated below with EMS data:
Real-time data memory format specification:
Key: being identified to measurement, and format is " ed&qy&rid ", wherein " ed " is identified as EMS system data, " qy " mark
For area, " rid " is to measure id；Value: gathering for map, under key identified time；Value identifies measuring value.Show
Example: " ed&bz&212023,201510210914=32.90289057791233 ".
Real-time data broadcast format specification:
Key: being identified to the time, and format is " region & measures the id& time "；Value: it is identified as measuring value.Example:
" bz&212023&201510210914,32.90289057791233 ".
Data distribution formula storage format specification:
Key: being identified to the time, and format is " region & measures the id& time "；Value: it is identified as measuring value.Example:
" bz&212023&201510210914,32.90289057791233 ".
Visual presentation in step (7) describes the data of the access of access application in real time in showing interface step (2)
With the monitoring data in step (3), using access data such as Energy Management System (EMS) in measuring value with time change song
Line chart is shown；Such as intuitive showing interface for showing Web service state and storage state of monitoring data.Monitoring is shown, main to dock
Enter network in application process, storage state shows, including real time monitoring interface and flow are shown.
Visualize: visualization mainly divides initial data displaying and monitoring to show that the present invention is illustrated below:
Initial data is shown, to data creation one intuitive understandable interface of access.EMS competence management system number
Show that interface main presentation measures the variation of point data, the curve that interface changes comprising time change according to middle metric data.
Monitor interface in real time, it is intuitive to show Web service state and storage state, green representative is normal, yellow represents warning,
Red represents catastrophe failure.Two processes of external communication and storage inside of monitoring, external communication include monitoring: service requests, is outer
Portion's network state；The state of storage inside monitoring write-in data.
Flow shows interface, and the data traffic being related to each request of data is monitored, and monitoring content includes completing
Time, request measure number, return measures number, insertion measures number, are finally shown with line chart.
The cross-platform operation system data of effective integration of the present invention, form the multi-source heterogeneous pattern of fusion power transmission and transformation of distributed storage
Device status information resource, solution PMS, EMS, status monitoring, GIS, meteorology, thunder and lightning etc. are multi-platform, apply, more communication protocols more
View, the mass data of more data structures can not be accessed accurately in real time the unified platform the problem of, meet the analysis of data distribution formula, place
Reason and visualizing requires, realize the uniform data access of cross-platform information, real time monitoring calculate, distributed storage and visual
Change and shows.
The present invention using webservice technology, socket socket mechanics of communication, business datum modeling and analytic technique,
Distributed message frame kafka, Hadoop distributed storage technology, provides interface data model specification, data access strategy is matched
It sets, data-interface parameter configuration, data check rule configuration, data access monitoring, data amended record, data distribution formula storage tube
A series of functions such as reason, visual presentation, obtain data by above functions from operation system automatically, realize cross-platform information
Uniform data access, real time monitoring calculate, distributed storage and visual presentation.
The present invention is that base has been established in integrated power system systematic data modeling, power industry and other inter-sectional data interactions
Plinth；Technical basis is provided for the cross-platform data acquisition of other later electric system；Data can be accessed to ensure from now on
Integrality is used for reference；The condition that provides the foundation is handled for power grid big data distributed analysis.
Above-mentioned, although the foregoing specific embodiments of the present invention is described with reference to the accompanying drawings, not protects model to the present invention
The limitation enclosed, those skilled in the art should understand that, based on the technical solutions of the present invention, those skilled in the art are not
Need to make the creative labor the various modifications or changes that can be made still within protection scope of the present invention.
Claims (9)
1. a kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method, characterized in that the following steps are included:
Interface protocol is established according to each operation system data characteristics and establishes operation of power networks environment and device data model specification；
Data access strategy, the configuration data verification rule for configuring each operation system, to realize access application in real time and history
Access application；
Access is using log information, fault message and the warning information generated in operational process in real time for monitoring, for what is monitored
Fault message carries out data amended record to the data lost during failure；
It establishes data broadcasting and caches the format specification of grid operation data, the online monitoring data, operation data, thunder of parsing
Electric data and meteorological data are broadcast to and are stored in buffered message queue, by period write-in Hadoop distributed storage text
Part or HBase；
The data of access and monitoring data are visualized；
The data access strategy for configuring each operation system, including regular agreement, the data source with each operation system interaction data
Address, interface form, wherein the regular agreement of the operation system interaction data mainly includes real-time data interaction protocol, goes through
History data interaction agreement；
Real-time, interactive agreement mainly describes explanation and the system data that system data sends required parameter to system data provider
The explanation for the data content attribute that provider accordingly returns, description trigger mechanism are the frequency for using real-time data interaction protocol；
Historical data interaction protocol and real-time data interaction protocol request and return identical, but non-periodically triggers；
Data source address mainly describes the address ip and port of system data provider's issuing service；
Interface form mainly describes the mode of the publication data of system data provider.
2. a kind of acquisition for merging multi-source heterogeneous electric network data as described in claim 1 and distributed storage method, feature
It is that the data model specification of foundation is the data attribute of each operation system, including Energy Management System model specification, thunder and lightning
Positioning system models specification and meteorological system model specification；
Lightning location system model specification: it formulates interface protocol and obtains lightning stroke time of origin, position, return stroke times；Establish thunder and lightning
System model specification includes: thunder and lightning time of origin, accuracy coordinate, latitude coordinate, current strength and return stroke times；
Meteorological system model specification: it is pre- that interface protocol acquisition weather data, radar diagram data, cloud atlas data and weather are formulated
Count off evidence；Establish meteorological system monitoring information model, radar information model, cloud atlas information model and weather forecast information model；
Energy Management System model specification: it formulates interface protocol and obtains electric current, voltage, active power, reactive power data；It establishes
EMS system model specification includes measuring id, measuring the time, measure type and measuring value.
3. a kind of acquisition for merging multi-source heterogeneous electric network data as described in claim 1 and distributed storage method, feature
It is that the prerinse that the data check rule of configuration is used to describe for the data obtained from system data provider is regular, including
Cleaning based on time series, the cleaning based on clustering algorithm, the cleaning based on SVM, account cleaning rule；
Wherein, access application in real time and history access are using integrated data access strategy and data check rule, according to data mould
Type specification realizes data access and the parsing of each operation system.
4. a kind of acquisition for merging multi-source heterogeneous electric network data as described in claim 1 and distributed storage method, feature
It is that the real-time access is the procedural information for accessing application in real time, record using the log information generated in operational process
WebService starts access, a reading data action message is completed or completed in WebService access；
Fault message is the malfunction of access application in real time, including network failure, service stopping failure and storage failure；
Warning information is described as accessing the alarm status of application, including access delay or storage delay in real time.
5. a kind of acquisition for merging multi-source heterogeneous electric network data as described in claim 1 and distributed storage method, feature
It is that the format specification established data broadcasting and cache grid operation data, format specification is intended to according to data model specification
Formulate meet buffered message queue storage format, the traffic table of differentiated service system and the data structure of searchable inquiry and
The method that function decomposition into analytic function goes out each system operation data message.
6. a kind of acquisition for merging multi-source heterogeneous electric network data as claimed in claim 1 or 5 and distributed storage method, special
Sign is that the data broadcasting is the real time data write-in buffered message queue that will be accessed, and as real-time analysis processing and applies function
The data source of energy；
The data of Hadoop distributed storage file storage regular persistence from caching message queue, as distribution point
Analyse the data source of processing.
7. a kind of acquisition for merging multi-source heterogeneous electric network data as described in claim 1 and distributed storage method, feature
It is that the data and monitoring data using access are accessed by showing interface in real time in the visual presentation, using access data
Middle measuring value is shown with the curve graph of time change；The boundary of Web service state and storage state is intuitively shown in monitoring data
Face is shown.
8. a kind of acquisition for merging multi-source heterogeneous electric network data as described in claim 1 and distributed storage method, feature
It is that access is applied in real time for the monitoring further include: network failure, service existing for monitoring identification access application in the process of running
Failure or storage failure, specifically:
It when totally service being requested mistake occur, disappears if mistake is not up to n times, is judged as that interruption stops servicing；It waits
Do not occur for overall request N-1 time wrong then releasing failure；
When totally service being requested mistake occur, if overall request reaches n times or more and mistake continuously occurs, web is judged at this time
It services whether ip is communication, is judged as service stopping if communication；It does not communicate, is judged as that external network is obstructed；Etc. to be serviced
Failure is released after having return；
Service monitoring request and the limitation of response duration are configured, if it exceeds the duration of setting, then be judged as request timed out, if N-
1 duration lower than setting then releases request timed out state；
When storage reports an error, then being judged as can not be written；Waiting releases failure after being successfully written；
Setting write-in duration limitation is judged as write latency if write-in is more than the duration of setting；Wait N-1 write-in low
Delaying state is then released in duration limitation；
To the data of gaps and omissions in failure logging table, the data lost during application recovery failure are accessed by history.
9. a kind of acquisition for merging multi-source heterogeneous electric network data as described in claim 1 and distributed storage method, feature
When being the data parsing for realizing each operation system, the data of acquisition are parsed according to model specification, comprising: according to energy pipe
System model specification is managed, operation data analysis of object Property Name is obtained and is matched with the specification of caching, is packed after matching
At operation data structured set object；According to lightning systems model specification, it is using XML component conversion lightning systems data
XML document object packs lightning data structured set object according to specification matching；According to meteorological system model specification, for
Picture file data is converted to the stream character string of BASE64 coding and is instantiated as object picture.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611177776.6A CN106777141B (en) | 2016-12-19 | 2016-12-19 | A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611177776.6A CN106777141B (en) | 2016-12-19 | 2016-12-19 | A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106777141A CN106777141A (en) | 2017-05-31 |
| CN106777141B true CN106777141B (en) | 2019-07-12 |
Family
ID=58890179
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611177776.6A Active CN106777141B (en) | 2016-12-19 | 2016-12-19 | A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106777141B (en) |
Families Citing this family (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107220360A (en) * | 2017-06-07 | 2017-09-29 | 云南电网有限责任公司信息中心 | A kind of Unified Modeling storage cut-in method based on magnanimity electric power monitoring data |
| CN107341205A (en) * | 2017-06-23 | 2017-11-10 | 国网上海市电力公司 | A kind of intelligent distribution system based on big data platform |
| CN107633004A (en) * | 2017-08-07 | 2018-01-26 | 国网新疆电力公司信息通信公司 | Data monitoring method and device |
| CN107656971A (en) * | 2017-09-02 | 2018-02-02 | 国网辽宁省电力有限公司 | A kind of intelligent grid collection Monitoring Data storage method based on Redis |
| CN107590749A (en) * | 2017-09-07 | 2018-01-16 | 北京国电通网络技术有限公司 | A kind of processing method and system with electricity consumption data |
| CN107769993A (en) * | 2017-09-19 | 2018-03-06 | 广西电网有限责任公司电力科学研究院 | Towards the data traffic monitoring method of power network big data distributed system |
| CN107730040B (en) * | 2017-09-30 | 2021-03-16 | 国网山东省电力公司电力科学研究院 | RBM-based log information comprehensive feature extraction method and device for power information system |
| CN107634963B (en) * | 2017-10-12 | 2020-03-10 | 中国铁塔股份有限公司 | On-site monitoring unit registration system and method |
| CN107943835A (en) * | 2017-10-26 | 2018-04-20 | 中国南方电网有限责任公司 | It is a kind of to report and submit data analysis and taxis system for electric system |
| CN107968840B (en) * | 2017-12-15 | 2020-10-09 | 华北电力大学（保定） | Real-time processing method and system for monitoring alarm data of large-scale power equipment |
| CN108229799B (en) * | 2017-12-20 | 2022-04-05 | 江苏瑞中数据股份有限公司 | Multi-source heterogeneous power grid operation real-time data access system and method |
| CN108074044A (en) * | 2018-01-19 | 2018-05-25 | 广东电网有限责任公司珠海供电局 | A kind of power equipment state monitoring evaluation and command methods |
| CN108319695A (en) * | 2018-02-02 | 2018-07-24 | 华自科技股份有限公司 | Power station fault data processing method, device, computer equipment and storage medium |
| CN108737550A (en) * | 2018-05-25 | 2018-11-02 | 山东鲁能智能技术有限公司 | A kind of robot interface's device and robot based on robot centralized control system |
| CN109033174A (en) * | 2018-06-21 | 2018-12-18 | 北京国网信通埃森哲信息技术有限公司 | A kind of power quality data cleaning method and device |
| CN109165207B (en) * | 2018-07-16 | 2021-11-26 | 华南农业大学 | Drinking water mass data storage management method and system based on Hadoop |
| CN110019488A (en) * | 2018-09-12 | 2019-07-16 | 国网浙江省电力有限公司嘉兴供电公司 | Multi-source heterogeneous data fusion multi-core classification method |
| CN109871375A (en) * | 2018-09-12 | 2019-06-11 | 国网浙江省电力有限公司嘉兴供电公司 | The information platform and its control method of distributed new scale access |
| CN109558397B (en) * | 2018-10-30 | 2023-08-22 | 深圳平安医疗健康科技服务有限公司 | Data processing method, device, server and computer storage medium |
| CN109413185A (en) * | 2018-10-31 | 2019-03-01 | 国网河南省电力公司焦作供电公司 | A kind of equipment routing inspection system and its Cloud Server design method |
| CN110502517B (en) * | 2019-08-23 | 2022-01-28 | 中国南方电网有限责任公司 | Distributed storage system for storing real-time operation data of power grid |
| CN111262915B (en) * | 2020-01-10 | 2020-09-22 | 北京东方金信科技有限公司 | Kafka cluster-crossing data conversion system and method |
| CN111506409B (en) * | 2020-04-20 | 2023-05-23 | 南方电网科学研究院有限责任公司 | Data processing method and system |
| CN111667143A (en) * | 2020-04-30 | 2020-09-15 | 广西电网有限责任公司 | Method for realizing fusion of real-time monitoring data and service management data in power grid dispatching |
| CN111798096A (en) * | 2020-06-02 | 2020-10-20 | 国网山西省电力公司电力科学研究院 | Visual analysis method, system and storage medium based on power grid dispatching data |
| CN111752723B (en) * | 2020-06-06 | 2021-05-04 | 中国科学院电子学研究所苏州研究院 | Visual multi-source service management system and implementation method thereof |
| CN112235361B (en) * | 2020-09-28 | 2022-12-27 | 青海绿能数据有限公司 | Photovoltaic power plant data switching platform |
| CN112308538B (en) * | 2020-12-03 | 2022-08-05 | 国网福建省电力有限公司厦门供电公司 | Active service data interaction method for multi-type park users |
| CN112650761A (en) * | 2021-02-23 | 2021-04-13 | 善智互联(北京)网络科技有限公司 | Structured extraction method and system for transformer fault information |
| CN113642850A (en) * | 2021-07-20 | 2021-11-12 | 国网江苏省电力有限公司南通供电分公司 | Data fusion method and terminal for power distribution network planning |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617557A (en) * | 2013-11-06 | 2014-03-05 | 广东电网公司电力科学研究院 | Multi-source heterogeneous power grid operation parameter analysis system |
| CN103941158A (en) * | 2014-04-17 | 2014-07-23 | 国家电网公司 | Power distribution network fault diagnosis system and method based on multi-source information |
| CN104240021A (en) * | 2014-09-04 | 2014-12-24 | 海南国能中海智能技术有限公司 | Visual, traceable and controllable power grid managing system and method |
| CN104852927A (en) * | 2015-06-01 | 2015-08-19 | 国家电网公司 | Safety comprehensive management system based on multi-source heterogeneous information |
| CN105184424A (en) * | 2015-10-19 | 2015-12-23 | 国网山东省电力公司菏泽供电公司 | Mapreduced short period load prediction method of multinucleated function learning SVM realizing multi-source heterogeneous data fusion |
| CN105335896A (en) * | 2014-08-13 | 2016-02-17 | 国家电网公司 | Multi-source heterogeneous data processing method and device used for power grid |
| CN105427188A (en) * | 2015-12-02 | 2016-03-23 | 国家电网公司 | Power supply control system and method based on multi-source heterogeneous information fusion |
| CN105574643A (en) * | 2015-11-23 | 2016-05-11 | 江苏瑞中数据股份有限公司 | Real-time data center and big data platform fusion method for power grid |
-
2016
- 2016-12-19 CN CN201611177776.6A patent/CN106777141B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617557A (en) * | 2013-11-06 | 2014-03-05 | 广东电网公司电力科学研究院 | Multi-source heterogeneous power grid operation parameter analysis system |
| CN103941158A (en) * | 2014-04-17 | 2014-07-23 | 国家电网公司 | Power distribution network fault diagnosis system and method based on multi-source information |
| CN105335896A (en) * | 2014-08-13 | 2016-02-17 | 国家电网公司 | Multi-source heterogeneous data processing method and device used for power grid |
| CN104240021A (en) * | 2014-09-04 | 2014-12-24 | 海南国能中海智能技术有限公司 | Visual, traceable and controllable power grid managing system and method |
| CN104852927A (en) * | 2015-06-01 | 2015-08-19 | 国家电网公司 | Safety comprehensive management system based on multi-source heterogeneous information |
| CN105184424A (en) * | 2015-10-19 | 2015-12-23 | 国网山东省电力公司菏泽供电公司 | Mapreduced short period load prediction method of multinucleated function learning SVM realizing multi-source heterogeneous data fusion |
| CN105574643A (en) * | 2015-11-23 | 2016-05-11 | 江苏瑞中数据股份有限公司 | Real-time data center and big data platform fusion method for power grid |
| CN105427188A (en) * | 2015-12-02 | 2016-03-23 | 国家电网公司 | Power supply control system and method based on multi-source heterogeneous information fusion |
Non-Patent Citations (3)
| Title |
|---|
| "基于大数据平台的电网防灾调度系统功能设计与系统架构";王逸飞 等;《电 网 技 术》;20161031;全文 |
| "智能电网大数据平台及其关键技术研究";孟祥君 等;《供用电》;20151231;第19-23页 |
| "面向多源异构电网数据的获取与转换技术研究";周爱华 等;《电力信息与通信技术》;20151231;22-25页、图1-图2 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106777141A (en) | 2017-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106777141B (en) | A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method | |
| RU2546320C2 (en) | Intelligent network | |
| CN111464336B (en) | High-concurrency data processing method and system based on electric power communication machine room | |
| CN111864898B (en) | Three-dimensional information system based on power transmission online monitoring data and control method thereof | |
| CN109146093A (en) | A kind of electric power equipment on-site exploration method based on study | |
| CN106603958B (en) | Distributed storage remote centralized monitoring system and method for transformer substation inspection robot | |
| CN103700031A (en) | Electric power warning information publishing method in regulation and control integration mode | |
| CN111275948B (en) | Electric fire fighting early warning system | |
| CN112186901B (en) | Panoramic sensing monitoring method and system for transformer substation | |
| CN109309733A (en) | A kind of Internet of Things energy monitoring system | |
| CN113868078A (en) | Wind power plant monitoring method based on cloud platform | |
| CN212779278U (en) | Optical cable state monitoring system | |
| CN110932405A (en) | Intelligent monitoring and analyzing system for power transformation equipment based on big data | |
| CN111327468A (en) | Operation method and system for edge computing platform of power system | |
| CN113849553A (en) | Substation data acquisition and processing method and system based on Internet of things equipment data acquisition | |
| CN114827140A (en) | Real-time data centralized management and control system for wind tunnel site | |
| Ju et al. | The use of edge computing-based internet of things big data in the design of power intelligent management and control platform | |
| CN103605767A (en) | Data access method and system of power transmission lines | |
| CN114065955A (en) | Power cable visual monitoring maintenance management system based on GIS | |
| CN203261356U (en) | Remote data monitoring platform for renewable energy building application system | |
| CN113064399A (en) | Industrial monitoring software prediction maintenance system based on big data distributed programming framework | |
| CN112947364A (en) | System and method for early warning of equipment faults of power distribution station based on big data | |
| CN110647070A (en) | Power environment monitoring system for super-large-scale data center | |
| CN114278517A (en) | Wind power plant monitoring system based on time sequence database | |
| CN114860839A (en) | Power distribution network data analysis and visualization system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 250002 Wang Yue Road, Ji'nan City, Shandong Province, No. 2000 Patentee after: ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID SHANDONG ELECTRIC POWER Co. Patentee after: STATE GRID SHANDONG ELECTRIC POWER Co. Patentee after: Shandong luruan Digital Technology Co.,Ltd. Patentee after: STATE GRID CORPORATION OF CHINA Address before: 250002 Wang Yue Road, Ji'nan City, Shandong Province, No. 2000 Patentee before: ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID SHANDONG ELECTRIC POWER Co. Patentee before: STATE GRID SHANDONG ELECTRIC POWER Co. Patentee before: SHANDONG LUNENG SOFTWARE TECHNOLOGY Co.,Ltd. Patentee before: State Grid Corporation of China |
|
| CP01 | Change in the name or title of a patent holder |