The content of the invention
It is to solve the deficiency that prior art is present, the invention discloses fusion operation of power networks environment and facility information across flat
Platform data acquisition and distributed storage method, the purpose of the present invention are obtaining data from operation system automatically with realization, realize
The uniform data of cross-platform information is accessed, monitor in real time is calculated, distributed storage and visual presentation.
To achieve the above object, concrete scheme of the invention is as follows:
A kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method, comprise the following steps:
Interface protocol is set up according to each operation system data characteristics and set up operation of power networks environment and device data model
Specification;
Configure each operation system data access strategy, configuration data verification rule so that realize in real time access application and
History accesses application;
Monitoring is accessed using log information, fault message and the warning information for producing in the process of running and passed through in real time,
For the fault message for monitoring, to failure during the data lost carry out data amended record;
The format specification of data broadcasting and caching operation of power networks data is set up, online monitoring data, operation number parsing
It is distributed by cycle write-in Hadoop in being broadcast to and stored buffered message queue according to, lightning data and meteorological data
Storage file or HBase;
The data and monitoring data of access are carried out into visual presentation.
Further, the data model specification of above-mentioned foundation is the data attribute of each operation system, including energy pipe
Reason system model specification, lightning location system model specification and meteorological system model specification;
Lightning location system model specification:Formulate interface protocol and obtain thunderbolt time of origin, position, return stroke times;Set up
Lightning systems model specification includes:Thunder and lightning time of origin, accuracy coordinate, latitude coordinate, current strength and return stroke times;
Meteorological system model specification:Formulate interface protocol and obtain weather data, radar diagram data, cloud atlas data and day
Gas forecast data;Set up meteorological system monitoring information model, radar information model, cloud atlas information model and weather forecast information mould
Type;
EMS model specification:Formulate interface protocol and obtain electric current, voltage, active power, reactive power data;
Setting up EMS system model specification includes measuring id, measurement time, measures type and measuring value.
Further, the data access strategy of each operation system, including the rule with each operation system interaction data are configured
Then agreement, data source address, interface shape, wherein, the regular agreement of the operation system interaction data mainly includes number in real time
According to interaction protocol, historical data interaction protocol;
The main descriptive system data of real-time, interactive agreement send explanation and the system of required parameter to system data provider
The explanation of the data content attribute that data providing is accordingly returned, description trigger mechanism is the frequency for using real-time data interaction protocol
Rate;Historical data interaction protocol and real-time data interaction protocol are asked and return identical, but non-periodically triggers;
The ip addresses and port of the main descriptive system data providing issuing service of data source address;
The mode of the issue data of the main descriptive system data providing of interface shape.
Further, the data check rule of configuration is pre- for the data that are obtained from system data provider for describing
Cleaning rule, including the cleaning based on time series, the cleaning based on clustering algorithm, the cleaning based on SVM, account cleaning rule
Then;
Wherein, application is accessed in real time and history accesses application integrated data access strategy and data check rule, according to number
Data access and the parsing of each operation system are realized according to model specification.
Further, the real-time access using the log information produced in running is the real-time process for accessing application
Information, record WebService starts access, WebService and accesses completion or complete a digital independent action message;
Fault message is to access the malfunction applied in real time, including network failure, service stopping failure and storage failure;
Warning information is described as accessing in real time the alarm status of application, including access delay or storage delay.
Further, the data amended record is described after the fault recovery for accessing application in real time, and history run connects
Enter the data lost during application accesses failure.
Further, the format specification for setting up data broadcasting and caching operation of power networks data, format specification is intended to root
According to the data for meeting buffered message queue storage format, differentiated service system and searchable inquiry of data model norm-setting
The method that the traffic table and function decomposition into analytic function of structure go out each system operation data message.
Further, the data broadcasting is that the real time data that will be accessed writes buffered message queue, used as analysis in real time
Treatment and the data source of application function;
The data of Hadoop distributed storages file storage regular persistence from caching message queue, as distribution
The data source of formula analyzing and processing.
Further, accessed in real time using the data and monitoring number for accessing by showing interface in the visual presentation
According to the curve map changed over time using measuring value in access data shows;Displaying Web service state directly perceived in monitoring data
With the showing interface of storage state.
Further, the monitoring accesses application and also includes in real time:Monitoring identification accesses application to be existed in the process of running
Network failure, service fault or storage failure, specially:
When mistake occurs in overall request service, disappeared if mistake is not up to n times, be then judged as that interruption stops service;
Wait overall request not occur mistake for N-1 times and then release failure;
When mistake occurs in overall request service, if overall request reaches n times and mistake continuously occurs in the above, now judge
Whether web services ip is communication, and service stopping is judged as if communication;Do not communicate, be judged as that external network is obstructed;Wait
Service releases failure after having return;
Configuration service monitoring request and the limitation of response duration, if it exceeds the duration for setting, then be judged as request timed out, such as
Fruit then releases request timed out state N-1 times less than the duration for setting;
When storage reports an error, then it is judged as writing;Wait releases failure after being successfully written;
Write-in duration limitation is set, if write-in exceedes the duration for setting, is judged as write latency;Wait N-1 times and writing
Enter and then release delaying state less than duration limitation;
To the data of gaps and omissions in failure logging table, the data lost during accessing application recovery failure by history.
Further, when realizing that the data of each operation system are parsed, the data to obtaining are parsed according to model specification, bag
Include:According to EMS model specification, obtain service data analysis of object Property Name and carried out with the specification of caching
Match somebody with somebody, service data structured set object is packaged into after matching;According to lightning systems model specification, thunder is changed using XML components
Electric system data are XML document object, and lightning data structured set object is packed according to specification matching;According to meteorological system mould
Type specification, for picture file data, is converted to the stream character string of BASE64 codings and is instantiated as object picture.
Beneficial effects of the present invention:
1st, it is that integrated power system systematic data modeling is established invention creates operation of power networks environment and facility information data model
Basis is determined, has been that power specialty application, power industry and other inter-sectional data interactions are laid a good foundation.
2nd, the present invention describes to set up the process of data access application, is that the cross-platform data acquisition of other power systems is carried
Technical basis are supplied, the application of data access monitoring means can be to ensure that the integrality for accessing data is offered reference from now on.
3rd, real time data is cached using Distributed Message Queue Kafka, number after message number reaches setting value
It is that power network big data distributed analysis process the condition that provides the foundation according to distributed file system is written into.
Specific embodiment:
The present invention is described in detail below in conjunction with the accompanying drawings:
As shown in figure 1, the cross-platform data of fusion operation of power networks environment and facility information is obtained and distributed storage method,
Comprise the following steps:
Step (1):Interface protocol is set up according to each operation system data characteristics and set up data model specification;
Step (2):Each system data access strategy, configuration data are configured according to the access protocol of each system in step (1)
Verification rule, realizes real-time data imputing system function and historical data access function;
Step (3):Log information, fault message, the announcement produced in real-time access function running in monitoring step (2)
Alert information etc.;
Step (4):The fault message monitored in analytical procedure (3), loses during resending request, amended record failure
Data;For warning information, alarm cause is analyzed, exclude hidden danger.
Step (5):The form rule of broadcast and caching operation of power networks data are set up according to the data model specification in step (1)
Model;
Step (6):According to the format specification in step (5), the online monitoring data, operation number that are accessed in step (2)
It is broadcast in buffered message queue according to, lightning data and meteorological data etc., Hadoop distributed storage files is written to by the cycle
Or HBase;
Step (7):According to the data model specification in step (1), the data for accessing application access in step (2) in real time
Visual presentation is carried out with the monitoring data in step (3).
Wherein in step (1), operation of power networks environment and device data model specification are set up:
Data acquisition is the safety using data-interface, data center be shared, under Network Isolation based on ESB
The modes such as file transmission, by configuring corresponding strategies, define related interface, cycle, call the parameters such as frequency and object, automatically
The extracted data from operation system, solution platform database is accessed, large data files across platform concurrently reads at a high speed, cross-platform
Security Data Transmission and the key issue such as synchronous.Data are mainly derived from the related business application system of power network, including power transmission and transformation
Equipment Condition Monitoring System, production management system PMS, EMS EMS, power grid GIS GIS, weather information
System, lightning location system, intelligent robot cruising inspection system etc..
Data model specification describes the business datum attribute of system, and model specification is advised including EMS model
Model, lightning location system model specification, meteorological system model specification etc..
To ensure the accuracy and uniformity of acquisition data, the origin system to operation of power networks data and facility information includes thunder
Electric alignment system, meteorological system, PMS and EMS etc., specification is embarked with reference to the data cases for itself describing, and conduct is obtained
The final explanation of the data for taking, such as:
Lightning location system specification:Interface protocol method is formulated, thunderbolt time of origin, position, return stroke times are obtained;Set up
Lightning systems model specification includes:Thunder and lightning time of origin, accuracy coordinate, latitude coordinate, current strength, return stroke times.
Meteorological system specification:Interface protocol is formulated to include:Obtain weather data method, radar map data method, cloud
Diagram data method, data of weather forecast method;Set up meteorological system monitoring information model (monitoring time, monitoring station, monitoring station institute
Category city, county where monitoring station, temperature, humidity, wind scale, wind speed, wind angle, extreme wind speed, very big angle, precipitation, energy
Degree of opinion, air pressure, issuing time, wind direction, very big wind wind scale, very big wind wind direction), radar information model (radar map date, latitude
Degree, longitude, radar map filestream data), cloud atlas information model (cloud atlas date, picture file flow data, centre coordinate), weather
(monitoring time, city belonging to monitoring, county where monitoring, forecast time span, the highest temperature, lowest temperature, weather to forecast information model
Situation, wind-force rank, wind direction code).
EMS EMSs:Interface protocol data capture method is formulated, electric current, voltage, active power, idle is obtained
The data such as power;Setting up EMS system model specification includes measuring ID, measurement time, measures type and measuring value.
In addition, the configuration system data access strategy in step (2) describes the rule with each system interaction data
Agreement, data source address, interface shape etc., the regular agreement of system interaction mainly include real-time data interaction protocol, history number
According to interaction protocol etc..The main descriptive system data of real-time, interactive agreement to system data provider send required parameter explanation and
The explanation of the data content attribute that system data provider accordingly returns, description trigger mechanism is to use real-time data interaction protocol
Frequency;Historical data interaction protocol and real-time data interaction protocol are asked and return identical, but non-periodically triggers;Data source ground
The ip addresses and port of the main descriptive system data providing issuing service in location;The main descriptive system data providing of interface shape
Issue data mode, such as webservice modes.
Access strategy description obtains regular agreement, data source address, interface shape of each system data etc., such as:
Lightning systems configuration data content:Thunderbolt time of origin, position, return stroke times;Frequency:In real time;Source data address:
Lightning data service IP address and port numbers;Interface shape:webservice.
Meteorological system configuration data content:Weather data, radar map, cloud atlas, weather forecast information;Frequency:In real time;
Source data address:Meteorological data service IP address and port numbers;Interface shape:webservice.
EMS system configuration data content:Electric current, voltage, active power, reactive power;Frequency:In real time;Source data address:
EMS data, services IP address and port numbers;Interface shape:webservice.
Configuration data verification rule:Data check rule configuration in the step (2) from each system data for providing
Side obtain data preprocessing rule, including the cleaning based on time series, the cleaning based on clustering algorithm, based on SVM's
Cleaning, equipment account cleaning etc., such as:
EMS data configurations are based on the cleaning rule of time series;
Meteorological data cleaning of the configuration based on clustering algorithm;
Lightning data cleaning of the configuration based on clustering algorithm.
In step (2), create and access data application:Integrated access strategy, data check rule, according in step (1)
Data model specification realizes operation system data access and parsing.
According to each operation system interface protocol, generated using the wsimport instruments of wsdl.jar and be based on webservice
Client, read the configuration of operation system access strategy, data check rule configuration parameter, such as:Lightning location system frequency is 5
Minute;EMS frequency is 1 minute and configures based on time series cleaning rule;Monitoring Data frequency in meteorological data
For 10 minutes and configure based on clustering algorithm cleaning rule, radar map data frequency be 5 minutes, cloud atlas data frequency be 1 hour,
Data of weather forecast frequency 24 hours.
The data of acquisition are parsed according to model specification, such as:According to EMS model specification, service data pair is obtained
Matched as parsing Property Name and with the specification of caching, service data structured set object is packaged into after matching;According to
Lightning systems model specification, it is XML document object to change lightning systems data using XML components, and thunder is packed according to specification matching
Electric data structured collection object;According to meteorological system model specification, for picture file data, BASE64 codings are converted to
Stream character string is simultaneously instantiated as object picture.
Monitoring accesses application:Monitoring identification is accessed using the network failure, service fault for existing in the process of running or deposited
Storage failure, method includes:
When mistake occurs in overall request service, disappeared if mistake is not up to n times, be then judged as that interruption stops service;
Wait overall request not occur mistake for N-1 times and then release failure.
When mistake occurs in overall request service, if overall request reaches n times and mistake continuously occurs in the above, now judge
Whether web services ip is communication, and service stopping is judged as if communication;Do not communicate, be judged as that external network is obstructed;Wait
Service releases failure after having return.
Configuration service monitoring request and the limitation of response duration, if it exceeds the duration for setting, then be judged as request timed out, such as
Fruit then releases request timed out state N-1 times less than the duration for setting.
When storage reports an error, then it is judged as writing;Wait releases failure after being successfully written.
Write-in duration limitation is set, if write-in exceedes the duration for setting, is judged as write latency;Wait N-1 times and writing
Enter and then release delaying state less than duration limitation.
The fault message of monitoring is sent to being used for for monitoring server by monitoring agent application by socket sockets
On socket servers, and by monitoring server storage to failure logging table
To the data of gaps and omissions in failure logging table, the data lost during accessing application recovery failure by history.
Log information in step (3) describes the procedural information of the real-time access application in step (2), record
WebService starts access, WebService and accesses the actions such as digital independent of completion or completion.
Fault message describes the malfunction of the real-time access application in step (2), including network failure, service stop
Only failure, storage failure etc..
Warning information describes the alarm status of the real-time access application in step (2), including access delay or storage
Postpone etc..
Data amended record in step (4) is described after the fault recovery of the real-time access application in step (2), fortune
History in row step (2) accesses the data lost during application accesses failure.
Format specification in step (5), it is intended to which system model norm-setting in step (1) meets buffered message
The traffic table and function decomposition into analytic function of the data structure of queue storage format, differentiated service system and searchable inquiry go out each system fortune
The method for designing of row data-message.
Data broadcasting in step (6) is that the real time data that will be accessed writes buffered message queue, at real-time analysis
The data source of reason and application function.
The data of Hadoop distributed storages file storage regular persistence from caching message queue, as distribution point
Analyse the data source for the treatment of.
Specifically, data broadcasting is shared and distributed storage:These three are big using Redis, kafka and Hadoop for this method
Data tool is used for power grid environment and service data broadcast, shared and distributed storage, and kafka realizes the distribution of high-throughput
News release is subscribed to, and redis realizes that data high-speed is cached, and Hadoop realizes mass data distributed storage, by parallel mechanism
To unify Message Processing on line and offline, consumption in real time is provided by cluster.Critical process data storage structure is focused on
Explain and be illustrated below with EMS data:
Real-time data memory format specification:
key:Measurement is identified to, its form is " ed&qy&rid ", wherein " ed " is designated EMS system data, " qy " mark
It is area, " rid " is measurement id;value:It is a map set, the key identified times under it;Value identifies measuring value.Show
Example:" ed&bz&212023,201510210914=32.90289057791233 ".
Real-time data broadcast format specification:
key:The time is identified to, its form is " region & measures the id& times ";value:It is designated measuring value.Example:
" bz&212023&201510210914,32.90289057791233 ".
Data distribution formula storage format specification:
key:The time is identified to, its form is " region & measures the id& times ";value:It is designated measuring value.Example:
" bz&212023&201510210914,32.90289057791233 ".
Visual presentation in step (7), describes to access the data that application is accessed in real time in showing interface step (2)
With the monitoring data in step (3), using accessing the song that measuring value in data such as EMS (EMS) changes over time
Line chart shows;Monitoring data such as displaying Web service state directly perceived and the showing interface of storage state.Monitoring displaying, main docking
Enter the network in application process, storage state to represent, including monitor in real time interface and flow displaying.
Visual presentation:Visualization mainly point initial data displaying and monitoring displaying, the present invention is illustrated below:
Initial data shows, to the understandable interface directly perceived of data creation one of access.EMS competence management system numbers
The change of point data, the curve that interface changes comprising time change are measured according to the displaying interface main presentation of middle metric data.
Monitor in real time interface, displaying Web service state and storage state directly perceived, green representative is normal, yellow represents warning,
Red represents catastrophe failure.Monitoring two processes of external communication and storage inside, external communication includes monitoring:It is service, request, outer
Portion's network state;The state of storage inside monitoring write-in data.
Flow shows interface, and the data traffic that every secondary data request is related to is monitored, and monitoring content includes completing
Time, request measurement number, return measure number, insertion and measure number, finally with broken line graph displaying.
The cross-platform operation system data of effective integration of the present invention, form the multi-source heterogeneous pattern of fusion power transmission and transformation of distributed storage
Status information of equipment resource, solves multi-platform, many applications, many communication protocols such as PMS, EMS, status monitoring, GIS, meteorology, thunder and lightning
View, the mass data of many data structures cannot accurately access the problem in the unified platform in real time, meet the analysis of data distribution formula, place
Reason and visual presentation requirement, realize that the uniform data of cross-platform information is accessed, monitor in real time is calculated, distributed storage and visual
Change displaying.
Present invention use webservice technologies, socket sockets mechanics of communication, business datum modeling and analytic technique,
Distributed message framework kafka, Hadoop distributed storage technology, there is provided interface data model specification, data access strategy are matched somebody with somebody
Put, data-interface parameter configuration, data check regular configuration, data access monitoring, data amended record, data distribution formula storage tube
A series of functions such as reason, visual presentation, data are obtained by function above from operation system automatically, realize cross-platform information
Uniform data access, monitor in real time is calculated, distributed storage and visual presentation.
The present invention has established base for integrated power system systematic data modeling, power industry and other inter-sectional data interactions
Plinth;Technical basis are provided for the cross-platform data of other later power systems is obtained;Can be to ensure to access data from now on
Integrality is used for reference;For power network big data distributed analysis process the condition that provides the foundation.
Although above-mentioned be described with reference to accompanying drawing to specific embodiment of the invention, not to present invention protection model
The limitation enclosed, one of ordinary skill in the art should be understood that on the basis of technical scheme those skilled in the art are not
Need the various modifications made by paying creative work or deformation still within protection scope of the present invention.