CN106777141A - A kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method - Google Patents

A kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method Download PDF

Info

Publication number
CN106777141A
CN106777141A CN201611177776.6A CN201611177776A CN106777141A CN 106777141 A CN106777141 A CN 106777141A CN 201611177776 A CN201611177776 A CN 201611177776A CN 106777141 A CN106777141 A CN 106777141A
Authority
CN
China
Prior art keywords
data
service
monitoring
time
failure
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201611177776.6A
Other languages
Chinese (zh)
Other versions
CN106777141B (en
Inventor
杨祎
苏建军
陈玉峰
郭志红
孟瑜
杜修明
王辉
马强
马艳
李程启
耿玉杰
林颖
白德盟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
State Grid Corp of China SGCC
Electric Power Research Institute of State Grid Shandong Electric Power Co Ltd
State Grid Shandong Electric Power Co Ltd
Shandong Luruan Digital Technology Co Ltd
Original Assignee
State Grid Corp of China SGCC
Electric Power Research Institute of State Grid Shandong Electric Power Co Ltd
State Grid Shandong Electric Power Co Ltd
Shandong Luneng Software Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by State Grid Corp of China SGCC, Electric Power Research Institute of State Grid Shandong Electric Power Co Ltd, State Grid Shandong Electric Power Co Ltd, Shandong Luneng Software Technology Co Ltd filed Critical State Grid Corp of China SGCC
Priority to CN201611177776.6A priority Critical patent/CN106777141B/en
Publication of CN106777141A publication Critical patent/CN106777141A/en
Application granted granted Critical
Publication of CN106777141B publication Critical patent/CN106777141B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/168Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
    • G06Q50/06Electricity, gas or water supply

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Economics (AREA)
  • Public Health (AREA)
  • Water Supply & Treatment (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Primary Health Care (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Human Computer Interaction (AREA)
  • Remote Monitoring And Control Of Power-Distribution Networks (AREA)

Abstract

The invention discloses a kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method, comprise the following steps:Interface protocol is set up according to each operation system data characteristics and set up operation of power networks environment and device data model specification;Configure data access strategy, the configuration data verification rule of each operation system;Monitoring accesses the information produced in application running in real time, for the fault message for monitoring, to failure during the data lost carry out data amended record;Set up the format specification of real-time message queue, data broadcasting and caching operation of power networks data, online monitoring data, service data, lightning data and meteorological data parsing are broadcast to and are stored in buffered message queue, and Hadoop distributed storage files are write by the cycle;The data and monitoring data of access are carried out into visual presentation.It is that integrated power system systematic data modeling is laid a good foundation invention creates operation of power networks environment and device data model specification.

Description

A kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method
Technical field
The present invention relates to the running environment and the access field of storage of facility information of power system, and in particular to fusion power network The cross-platform data of running environment and facility information is obtained and distributed storage method.
Background technology
Power transmission and transformation equipment state relevant information is valuator device state, diagnostic device failure, the load of dynamic adjusting device, comments Estimate the important evidence that power grid risk influences on equipment, it is accurate in real time to obtain the status related informations such as power transmission and transforming equipment, operation, environment It is to analyse in depth equipment state and the accurate key foundation for controlling capacity of equipment level.
The data such as equipment, operation, environment at present, involved by power transmission and transformation equipment state relevant information are mainly derived from energy Management system (EMS), meteorological system, production management system (PMS), power transmission and transforming equipment on-line monitoring system, power network spatial information The different business systems such as service platform (GIS), lightning location system, robot used for intelligent substation patrol system.
Built because each operation system is based on different platform, different application target, different agreement, different pieces of information structure, Present highly discrete, spatial and temporal distributions the characteristics of, exist between cross-platform data inconsistent data-interface specification, structuring and Unstructured data storage mode differs, it is weak related the problems such as;Secondly, the data of each operation system are substantially all according to relation Database or file mode are stored, and are not easy to be analyzed using Hadoop distributed technologies Develop Data and processed;Again, for The real-time streams such as EMS, meteorology, on-line monitoring, the real-time and accuracy of its access way, are quickly to judge equipment state Important step is, it is necessary to set up a real-time data broadcast and Data Share System.
The content of the invention
It is to solve the deficiency that prior art is present, the invention discloses fusion operation of power networks environment and facility information across flat Platform data acquisition and distributed storage method, the purpose of the present invention are obtaining data from operation system automatically with realization, realize The uniform data of cross-platform information is accessed, monitor in real time is calculated, distributed storage and visual presentation.
To achieve the above object, concrete scheme of the invention is as follows:
A kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method, comprise the following steps:
Interface protocol is set up according to each operation system data characteristics and set up operation of power networks environment and device data model Specification;
Configure each operation system data access strategy, configuration data verification rule so that realize in real time access application and History accesses application;
Monitoring is accessed using log information, fault message and the warning information for producing in the process of running and passed through in real time, For the fault message for monitoring, to failure during the data lost carry out data amended record;
The format specification of data broadcasting and caching operation of power networks data is set up, online monitoring data, operation number parsing It is distributed by cycle write-in Hadoop in being broadcast to and stored buffered message queue according to, lightning data and meteorological data Storage file or HBase;
The data and monitoring data of access are carried out into visual presentation.
Further, the data model specification of above-mentioned foundation is the data attribute of each operation system, including energy pipe Reason system model specification, lightning location system model specification and meteorological system model specification;
Lightning location system model specification:Formulate interface protocol and obtain thunderbolt time of origin, position, return stroke times;Set up Lightning systems model specification includes:Thunder and lightning time of origin, accuracy coordinate, latitude coordinate, current strength and return stroke times;
Meteorological system model specification:Formulate interface protocol and obtain weather data, radar diagram data, cloud atlas data and day Gas forecast data;Set up meteorological system monitoring information model, radar information model, cloud atlas information model and weather forecast information mould Type;
EMS model specification:Formulate interface protocol and obtain electric current, voltage, active power, reactive power data; Setting up EMS system model specification includes measuring id, measurement time, measures type and measuring value.
Further, the data access strategy of each operation system, including the rule with each operation system interaction data are configured Then agreement, data source address, interface shape, wherein, the regular agreement of the operation system interaction data mainly includes number in real time According to interaction protocol, historical data interaction protocol;
The main descriptive system data of real-time, interactive agreement send explanation and the system of required parameter to system data provider The explanation of the data content attribute that data providing is accordingly returned, description trigger mechanism is the frequency for using real-time data interaction protocol Rate;Historical data interaction protocol and real-time data interaction protocol are asked and return identical, but non-periodically triggers;
The ip addresses and port of the main descriptive system data providing issuing service of data source address;
The mode of the issue data of the main descriptive system data providing of interface shape.
Further, the data check rule of configuration is pre- for the data that are obtained from system data provider for describing Cleaning rule, including the cleaning based on time series, the cleaning based on clustering algorithm, the cleaning based on SVM, account cleaning rule Then;
Wherein, application is accessed in real time and history accesses application integrated data access strategy and data check rule, according to number Data access and the parsing of each operation system are realized according to model specification.
Further, the real-time access using the log information produced in running is the real-time process for accessing application Information, record WebService starts access, WebService and accesses completion or complete a digital independent action message;
Fault message is to access the malfunction applied in real time, including network failure, service stopping failure and storage failure;
Warning information is described as accessing in real time the alarm status of application, including access delay or storage delay.
Further, the data amended record is described after the fault recovery for accessing application in real time, and history run connects Enter the data lost during application accesses failure.
Further, the format specification for setting up data broadcasting and caching operation of power networks data, format specification is intended to root According to the data for meeting buffered message queue storage format, differentiated service system and searchable inquiry of data model norm-setting The method that the traffic table and function decomposition into analytic function of structure go out each system operation data message.
Further, the data broadcasting is that the real time data that will be accessed writes buffered message queue, used as analysis in real time Treatment and the data source of application function;
The data of Hadoop distributed storages file storage regular persistence from caching message queue, as distribution The data source of formula analyzing and processing.
Further, accessed in real time using the data and monitoring number for accessing by showing interface in the visual presentation According to the curve map changed over time using measuring value in access data shows;Displaying Web service state directly perceived in monitoring data With the showing interface of storage state.
Further, the monitoring accesses application and also includes in real time:Monitoring identification accesses application to be existed in the process of running Network failure, service fault or storage failure, specially:
When mistake occurs in overall request service, disappeared if mistake is not up to n times, be then judged as that interruption stops service; Wait overall request not occur mistake for N-1 times and then release failure;
When mistake occurs in overall request service, if overall request reaches n times and mistake continuously occurs in the above, now judge Whether web services ip is communication, and service stopping is judged as if communication;Do not communicate, be judged as that external network is obstructed;Wait Service releases failure after having return;
Configuration service monitoring request and the limitation of response duration, if it exceeds the duration for setting, then be judged as request timed out, such as Fruit then releases request timed out state N-1 times less than the duration for setting;
When storage reports an error, then it is judged as writing;Wait releases failure after being successfully written;
Write-in duration limitation is set, if write-in exceedes the duration for setting, is judged as write latency;Wait N-1 times and writing Enter and then release delaying state less than duration limitation;
To the data of gaps and omissions in failure logging table, the data lost during accessing application recovery failure by history.
Further, when realizing that the data of each operation system are parsed, the data to obtaining are parsed according to model specification, bag Include:According to EMS model specification, obtain service data analysis of object Property Name and carried out with the specification of caching Match somebody with somebody, service data structured set object is packaged into after matching;According to lightning systems model specification, thunder is changed using XML components Electric system data are XML document object, and lightning data structured set object is packed according to specification matching;According to meteorological system mould Type specification, for picture file data, is converted to the stream character string of BASE64 codings and is instantiated as object picture.
Beneficial effects of the present invention:
1st, it is that integrated power system systematic data modeling is established invention creates operation of power networks environment and facility information data model Basis is determined, has been that power specialty application, power industry and other inter-sectional data interactions are laid a good foundation.
2nd, the present invention describes to set up the process of data access application, is that the cross-platform data acquisition of other power systems is carried Technical basis are supplied, the application of data access monitoring means can be to ensure that the integrality for accessing data is offered reference from now on.
3rd, real time data is cached using Distributed Message Queue Kafka, number after message number reaches setting value It is that power network big data distributed analysis process the condition that provides the foundation according to distributed file system is written into.
Brief description of the drawings
The cross-platform data of Fig. 1 fusion operation of power networks environment of the invention and facility information is obtained and distributed storage method Flow chart.
Specific embodiment:
The present invention is described in detail below in conjunction with the accompanying drawings:
As shown in figure 1, the cross-platform data of fusion operation of power networks environment and facility information is obtained and distributed storage method, Comprise the following steps:
Step (1):Interface protocol is set up according to each operation system data characteristics and set up data model specification;
Step (2):Each system data access strategy, configuration data are configured according to the access protocol of each system in step (1) Verification rule, realizes real-time data imputing system function and historical data access function;
Step (3):Log information, fault message, the announcement produced in real-time access function running in monitoring step (2) Alert information etc.;
Step (4):The fault message monitored in analytical procedure (3), loses during resending request, amended record failure Data;For warning information, alarm cause is analyzed, exclude hidden danger.
Step (5):The form rule of broadcast and caching operation of power networks data are set up according to the data model specification in step (1) Model;
Step (6):According to the format specification in step (5), the online monitoring data, operation number that are accessed in step (2) It is broadcast in buffered message queue according to, lightning data and meteorological data etc., Hadoop distributed storage files is written to by the cycle Or HBase;
Step (7):According to the data model specification in step (1), the data for accessing application access in step (2) in real time Visual presentation is carried out with the monitoring data in step (3).
Wherein in step (1), operation of power networks environment and device data model specification are set up:
Data acquisition is the safety using data-interface, data center be shared, under Network Isolation based on ESB The modes such as file transmission, by configuring corresponding strategies, define related interface, cycle, call the parameters such as frequency and object, automatically The extracted data from operation system, solution platform database is accessed, large data files across platform concurrently reads at a high speed, cross-platform Security Data Transmission and the key issue such as synchronous.Data are mainly derived from the related business application system of power network, including power transmission and transformation Equipment Condition Monitoring System, production management system PMS, EMS EMS, power grid GIS GIS, weather information System, lightning location system, intelligent robot cruising inspection system etc..
Data model specification describes the business datum attribute of system, and model specification is advised including EMS model Model, lightning location system model specification, meteorological system model specification etc..
To ensure the accuracy and uniformity of acquisition data, the origin system to operation of power networks data and facility information includes thunder Electric alignment system, meteorological system, PMS and EMS etc., specification is embarked with reference to the data cases for itself describing, and conduct is obtained The final explanation of the data for taking, such as:
Lightning location system specification:Interface protocol method is formulated, thunderbolt time of origin, position, return stroke times are obtained;Set up Lightning systems model specification includes:Thunder and lightning time of origin, accuracy coordinate, latitude coordinate, current strength, return stroke times.
Meteorological system specification:Interface protocol is formulated to include:Obtain weather data method, radar map data method, cloud Diagram data method, data of weather forecast method;Set up meteorological system monitoring information model (monitoring time, monitoring station, monitoring station institute Category city, county where monitoring station, temperature, humidity, wind scale, wind speed, wind angle, extreme wind speed, very big angle, precipitation, energy Degree of opinion, air pressure, issuing time, wind direction, very big wind wind scale, very big wind wind direction), radar information model (radar map date, latitude Degree, longitude, radar map filestream data), cloud atlas information model (cloud atlas date, picture file flow data, centre coordinate), weather (monitoring time, city belonging to monitoring, county where monitoring, forecast time span, the highest temperature, lowest temperature, weather to forecast information model Situation, wind-force rank, wind direction code).
EMS EMSs:Interface protocol data capture method is formulated, electric current, voltage, active power, idle is obtained The data such as power;Setting up EMS system model specification includes measuring ID, measurement time, measures type and measuring value.
In addition, the configuration system data access strategy in step (2) describes the rule with each system interaction data Agreement, data source address, interface shape etc., the regular agreement of system interaction mainly include real-time data interaction protocol, history number According to interaction protocol etc..The main descriptive system data of real-time, interactive agreement to system data provider send required parameter explanation and The explanation of the data content attribute that system data provider accordingly returns, description trigger mechanism is to use real-time data interaction protocol Frequency;Historical data interaction protocol and real-time data interaction protocol are asked and return identical, but non-periodically triggers;Data source ground The ip addresses and port of the main descriptive system data providing issuing service in location;The main descriptive system data providing of interface shape Issue data mode, such as webservice modes.
Access strategy description obtains regular agreement, data source address, interface shape of each system data etc., such as:
Lightning systems configuration data content:Thunderbolt time of origin, position, return stroke times;Frequency:In real time;Source data address: Lightning data service IP address and port numbers;Interface shape:webservice.
Meteorological system configuration data content:Weather data, radar map, cloud atlas, weather forecast information;Frequency:In real time; Source data address:Meteorological data service IP address and port numbers;Interface shape:webservice.
EMS system configuration data content:Electric current, voltage, active power, reactive power;Frequency:In real time;Source data address: EMS data, services IP address and port numbers;Interface shape:webservice.
Configuration data verification rule:Data check rule configuration in the step (2) from each system data for providing Side obtain data preprocessing rule, including the cleaning based on time series, the cleaning based on clustering algorithm, based on SVM's Cleaning, equipment account cleaning etc., such as:
EMS data configurations are based on the cleaning rule of time series;
Meteorological data cleaning of the configuration based on clustering algorithm;
Lightning data cleaning of the configuration based on clustering algorithm.
In step (2), create and access data application:Integrated access strategy, data check rule, according in step (1) Data model specification realizes operation system data access and parsing.
According to each operation system interface protocol, generated using the wsimport instruments of wsdl.jar and be based on webservice Client, read the configuration of operation system access strategy, data check rule configuration parameter, such as:Lightning location system frequency is 5 Minute;EMS frequency is 1 minute and configures based on time series cleaning rule;Monitoring Data frequency in meteorological data For 10 minutes and configure based on clustering algorithm cleaning rule, radar map data frequency be 5 minutes, cloud atlas data frequency be 1 hour, Data of weather forecast frequency 24 hours.
The data of acquisition are parsed according to model specification, such as:According to EMS model specification, service data pair is obtained Matched as parsing Property Name and with the specification of caching, service data structured set object is packaged into after matching;According to Lightning systems model specification, it is XML document object to change lightning systems data using XML components, and thunder is packed according to specification matching Electric data structured collection object;According to meteorological system model specification, for picture file data, BASE64 codings are converted to Stream character string is simultaneously instantiated as object picture.
Monitoring accesses application:Monitoring identification is accessed using the network failure, service fault for existing in the process of running or deposited Storage failure, method includes:
When mistake occurs in overall request service, disappeared if mistake is not up to n times, be then judged as that interruption stops service; Wait overall request not occur mistake for N-1 times and then release failure.
When mistake occurs in overall request service, if overall request reaches n times and mistake continuously occurs in the above, now judge Whether web services ip is communication, and service stopping is judged as if communication;Do not communicate, be judged as that external network is obstructed;Wait Service releases failure after having return.
Configuration service monitoring request and the limitation of response duration, if it exceeds the duration for setting, then be judged as request timed out, such as Fruit then releases request timed out state N-1 times less than the duration for setting.
When storage reports an error, then it is judged as writing;Wait releases failure after being successfully written.
Write-in duration limitation is set, if write-in exceedes the duration for setting, is judged as write latency;Wait N-1 times and writing Enter and then release delaying state less than duration limitation.
The fault message of monitoring is sent to being used for for monitoring server by monitoring agent application by socket sockets On socket servers, and by monitoring server storage to failure logging table
To the data of gaps and omissions in failure logging table, the data lost during accessing application recovery failure by history.
Log information in step (3) describes the procedural information of the real-time access application in step (2), record WebService starts access, WebService and accesses the actions such as digital independent of completion or completion.
Fault message describes the malfunction of the real-time access application in step (2), including network failure, service stop Only failure, storage failure etc..
Warning information describes the alarm status of the real-time access application in step (2), including access delay or storage Postpone etc..
Data amended record in step (4) is described after the fault recovery of the real-time access application in step (2), fortune History in row step (2) accesses the data lost during application accesses failure.
Format specification in step (5), it is intended to which system model norm-setting in step (1) meets buffered message The traffic table and function decomposition into analytic function of the data structure of queue storage format, differentiated service system and searchable inquiry go out each system fortune The method for designing of row data-message.
Data broadcasting in step (6) is that the real time data that will be accessed writes buffered message queue, at real-time analysis The data source of reason and application function.
The data of Hadoop distributed storages file storage regular persistence from caching message queue, as distribution point Analyse the data source for the treatment of.
Specifically, data broadcasting is shared and distributed storage:These three are big using Redis, kafka and Hadoop for this method Data tool is used for power grid environment and service data broadcast, shared and distributed storage, and kafka realizes the distribution of high-throughput News release is subscribed to, and redis realizes that data high-speed is cached, and Hadoop realizes mass data distributed storage, by parallel mechanism To unify Message Processing on line and offline, consumption in real time is provided by cluster.Critical process data storage structure is focused on Explain and be illustrated below with EMS data:
Real-time data memory format specification:
key:Measurement is identified to, its form is " ed&qy&rid ", wherein " ed " is designated EMS system data, " qy " mark It is area, " rid " is measurement id;value:It is a map set, the key identified times under it;Value identifies measuring value.Show Example:" ed&bz&212023,201510210914=32.90289057791233 ".
Real-time data broadcast format specification:
key:The time is identified to, its form is " region & measures the id& times ";value:It is designated measuring value.Example: " bz&212023&201510210914,32.90289057791233 ".
Data distribution formula storage format specification:
key:The time is identified to, its form is " region & measures the id& times ";value:It is designated measuring value.Example: " bz&212023&201510210914,32.90289057791233 ".
Visual presentation in step (7), describes to access the data that application is accessed in real time in showing interface step (2) With the monitoring data in step (3), using accessing the song that measuring value in data such as EMS (EMS) changes over time Line chart shows;Monitoring data such as displaying Web service state directly perceived and the showing interface of storage state.Monitoring displaying, main docking Enter the network in application process, storage state to represent, including monitor in real time interface and flow displaying.
Visual presentation:Visualization mainly point initial data displaying and monitoring displaying, the present invention is illustrated below:
Initial data shows, to the understandable interface directly perceived of data creation one of access.EMS competence management system numbers The change of point data, the curve that interface changes comprising time change are measured according to the displaying interface main presentation of middle metric data.
Monitor in real time interface, displaying Web service state and storage state directly perceived, green representative is normal, yellow represents warning, Red represents catastrophe failure.Monitoring two processes of external communication and storage inside, external communication includes monitoring:It is service, request, outer Portion's network state;The state of storage inside monitoring write-in data.
Flow shows interface, and the data traffic that every secondary data request is related to is monitored, and monitoring content includes completing Time, request measurement number, return measure number, insertion and measure number, finally with broken line graph displaying.
The cross-platform operation system data of effective integration of the present invention, form the multi-source heterogeneous pattern of fusion power transmission and transformation of distributed storage Status information of equipment resource, solves multi-platform, many applications, many communication protocols such as PMS, EMS, status monitoring, GIS, meteorology, thunder and lightning View, the mass data of many data structures cannot accurately access the problem in the unified platform in real time, meet the analysis of data distribution formula, place Reason and visual presentation requirement, realize that the uniform data of cross-platform information is accessed, monitor in real time is calculated, distributed storage and visual Change displaying.
Present invention use webservice technologies, socket sockets mechanics of communication, business datum modeling and analytic technique, Distributed message framework kafka, Hadoop distributed storage technology, there is provided interface data model specification, data access strategy are matched somebody with somebody Put, data-interface parameter configuration, data check regular configuration, data access monitoring, data amended record, data distribution formula storage tube A series of functions such as reason, visual presentation, data are obtained by function above from operation system automatically, realize cross-platform information Uniform data access, monitor in real time is calculated, distributed storage and visual presentation.
The present invention has established base for integrated power system systematic data modeling, power industry and other inter-sectional data interactions Plinth;Technical basis are provided for the cross-platform data of other later power systems is obtained;Can be to ensure to access data from now on Integrality is used for reference;For power network big data distributed analysis process the condition that provides the foundation.
Although above-mentioned be described with reference to accompanying drawing to specific embodiment of the invention, not to present invention protection model The limitation enclosed, one of ordinary skill in the art should be understood that on the basis of technical scheme those skilled in the art are not Need the various modifications made by paying creative work or deformation still within protection scope of the present invention.

Claims (10)

1. a kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method, it is characterized in that, comprise the following steps:
Interface protocol is set up according to each operation system data characteristics and set up operation of power networks environment and device data model specification;
Data access strategy, the configuration data verification rule of each operation system are configured, so as to realize accessing application and history in real time Access application;
Monitoring accesses log information, fault message and the warning information produced in application running in real time, for what is monitored Fault message, to failure during lose data carry out data amended record;
The format specification of data broadcasting and caching operation of power networks data is set up, online monitoring data, service data, the thunder of parsing Electric data and meteorological data are broadcast to and are stored in buffered message queue, by cycle write-in Hadoop distributed storage text Part or HBase;
The data and monitoring data of access are carried out into visual presentation.
2. a kind of acquisition for merging multi-source heterogeneous electric network data as claimed in claim 1 and distributed storage method, its feature It is that the data model specification of foundation is the data attribute of each operation system, including EMS model specification, thunder and lightning Positioning system models specification and meteorological system model specification;
Lightning location system model specification:Formulate interface protocol and obtain thunderbolt time of origin, position, return stroke times;Set up thunder and lightning System model specification includes:Thunder and lightning time of origin, accuracy coordinate, latitude coordinate, current strength and return stroke times;
Meteorological system model specification:Formulate interface protocol acquisition weather data, radar diagram data, cloud atlas data and weather pre- Count off evidence;Set up meteorological system monitoring information model, radar information model, cloud atlas information model and weather forecast information model;
EMS model specification:Formulate interface protocol and obtain electric current, voltage, active power, reactive power data;Set up EMS system model specification includes measuring id, measures the time, measures type and measuring value.
3. a kind of acquisition for merging multi-source heterogeneous electric network data as claimed in claim 1 and distributed storage method, its feature It is to configure the data access strategy of each operation system, including regular agreement, data source ground with each operation system interaction data Location, interface shape, wherein, the regular agreement of the operation system interaction data mainly includes real-time data interaction protocol, history Data interaction agreement;
The main descriptive system data of real-time, interactive agreement send explanation and the system data of required parameter to system data provider The explanation of the data content attribute that provider accordingly returns, description trigger mechanism is the frequency for using real-time data interaction protocol; Historical data interaction protocol and real-time data interaction protocol are asked and return identical, but non-periodically triggers;
The ip addresses and port of the main descriptive system data providing issuing service of data source address;
The mode of the issue data of the main descriptive system data providing of interface shape.
4. a kind of acquisition for merging multi-source heterogeneous electric network data as claimed in claim 1 and distributed storage method, its feature It is that the data check rule of configuration is used to describe the prerinse rule for the data obtained from system data provider, including Cleaning based on time series, the cleaning based on clustering algorithm, the cleaning based on SVM, account cleaning rule;
Wherein, application is accessed in real time and history accesses application integrated data access strategy and data check rule, according to data mould Type specification realizes data access and the parsing of each operation system.
5. a kind of acquisition for merging multi-source heterogeneous electric network data as claimed in claim 1 and distributed storage method, its feature It is that real-time the access using the log information produced in running is the real-time procedural information for accessing application, record WebService starts access, WebService and accesses completion or complete a digital independent action message;
Fault message is to access the malfunction applied in real time, including network failure, service stopping failure and storage failure;
Warning information is described as accessing in real time the alarm status of application, including access delay or storage delay.
6. a kind of acquisition for merging multi-source heterogeneous electric network data as claimed in claim 1 and distributed storage method, its feature It is that the format specification for setting up data broadcasting and caching operation of power networks data, format specification is intended to according to data model specification Formulate the data structure for meeting buffered message queue storage format, differentiated service system and searchable inquiry traffic table and The method that function decomposition into analytic function goes out each system operation data message.
7. a kind of acquisition of the multi-source heterogeneous electric network data of fusion as described in claim 1 or 6 and distributed storage method, it is special Levying is, the data broadcasting is the real time data write-in buffered message queue that will be accessed, used as analyzing and processing and apply work(in real time The data source of energy;
The data of Hadoop distributed storages file storage regular persistence from caching message queue, as distribution point Analyse the data source for the treatment of.
8. a kind of acquisition for merging multi-source heterogeneous electric network data as claimed in claim 1 and distributed storage method, its feature It is to be accessed in real time using the data and monitoring data for accessing by showing interface in the visual presentation, using access data The curve map displaying that middle measuring value changes over time;The boundary of displaying Web service state directly perceived and storage state in monitoring data Face shows.
9. a kind of acquisition for merging multi-source heterogeneous electric network data as claimed in claim 1 and distributed storage method, its feature It is that the monitoring accesses application in real time also to be included:Monitoring identification accesses network failure, the service that application exists in the process of running Failure or storage failure, specially:
When mistake occurs in overall request service, disappeared if mistake is not up to n times, be then judged as that interruption stops service;Wait Overall request does not occur mistake for N-1 times and then releases failure;
When mistake occurs in overall request service, if overall request reaches n times and mistake continuously occurs in the above, web is now judged Whether service ip is communication, and service stopping is judged as if communication;Do not communicate, be judged as that external network is obstructed;Etc. to be serviced Failure is released after having return;
Configuration service monitoring request and the limitation of response duration, if it exceeds the duration for setting, then be judged as request timed out, if N- Request timed out state is then released less than the duration for setting 1 time;
When storage reports an error, then it is judged as writing;Wait releases failure after being successfully written;
Write-in duration limitation is set, if write-in exceedes the duration for setting, is judged as write latency;Wait N-1 write-in low Delaying state is then released in duration limitation;
To the data of gaps and omissions in failure logging table, the data lost during accessing application recovery failure by history.
10. a kind of acquisition for merging multi-source heterogeneous electric network data as claimed in claim 1 and distributed storage method, its feature When being the data parsing for realizing each operation system, the data to obtaining are parsed according to model specification, including:According to energy pipe Reason system model specification, obtains service data analysis of object Property Name and is matched with the specification of caching, is packed after matching Into service data structured set object;According to lightning systems model specification, changing lightning systems data using XML components is XML document object, lightning data structured set object is packed according to specification matching;According to meteorological system model specification, for Picture file data, is converted to the stream character string of BASE64 codings and is instantiated as object picture.
CN201611177776.6A 2016-12-19 2016-12-19 A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method Active CN106777141B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201611177776.6A CN106777141B (en) 2016-12-19 2016-12-19 A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201611177776.6A CN106777141B (en) 2016-12-19 2016-12-19 A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method

Publications (2)

Publication Number Publication Date
CN106777141A true CN106777141A (en) 2017-05-31
CN106777141B CN106777141B (en) 2019-07-12

Family

ID=58890179

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201611177776.6A Active CN106777141B (en) 2016-12-19 2016-12-19 A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method

Country Status (1)

Country Link
CN (1) CN106777141B (en)

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107220360A (en) * 2017-06-07 2017-09-29 云南电网有限责任公司信息中心 A kind of Unified Modeling storage cut-in method based on magnanimity electric power monitoring data
CN107341205A (en) * 2017-06-23 2017-11-10 国网上海市电力公司 A kind of intelligent distribution system based on big data platform
CN107590749A (en) * 2017-09-07 2018-01-16 北京国电通网络技术有限公司 A kind of processing method and system with electricity consumption data
CN107634963A (en) * 2017-10-12 2018-01-26 中国铁塔股份有限公司 A kind of field terminal unit Accreditation System and method
CN107633004A (en) * 2017-08-07 2018-01-26 国网新疆电力公司信息通信公司 Data monitoring method and device
CN107656971A (en) * 2017-09-02 2018-02-02 国网辽宁省电力有限公司 A kind of intelligent grid collection Monitoring Data storage method based on Redis
CN107730040A (en) * 2017-09-30 2018-02-23 国网山东省电力公司电力科学研究院 Power information system log information comprehensive characteristics extracting method and device based on RBM
CN107769993A (en) * 2017-09-19 2018-03-06 广西电网有限责任公司电力科学研究院 Towards the data traffic monitoring method of power network big data distributed system
CN107943835A (en) * 2017-10-26 2018-04-20 中国南方电网有限责任公司 It is a kind of to report and submit data analysis and taxis system for electric system
CN107968840A (en) * 2017-12-15 2018-04-27 华北电力大学(保定) A kind of extensive power equipment monitoring, alarming Real-time Data Processing Method and system
CN108074044A (en) * 2018-01-19 2018-05-25 广东电网有限责任公司珠海供电局 A kind of power equipment state monitoring evaluation and command methods
CN108229799A (en) * 2017-12-20 2018-06-29 江苏瑞中数据股份有限公司 A kind of multi-source heterogeneous operation of power networks real-time data imputing system system and method
CN108319695A (en) * 2018-02-02 2018-07-24 华自科技股份有限公司 Power station fault data processing method, device, computer equipment and storage medium
CN108737550A (en) * 2018-05-25 2018-11-02 山东鲁能智能技术有限公司 A kind of robot interface's device and robot based on robot centralized control system
CN109033174A (en) * 2018-06-21 2018-12-18 北京国网信通埃森哲信息技术有限公司 A kind of power quality data cleaning method and device
CN109165207A (en) * 2018-07-16 2019-01-08 华南农业大学 Drinking water mass data storage management method and system based on Hadoop
CN109413185A (en) * 2018-10-31 2019-03-01 国网河南省电力公司焦作供电公司 A kind of equipment routing inspection system and its Cloud Server design method
CN109558397A (en) * 2018-10-30 2019-04-02 平安医疗健康管理股份有限公司 A kind of data processing method, device, server and computer storage medium
CN109871375A (en) * 2018-09-12 2019-06-11 国网浙江省电力有限公司嘉兴供电公司 The information platform and its control method of distributed new scale access
CN110019488A (en) * 2018-09-12 2019-07-16 国网浙江省电力有限公司嘉兴供电公司 Multi-source heterogeneous data fusion multi-core classification method
CN110502517A (en) * 2019-08-23 2019-11-26 中国南方电网有限责任公司 It is a kind of for storing the distributed memory system of power grid real-time running data
CN111262915A (en) * 2020-01-10 2020-06-09 北京东方金信科技有限公司 Kafka cluster-crossing data conversion system and method
CN111506409A (en) * 2020-04-20 2020-08-07 南方电网科学研究院有限责任公司 Data processing method and system
CN111667143A (en) * 2020-04-30 2020-09-15 广西电网有限责任公司 Method for realizing fusion of real-time monitoring data and service management data in power grid dispatching
CN111752723A (en) * 2020-06-06 2020-10-09 中国科学院电子学研究所苏州研究院 Visual multi-source service management system and implementation method thereof
CN111798096A (en) * 2020-06-02 2020-10-20 国网山西省电力公司电力科学研究院 Visual analysis method, system and storage medium based on power grid dispatching data
CN112235361A (en) * 2020-09-28 2021-01-15 青海绿能数据有限公司 Photovoltaic power plant data switching platform
CN112308538A (en) * 2020-12-03 2021-02-02 国网福建省电力有限公司厦门供电公司 Active service data interaction method for multi-type park users
CN112650761A (en) * 2021-02-23 2021-04-13 善智互联(北京)网络科技有限公司 Structured extraction method and system for transformer fault information
CN113642850A (en) * 2021-07-20 2021-11-12 国网江苏省电力有限公司南通供电分公司 Data fusion method and terminal for power distribution network planning

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103617557A (en) * 2013-11-06 2014-03-05 广东电网公司电力科学研究院 Multi-source heterogeneous power grid operation parameter analysis system
CN103941158A (en) * 2014-04-17 2014-07-23 国家电网公司 Power distribution network fault diagnosis system and method based on multi-source information
CN104240021A (en) * 2014-09-04 2014-12-24 海南国能中海智能技术有限公司 Visual, traceable and controllable power grid managing system and method
CN104852927A (en) * 2015-06-01 2015-08-19 国家电网公司 Safety comprehensive management system based on multi-source heterogeneous information
CN105184424A (en) * 2015-10-19 2015-12-23 国网山东省电力公司菏泽供电公司 Mapreduced short period load prediction method of multinucleated function learning SVM realizing multi-source heterogeneous data fusion
CN105335896A (en) * 2014-08-13 2016-02-17 国家电网公司 Multi-source heterogeneous data processing method and device used for power grid
CN105427188A (en) * 2015-12-02 2016-03-23 国家电网公司 Power supply control system and method based on multi-source heterogeneous information fusion
CN105574643A (en) * 2015-11-23 2016-05-11 江苏瑞中数据股份有限公司 Real-time data center and big data platform fusion method for power grid

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103617557A (en) * 2013-11-06 2014-03-05 广东电网公司电力科学研究院 Multi-source heterogeneous power grid operation parameter analysis system
CN103941158A (en) * 2014-04-17 2014-07-23 国家电网公司 Power distribution network fault diagnosis system and method based on multi-source information
CN105335896A (en) * 2014-08-13 2016-02-17 国家电网公司 Multi-source heterogeneous data processing method and device used for power grid
CN104240021A (en) * 2014-09-04 2014-12-24 海南国能中海智能技术有限公司 Visual, traceable and controllable power grid managing system and method
CN104852927A (en) * 2015-06-01 2015-08-19 国家电网公司 Safety comprehensive management system based on multi-source heterogeneous information
CN105184424A (en) * 2015-10-19 2015-12-23 国网山东省电力公司菏泽供电公司 Mapreduced short period load prediction method of multinucleated function learning SVM realizing multi-source heterogeneous data fusion
CN105574643A (en) * 2015-11-23 2016-05-11 江苏瑞中数据股份有限公司 Real-time data center and big data platform fusion method for power grid
CN105427188A (en) * 2015-12-02 2016-03-23 国家电网公司 Power supply control system and method based on multi-source heterogeneous information fusion

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
周爱华 等: ""面向多源异构电网数据的获取与转换技术研究"", 《电力信息与通信技术》 *
孟祥君 等: ""智能电网大数据平台及其关键技术研究"", 《供用电》 *
王逸飞 等: ""基于大数据平台的电网防灾调度系统功能设计与系统架构"", 《电 网 技 术》 *

Cited By (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107220360A (en) * 2017-06-07 2017-09-29 云南电网有限责任公司信息中心 A kind of Unified Modeling storage cut-in method based on magnanimity electric power monitoring data
CN107341205A (en) * 2017-06-23 2017-11-10 国网上海市电力公司 A kind of intelligent distribution system based on big data platform
CN107633004A (en) * 2017-08-07 2018-01-26 国网新疆电力公司信息通信公司 Data monitoring method and device
CN107656971A (en) * 2017-09-02 2018-02-02 国网辽宁省电力有限公司 A kind of intelligent grid collection Monitoring Data storage method based on Redis
CN107590749A (en) * 2017-09-07 2018-01-16 北京国电通网络技术有限公司 A kind of processing method and system with electricity consumption data
CN107769993A (en) * 2017-09-19 2018-03-06 广西电网有限责任公司电力科学研究院 Towards the data traffic monitoring method of power network big data distributed system
CN107730040A (en) * 2017-09-30 2018-02-23 国网山东省电力公司电力科学研究院 Power information system log information comprehensive characteristics extracting method and device based on RBM
CN107634963A (en) * 2017-10-12 2018-01-26 中国铁塔股份有限公司 A kind of field terminal unit Accreditation System and method
CN107634963B (en) * 2017-10-12 2020-03-10 中国铁塔股份有限公司 On-site monitoring unit registration system and method
CN107943835A (en) * 2017-10-26 2018-04-20 中国南方电网有限责任公司 It is a kind of to report and submit data analysis and taxis system for electric system
CN107968840B (en) * 2017-12-15 2020-10-09 华北电力大学(保定) Real-time processing method and system for monitoring alarm data of large-scale power equipment
CN107968840A (en) * 2017-12-15 2018-04-27 华北电力大学(保定) A kind of extensive power equipment monitoring, alarming Real-time Data Processing Method and system
CN108229799A (en) * 2017-12-20 2018-06-29 江苏瑞中数据股份有限公司 A kind of multi-source heterogeneous operation of power networks real-time data imputing system system and method
CN108229799B (en) * 2017-12-20 2022-04-05 江苏瑞中数据股份有限公司 Multi-source heterogeneous power grid operation real-time data access system and method
CN108074044A (en) * 2018-01-19 2018-05-25 广东电网有限责任公司珠海供电局 A kind of power equipment state monitoring evaluation and command methods
CN108319695A (en) * 2018-02-02 2018-07-24 华自科技股份有限公司 Power station fault data processing method, device, computer equipment and storage medium
CN108737550A (en) * 2018-05-25 2018-11-02 山东鲁能智能技术有限公司 A kind of robot interface's device and robot based on robot centralized control system
CN109033174A (en) * 2018-06-21 2018-12-18 北京国网信通埃森哲信息技术有限公司 A kind of power quality data cleaning method and device
CN109165207A (en) * 2018-07-16 2019-01-08 华南农业大学 Drinking water mass data storage management method and system based on Hadoop
CN109871375A (en) * 2018-09-12 2019-06-11 国网浙江省电力有限公司嘉兴供电公司 The information platform and its control method of distributed new scale access
CN110019488A (en) * 2018-09-12 2019-07-16 国网浙江省电力有限公司嘉兴供电公司 Multi-source heterogeneous data fusion multi-core classification method
CN109558397A (en) * 2018-10-30 2019-04-02 平安医疗健康管理股份有限公司 A kind of data processing method, device, server and computer storage medium
CN109558397B (en) * 2018-10-30 2023-08-22 深圳平安医疗健康科技服务有限公司 Data processing method, device, server and computer storage medium
CN109413185A (en) * 2018-10-31 2019-03-01 国网河南省电力公司焦作供电公司 A kind of equipment routing inspection system and its Cloud Server design method
CN110502517B (en) * 2019-08-23 2022-01-28 中国南方电网有限责任公司 Distributed storage system for storing real-time operation data of power grid
CN110502517A (en) * 2019-08-23 2019-11-26 中国南方电网有限责任公司 It is a kind of for storing the distributed memory system of power grid real-time running data
CN111262915A (en) * 2020-01-10 2020-06-09 北京东方金信科技有限公司 Kafka cluster-crossing data conversion system and method
CN111262915B (en) * 2020-01-10 2020-09-22 北京东方金信科技有限公司 Kafka cluster-crossing data conversion system and method
CN111506409A (en) * 2020-04-20 2020-08-07 南方电网科学研究院有限责任公司 Data processing method and system
CN111667143A (en) * 2020-04-30 2020-09-15 广西电网有限责任公司 Method for realizing fusion of real-time monitoring data and service management data in power grid dispatching
CN111798096A (en) * 2020-06-02 2020-10-20 国网山西省电力公司电力科学研究院 Visual analysis method, system and storage medium based on power grid dispatching data
CN111752723A (en) * 2020-06-06 2020-10-09 中国科学院电子学研究所苏州研究院 Visual multi-source service management system and implementation method thereof
CN112235361A (en) * 2020-09-28 2021-01-15 青海绿能数据有限公司 Photovoltaic power plant data switching platform
CN112235361B (en) * 2020-09-28 2022-12-27 青海绿能数据有限公司 Photovoltaic power plant data switching platform
CN112308538A (en) * 2020-12-03 2021-02-02 国网福建省电力有限公司厦门供电公司 Active service data interaction method for multi-type park users
CN112650761A (en) * 2021-02-23 2021-04-13 善智互联(北京)网络科技有限公司 Structured extraction method and system for transformer fault information
CN113642850A (en) * 2021-07-20 2021-11-12 国网江苏省电力有限公司南通供电分公司 Data fusion method and terminal for power distribution network planning

Also Published As

Publication number Publication date
CN106777141B (en) 2019-07-12

Similar Documents

Publication Publication Date Title
CN106777141B (en) A kind of acquisition that merging multi-source heterogeneous electric network data and distributed storage method
CN107330056B (en) Wind power plant SCADA system based on big data cloud computing platform and operation method thereof
CN111464336B (en) High-concurrency data processing method and system based on electric power communication machine room
CN103150687B (en) Electric network structure vulnerability real-time evaluation system
CN109146093A (en) A kind of electric power equipment on-site exploration method based on study
CN108718345A (en) A kind of digitlization workshop industrial data Network Transmitting system
CN106603958B (en) Distributed storage remote centralized monitoring system and method for transformer substation inspection robot
CN105790441A (en) Intelligent transformer substation relay protection inspection system
CN111275948B (en) Electric fire fighting early warning system
CN103700031A (en) Electric power warning information publishing method in regulation and control integration mode
CN111327468A (en) Operation method and system for edge computing platform of power system
CN113436029B (en) Wind power data acquisition system and method
CN112186901B (en) Panoramic sensing monitoring method and system for transformer substation
CN110941558B (en) Intelligent office remote operation and maintenance method and system
CN109309733A (en) A kind of Internet of Things energy monitoring system
CN113868078A (en) Wind power plant monitoring method based on cloud platform
CN107742883B (en) Spark-based power system topology island rapid identification system and method
CN111586608A (en) Intelligent health service system of power supply vehicle and data transmission method thereof
Ju et al. The use of edge computing-based internet of things big data in the design of power intelligent management and control platform
CN104579784A (en) Electric power industry control system network management method based on multi-dimensional virtual link
CN103605767A (en) Data access method and system of power transmission lines
CN203261356U (en) Remote data monitoring platform for renewable energy building application system
CN112947364A (en) System and method for early warning of equipment faults of power distribution station based on big data
CN115118578B (en) SCADA system based on WEB
CN110647070A (en) Power environment monitoring system for super-large-scale data center

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CP01 Change in the name or title of a patent holder
CP01 Change in the name or title of a patent holder

Address after: 250002 Wang Yue Road, Ji'nan City, Shandong Province, No. 2000

Patentee after: ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID SHANDONG ELECTRIC POWER Co.

Patentee after: STATE GRID SHANDONG ELECTRIC POWER Co.

Patentee after: Shandong luruan Digital Technology Co.,Ltd.

Patentee after: STATE GRID CORPORATION OF CHINA

Address before: 250002 Wang Yue Road, Ji'nan City, Shandong Province, No. 2000

Patentee before: ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID SHANDONG ELECTRIC POWER Co.

Patentee before: STATE GRID SHANDONG ELECTRIC POWER Co.

Patentee before: SHANDONG LUNENG SOFTWARE TECHNOLOGY Co.,Ltd.

Patentee before: State Grid Corporation of China