CN110022226A - A kind of data collection system and acquisition method based on object-oriented - Google Patents

A kind of data collection system and acquisition method based on object-oriented Download PDF

Info

Publication number
CN110022226A
CN110022226A CN201910165447.7A CN201910165447A CN110022226A CN 110022226 A CN110022226 A CN 110022226A CN 201910165447 A CN201910165447 A CN 201910165447A CN 110022226 A CN110022226 A CN 110022226A
Authority
CN
China
Prior art keywords
data
acquisition
topic
node
message
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910165447.7A
Other languages
Chinese (zh)
Other versions
CN110022226B (en
Inventor
郑安刚
巫钟兴
王伟峰
汪岳荣
顾春云
江婷
骆云江
郁春雷
麻吕斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
State Grid Corp of China SGCC
State Grid Zhejiang Electric Power Co Ltd
China Electric Power Research Institute Co Ltd CEPRI
Zhejiang Huayun Information Technology Co Ltd
Original Assignee
State Grid Corp of China SGCC
State Grid Zhejiang Electric Power Co Ltd
China Electric Power Research Institute Co Ltd CEPRI
Zhejiang Huayun Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by State Grid Corp of China SGCC, State Grid Zhejiang Electric Power Co Ltd, China Electric Power Research Institute Co Ltd CEPRI, Zhejiang Huayun Information Technology Co Ltd filed Critical State Grid Corp of China SGCC
Publication of CN110022226A publication Critical patent/CN110022226A/en
Application granted granted Critical
Publication of CN110022226B publication Critical patent/CN110022226B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/02Standardisation; Integration
    • H04L41/0233Object-oriented techniques, for representation of network management data, e.g. common object request broker architecture [CORBA]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/02Standardisation; Integration
    • H04L41/024Standardisation; Integration using relational databases for representation of network management data, e.g. managing via structured query language [SQL]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06Management of faults, events, alarms or notifications
    • H04L41/0631Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/02Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H04L67/025Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1001Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004Server selection for load balancing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1097Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/12Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/568Storing data temporarily at an intermediate stage, e.g. caching
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22Parsing or analysis of headers

Abstract

The invention discloses a kind of data collection system and acquisition method based on object-oriented, are related to power system of data acquisition field.Currently, insufficient in terms of the expansion occurred during electric energy acquisition, reusability and flexibility.The present invention includes gateway cluster, the preposition cluster of communication, operational processor cluster, data/address bus, storage service module, mass data analysis module, data memory module;The technical program uses distributed elastic architecture design, with technologies such as stream process, message-oriented middleware, distributed storage and parallel computations, reconstruct electric power data acquisition system, it is greatly promoted from storage capacity, calculated performance, data processing speed and intellectual analysis etc., provides powerful guarantee for support Power marketing intellectual analysis, service business innovation, expansion professional application and raising electric service level etc..

Description

A kind of data collection system and acquisition method based on object-oriented
Technical field
The present invention relates to power system of data acquisition field more particularly to a kind of data collection systems based on object-oriented And acquisition method.
Background technique
The communication protocol of acquisition system very disunity due to various expansions at present, causes to increase in electric energy acquisition communication process A large amount of unnecessary specification conversion works seriously constrain electric under the following smart grid to realizing that interoperability brings difficulty Measure the height intelligent Application of information.Meanwhile traditional communication protocol is mainly the data type agreement of service-oriented, for electric energy The acquisition tasks demand to become more diverse in acquisition, gradually shown in terms of its expansion, reusability and flexibility it is some not Foot.
When acquiring data, lower of traditional 1376.1 specifications allow to freeze according to regular progress data acquisition, such as day The positive active total electric flux of knot, it is necessary to it goes to acquire by defined 0DF005 coding, the expansion that occurs during electric energy acquisition, It is insufficient in terms of reusability and flexibility.
Summary of the invention
The technical problem to be solved in the present invention and the technical assignment of proposition are to be improved and improved to prior art, A kind of data collection system and acquisition method based on object-oriented is provided, improves expansion, reusability and flexibility to reach Purpose.For this purpose, the present invention takes following technical scheme.
A kind of data collection system based on object-oriented, including gateway cluster, the preposition cluster of communication, operational processor collection Group, data/address bus, storage service module, mass data analysis module, data memory module;
Gateway cluster accesses electric power data acquisition system, and maintenance terminal communication link and original for that will acquire equipment The transmitting-receiving of message, the acquisition equipment include specially becoming negative control terminal, Distribution transformer, low pressure concentrator;
Preposition cluster is communicated, is connected with gateway cluster, for the distribution scheduling of initial data message, and by original datagram Text pushes in Distributed Message Queue;It communicates preposition cluster and device address domain algorithm realization plan is based on to the distribution scheduling of message Slightly distribute, realizes that the dynamic of distribution policy adjusts by the operation conditions of each node of monitoring business processor cluster;Pass through heartbeat The operation conditions of each node of handshake mechanism monitoring front operational processor, for node, newly-increased, node failure and malfunctioning node are extensive Multiple three kinds of scenes are respectively according to " newly-increased node distribution policy ", " distribution policy when node failure " and " malfunctioning node recovery time-division Hair strategy carries out dynamic adjustment, and the terminal address on this node is distributed to specified services processor section by address field algorithm Point reduces program to server memory configuration requirement with reduction and load balance system file;
Operational processor cluster is connected with preposition module is communicated, parses for communication protocol, and and Distributed Message Queue It interacts;Downbound request and composition downlink frame are obtained from message queue, the initial data message for communicating preposition is advised It about parses and pushes to parsing result in Distributed Message Queue,
Data bus module is used to support the timing and persistence of uplink and downlink communication interactive information;Using high-throughput Distributed Kafka message queue, make full use of Kafka service theme and theme subregion, publisher's theme and main website are answered It is associated with cluster, operational processor cluster, the downbound request data and terminal uplink data that unified management master station application generates Transmit/receive;
It is put in storage service module, is stored in relevant database for obtaining batch data from message queue;Using distributed big The mode that data framework Hadoop and traditional Relational DataBase Oracle are combined is to adapt to the analysis and storage of mass data;
Mass data analysis module realizes the real-time of business datum by the big data frame based on distributed file system It calculates, off-line analysis, provides technical support for the excavation of further depth;
Data memory module provides basic number for storing whole business datums, file data, initial data for system According to support and calculate service;It is divided into main storage facility located at processing plant, calamity for library, history library, data publication library, and does by business and storage time limit Point library strategy is to reduce the access pressure in one point data library.
Storage to mass data, according to different data using attribute by relevant database divide for main storage facility located at processing plant, Calamity ensures that acquisition data safety is stable, reduces the data access pressure of storage facility located at processing plant, promotes number for library, history library, data publication library Point library strategy is done according to efficiency for issuing, and by business and storage time limit to reduce the access pressure in one point data library.
As optimization technique means: prepositive communication computer, which draws all terminal device address domains in scene by certain regular partition, is Multiple sections, device address obtain corresponding group address domain section by the quantity modulus of downlink Topic;Downlink Topic and address field There are mapping relations, management of the front service processor node to address field section, namely the pipes to downlink Topic between section Reason;Initialization address domain allocation strategy presses the preposition processing service node quantity modulus of business, and node increases newly, node failure and failure Respectively according to " newly-increased node distribution policy ", " distribution policy when node failure " and " malfunctioning node recovery time-division when node restores Hair strategy " realizes that dynamic adjusts, and distribution information is timely updated to Zookeeper distribution service, to reduce in program Load is deposited, program cluster extended capability is improved.
As optimization technique means: A) node is newly-increased/dilatation when distribution policy are as follows:
A01) Topic distributed each operational processor node is ranked up according to Topic coding, and calculates the section The currently processed Topic sum of point;
A02) operational processor node is ranked up according to Topic sum;
A03 the average value of Topic can be handled by) calculating each operational processor node, and Topic sum is divided by operational processor Node total number;
A04) the extra Topic of all operational processor nodes by Topic quantity in node greater than AvgTopic takes out, Take out rule: A05) preferential selection step A02) Topic encodes biggish Topic in the biggish operational processor node of sequence;
A06 the Topic taken out in step A04) preferentially) is distributed into newly-increased operational processor node, makes newly-increased node Topic number about average value;Modulus distribution is carried out to all nodes if still having unappropriated Topic;
A07 the Topic information being assigned away) is deleted in other nodes.
As optimization technique means: B) node failure when distribution policy are as follows:
B01) front service processor node is ranked up according to Topic quantity;
B02) Topic sum obtains currently running each divided by currently running front service processor node sum The average value of operational processor node processing Topic;
B03 it) the Topic quantity that can be increased newly according to the currently running each operational processor node of mean value calculation: is counted by b The counted existing Topic number of mean value-;
B04) by because Topic to be distributed caused by node failure according to node it is newly-increased/dilatation when distribution policy calculating Value is successively distributed to the small operational processor node that sorts.
As optimization technique means: C) malfunctioning node restore when distribution policy are as follows:
C01 allocation strategy when) the operational processor node restored is according to initialization loads corresponding Topic;
C02 these Topic information for being returned to recovery nodes) are deleted from other operational processor node timings.
As optimization technique means: storage service module is adopted i.e. school, the in real time mode repaired, fortune to acquisition data It with stream process technology, realizes and acquisition load, electric energy indicating value data is checked in real time, are verified, and problem data is marked Note, repairs abnormal load data;For problem data by power estimation, ARIMA algorithm and marketing distribution electricity into Row is repaired, and guarantees the reasonability, consistency, logicality of data, by finding, marking invalid and distortion data in time, improves system The system quality of data;The real time monitoring and analyzing of electric energy data, alarm event is realized with stream process technology;The stream process skill Art is using the frame calculated in real time, and using Hbase+Storm, the real-time Computational frame of Storm is responsible for obtaining from message queue former Beginning data and message data enter HBase distributed data base.
As optimization technique means: mass data analysis module is real by big data distributed memory parallel computation frame Now by the hour to acquisition success rate index, all types of user electricity and load, line loss calculation, distribution transforming operational monitoring, mobile operator Channel quality monitoring, the online rate of terminal are counted, to meet unit business control demands at different levels;Quasi real time analytical framework uses Hive+Spark, Spark off-line calculation frame, which are realized, imports the statistics that Hive data warehouse executes mass data for initial data Analysis business and data mining.
Another goal of the invention of the invention is to provide a kind of collecting method based on object-oriented, and feature exists In:
1) when acquisition main website needs to be configured terminal, measurement point, calls survey operation together, comprising the following steps:
101) the main storage facility located at processing plant of Oracle is main to store whole business datums, archives number from marketing system synchronous foundation data According to, initial data, data query is provided for acquisition main website;
102) acquisition main website initiates downbound request, and different Key can be arranged according to different operation type and be published to Kafka service Downlink Topic in, and by operational order id deposit Redis caching in;
103) message in downlink Topic is partitioned storage according to Key and algorithm, and different subregions can define different excellent Class downlink is arranged in first grade, the multidomain treat-ment as the highest multidomain treat-ment control class downbound request of configuration preference level, priority are taken second place It requests, the multidomain treat-ment of other priority calls survey/relaying class downbound request together;
104) operational processor node loaded from Redis cache server with the archive information of synchronous designated terminal, from The message of Kafka service subscription down queue is executed according to different Partition priority, forms downbound request message frame, It is distributed to the preposition cluster of communication, and downlink message is pushed in the message Topic in Kafka;
105) it communicates preposition cluster and is sent to communication gate cluster by scheduling distribution policy;
106) downbound request is sent to terminal device by communication gate;
107) terminal returns to operating result, carries out packet parsing by operational processor through communication gate, the preposition cluster of communication And operating result is returned into the corresponding operational order id of the terminal in Redis;
108) acquisition main website obtains operating result according to the corresponding operational order id of the terminal from Redis;
2) when needing data acquisition for electric energy, comprising the following steps:
201) by task class data, abnormal events, the formal of message is sent to gateway cluster accordingly;
202 gateway clusters are distributed to the preposition cluster of communication by load balancing;
203) original message data is distributed to operational processor cluster according to scheduling distribution policy;
204) device node loads and the archive information of synchronous designated terminal, parsing uplink original from Redis cache server Beginning message data, and the information such as parsing result, original message data are pushed in corresponding Kafka message queue;That is, parsing As a result it pushes in reported data Topic, original message data pushes to message Topic;
205) library service obtains former from Kafka service subscription message, the real-time Computational frame of Storm from Kafka message queue Beginning message data, electric energy data etc. are stored in HBase distributed data base;Spark off-line calculation frame imports initial data Hive data warehouse executes complicated statistical analysis and data mining;Data loading service is by original message data, electric flux number Relevant database is stored according to batch;
206) from cloud platform quick search data acquisition for electric energy detail, acquisition success rate etc.;
207) the main storage facility located at processing plant of le is main to store whole business datums, file data, original from marketing system synchronous foundation data Beginning data provide data query for acquisition main website;
3) when the completion of Yao Jinhang electric energy data, include the following steps
301) electric energy data is sent with message in form to communication gate cluster by multiple communication modes;
302) communication gate cluster is sent to the preposition cluster of communication according to load balancing distribution policy;
303) preposition cluster is distributed to operational processor cluster by dispatching distribution policy;
304) operational processor node loads and the archive information of synchronous designated terminal, solution from Redis cache server Uplink original message data is analysed, and the information such as parsing result, original message data are pushed into corresponding Kafka message queue In;That is, parsing result pushes in reported data Topic, original message data pushes to message Topic;
305) stream calculation services Storm from Kafka service subscription message, obtains electric energy data in real time, is stored in real time Task data in HBase distributed data gets table ready;
306) when mending trick for real-time leak source, Spark RDD timing executes leak source audit task, i.e., is mended according to leak source and recruit plan Leak source audit slightly is carried out to getting table in HBase ready, and corresponding leak source is formed according to terminal called state and is requested, and pushes to Kafka It issues in the downlink Topic of service for operational processor acquisition, is recruited to realize that real-time leak source is mended;
307) when mending trick for manual leak source, Spark RDD timed task reads benefit from oracle database and recruits plan Leak source audit task is slightly executed afterwards, and corresponding leak source is formed according to terminal called situation and is requested, and pushes to the confession of Kafka messaging service Operational processor acquisition issues, and realizes that leak source is mended and recruits.
As optimization technique means: in the acquisition for terminal affair, being defined according to the order of importance and emergency of event not at the same level Other acquisition module specifies and different reports frequency;Channel resource can not only be more reasonably distributed, reduces terminal handler not Necessary expense, while administrative staff's analysis can be assisted, processing anomalous event, promote the efficiency of management.
As optimization technique means: data acquisition include the acquisition to special parameter evidence, low pressure I type concentrator data acquisition, The acquisition of low pressure II type concentrator data;
One) special parameter is according to acquisition:
When specially day is freezed active energy in change acquisition, it is 1 that terminal, which acquires ammeter and executes frequency, the frequency of terminal reported data It is 12 hours, data classification is day freezing data, and data item includes working as previous quadrant reactive energy indicating value data block, current four-quadrant Limit reactive energy indicating value data block, current forward direction active energy indicating value data block, current reversed active energy indicating value data block;
When specially becoming 96 point load curve of acquisition, it is 15 minutes that terminal, which acquires ammeter and the execution frequency of reported data, data It is classified as real time data, data item includes voltage data block, current data block, active power, when previous quadrant reactive energy shows Value Data block, current four-quadrant reactive energy indicating value data block, positive active total electric flux, power factor;
Two) low pressure I type concentrator data acquire:
When day is freezed active energy in the acquisition of low pressure I type concentrator, it is 1 that terminal, which acquires ammeter and executes frequency, and terminal reports The frequency of data is 12 hours, and data classification is day freezing data, and acquisition and reported data item include current positive active energy Indicating value data block, current reversed active energy indicating value data block;
When low pressure I type concentrator acquires 96 point load curve, it is 6 small that terminal, which acquires ammeter and the execution frequency of reported data, When, data classification is minute freezing data, if installation is three-phase electric energy meter, data item is voltage data block, current data Block, power factor (PF), active power, positive active total electric flux, if installation is single-phase electric energy meter, data item be A phase voltage, A phase current, power factor (PF), positive active total electric flux, active power, N line current;
Three) low pressure II type concentrator data acquire:
It is same with I type concentrator when day is freezed active energy in the acquisition of low pressure II type concentrator, that is, issue same acquisition scheme Template and report plan template;
When low pressure II type concentrator acquires 96 point load curve, it is 15 that terminal, which acquires ammeter and the execution frequency of reported data, Minute, data classification is real time data, if installation is three-phase electric energy meter, data item be voltage data block, current data block, Power factor (PF), active power, positive active total electric flux, if installation is single-phase electric energy meter, data item is A phase voltage, A phase Electric current, power factor (PF), positive active total electric flux, active power, N line current.
The utility model has the advantages that
One, the technical program uses distributed elastic architecture design, with stream process, message-oriented middleware, distributed storage With the technologies such as parallel computation, electric power data acquisition system is reconstructed, from storage capacity, calculated performance, data processing speed and intelligence Analysis etc. is greatly promoted, and for support Power marketing intellectual analysis, service business innovation, is expanded professional application and is mentioned High electric service level etc. provides powerful guarantee.
Two, the technical program is based on object-oriented communications protocol characteristic, more sets of data acquisition methods can be set, relative to tradition Acquisition mode be obviously improved effect in terms of the efficiency of data acquisition, flexibility and loss:
1, when the acquisition scheme of basic data is divided into acquisition scheme and reports scheme, terminal acquisition ammeter is defined respectively Rule and the rule that reports of terminal data, have the advantages that following two aspect:
(1) flexible data acquisition modes (adopt in real time, be packaged and adopt), the period of reported data and frequency are realized, is realized Local communication traffic, which avoids the peak hour, reduces acquisition leak source obvious action;
(2) alternative configuration data acquisition and data report, for the partial data item (example of support equipment local service As clock patrols survey, local mend is copied), it can accomplish only to adopt not reporting, effectively improve the diversity and data acquisition of field device service Quality.
2, different acquisition data item, such as three-phase meter acquisition current data block and voltage be can configure for electric energy meter type Data block (A, B, C), single-phase meter only acquire A phase current and A phase voltage, all acquire A, B, C three-phase electricity relative to all tables of tradition Pressure and electric current, effectively reduce the loss of flow.
3, for the acquisition of terminal affair, the acquisition module of different stage can be defined according to the order of importance and emergency of event, specified Different reports frequency, can not only more reasonably distribute channel resource, reduces the unnecessary expense of terminal handler, while can To assist administrative staff's analysis, processing anomalous event, promote the efficiency of management.
The technical program realizes that dynamic adjusts, and with reduction and load balance system file, reduces program to server memory Configuration requirement.
Mass data storage framework focuses on " memorization, Yun Hua, the division of labor are specialized ", realize electric energy data integrated fusion and Efficiently management.
Analysis to mass data, it is real by the big data frame based on distributed file system using big data cloud platform The real-time calculating of existing business datum, off-line analysis.
Off-line analysis frame preferably uses Hive+Spark, and the realization of Spark off-line calculation frame imports initial data The statistical analysis business and data mining of Hive data warehouse execution mass data.
Storage to mass data, according to different data using attribute by relevant database divide for main storage facility located at processing plant, Calamity ensures that acquisition data safety is stable, reduces the data access pressure of storage facility located at processing plant, promotes number for library, history library, data publication library Point library strategy is done according to efficiency for issuing, and by business and storage time limit to reduce the access pressure in one point data library.
Detailed description of the invention
Fig. 1 is structure of the invention block diagram;
Fig. 2 is main website setting of the invention, the flow chart for calling survey together;
Fig. 3 is the flow chart of data acquisition for electric energy of the invention;
Fig. 4 is the flow chart of the invention to electric energy data completion;
Specific embodiment
Technical solution of the present invention is described in further detail below in conjunction with Figure of description.
As shown in Figure 1, a kind of data collection system based on object-oriented includes gateway cluster, the preposition cluster of communication, industry Business processor cluster, data/address bus, storage service module, mass data analysis module, data memory module.
The technical program use distributed elastic architecture design, with stream process, message-oriented middleware, distributed storage with simultaneously The technologies such as row calculating, reconstruct electric power data acquisition system, from storage capacity, calculated performance, data processing speed and intellectual analysis Etc. greatly promoted, for support Power marketing intellectual analysis, service business innovation, expand professional application and improve supply Electric service level etc. provides powerful guarantee.
The technical program has the following characteristics that
1. reconstructing signal procedure using resilient infrastructure, meets and constantly increase userbase and acquisition demand: using big data Technology redesigns the framework of power information acquisition system, using distributed elastic architecture design: first is that communication gate It is cached using message as bus with acquisition front end processor, carries out message communication;Second is that front end processor operational processor only to message into Professional etiquette about parses, and writes data into message caching;Third is that storage architecture focuses on " memorization, Yun Hua, the division of labor are specialized ", introduce NOSQL storage, data volume is big, storage management ability is various, and cloud platform streaming computing and off-line analysis service cluster carry out event The operation such as analysis, data check, reparation, relational database use Attribute transposition standby for main storage facility located at processing plant, calamity according to different data Library, history library, data publication library.Fourth is that all acquisition data are put in storage by storage service cluster is unified.
Communication gate service is mainly responsible for the specially acquisition equipment such as change negative control terminal, Distribution transformer, low pressure concentrator and connects Enter electric power data acquisition system, and the transmitting-receiving of maintenance terminal communication link and original message.
Communicate the preposition distribution scheduling for servicing and being responsible for initial data message.
It communicates preposition service to distribute the distribution scheduling of message based on device address domain algorithm implementation strategy, before monitoring The operation conditions for setting each node of operational processor realizes the dynamic adjustment of distribution policy.Its specific algorithm is as follows:
A. all terminal device address domains in scene are drawn by certain regular partition is several sections, for example device address is pressed Quantity (100) modulus of row Topic obtains 100 group address domain sections.In this way, between downlink Topic and address field section There are mapping relations, management of the front service processor node to the management in address field section namely to downlink Topic.
B. initialization address domain allocation strategy presses the preposition processing service node quantity modulus of business, and node is newly-increased, node is former Respectively according to " newly-increased node distribution policy ", " distribution policy when node failure " and " malfunctioning node when barrier and malfunctioning node restore Distribution policy when recovery " realizes that dynamic adjusts, and distribution information is timely updated to Zookeeper distribution service.This Distribution policy purpose is to reduce program internal memory load, raising program cluster extended capability.
C. " distribution policy when node increases (dilatation) newly ":
1) Topic each operational processor node distributed be ranked up according to Topic coding (such as: by greatly to It is small), and calculate the currently processed Topic sum of the node.
2) operational processor node (without newly-increased node) is ranked up (such as: descending) according to Topic sum;
3) average value (assuming that the value is labeled as AvgTopic) of Topic can be handled by calculating each operational processor node: Topic sum divided by operational processor node total number (containing newly-increased node), cast out by decimal place.
4) the extra Topic of all operational processor nodes by Topic quantity in node greater than AvgTopic takes out, and takes Regular out: preferential selection step 2 Topic in biggish operational processor node that sorts encodes biggish Topic.
5) Topic taken out in step 4 is preferentially distributed to newly-increased operational processor node, makes newly-increased node Topic number about average value;Modulus distribution is carried out to all nodes if still having unappropriated Topic.
6) the Topic information being assigned away is deleted in other nodes.
D. " distribution policy when node failure ":
1) (such as descending) is ranked up according to Topic quantity to front service processor node.
2) Topic sum obtains currently running each industry divided by currently running front service processor node sum The average value of business processor node processing Topic;
3) it the Topic quantity that can be increased newly according to the currently running each operational processor node of mean value calculation: is calculated by b The existing Topic number of mean value-obtained;
4) according to the calculated value of step c the small industry that sorts will be successively distributed to because of Topic to be distributed caused by node failure Business processor node;
E. " distribution policy when malfunctioning node restores ":
1) allocation strategy when operational processor node restored is according to initialization loads corresponding Topic;
2) these Topic information for being returned to recovery nodes are deleted from other operational processor node timings.
Front service processing, which services, is responsible for communication protocol parsing, and interacts with Distributed Message Queue.I.e. from message Queue obtains downbound request and composition downlink frame, will communicate preposition initial data message and carry out specification parsing and by parsing result It pushes in Distributed Message Queue.
Preposition service node is communicated to be shaken hands by timing heartbeat between operational processor node to obtain all business processings The operating status of device node, and the terminal address on this node is distributed to specified services processor node by address field algorithm.
Distributed Message Queue is responsible for supporting the timing and persistence of uplink and downlink communication interactive information as data/address bus. The specific distributed Kafka message queue for using high-throughput, the theme and theme subregion for making full use of Kafka to service will Publisher's theme and master station application service cluster, operational processor service cluster are associated, and unified management master station application generates Downbound request data and terminal uplink data transmit/receive.
Service cluster is put in storage to be responsible for obtaining batch data deposit relevant database from message queue.
Processing cluster realizes industry by the big data frame based on distributed file system using big data cloud platform in real time The real-time calculating for data of being engaged in, off-line analysis provide technical support for the excavation of further depth.
2, " adopting i.e. school " carried out to acquisition data, repaired in real time, promoted the quality of data: using stream process technology, realize Acquisition load, electric energy indicating value data are checked in real time, are verified, and problem data is marked, to abnormal load data It is repaired;It is repaired for problem data by power estimation, ARIMA algorithm and marketing distribution electricity, guarantees data Reasonability, consistency, logicality improve system data quality by finding, marking invalid and distortion data in time.Meanwhile it transporting The real time monitoring and analyzing of electric energy data, alarm event is realized with stream process technology.The stream process technology calculates in real time Frame, preferably use Hbase+Storm, the real-time Computational frame of Storm be responsible for from message queue obtain initial data and Message data enters HBase distributed data base;
3, by distributive parallel computation framework, realize that mass data quasi real time counts: by big data distributed memory Parallel computation frame is realized and is run by the hour to acquisition success rate index, all types of user electricity and load, line loss calculation, distribution transforming Monitoring, the monitoring of mobile operator channel quality, the online rate of terminal etc. are counted, and unit business control demands at different levels are met.It is quasi- Real-time analytical framework preferably uses Hive+Spark, and Spark off-line calculation frame, which is realized, imports Hive data for initial data The statistical analysis business and data mining of warehouse execution mass data.
4, flexible data store strategy is constructed, realize " storage on demand ", meet various dimensions query demand: analysis is not of the same trade or business Be engaged in data application demand, play commercial data base (Oracle), cache database (Redis), distributed data base (HBase) and Data warehouse (Hive) strong point designs multi-level storage machine system, improves query performance, promotes data application efficiency.
Commercial data base uses Oracle12c database version, and in conjunction with InfiniBand high speed network and SSD, (solid-state is hard Disk) data storing platform for building OLTP (Transaction Processing) business of support high-throughput high concurrent is stored, it is mainly responsible for Whole business datums, file data, initial data are stored, master data is provided for system and supports and calculate service.Relationship type number It can segment as main storage facility located at processing plant, calamity according to library for library, history library, data publication library, ensure that acquisition data safety is stable, reduce storage facility located at processing plant Data access pressure, promoted data publication efficiency, and by business and storage the time limit do point library strategy to reduce one point data library Access pressure.
Cache database (Redis) is a high performance Key-Value database, and the high-Redis of performance can support to surpass Cross 100K+ read-write frequency per second.Not only support the data of simple Key-Value type, at the same also provide list, set, The storage of the data structures such as zset, hash.
Distributed data base HBase is a high reliability, high-performance, towards column, telescopic distributed memory system.
Hive is a Tool for Data Warehouse based on Hadoop, the data file of structuring can be mapped as a number According to library table, simple MapReduce is fast implemented by class SQL statement and is counted, the statistical analysis of data warehouse is very suitable for.
Data between cloud data platform and relevant database synchronize preferably real using Sqoop data transfer tool It is existing.
5, it is based on object-oriented communications protocol characteristic, more sets of data acquisition methods can be devised, relative to traditional acquisition Mode is obviously improved effect in terms of the efficiency of data acquisition, flexibility and loss, described in detail below:
The acquisition scheme of basic data is divided into acquisition scheme and reports scheme by a, defines terminal acquisition ammeter respectively The rule that rule and terminal data report, has the advantages that following two aspect:
(1) flexible data acquisition modes (adopt in real time, be packaged and adopt), the period of reported data and frequency are realized, is realized Local communication traffic, which avoids the peak hour, reduces acquisition leak source obvious action;
(2) alternative configuration data acquisition and data report, for the partial data item (example of support equipment local service As clock patrols survey, local mend is copied), it can accomplish only to adopt not reporting, effectively improve the diversity and data acquisition of field device service Quality.
B can configure different acquisition data item, such as three-phase meter acquisition current data block and voltage for electric energy meter type Data block (A, B, C), single-phase meter only acquire A phase current and A phase voltage, all acquire A, B, C three-phase electricity relative to all tables of tradition Pressure and electric current, effectively reduce the loss of flow.
Acquisition of the c for terminal affair can define the acquisition module of different stage according to the order of importance and emergency of event, specify not With report frequency, can not only more reasonably distribute channel resource, reduce the unnecessary expense of terminal handler, while can be with It assists administrative staff's analysis, processing anomalous event, promote the efficiency of management.
The data acquisition modes of object-oriented protocol terminal can be divided into acquisition scheme and report scheme, and acquisition scheme defines Terminal acquires the rule of ammeter, the rule of schema definition terminal reported data is reported, shown in template sample following table.
Table 1
Object-oriented terminal affair is divided into each rank according to event significance level, and each rank defines respective acquisition Frequency and reported data item, template sample are as shown in the table.
The collecting method of data collection system based on object-oriented, comprising:
One: acquisition main website is configured terminal, measurement point, calls the process for surveying operation together, as shown in Figure 2.
1) the main storage facility located at processing plant of .Oracle is main to store whole business datums, archives number from marketing system synchronous foundation data According to, initial data, data query is provided for acquisition main website.
2) acquires main website and initiates downbound request, and different Key can be arranged according to different operation type and be published to Kafka service Downlink Topic in, and by operational order id deposit Redis caching in.
3) message in downlink Topic is partitioned storage according to Key and algorithm, and different subregions can define different excellent Class downlink is arranged in first grade, the multidomain treat-ment as the highest multidomain treat-ment control class downbound request of configuration preference level, priority are taken second place It requests, the multidomain treat-ment of other priority calls survey/relaying class downbound request together.
4) operational processor node loaded from Redis cache server with the archive information of synchronous designated terminal, from The message of Kafka service subscription down queue is executed according to different Partition priority, forms downbound request message frame, It is distributed to the preposition cluster of communication, and downlink message is pushed in the message Topic in Kafka.
5) communicates preposition cluster and is sent to communication gate cluster by scheduling distribution policy.
6) downbound request is sent to terminal device by communication gate.
7) terminal returns to operating result, carries out packet parsing simultaneously by operational processor through communication gate, the preposition cluster of communication Operating result is returned into the corresponding operational order id of the terminal in Redis.
8) acquires main website and obtains operating result according to the corresponding operational order id of the terminal from Redis.
Two: the process of data acquisition for electric energy, as shown in Figure 3;
1) by task class data, abnormal events, the formal of message is sent to gateway cluster acquisition terminal accordingly;
2) gateway cluster is distributed to the preposition cluster of communication by load balancing;
3) communication is preposition is distributed to operational processor cluster for original message data according to scheduling distribution policy;
4) operational processor node loads and the archive information of synchronous designated terminal, parsing from Redis cache server Uplink original message data, and the information such as parsing result, original message data are pushed in corresponding Kafka message queue; That is, parsing result pushes in reported data Topic, original message data pushes to message Topic.
5) service of stream process storage is from Kafka service subscription message, and the real-time Computational frame of Storm is from Kafka message queue Obtain the deposit HBase distributed data base such as original message data, electric energy data.Spark off-line calculation frame is by original number Complicated statistical analysis and data mining are executed according to Hive data warehouse is imported.Data loading service is by original message data, electricity Energy datum batch deposit relevant database.
6) acquires main website from cloud platform quick search data acquisition for electric energy detail, acquisition success rate etc..
7) the main storage facility located at processing plant of .Oracle is main to store whole business datums, archives number from marketing system synchronous foundation data According to, initial data, data query is provided for acquisition main website.
Performance indicator:
Calculate time-consuming index: all kinds of off-line calculation business of big data cloud platform complete (such as acquisition quality point within half an hour The typical services such as analysis, the analysis of industry load trend);All kinds of real-time stream calculation business treatment scales per second reach 20,000 (such as loads The typical services such as specificity analysis, the maintenance of terminal communications status).
Communication process index communicates preposition cluster single node TCP link access amount and is up to 400,000;Acquire preposition cluster Single node distribution processor message per second is up to 30,000;Data storage service overall data warehouse-in efficiency reaches 60,000 per second.
Supplementing Data can be carried out by mending the strategy recruited for electric energy data missing, improve acquisition success rate;By big Data cloud platform can realize the quick completion of leak source.Benefit trick can be divided into real-time leak source and mend trick and main website manual leak source benefit trick two Point, the specific process such as to electric energy data completion.
Three: to the flow chart of electric energy data completion, as shown in Figure 4.
1) acquisition terminal is sent electric energy data with message to communication gate cluster by multiple communication modes in form.
2) communication gate cluster is sent to the preposition cluster of communication according to load balancing distribution policy.
3) communicates preposition cluster and is distributed to operational processor cluster by dispatching distribution policy.
4) operational processor node loads and the archive information of synchronous designated terminal, parsing from Redis cache server Uplink original message data, and the information such as parsing result, original message data are pushed in corresponding Kafka message queue; That is, parsing result pushes in reported data Topic, original message data pushes to message Topic.
5) stream calculation services Storm from Kafka service subscription message, obtains electric energy data in real time, is stored in real time Task data in HBase distributed data gets table ready.
6) .Spark RDD timing execute leak source audit task, i.e., according to leak source mend recruit strategy to got ready in HBase table into The audit of row leak source, and corresponding leak source is formed according to terminal called state and is requested, it pushes in the downlink Topic of Kafka service and supplies Operational processor acquisition issues, and recruits to realize that real-time leak source is mended.
On the other hand, acquisition main website also can trigger manual leak source and mend trick.
1), which will be mended, recruits in tactful (such as districts and cities' unit, user type, data type) deposit oracle database.
2) .Spark RDD timed task executes leak source audit task after reading benefit trick strategy in oracle database, And corresponding leak source is formed according to terminal called situation and is requested, it pushes to Kafka messaging service and is issued for operational processor acquisition, it is real Existing leak source, which is mended, recruits.
Four: the technical program can be according to terminal type, device type, the type (electricity, load etc.) for acquiring data, electric energy The difference of table type, we need to select different acquisition methods when implementation.
1) specially become collecting method
When specially day is freezed active energy in change acquisition, it is 1 that terminal, which acquires ammeter and executes frequency, the frequency of terminal reported data It is 12 hours, data classification is day freezing data, and data item includes working as previous quadrant reactive energy indicating value data block, current four-quadrant Limit reactive energy indicating value data block, current forward direction active energy indicating value data block, current reversed active energy indicating value data block.
When specially becoming 96 point load curve of acquisition, it is 15 minutes that terminal, which acquires ammeter and the execution frequency of reported data, data It is classified as real time data, data item includes voltage data block, current data block, active power, when previous quadrant reactive energy shows Value Data block, current four-quadrant reactive energy indicating value data block, positive active total electric flux, power factor.
2) low pressure I type concentrator collecting method
When day is freezed active energy in the acquisition of low pressure I type concentrator, it is 1 that terminal, which acquires ammeter and executes frequency, and terminal reports The frequency of data is 12 hours, and data classification is day freezing data, and acquisition and reported data item include current positive active energy Indicating value data block, current reversed active energy indicating value data block.
When low pressure I type concentrator acquires 96 point load curve, it is 6 small that terminal, which acquires ammeter and the execution frequency of reported data, When, data classification is minute freezing data, if installation is three-phase electric energy meter, data item is voltage data block, current data Block, power factor (PF), active power, positive active total electric flux, if installation is single-phase electric energy meter, data item be A phase voltage, A phase current, power factor (PF), positive active total electric flux, active power, N line current.
3) low pressure II type concentrator collecting method
It is completely the same with I type concentrator when day is freezed active energy in the acquisition of low pressure II type concentrator, that is, it issues and similarly adopts Collect plan template and reports plan template.
When low pressure II type concentrator acquires 96 point load curve, it is 15 that terminal, which acquires ammeter and the execution frequency of reported data, Minute, data classification is real time data, if installation is three-phase electric energy meter, data item be voltage data block, current data block, Power factor (PF), active power, positive active total electric flux, if installation is single-phase electric energy meter, data item is A phase voltage, A phase Electric current, power factor (PF), positive active total electric flux, active power, N line current.
4) event scheme designs
The event scheme designed in the present invention can almost cover all terminal affairs, select three events in the present embodiment As case:
A kind of collecting method based on object-oriented shown in figure 1 above -4 is specific embodiments of the present invention, Substantive distinguishing features of the present invention and progress are embodied, it can be carried out under the inspiration of the present invention using needs according to actual The equivalent modifications of shape, structure etc., the column in the protection scope of this programme.

Claims (10)

1. a kind of data collection system based on object-oriented, it is characterised in that: including gateway cluster, the preposition cluster of communication, industry Business processor cluster, data/address bus, storage service module, mass data analysis module, data memory module;
Gateway cluster, for equipment access electric power data acquisition system, and maintenance terminal communication link and original message will to be acquired Transmitting-receiving, the acquisition equipment include specially become negative control terminal, Distribution transformer, low pressure concentrator;
Preposition cluster is communicated, is connected with gateway cluster, is pushed away for the distribution scheduling of initial data message, and by initial data message It send into Distributed Message Queue;Preposition cluster is communicated to the distribution scheduling of message based on device address domain algorithm implementation strategy point Hair realizes that the dynamic of distribution policy adjusts by the operation conditions of each node of monitoring business processor cluster;It is shaken hands by heartbeat The operation conditions of each node of mechanism monitors front service processor, for node, newly-increased, node failure and malfunctioning node restore three Kind scene according to " newly-increased node distribution policy ", " distribution policy when node failure " and " distributes plan when malfunctioning node recovery respectively Dynamic adjustment is slightly carried out, and the terminal address on this node is distributed to specified services processor node by address field algorithm, with Reduction and load balance system file reduce program to server memory configuration requirement;
Operational processor cluster is connected with preposition module is communicated, parses for communication protocol, and carries out with Distributed Message Queue Interaction;Downbound request and composition downlink frame are obtained from message queue, preposition initial data message will be communicated and carry out specification solution It analyses and pushes to parsing result in Distributed Message Queue,
Data bus module is used to support the timing and persistence of uplink and downlink communication interactive information;Using point of high-throughput Cloth Kafka message queue, the theme and theme subregion for making full use of Kafka to service, by publisher's theme and master station application collection Group, operational processor cluster are associated, and the receipts of downbound request data and terminal uplink data that unified management master station application generates/ Hair;
It is put in storage service module, is stored in relevant database for obtaining batch data from message queue;Using distributed big data The mode that frame Hadoop and traditional Relational DataBase Oracle are combined is to adapt to the analysis and storage of mass data;
Mass data analysis module, by by the big data frame of distributed file system realize business datum it is real-time based on It calculates, off-line analysis, provides technical support for the excavation of further depth;
Data memory module provides master data branch for storing whole business datums, file data, initial data for system Hold and calculate service;It is divided into main storage facility located at processing plant, calamity for library, history library, data publication library, and does a point library by business and storage time limit Strategy is to reduce the access pressure in one point data library.
2. a kind of data collection system based on object-oriented according to claim 1, it is characterised in that: prepositive communication computer It is multiple sections that all terminal device address domains in scene, which are drawn by certain regular partition, and the quantity of downlink Topic is pressed in device address Modulus obtains corresponding group address domain section;There are mapping relations, front service processors between downlink Topic and address field section Management of the node to address field section, namely the management to downlink Topic;Initialization address domain allocation strategy presses the preposition place of business Service node quantity modulus is managed, respectively according to " newly-increased node distributes plan when node is newly-increased, node failure and malfunctioning node restore Slightly ", " distribution policy when node failure " and " distribution policy when malfunctioning node restores " realizes that dynamic adjusts, timely by information is distributed It is updated to Zookeeper distribution service, to reduce program internal memory load, improves program cluster extended capability.
3. a kind of data collection system based on object-oriented according to claim 2, it is characterised in that: A) node is new Distribution policy when increasing/dilatation are as follows:
A01) Topic distributed each operational processor node is ranked up according to Topic coding, and is calculated the node and worked as The Topic sum of pre-treatment;
A02) operational processor node is ranked up according to Topic sum;
A03 the average value of Topic can be handled by) calculating each operational processor node, and Topic sum is divided by operational processor node Sum;
A04) the extra Topic of all operational processor nodes by Topic quantity in node greater than AvgTopic takes out, and takes out Rule: A05) preferential selection step A02) Topic encodes biggish Topic in the biggish operational processor node of sequence;
A06 the Topic taken out in step A04) preferentially) is distributed into newly-increased operational processor node, makes newly-increased node Topic number about average value;Modulus distribution is carried out to all nodes if still having unappropriated Topic;
A07 the Topic information being assigned away) is deleted in other nodes.
4. a kind of data collection system based on object-oriented according to claim 3, it is characterised in that: B) node failure When distribution policy are as follows:
B01) front service processor node is ranked up according to Topic quantity;
B02) Topic sum obtains currently running each business divided by currently running front service processor node sum The average value of processor node processing Topic;
B03 it) the Topic quantity that can be increased newly according to the currently running each operational processor node of mean value calculation: is calculated by b The existing Topic number of mean value-;
B04) by because Topic to be distributed caused by node failure according to node it is newly-increased/dilatation when distribution policy calculated value according to It is secondary to be distributed to the small operational processor node that sorts.
5. a kind of data collection system based on object-oriented according to claim 4, it is characterised in that: C) malfunctioning node Distribution policy when recovery are as follows:
C01 allocation strategy when) the operational processor node restored is according to initialization loads corresponding Topic;
C02 these Topic information for being returned to recovery nodes) are deleted from other operational processor node timings.
6. a kind of data collection system based on object-oriented according to claim 5, it is characterised in that: storage service mould Block adopts acquisition data, and i.e. school, the mode repaired is realized and is shown acquisition load, electric energy with stream process technology in real time Value Data is checked in real time, is verified, and problem data is marked, and is repaired to abnormal load data;For problem Data are repaired by power estimation, ARIMA algorithm and marketing distribution electricity, guarantee the reasonability, consistency, logic of data Property, by finding, marking invalid and distortion data in time, improve system data quality;Electric flux is realized with stream process technology The real time monitoring and analyzing of data, alarm event;The stream process technology is using the frame calculated in real time, using Hbase+ The real-time Computational frame of Storm, Storm is responsible for obtaining initial data from message queue and message data enters HBase distribution number According to library.
7. a kind of data collection system based on object-oriented according to claim 6, it is characterised in that: mass data point Module is analysed by big data distributed memory parallel computation frame, is realized by the hour to acquisition success rate index, all types of user electricity Amount and load, line loss calculation, distribution transforming operational monitoring, the monitoring of mobile operator channel quality, the online rate of terminal are counted, with full Foot unit business at different levels manage demand;Quasi real time analytical framework uses Hive+Spark, and the realization of Spark off-line calculation frame will be former Beginning data import the statistical analysis business and data mining that Hive data warehouse executes mass data.
8. a kind of data of the data collection system based on object-oriented described in -7 any claims are adopted according to claim 1 Set method, it is characterised in that:
1) when acquisition main website needs to be configured terminal, measurement point, calls survey operation together, comprising the following steps:
101) the main storage facility located at processing plant of Oracle is main to store whole business datums, file data, original from marketing system synchronous foundation data Beginning data provide data query for acquisition main website;
102) acquisition main website initiates downbound request, and different Key can be arranged according to different operation type and be published under Kafka service In row Topic, and will be in operational order id deposit Redis caching;
103) message in downlink Topic is partitioned storage according to Key and algorithm, and different subregions can define different preferential Grade, the multidomain treat-ment setting class downlink as the highest multidomain treat-ment control class downbound request of configuration preference level, priority are taken second place are asked It asks, the multidomain treat-ment of other priority calls survey/relaying class downbound request together;
104) operational processor node loaded from Redis cache server with the archive information of synchronous designated terminal, from Kafka The message of service subscription down queue is executed according to different Partition priority, forms downbound request message frame, distribution It gives communication preposition cluster, and downlink message is pushed in the message Topic in Kafka;
105) it communicates preposition cluster and is sent to communication gate cluster by scheduling distribution policy;
106) downbound request is sent to terminal device by communication gate;
107) terminal returns to operating result, carries out packet parsing by operational processor through communication gate, the preposition cluster of communication and incites somebody to action Operating result returns to the corresponding operational order id of the terminal in Redis;
108) acquisition main website obtains operating result according to the corresponding operational order id of the terminal from Redis;
2) when needing data acquisition for electric energy, comprising the following steps:
201) by task class data, abnormal events, the formal of message is sent to gateway cluster accordingly;
202 gateway clusters are distributed to the preposition cluster of communication by load balancing;
203) original message data is distributed to operational processor cluster according to scheduling distribution policy;
204) device node loads and the archive information of synchronous designated terminal, the parsing original report of uplink from Redis cache server Literary data, and the information such as parsing result, original message data are pushed in corresponding Kafka message queue;That is, parsing result It pushes in reported data Topic, original message data pushes to message Topic;
205) library service obtains original report from Kafka message queue from Kafka service subscription message, the real-time Computational frame of Storm Literary data, electric energy data etc. are stored in HBase distributed data base;Initial data is imported Hive by Spark off-line calculation frame Data warehouse executes complicated statistical analysis and data mining;Original message data, electric energy data are criticized in data loading service Amount deposit relevant database;
206) from cloud platform quick search data acquisition for electric energy detail, acquisition success rate etc.;
207) the main storage facility located at processing plant of le is main to store whole business datums, file data, original number from marketing system synchronous foundation data According to, for acquisition main website data query is provided;
3) when the completion of Yao Jinhang electric energy data, include the following steps
301) electric energy data is sent with message in form to communication gate cluster by multiple communication modes;
302) communication gate cluster is sent to the preposition cluster of communication according to load balancing distribution policy;
303) preposition cluster is distributed to operational processor cluster by dispatching distribution policy;
304) operational processor node loaded from Redis cache server with the archive information of synchronous designated terminal, in parsing Row original message data, and the information such as parsing result, original message data are pushed in corresponding Kafka message queue;That is, Parsing result pushes in reported data Topic, and original message data pushes to message Topic;
305) stream calculation services Storm from Kafka service subscription message, obtains electric energy data in real time, is stored in HBase points in real time Task data in cloth data gets table ready;
306) when mending trick for real-time leak source, Spark RDD timing executes leak source audit task, i.e., it is tactful right to be recruited according to leak source benefit It gets table in HBase ready and carries out leak source audit, and corresponding leak source is formed according to terminal called state and is requested, push to Kafka service Downlink Topic in issued for operational processor acquisition, recruited to realize that real-time leak source is mended;
307) when mending trick for manual leak source, Spark RDD timed task is after reading benefit trick strategy in oracle database Leak source audit task is executed, and corresponding leak source is formed according to terminal called situation and is requested, pushes to Kafka messaging service for business Processor acquisition issues, and realizes that leak source is mended and recruits.
9. a kind of collecting method based on object-oriented according to claim 8, it is characterised in that: for terminal When the acquisition of event, the acquisition module of different stage is defined according to the order of importance and emergency of event, is specified and different is reported frequency.
10. a kind of collecting method based on object-oriented according to claim 8, it is characterised in that: data acquisition Including acquisition, the acquisition of low pressure I type concentrator data, the acquisition of low pressure II type concentrator data to special parameter evidence;
One) special parameter is according to acquisition:
When specially day is freezed active energy in change acquisition, it is 1 that terminal, which acquires ammeter and executes frequency, and the frequency of terminal reported data is 12 Hour, data classification be day freezing data, data item include work as previous quadrant reactive energy indicating value data block, current four-quadrant without Function electric energy indicating value data block, current forward direction active energy indicating value data block, current reversed active energy indicating value data block;
When specially becoming 96 point load curve of acquisition, it is 15 minutes that terminal, which acquires ammeter and the execution frequency of reported data, data classification For real time data, data item includes voltage data block, current data block, active power, when previous quadrant reactive energy indicating value number According to block, current four-quadrant reactive energy indicating value data block, positive active total electric flux, power factor;
Two) low pressure I type concentrator data acquire:
When day is freezed active energy in the acquisition of low pressure I type concentrator, it is 1 that terminal, which acquires ammeter and executes frequency, terminal reported data Frequency be 12 hours, data classification is day freezing data, and acquisition and reported data item include current positive active energy indicating value Data block, current reversed active energy indicating value data block;
When low pressure I type concentrator acquires 96 point load curve, it is 6 hours that terminal, which acquires ammeter and the execution frequency of reported data, Data classification is minute freezing data, if installation is three-phase electric energy meter, data item be voltage data block, current data block, Power factor (PF), active power, positive active total electric flux, if installation is single-phase electric energy meter, data item is A phase voltage, A phase Electric current, power factor (PF), positive active total electric flux, active power, N line current;
Three) low pressure II type concentrator data acquire:
It is same with I type concentrator when day is freezed active energy in the acquisition of low pressure II type concentrator, that is, issue same acquisition scheme template With report plan template;
When low pressure II type concentrator acquires 96 point load curve, it is 15 points that terminal, which acquires ammeter and the execution frequency of reported data, Clock, data classification are real time data, if installation is three-phase electric energy meter, data item is voltage data block, current data block, function Rate factor, active power, positive active total electric flux, if installation is single-phase electric energy meter, data item is A phase voltage, A phase electricity Stream, power factor (PF), positive active total electric flux, active power, N line current.
CN201910165447.7A 2019-01-04 2019-03-05 Object-oriented data acquisition system and acquisition method Active CN110022226B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910009543 2019-01-04
CN2019100095432 2019-01-04

Publications (2)

Publication Number Publication Date
CN110022226A true CN110022226A (en) 2019-07-16
CN110022226B CN110022226B (en) 2023-04-04

Family

ID=67189296

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910165447.7A Active CN110022226B (en) 2019-01-04 2019-03-05 Object-oriented data acquisition system and acquisition method

Country Status (1)

Country Link
CN (1) CN110022226B (en)

Cited By (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110502559A (en) * 2019-07-25 2019-11-26 浙江公共安全技术研究院有限公司 A kind of data/address bus and transmission method of credible and secure cross-domain data exchange
CN110535909A (en) * 2019-07-29 2019-12-03 广东电网有限责任公司信息中心 Big data and cloud computing unified platform system towards energy Internet application
CN110636116A (en) * 2019-08-29 2019-12-31 武汉烽火众智数字技术有限责任公司 Multidimensional data acquisition system and method
CN110851447A (en) * 2019-11-07 2020-02-28 积成电子股份有限公司 Flexible storage method of II-type concentrator
CN110941530A (en) * 2019-11-11 2020-03-31 南方电网财务有限公司 Method and device for acquiring monitoring data, computer equipment and storage medium
CN110987083A (en) * 2019-12-23 2020-04-10 北京蜂云科创信息技术有限公司 Method and equipment for monitoring vehicle emission data based on Internet of vehicles
CN111177276A (en) * 2020-01-06 2020-05-19 浙江中烟工业有限责任公司 Spark calculation framework-based kinetic energy data processing system and method
CN111211919A (en) * 2019-12-23 2020-05-29 南京壹格软件技术有限公司 Configuration method of special Internet of things intelligent gateway for data center machine room
CN111432295A (en) * 2020-03-18 2020-07-17 北京科东电力控制系统有限责任公司 Power consumption information acquisition master station system based on distributed technology
CN111629029A (en) * 2020-04-17 2020-09-04 金蝶软件(中国)有限公司 Service publishing method and system
CN111680074A (en) * 2019-12-31 2020-09-18 国网浙江省电力有限公司 Clustering algorithm-based electric power collection load leakage point feature mining method
CN111881105A (en) * 2020-07-30 2020-11-03 北京智能工场科技有限公司 Service data labeling model and model training method thereof
CN111966663A (en) * 2020-08-07 2020-11-20 广东卓维网络有限公司 Multi-user-side comprehensive energy data service system
CN112015766A (en) * 2020-08-24 2020-12-01 京东数字科技控股股份有限公司 Data processing method and device based on pipelining and data processing system
CN112235142A (en) * 2020-10-15 2021-01-15 国网江苏省电力有限公司营销服务中心 Power utilization information acquisition system capable of realizing key business disaster tolerance and operation method thereof
CN112260398A (en) * 2020-09-18 2021-01-22 许继集团有限公司 Power grid monitoring system supporting dynamic expansion
CN112286962A (en) * 2020-10-26 2021-01-29 积成电子股份有限公司 Electricity consumption information acquisition terminal meter reading success rate statistical method and system
CN112308731A (en) * 2020-09-24 2021-02-02 国网天津市电力公司营销服务中心 Cloud computing method and system for multitask concurrent processing of acquisition system
CN112306674A (en) * 2020-09-24 2021-02-02 国网天津市电力公司营销服务中心 Energy equipment information acquisition task cooperative scheduling method and system
CN112397193A (en) * 2020-11-16 2021-02-23 康键信息技术(深圳)有限公司 Data reporting method, device, equipment and storage medium
CN112395318A (en) * 2020-11-24 2021-02-23 福州大学 Distributed storage middleware based on HBase + Redis
CN112468578A (en) * 2020-11-25 2021-03-09 常州微亿智造科技有限公司 Real-time industrial data acquisition system and method
CN112492024A (en) * 2020-11-26 2021-03-12 国网湖南省电力有限公司 Real-time data sharing system for user electricity utilization information acquisition system
CN112765410A (en) * 2020-12-31 2021-05-07 山西省交通科技研发有限公司 Layered design platform architecture adopting end cloud architecture
CN112819282A (en) * 2021-01-04 2021-05-18 傲普(上海)新能源有限公司 Multi-device energy storage EMS data storage system
CN113010565A (en) * 2021-03-25 2021-06-22 腾讯科技(深圳)有限公司 Server cluster-based server real-time data processing method and system
CN113177088A (en) * 2021-04-02 2021-07-27 北京科技大学 Multi-scale simulation big data management system for material irradiation damage
CN113190571A (en) * 2021-05-24 2021-07-30 深圳市坤同智能仓储科技有限公司 System based on message acquisition and multi-dimensional distribution
CN113219235A (en) * 2021-05-10 2021-08-06 南京海兴电网技术有限公司 High-real-time data updating method applied to electricity utilization acquisition system
CN113242315A (en) * 2021-05-31 2021-08-10 中富通集团股份有限公司 Monitoring feedback data receiving and transmitting system and method for operating environment of machine room power equipment
CN113422842A (en) * 2021-08-20 2021-09-21 国网江西省电力有限公司供电服务管理中心 Distributed power utilization information data acquisition system considering network load
CN113596055A (en) * 2021-08-11 2021-11-02 傲普(上海)新能源有限公司 Multi-protocol equipment access method of energy storage EMS system
WO2021218036A1 (en) * 2020-04-28 2021-11-04 武汉旷视金智科技有限公司 Information monitoring method and system, and computer-readable storage medium
CN113672687A (en) * 2021-10-25 2021-11-19 北京值得买科技股份有限公司 E-commerce big data processing method, device, equipment and storage medium
CN113991876A (en) * 2021-12-28 2022-01-28 浙江正泰仪器仪表有限责任公司 Monitoring method and system of power terminal
CN114064741A (en) * 2022-01-17 2022-02-18 天津所托瑞安汽车科技有限公司 Method, device and equipment for acquiring prepositive data and storage medium
CN114928624A (en) * 2022-03-16 2022-08-19 国网河北省电力有限公司营销服务中心 Electricity consumption information acquisition protocol extension method and device, storage medium and terminal
CN115242893A (en) * 2022-05-27 2022-10-25 国电南瑞科技股份有限公司 Multi-protocol data acquisition system and method suitable for power distribution Internet of things management system
CN115617763A (en) * 2022-09-23 2023-01-17 中电金信软件有限公司 Data processing method and device, electronic equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001033349A2 (en) * 1999-11-03 2001-05-10 Accenture Llp Architectures for netcentric computing systems
CN106326331A (en) * 2016-06-29 2017-01-11 河南许继仪表有限公司 Intelligent power utilization data service system based on cloud computation
CN106502772A (en) * 2016-10-09 2017-03-15 国网浙江省电力公司信息通信分公司 Electric quantity data batch high speed processing method and system based on distributed off-line technology
CN106651633A (en) * 2016-10-09 2017-05-10 国网浙江省电力公司信息通信分公司 Power utilization information acquisition system and method based on big data technology
US20170201556A1 (en) * 2016-01-12 2017-07-13 International Business Machines Corporation Scalable event stream data processing using a messaging system
CN108011915A (en) * 2017-07-05 2018-05-08 国网浙江省电力公司 A kind of collection front-end system based on cloud communication

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001033349A2 (en) * 1999-11-03 2001-05-10 Accenture Llp Architectures for netcentric computing systems
US20170201556A1 (en) * 2016-01-12 2017-07-13 International Business Machines Corporation Scalable event stream data processing using a messaging system
CN106326331A (en) * 2016-06-29 2017-01-11 河南许继仪表有限公司 Intelligent power utilization data service system based on cloud computation
CN106502772A (en) * 2016-10-09 2017-03-15 国网浙江省电力公司信息通信分公司 Electric quantity data batch high speed processing method and system based on distributed off-line technology
CN106651633A (en) * 2016-10-09 2017-05-10 国网浙江省电力公司信息通信分公司 Power utilization information acquisition system and method based on big data technology
CN108011915A (en) * 2017-07-05 2018-05-08 国网浙江省电力公司 A kind of collection front-end system based on cloud communication

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
JAYAKUMAR LOGANATHAN: "Distrubuted resource management scheme using enhanced artificial bee-colony in P2P", 《2015 2ND INTERNATIONAL CONFERENCE ON ELECTRONICS AND COMMUNICATION SYSTEMS》 *
黄彬: "基于spark的实时海量数据处理分析与优化", 《中国优秀硕士学位论文数据库》 *

Cited By (49)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110502559A (en) * 2019-07-25 2019-11-26 浙江公共安全技术研究院有限公司 A kind of data/address bus and transmission method of credible and secure cross-domain data exchange
CN110535909A (en) * 2019-07-29 2019-12-03 广东电网有限责任公司信息中心 Big data and cloud computing unified platform system towards energy Internet application
CN110636116B (en) * 2019-08-29 2022-05-10 武汉烽火众智数字技术有限责任公司 Multidimensional data acquisition system and method
CN110636116A (en) * 2019-08-29 2019-12-31 武汉烽火众智数字技术有限责任公司 Multidimensional data acquisition system and method
CN110851447A (en) * 2019-11-07 2020-02-28 积成电子股份有限公司 Flexible storage method of II-type concentrator
CN110851447B (en) * 2019-11-07 2022-04-15 积成电子股份有限公司 Flexible storage method of II-type concentrator
CN110941530A (en) * 2019-11-11 2020-03-31 南方电网财务有限公司 Method and device for acquiring monitoring data, computer equipment and storage medium
CN110941530B (en) * 2019-11-11 2023-09-26 南方电网财务有限公司 Method, device, computer equipment and storage medium for acquiring monitoring data
CN110987083A (en) * 2019-12-23 2020-04-10 北京蜂云科创信息技术有限公司 Method and equipment for monitoring vehicle emission data based on Internet of vehicles
CN111211919A (en) * 2019-12-23 2020-05-29 南京壹格软件技术有限公司 Configuration method of special Internet of things intelligent gateway for data center machine room
CN111680074A (en) * 2019-12-31 2020-09-18 国网浙江省电力有限公司 Clustering algorithm-based electric power collection load leakage point feature mining method
CN111680074B (en) * 2019-12-31 2023-07-04 国网浙江省电力有限公司 Clustering algorithm-based power acquisition load leakage point feature mining method
CN111177276A (en) * 2020-01-06 2020-05-19 浙江中烟工业有限责任公司 Spark calculation framework-based kinetic energy data processing system and method
CN111177276B (en) * 2020-01-06 2023-10-20 浙江中烟工业有限责任公司 Spark computing framework-based kinetic energy data processing system and method
CN111432295A (en) * 2020-03-18 2020-07-17 北京科东电力控制系统有限责任公司 Power consumption information acquisition master station system based on distributed technology
CN111629029A (en) * 2020-04-17 2020-09-04 金蝶软件(中国)有限公司 Service publishing method and system
WO2021218036A1 (en) * 2020-04-28 2021-11-04 武汉旷视金智科技有限公司 Information monitoring method and system, and computer-readable storage medium
CN111881105A (en) * 2020-07-30 2020-11-03 北京智能工场科技有限公司 Service data labeling model and model training method thereof
CN111881105B (en) * 2020-07-30 2024-02-09 北京智能工场科技有限公司 Labeling model of business data and model training method thereof
CN111966663A (en) * 2020-08-07 2020-11-20 广东卓维网络有限公司 Multi-user-side comprehensive energy data service system
CN112015766A (en) * 2020-08-24 2020-12-01 京东数字科技控股股份有限公司 Data processing method and device based on pipelining and data processing system
CN112260398A (en) * 2020-09-18 2021-01-22 许继集团有限公司 Power grid monitoring system supporting dynamic expansion
CN112308731A (en) * 2020-09-24 2021-02-02 国网天津市电力公司营销服务中心 Cloud computing method and system for multitask concurrent processing of acquisition system
CN112306674A (en) * 2020-09-24 2021-02-02 国网天津市电力公司营销服务中心 Energy equipment information acquisition task cooperative scheduling method and system
CN112235142A (en) * 2020-10-15 2021-01-15 国网江苏省电力有限公司营销服务中心 Power utilization information acquisition system capable of realizing key business disaster tolerance and operation method thereof
CN112286962A (en) * 2020-10-26 2021-01-29 积成电子股份有限公司 Electricity consumption information acquisition terminal meter reading success rate statistical method and system
CN112286962B (en) * 2020-10-26 2023-06-02 积成电子股份有限公司 Meter reading success rate statistics method and system for electricity consumption information acquisition terminal
CN112397193A (en) * 2020-11-16 2021-02-23 康键信息技术(深圳)有限公司 Data reporting method, device, equipment and storage medium
CN112395318A (en) * 2020-11-24 2021-02-23 福州大学 Distributed storage middleware based on HBase + Redis
CN112468578A (en) * 2020-11-25 2021-03-09 常州微亿智造科技有限公司 Real-time industrial data acquisition system and method
CN112468578B (en) * 2020-11-25 2021-12-17 常州微亿智造科技有限公司 Real-time industrial data acquisition system and method
CN112492024A (en) * 2020-11-26 2021-03-12 国网湖南省电力有限公司 Real-time data sharing system for user electricity utilization information acquisition system
CN112765410A (en) * 2020-12-31 2021-05-07 山西省交通科技研发有限公司 Layered design platform architecture adopting end cloud architecture
CN112819282A (en) * 2021-01-04 2021-05-18 傲普(上海)新能源有限公司 Multi-device energy storage EMS data storage system
CN113010565A (en) * 2021-03-25 2021-06-22 腾讯科技(深圳)有限公司 Server cluster-based server real-time data processing method and system
CN113177088A (en) * 2021-04-02 2021-07-27 北京科技大学 Multi-scale simulation big data management system for material irradiation damage
CN113219235B (en) * 2021-05-10 2024-01-19 南京海兴电网技术有限公司 High-instantaneity data updating method applied to electricity acquisition system
CN113219235A (en) * 2021-05-10 2021-08-06 南京海兴电网技术有限公司 High-real-time data updating method applied to electricity utilization acquisition system
CN113190571A (en) * 2021-05-24 2021-07-30 深圳市坤同智能仓储科技有限公司 System based on message acquisition and multi-dimensional distribution
CN113242315A (en) * 2021-05-31 2021-08-10 中富通集团股份有限公司 Monitoring feedback data receiving and transmitting system and method for operating environment of machine room power equipment
CN113596055A (en) * 2021-08-11 2021-11-02 傲普(上海)新能源有限公司 Multi-protocol equipment access method of energy storage EMS system
CN113422842A (en) * 2021-08-20 2021-09-21 国网江西省电力有限公司供电服务管理中心 Distributed power utilization information data acquisition system considering network load
CN113672687A (en) * 2021-10-25 2021-11-19 北京值得买科技股份有限公司 E-commerce big data processing method, device, equipment and storage medium
CN113991876A (en) * 2021-12-28 2022-01-28 浙江正泰仪器仪表有限责任公司 Monitoring method and system of power terminal
CN114064741B (en) * 2022-01-17 2022-05-24 天津所托瑞安汽车科技有限公司 Method, device and equipment for acquiring prepositive data and storage medium
CN114064741A (en) * 2022-01-17 2022-02-18 天津所托瑞安汽车科技有限公司 Method, device and equipment for acquiring prepositive data and storage medium
CN114928624A (en) * 2022-03-16 2022-08-19 国网河北省电力有限公司营销服务中心 Electricity consumption information acquisition protocol extension method and device, storage medium and terminal
CN115242893A (en) * 2022-05-27 2022-10-25 国电南瑞科技股份有限公司 Multi-protocol data acquisition system and method suitable for power distribution Internet of things management system
CN115617763A (en) * 2022-09-23 2023-01-17 中电金信软件有限公司 Data processing method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN110022226B (en) 2023-04-04

Similar Documents

Publication Publication Date Title
CN110022226A (en) A kind of data collection system and acquisition method based on object-oriented
CN110047014A (en) A kind of user's electricity data restorative procedure based on load curve and history electricity
CN110225074B (en) Communication message distribution system and method based on equipment address domain
CN107733986B (en) Protection operation big data supporting platform supporting integrated deployment and monitoring
WO2017036244A1 (en) Digital simulation system of power distribution network
CN109739919B (en) Front-end processor and acquisition system for power system
CN106339509A (en) Power grid operation data sharing system based on large data technology
CN103581332B (en) HDFS framework and pressure decomposition method for NameNodes in HDFS framework
CN107330056A (en) Wind power plant SCADA system and its operation method based on big data cloud computing platform
CN108183869B (en) Electric quantity data acquisition system based on distributed message queue
CN112860695B (en) Monitoring data query method, device, equipment, storage medium and program product
CN104599032A (en) Distributed memory power grid construction method and system for resource management
CN103955509A (en) Quick search method for massive electric power metering data
CN105405070A (en) Distributed memory power grid system construction method
CN105045856A (en) Hadoop-based data processing system for big-data remote sensing satellite
CN108573029A (en) A kind of method, apparatus and storage medium obtaining network access relational data
CN113129063A (en) Electricity charge calculation and distribution method and system based on cloud platform and data center
CN103530335B (en) In-stockroom operation method and device of electric power measurement acquisition system
CN113923540A (en) Hierarchical cluster processing method for Internet of things (IOT) collection based on NB-lot network
CN111049898A (en) Method and system for realizing cross-domain architecture of computing cluster resources
CN111260146A (en) Method, device, equipment and medium for locating power system edge cloud data center
CN109508354A (en) A kind of parallel processing system (PPS)
CN102843354B (en) A kind of electric charge calculating method, Control Server, calculation server and system
Wang et al. A global data model for electric power data centers
CN113177084A (en) Database storage method based on Internet of things

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant