CN105577422A - Energy internet real-time data analysis system and method thereof - Google Patents
Energy internet real-time data analysis system and method thereof Download PDFInfo
- Publication number
- CN105577422A CN105577422A CN201510766730.7A CN201510766730A CN105577422A CN 105577422 A CN105577422 A CN 105577422A CN 201510766730 A CN201510766730 A CN 201510766730A CN 105577422 A CN105577422 A CN 105577422A
- Authority
- CN
- China
- Prior art keywords
- real
- data
- computation module
- time
- memory module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 title claims abstract description 16
- 238000004364 calculation method Methods 0.000 claims abstract description 24
- 230000007246 mechanism Effects 0.000 claims abstract description 6
- 238000004891 communication Methods 0.000 claims description 24
- 238000007619 statistical method Methods 0.000 claims description 4
- 238000004458 analytical method Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000000712 assembly Effects 0.000 description 2
- 238000000429 assembly Methods 0.000 description 2
- 238000010205 computational analysis Methods 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 238000010223 real-time analysis Methods 0.000 description 2
- 230000032683 aging Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000003750 conditioning effect Effects 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/14—Arrangements for monitoring or testing data switching networks using software, i.e. software packages
Abstract
The invention discloses an energy internet real-time data analysis system and method thereof, which are suitable for real-time stream calculation, the timeliness requirement of which is high, and precise off-line calculation, the timeliness requirement of which is not high. The front end of the system adopts a distributed cache mechanism to realize large-scale concurrent data high-speed acquisition; and the back end of the system adopts a distributed computation framework mechanism to realize rapid realization of a service computation method. The system and method realize a general and efficient computation framework meeting both real-time stream calculation and off-line calculation, and meanwhile, support horizontal expansion of service computation.
Description
Technical field
The present invention relates to real-time data analysis field, particularly a kind of real-time data analysis system of energy the Internet and method thereof.
Background technology
Along with the development of mobile interchange, technology of Internet of things, the energy internet, applications be based upon on technology of Internet of things basis produces the time series data of multi-source, magnanimity in generative process.Its principal character is: (1) is distributed in the whole network acquisition node and gets more and more; (2) acquisition node data generation frequency is low, and part of nodes data are level second; (3) for the data between single acquisition node or multiple acquisition node, real-time analysis is carried out; (4) for the historical data of single acquisition node or the historical data of multiple acquisition node, statistical analysis is carried out; (5) reliable computational analysis is carried out for real time data; (6) for new business demand, by horizontal extension mode, new computational analysis demand is realized fast.
Summary of the invention
For problems of the prior art, the invention provides a kind of can real-time analysis multi-source, magnanimity the system of time series data to realize the dynamic conditioning of the energy, the high speed acquisition that can realize data with realize real-time streams fast and calculate and the real-time data analysis system of calculated off-line business and method thereof.
Object of the present invention is achieved through the following technical solutions.
A kind of real-time data analysis system of energy the Internet, comprise caching component, computation module, coordination service assembly, described caching component is to computation module one-way communication, two-way communication between described caching component and coordination service assembly, two-way communication between described computation module and coordination service assembly, described caching component is connected with api interface, two-way communication between described api interface and coordination service assembly, described computation module is connected with memory module, two-way communication between described memory module and computation module.
Described computation module comprises data source unit and computing unit, and described data source unit is to computing unit one-way communication.
Described memory module is to data source unit one-way communication, and described computing unit is to memory module one-way communication.
Described api interface be real time data load api interface and or real-time messages load api interface; Described memory module comprises real-time data base, relational database and HDFS.
A real-time data analysis method for energy the Internet, performs real-time streams and calculates under real-time calculation and analysis scene; Calculated off-line is performed under historical data statistical analysis scene.
Described real-time streams calculation procedure comprises:
1) application program loads real time data by api interface to caching component;
2) data source unit of computation module reads real time data from cache layer;
3) Data Computation Unit of computation module, completes the calculating of real-time streaming data, and result of calculation is stored in memory module.
Described calculated off-line is divided into timing cycle calculated off-line and message trigger calculated off-line, and described timing cycle calculated off-line step comprises: 1) data source unit of computation module is by timing mechanism, from memory module, periodically obtain historical data; 2) the computing unit finishing service of computation module calculates, and result of calculation is stored in memory module, and described message trigger calculated off-line step comprises: 1) application program is by api interface, and Trigger message is sent to caching component; 2) data source unit of computation module, obtains message from caching component, and after data source unit acquisition message, then reads historical data from memory module; 3) the computing unit finishing service of computation module calculates, and result of calculation is stored in memory module.
Compared to prior art, the invention has the advantages that: solve real time data high speed distributed capture; Provide the programmable computer system under real-time streams calculating scene; Provide for the high and ageing not high off-line programmable computer system of calculating accuracy.
Accompanying drawing explanation
Fig. 1 is the structural representation of real-time data analysis of the present invention.
Fig. 2 is real-time streams calculation mechanism schematic diagram of the present invention.
Fig. 3 is timing cycle calculated off-line structure chart of the present invention.
Fig. 4 is message trigger calculated off-line structure chart of the present invention.
Embodiment
Below in conjunction with Figure of description and specific embodiment, the present invention is described in detail.
As shown in Figure 1, a kind of real-time data analysis system of energy the Internet, comprise caching component, computation module, coordination service assembly, above assembly all supports that distributed type assemblies is disposed.Described caching component is to computation module one-way communication, two-way communication between described caching component and coordination service assembly, the function of caching component is the high speed acquisition realizing data, and it is mainly divided into two type queues: data buffer storage queue, message buffer queue.In order to the highly effective gathering of real time data, data buffer storage can create multiple data queue, and then promotes the concurrency performance of real time data loading.Meanwhile, from data reliability angle, data/message buffer queue creates backup queue, and is deployed on the clustered node different from former queue by backup queue.The function of coordination service assembly is the coordination service solving caching component and computation module cluster, and the metadata management of application.Two-way communication between described computation module and coordination service assembly, the function of computation module is to provide the fundamental system of real time/off-line calculating, described caching component is connected with api interface, two-way communication between described api interface and coordination service assembly, described computation module is connected with memory module, two-way communication between described memory module and computation module.
Described computation module comprises data source unit and computing unit, and described data source unit is to computing unit one-way communication.
Described memory module is to data source unit one-way communication, and described computing unit is to memory module one-way communication.
Described api interface be real time data load api interface and or real-time messages load api interface; Described memory module comprises real-time data base, relational database and HDFS.Computation module, by the interface of memory module, obtains data or result of calculation is stored in memory module from memory module.The metadata management of caching component, computation module is responsible for by coordination service assembly, and then realizes the coordinated management of each clustered node.Coordination service assembly also can be user and provides metadata management, such as: caching component IP and port numbers, memory module IP and port numbers etc.In the application, whole system system only provides IP and the port numbers of coordination service system to outside, the metadata information of other assemblies is all stored in coordination service assembly.
A real-time data analysis method for energy the Internet, performs real-time streams and calculates under real-time calculation and analysis scene; Calculated off-line is performed under historical data statistical analysis scene.
As shown in Figure 2, described real-time streams calculation procedure comprises:
1) application program loads real time data by api interface to caching component;
2) data source unit of computation module reads real time data from cache layer;
3) Data Computation Unit of computation module, completes the calculating of real-time streaming data, and result of calculation is stored in memory module.
Under this kind of application scenarios, caching component purpose of design: (1) solves the problem of the data buffer storage that application data is submitted at a high speed; (2) persistent storage of data, solves the reliability of data buffer storage; (3) support clustered deploy(ment), solve the problem that Data Concurrent loads.Computation module purpose of design: (1) provides the real-time streams computing system on basis, makes user only pay close attention to service computation; (2) support clustered deploy(ment), solve data parallel and calculate reliably.
When the change of historical data, and then have impact on the correctness of real-time streams result of calculation.For this problem, the present invention, on original system basis, provides the basic system of historical data calculated off-line, and then promotes the completeness of whole system.Calculated off-line provides two kinds of account forms: timing cycle calculated off-line, message trigger calculated off-line.
Described calculated off-line is divided into timing cycle calculated off-line and message trigger calculated off-line, and as shown in Figure 3, timing cycle calculates and do not comprise caching component.Described timing cycle calculated off-line step comprises: 1) data source unit of computation module is by timing mechanism, from memory module, periodically obtain historical data; 2) the computing unit finishing service of computation module calculates, and result of calculation is stored in memory module, and as shown in Figure 4, described message trigger calculated off-line step comprises: 1) application program is by api interface, and Trigger message is sent to caching component; 2) data source unit of computation module, obtains message from caching component, and after data source unit acquisition message, then reads historical data from memory module; 3) the computing unit finishing service of computation module calculates, and result of calculation is stored in memory module.
Embodiment 1: " profile data " (data acquisition system of multiple acquisition node of a certain moment) is respectively loaded on real-time data base, relational database, and each acquisition node of real-time statistics data volume of a day.The caching component of real-time data analysis system, computation module, coordination component adopt respectively: kafka, storm, zookeeper technology.
A kind of typical real-time streams calculates scene, as shown in Figure 2.Step is as follows:
(1) profile data is by Data import api interface, is loaded on caching component;
(2) data source unit of computation module, reads real-time profile data from caching component;
(3) computing unit of computation module, completes real time data and is loaded on real-time data base;
(4) computing unit of computation module, completes relational database when real time data loads;
(5) computing unit of computation module, completes each acquisition node data volume every day statistics, and statistics is loaded into relational database.
Embodiment 2: after considering real-time statistics, acquisition node data are deleted, increase etc., cause the incorrectness of real-time statistics result.In order to head it off, off-line is needed to add up each acquisition node data volume every day.
A kind of typical calculated off-line scene, this scene can be divided into two kinds of different implementations: timing cycle calculated off-line, message trigger calculated off-line.
As shown in Figure 3, the step of timing cycle calculated off-line:
(1) data source unit of computation module is periodically from relational database, reads historical data;
(2) computing unit of computation module completes statistical computation, and result of calculation is stored in relational database.
As shown in Figure 4, the step of message trigger calculating:
(1) Trigger message is by api interface, is loaded on message buffering assembly;
(2) data source unit of computation module, from reading Trigger message.After reading message, data source unit reads by calculated data from relational database;
(3) computing unit of computation module completes statistical computation, and result of calculation is stored in relational database.
The invention is not restricted to above-described embodiment, all technical schemes adopting equivalent replacement or equivalence replacement to be formed all belong to the scope of protection of present invention.
Claims (7)
1. the real-time data analysis system of an energy the Internet, comprise caching component, computation module, coordination service assembly, it is characterized in that described caching component is to computation module one-way communication, two-way communication between described caching component and coordination service assembly, two-way communication between described computation module and coordination service assembly, described caching component is connected with api interface, two-way communication between described api interface and coordination service assembly, described computation module is connected with memory module, two-way communication between described memory module and computation module.
2. the real-time data analysis system of a kind of energy the Internet according to claim 1, it is characterized in that described computation module comprises data source unit and computing unit, described data source unit is to computing unit one-way communication.
3. the real-time data analysis system of a kind of energy the Internet according to claim 2, it is characterized in that described memory module is to data source unit one-way communication, described computing unit is to memory module one-way communication.
4. the real-time data analysis system of a kind of energy the Internet according to claim 1 and 2, it is characterized in that described api interface be real time data load api interface and or real-time messages load api interface; Described memory module comprises real-time data base, relational database and HDFS.
5. a real-time data analysis method for energy the Internet, is characterized in that under real-time calculation and analysis scene, perform real-time streams calculates; Calculated off-line is performed under historical data statistical analysis scene.
6. the real-time data analysis method of a kind of energy the Internet according to claim 5, is characterized in that described real-time streams calculation procedure comprises:
1) application program loads real time data by api interface to caching component;
2) data source unit of computation module reads real time data from cache layer;
3) Data Computation Unit of computation module, completes the calculating of real-time streaming data, and result of calculation is stored in memory module.
7. the real-time data analysis method of a kind of energy the Internet according to claim 5, it is characterized in that described calculated off-line is divided into timing cycle calculated off-line and message trigger calculated off-line, described timing cycle calculated off-line step comprises: 1) data source unit of computation module is by timing mechanism, from memory module, periodically obtain historical data; 2) the computing unit finishing service of computation module calculates, and result of calculation is stored in memory module, and described message trigger calculated off-line step comprises: 1) application program is by api interface, and Trigger message is sent to caching component; 2) data source unit of computation module, obtains message from caching component, and after data source unit acquisition message, then reads historical data from memory module; 3) the computing unit finishing service of computation module calculates, and result of calculation is stored in memory module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510766730.7A CN105577422A (en) | 2015-11-11 | 2015-11-11 | Energy internet real-time data analysis system and method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510766730.7A CN105577422A (en) | 2015-11-11 | 2015-11-11 | Energy internet real-time data analysis system and method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105577422A true CN105577422A (en) | 2016-05-11 |
Family
ID=55887119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510766730.7A Pending CN105577422A (en) | 2015-11-11 | 2015-11-11 | Energy internet real-time data analysis system and method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105577422A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112035333A (en) * | 2020-08-21 | 2020-12-04 | 广东电网有限责任公司 | Energy internet service system, energy internet application system, and electronic device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5657450A (en) * | 1995-11-15 | 1997-08-12 | Xerox Corporation | Method and apparatus for time estimation and progress feedback on distal access operations |
| CN102902775A (en) * | 2012-09-27 | 2013-01-30 | 新浪网技术(中国)有限公司 | Internet real-time computing method and internet real-time computing system |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN104618343A (en) * | 2015-01-06 | 2015-05-13 | 中国科学院信息工程研究所 | Method and system for detecting website threat based on real-time log |
-
2015
- 2015-11-11 CN CN201510766730.7A patent/CN105577422A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5657450A (en) * | 1995-11-15 | 1997-08-12 | Xerox Corporation | Method and apparatus for time estimation and progress feedback on distal access operations |
| CN102902775A (en) * | 2012-09-27 | 2013-01-30 | 新浪网技术(中国)有限公司 | Internet real-time computing method and internet real-time computing system |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN104618343A (en) * | 2015-01-06 | 2015-05-13 | 中国科学院信息工程研究所 | Method and system for detecting website threat based on real-time log |
Non-Patent Citations (1)
| Title |
|---|
| 王利卿 等: "分布式计算框架的研究", 《信息技术与信息化》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112035333A (en) * | 2020-08-21 | 2020-12-04 | 广东电网有限责任公司 | Energy internet service system, energy internet application system, and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102930062B (en) | The method of the quick horizontal extension of a kind of database | |
| CN109951463A (en) | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column | |
| Xiang et al. | An adaptive cloudlet placement method for mobile applications over GPS big data | |
| CN107038162A (en) | Real time data querying method and system based on database journal | |
| CN109710731A (en) | A kind of multidirectional processing system of data flow based on Flink | |
| CN107332719A (en) | A kind of method that daily record is analyzed in real time in CDN system | |
| CN106357463B (en) | The access link tracking implementation method and its system of non-invasive | |
| CN104639658A (en) | Realization method for accessing object storage by file system mounting | |
| CN102254024A (en) | Mass data processing system and method | |
| CN105956068A (en) | Webpage URL repetition elimination method based on distributed database | |
| CN103399894A (en) | Distributed transaction processing method on basis of shared storage pool | |
| CN104699723A (en) | Data exchange adapter and system and method for synchronizing data among heterogeneous systems | |
| CN105405070A (en) | Distributed memory power grid system construction method | |
| CN103645904A (en) | Cache realization method of interface calling | |
| CN104636395A (en) | Count processing method and device | |
| CN102999608A (en) | System and method for tree table demonstration of large data | |
| CN108573029A (en) | A kind of method, apparatus and storage medium obtaining network access relational data | |
| CN109271449A (en) | A kind of distributed storage inquiry system file-based and querying method | |
| Kchaou et al. | Towards an offloading framework based on big data analytics in mobile cloud computing environments | |
| CN102404411A (en) | Data synchronization method of cloud storage system | |
| CN110209674A (en) | A kind of the cloud database statistical method and device of industrial environment dust control wechat small routine | |
| CN103294912B (en) | A kind of facing mobile apparatus is based on the cache optimization method of prediction | |
| CN105610899A (en) | Text file parallel uploading method and device | |
| CN104283966A (en) | Data distribution algorithm and device of cloud storage system | |
| CN107203437A (en) | The methods, devices and systems for preventing internal storage data from losing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160511 |