CN109165203A - Large public building energy consumption data based on Hadoop framework stores analysis method - Google Patents
Large public building energy consumption data based on Hadoop framework stores analysis method Download PDFInfo
- Publication number
- CN109165203A CN109165203A CN201810866259.2A CN201810866259A CN109165203A CN 109165203 A CN109165203 A CN 109165203A CN 201810866259 A CN201810866259 A CN 201810866259A CN 109165203 A CN109165203 A CN 109165203A
- Authority
- CN
- China
- Prior art keywords
- data
- energy consumption
- hadoop
- building
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005265 energy consumption Methods 0.000 title claims abstract description 43
- 238000004458 analytical method Methods 0.000 title claims abstract description 21
- 238000007405 data analysis Methods 0.000 claims abstract description 10
- 238000013500 data storage Methods 0.000 claims abstract description 8
- 238000005516 engineering process Methods 0.000 claims description 11
- 238000009825 accumulation Methods 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000005094 computer simulation Methods 0.000 claims description 3
- 238000013480 data collection Methods 0.000 claims description 3
- 238000007418 data mining Methods 0.000 claims description 3
- 238000013079 data visualisation Methods 0.000 claims description 3
- 230000004907 flux Effects 0.000 claims description 3
- 238000005194 fractionation Methods 0.000 claims description 3
- 238000005201 scrubbing Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 241000132092 Aster Species 0.000 description 1
- 241001269238 Data Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000009975 flexible effect Effects 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Business, Economics & Management (AREA)
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Economics (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention proposes a kind of large public building energy consumption data storage analysis method based on Hadoop framework, it is easy to operate, effectively improve the utilization rate of resource, it can be realized the distributed storage of building energy consumption big data, energy consumption data can be analyzed based on parallel computation with data analysis layer simultaneously, there is stronger retractility and stability.
Description
Technical field
The invention belongs to big datas to store analysis field, especially a kind of large public building energy based on Hadoop framework
It consumes data and stores analysis method.
Background technique
At present for big data processing technique include parallel database, MapReduce technology, parallel database and
MapReduce technology mixed architecture, wherein for big data be uniformly processed technology belong to third class parallel database and
MapReduce technology mixed architecture, the technology be divided into parallel database leading type, MapReduce leading type, parallel database and
MapReduce integrated-type.Parallel database leading type is the data processing function for enhancing parallel database using MapReduce
Can, such as the Greenplum of EMC, Aster Data, but its scalability and fault-tolerant ability and have not been changed；MapReduce leading type
It is using SQL (Structure Query Language, the structured query language) interface of relational database and to mode
Support to improve the ease for use of MapReduce, such as Hive, Pig Latin, but in terms of its real-time for data processing still without
Method meet demand；Parallel database and MapReduce integrated-type are that preferable fault-tolerance and right is obtained by Hadoop frame
The support of isomerous environment, while the performance advantage of relevant database is obtained, but application case is had no at present, it traces it to its cause
In work can not being pushed to suitable enforcement engine.
To sum up, in existing big data memory technology, parallel database leading type scalability and fault-tolerant ability are bad；
Demand is still unable to satisfy in terms of the real-time of MapReduce leading type data processing；Parallel database and MapReduce integrated-type
Work can not be pushed to suitable enforcement engine.
Summary of the invention
Technical problem solved by the invention is to provide a kind of large public building energy consumption number based on Hadoop framework
According to storage analysis method, the distributed storage of building energy consumption big data is realized, there is stronger retractility and stability.
The technical solution for realizing the aim of the invention is as follows:
Large public building energy consumption data based on Hadoop framework stores analysis method, comprising the following steps:
Step 1: in data collection layer, the energy consumption being distributed in building building acquires equipment for collected energy consumption data
Summarize and is sent to each acquisition terminal；
Step 2: data backup is stored in locally one week by each acquisition terminal by embedded memory technology, then passes through net
Network technology is completed to building on time to remote server transmission data packet in real time with the initial acquisition of energy situation；
Step 3: acquisition data being input to general data accumulation layer, carry out data scrubbing, conversion, merger and fractionation, structure
Build common architectural energy consumption data library；
Step 4: common architectural energy consumption data library is distributed by the Hadoop for being deployed in Hadoop data storage layer bottom
File system constructs data warehouse, then constructs Hadoop general data accumulation layer with business demand for guiding；
Step 5: based on the storage layer building parallel computation of Hadoop general data and data analysis layer, parallel computation and data
Process layer includes parallel computational model MapReduce, distributed computing framework Spark, Tool for Data Warehouse Hive, structuring number
According to processing module SparkSQL；
Step 6: application layer calls Hadoop general data accumulation layer and parallel computation and data processing according to user demand
Layer carries out data visualization, data mining and data analysis, user right authenticates, cluster operation monitors.
Further, large public building energy consumption data of the invention based on Hadoop stores analysis method, in step 1
The data of acquisition include: acquisition time, measuring point mark, the electric flux of ammeter, ammeter electric current etc..
Further, large public building energy consumption data of the invention based on Hadoop stores analysis method, in step 1
Acquisition terminal includes data collector, data gateway equipment.
Further, large public building energy consumption data of the invention based on Hadoop stores analysis method, in step 4
Data warehouse include system operational parameters library, subitem energy consumption data library, architectural environment parameter library and building information library.
The invention adopts the above technical scheme compared with prior art, has following technical effect that
1, the large public building energy consumption data storage analysis method of the invention based on Hadoop is easy to operate, improves money
The utilization rate in source；
2, the large public building energy consumption data storage analysis method of the invention based on Hadoop realizes that building energy consumption is big
The distributed storage of data, while energy consumption data can be analyzed based on parallel computation with data analysis layer；
3, the large public building energy consumption data storage analysis method of the invention based on Hadoop has stronger flexible
Property and stability.
Detailed description of the invention
Fig. 1 is the large public building energy consumption data storage analysis method flow chart of the invention based on Hadoop.
Specific embodiment
Embodiments of the present invention are described below in detail, the example of the embodiment is shown in the accompanying drawings, wherein from beginning
Same or similar element or element with the same or similar functions are indicated to same or similar label eventually.Below by ginseng
The embodiment for examining attached drawing description is exemplary, and for explaining only the invention, and is not construed as limiting the claims.
Large public building energy consumption data based on Hadoop framework stores analysis method, as shown in Figure 1, including following step
It is rapid:
Step 1: in data collection layer, the energy consumption being distributed in building building acquires equipment for collected energy consumption data
Summarize and is sent to each acquisition terminal.Wherein, energy consumption acquisition equipment includes that ammeter, water meter, gas meter, flow meter and Hygrothermograph scale etc. are each
Class intelligent gauge.Energy consumption data includes: acquisition time, measuring point mark, the electric flux of ammeter, ammeter electric current etc..Acquisition terminal packet
Include data collector, data gateway equipment.
Step 2: data backup is stored in locally one week by each acquisition terminal by embedded memory technology, then passes through net
Network technology is completed to building on time to remote server transmission data packet in real time with the initial acquisition of energy situation.
Step 3: acquisition data being input to general data accumulation layer, carry out data scrubbing, conversion, merger and fractionation, structure
Build common architectural energy consumption data library.General data accumulation layer is deployed in provincial energy consumption data center, the predominantly existing energy at different levels
Management system provides support, and the immediate data source as entire Hadoop platform, common store layer by sensor network with it is whole
A Hadoop platform is kept apart, and ensure that the operation of existing system, at the same improve Hadoop data source the quality of data and
Collecting efficiency.
Step 4: common architectural energy consumption data library is distributed by the Hadoop for being deployed in Hadoop data storage layer bottom
File system (HDFS) constructs data warehouse, and HDFS can reliably store the file of magnanimity across machine, by the storage of each file at
An equal amount of sequence of blocks of data, data warehouse include system operational parameters library, subitem energy consumption data library, architectural environment parameter library
With building information library, then with business demand be guiding building Hadoop general data accumulation layer, for run parallel computation frame
Support is provided with building energy consumption model.
Step 5: based on the storage layer building parallel computation of Hadoop general data and data analysis layer, parallel computation and data
Process layer includes parallel computational model MapReduce, distributed computing framework Spark, Tool for Data Warehouse Hive, structuring number
According to processing module SparkSQL.According to the difference of analysis task, appropriate component is selected to carry out data processing.
Step 6: application layer directly provides service for user, possesses unified system image conversion interface, according to user demand
Hadoop general data accumulation layer and parallel computation and data analysis layer are called, data visualization, data mining and data are carried out
Analysis, user right certification, cluster operation monitoring.
This method can be realized the distributed storage of building energy consumption big data, have stronger retractility and stability, together
When energy consumption data can be analyzed with data analysis layer based on parallel computation.
The above is only some embodiments of the invention, it is noted that for the ordinary skill people of the art
For member, without departing from the principle of the present invention, several improvement can also be made, these improvement should be regarded as guarantor of the invention
Protect range.
Claims (4)
1. the large public building energy consumption data based on Hadoop framework stores analysis method, which is characterized in that including following step
It is rapid:
Step 1: in data collection layer, the energy consumption acquisition equipment being distributed in building building summarizes collected energy consumption data
It is sent to each acquisition terminal；
Step 2: data backup is stored in locally one week by each acquisition terminal by embedded memory technology, then passes through network skill
Art is completed to building on time to remote server transmission data packet in real time with the initial acquisition of energy situation；
Step 3: acquisition data being input to general data accumulation layer, carry out data scrubbing, conversion, merger and fractionation, building is logical
With building energy consumption database；
Step 4: common architectural energy consumption data library is by being deployed in the Hadoop distributed document of Hadoop data storage layer bottom
System constructs data warehouse, then constructs Hadoop general data accumulation layer with business demand for guiding；
Step 5: based on the storage layer building parallel computation of Hadoop general data and data analysis layer, parallel computation and data processing
Layer include parallel computational model MapReduce, distributed computing framework Spark, Tool for Data Warehouse Hive, at structural data
Manage module SparkSQL；
Step 6: application layer calls Hadoop general data accumulation layer and parallel computation and data analysis layer according to user demand, into
Row data visualization, data mining and data analysis, user right certification, cluster operation monitor.
2. the large public building energy consumption data according to claim 1 based on Hadoop stores analysis method, feature
It is, the data acquired in step 1 include: acquisition time, measuring point mark, the electric flux of ammeter, ammeter electric current etc..
3. the large public building energy consumption data according to claim 1 based on Hadoop stores analysis method, feature
It is, acquisition terminal includes data collector, data gateway equipment in step 1.
4. the large public building energy consumption data according to claim 1 based on Hadoop stores analysis method, feature
It is, the data warehouse in step 4 includes system operational parameters library, subitem energy consumption data library, architectural environment parameter library and building
Object information bank.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810866259.2A CN109165203A (en) | 2018-08-01 | 2018-08-01 | Large public building energy consumption data based on Hadoop framework stores analysis method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810866259.2A CN109165203A (en) | 2018-08-01 | 2018-08-01 | Large public building energy consumption data based on Hadoop framework stores analysis method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109165203A true CN109165203A (en) | 2019-01-08 |
Family
ID=64898627
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810866259.2A Pending CN109165203A (en) | 2018-08-01 | 2018-08-01 | Large public building energy consumption data based on Hadoop framework stores analysis method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109165203A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112364081A (en) * | 2020-11-16 | 2021-02-12 | 广东百德朗科技有限公司 | Public building energy consumption data acquisition and analysis method and system based on big data |
| CN112686771A (en) * | 2020-12-07 | 2021-04-20 | 国网新疆电力有限公司 | Cloud-edge cooperative electric power metering data acquisition and analysis system and method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104699985A (en) * | 2015-03-26 | 2015-06-10 | 西安电子科技大学 | Medical big-data acquisition and analysis system and method |
| CN106951497A (en) * | 2017-03-15 | 2017-07-14 | 深圳市德信软件有限公司 | A kind of method and system based on Hadoop framework data analysis diagrammatic representation |
| CN107424079A (en) * | 2017-06-29 | 2017-12-01 | 广州智慧城市发展研究院 | A kind of heavy construction energy consumption management system and method based on big data platform |
-
2018
- 2018-08-01 CN CN201810866259.2A patent/CN109165203A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104699985A (en) * | 2015-03-26 | 2015-06-10 | 西安电子科技大学 | Medical big-data acquisition and analysis system and method |
| CN106951497A (en) * | 2017-03-15 | 2017-07-14 | 深圳市德信软件有限公司 | A kind of method and system based on Hadoop framework data analysis diagrammatic representation |
| CN107424079A (en) * | 2017-06-29 | 2017-12-01 | 广州智慧城市发展研究院 | A kind of heavy construction energy consumption management system and method based on big data platform |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112364081A (en) * | 2020-11-16 | 2021-02-12 | 广东百德朗科技有限公司 | Public building energy consumption data acquisition and analysis method and system based on big data |
| CN112686771A (en) * | 2020-12-07 | 2021-04-20 | 国网新疆电力有限公司 | Cloud-edge cooperative electric power metering data acquisition and analysis system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105069703B (en) | A kind of electrical network mass data management method | |
| CN106651633B (en) | Power utilization information acquisition system based on big data technology and acquisition method thereof | |
| CN104268695B (en) | A kind of multicenter basin water environment distributed type assemblies management system and method | |
| CN108446293A (en) | A method of based on urban multi-source isomeric data structure city portrait | |
| CN111586091B (en) | Edge computing gateway system for realizing computing power assembly | |
| CN111400326A (en) | Smart city data management system and method thereof | |
| CN104820670A (en) | Method for acquiring and storing big data of power information | |
| CN106547882A (en) | A kind of real-time processing method and system of big data of marketing in intelligent grid | |
| CN105069025A (en) | Intelligent aggregation visualization and management and control system for big data | |
| CN106126601A (en) | A kind of social security distributed preprocess method of big data and system | |
| CN106293941A (en) | A kind of distribution method of Database Systems resource | |
| CN109344207B (en) | Big data platform of integrative frequency spectrum all over the sky based on big dipper scanning | |
| CN106202566A (en) | A kind of magnanimity electricity consumption data mixing based on big data storage system and method | |
| CN112181960A (en) | Intelligent operation and maintenance framework system based on AIOps | |
| CN104899258A (en) | Interactive visualized analysis system structure facing massive document information | |
| CN106534784A (en) | Acquisition analysis storage statistical system for video analysis data result set | |
| CN104113605A (en) | Enterprise cloud application development monitoring processing method | |
| CN112559634A (en) | Big data management system based on computer cloud computing | |
| CN108228683A (en) | A kind of distributed intelligence electric network data analysis platform based on cloud computing | |
| CN112883001A (en) | Data processing method, device and medium based on marketing and distribution through data visualization platform | |
| CN108268569A (en) | The acquisition of water resource monitoring data and analysis system and method based on big data technology | |
| CN104298669A (en) | Person geographic information mining model based on social network | |
| CN109165203A (en) | Large public building energy consumption data based on Hadoop framework stores analysis method | |
| Ali et al. | From big data to smart data-centric software architectures for city analytics: the case of the pell smart city platform | |
| CN104991849A (en) | Method for monitoring system resource occupation of Linux process through zabbix |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190108 |