CN105321124A - Hadoop-based electric power cloud platform design scheme - Google Patents

Abstract

The present invention provides a Hadoop-based power cloud platform design scheme. By using HDFS and HBase, the efficiency of data storage and indexing is improved, and MapReduce is used for off-line analysis of data to efficiently process data. Rest standard Web? Service is convenient for uploading user data, and it also brings convenience to third-party development. This solution solves the shortcomings of low efficiency and poor performance in the traditional data processing process. In the case of traditional stand-alone data processing, the low efficiency solves the problem of data consistency, ensures the stable operation of the cluster, has high throughput, and improves A higher standard data interface is established to avoid a lot of cost caused by repeated development.

Description

A Hadoop-based power cloud platform design scheme

technical field

The invention relates to distributed computing technology and distributed storage technology, and belongs to the field of cloud computing, in particular to a Hadoop-based power cloud platform design scheme.

Background technique

With the increase of various electrical equipment, power data is becoming larger and more complex. Traditional data issuance and processing methods are difficult to handle TB or even PB-level data. Even if they can be processed correctly, the efficiency is very low. Secondly, data consistency and reliability cannot be guaranteed, and even with a complete system, the throughput of the system The amount is also limited, and there is no suitable interface that can provide third-party data access.

Cloud computing has been extensively studied not only in the academic field but also in the industrial field. Due to its large scale, virtualization, control between failed connection components, and asynchronous communication, it has unique advantages. Distributed system requirements and on-demand services, computing power, and storage resources are becoming increasingly pressing. MapReduce, MapReduce, GFS and BigTable proposed by Google, have solved the difficulties of traditional power data processing. MapReduce has always been the most popular distribution programming model in the cloud environment. Using MapReduce in cloud infrastructure, these datasets can be processed easily and efficiently.

A Hadoop-based power cloud platform design scheme, comprehensively using Apache Hadoop, flexibly using MapReduce, Hadoop Distributed File System (HDFS), HBase, etc., can efficiently process data and solve the problem of data consistency.

Contents of the invention

The purpose of the present invention is to provide a Hadoop-based power cloud platform design solution, which solves the shortcomings of low efficiency and poor performance in the traditional data processing process. In the case of a traditional single machine, the data processing process is inefficient. The consistency of data and the stability of the cluster are also difficult to guarantee the disadvantages of poor performance.

Technical solutions:

A Hadoop-based power cloud platform design scheme provided by the present invention mainly includes a plurality of modules: HDFS, MapReduce, HBase, WebService and website. DFS is used as a distributed file storage system for general storage of power data; at the same time, some data is convenient for query and stored in HBase; MapReduce can read data from HBase or HDFS for processing and write back; secondly, the website can also be accessed from HBase performs read and write operations. Both APP and power data can interact with HBase through WebService.

The specific operation process is as follows:

(1) Data upload:

The data of power sensors is uploaded to our cluster through WebService, and the data collected by users or power departments can also be directly uploaded through WebService.

(2) Data processing:

After the data is uploaded, some classification and statistical calculations will be performed by default, and some power consumption predictions will be performed later. These steps consist of multiple MapReduce programs. They are respectively responsible for some operations on the data, and will finally be written to HBase.

(3) Data display and management:

The display of data is carried out through the website, and at the same time, by calling WebService to develop an APP, both the website and the APP can display some recent power consumption, speculate on some similar power consumption behaviors, and recommend some relatively power-saving devices.

The specific functions of each module are as follows

(1) HDFS: In the cloud environment, the traditional file system can no longer meet the needs of users for data disaster recovery and data consistency. HDFS is a distributed file system running on general-purpose hardware. It has a lot in common with existing distributed file systems. But at the same time, it is a highly fault-tolerant system suitable for deployment on inexpensive machines. In this design scheme, HDFS is used as the basic file system to provide high-throughput data access, and it can interact with MapReduce, HBase, etc. for data.

(2) MapReduce: Due to the high frequency of power data transmission, the large amount of data, and the large amount of user data, the traditional processing program can no longer meet the demand. MapReduce is a programming model for parallel operations on large-scale data sets (greater than 1TB). In this solution, MapReduce is responsible for processing some data in HDFS and HBase, and writes the data back to HBase regularly. The power cloud platform mainly includes some programs based on MapReduce statistics, SVM predictive model programs and other programs. The specific processing operations are:

2-1) Statistics of the total power consumption of a single user;

2-2) Classification of user electricity consumption;

2-3) Prediction of user electricity consumption in the next 24 hours;

2-4) Classification of electrical equipment (according to power consumption);

2-5) Historical electricity consumption analysis of electrical equipment;

2-6) Some other statistical operations.

(3) HBase: HBase is a distributed, column-oriented open source database, an open source implementation of Google Bigtable, and a distributed storage system for structured data. HBase is a columnar storage database, which belongs to a type of NOSQL database. It is a database suitable for unstructured data storage. In this design scheme, HBase replaces the traditional database, and has functions such as disaster recovery and data consistency, etc., and can interact well with HDFS. The power data in HDFS can be freely handed over to HDFS for operation , HBase can also interact with some power data processing programs written based on MapReduce.

(4) Website and APP: The website is written based on JSP and J2EE. The APP includes Android version and IOS version, which are provided for users to access, graphically display some data, and provide some customized services, specifically including the following functions:

4-1) Monthly power consumption reminder;

4-2) Reminder mode setting;

4-3) Recommendations for power-saving electrical appliances;

4-4) Electrical aging prediction.

(5) WebService: A WebService that opens the Rest standard. As the source of our data, the data of the power sensor is uploaded to our cluster through the WebService, and then processed by multiple MapReduce programs. At the same time, WebService also supports the third user to directly upload a large amount of data, or directly obtain data from the power department through WebService. In addition to this, rest standard services support the following operations:

5-1)PUT

Corresponding to the cloud platform for uploading power data, the specific usage is as follows:

PUT http://aaa.com/user1/device1

5-2) DELETE

Corresponding to the deletion of power data, such as deleting the data of 20150101, the specific usage is as follows:

DELETE http://aaa.com/user1/device1/20150101

5-3) GET

Obtaining power data, such as the total amount of data in a certain day, such as obtaining the data of 20150101, the specific usage is as follows:

GET http://aaa.com/user1/device1/20150101

5-4) POST

You can modify the data in the cloud platform. If you want to modify the data of light1, the specific usage is as follows:

POST http://aaa.com/user1/device1/light1

Beneficial effect

The Hadoop-based power cloud platform design scheme can efficiently process data, solve the problem of data consistency, ensure the stable operation of the cluster, and have high throughput. At the same time, it improves the data interface of a higher standard and supports the third-party data connection. Income, to avoid a lot of costs caused by repeated development.

Description of drawings

Figure 1 is a structural diagram of a Hadoop-based power cloud platform design scheme.

Figure 2 is a data flow diagram realized by a Hadoop-based power cloud platform design scheme.

detailed description

The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. The present invention will be described below in combination with an actual deployment situation as an example.

The platform structure is shown in Figure 1: the design of the power cloud platform includes: HDFS, MapReduce, HBase, WebService and website. DFS is used as a distributed file storage system for general storage of power data; at the same time, some data is convenient for query and stored in HBase; MapReduce can read data from HBase or HDFS for processing and write back; secondly, the website can also be accessed from HBase performs read and write operations. Both APP and power data can interact with HBase through WebService.

Further, the cloud platform is composed of 1 master node, 1 second master node, and 20 slave nodes, and 15 slave nodes have 15 physical machines with a hard disk of 500GB and a memory of 4GB, and 5 virtual machines. Machine composition, the virtual machine is realized by the OracleVMVirtualBox software of Oracle Corporation. 2 sets are configured as 200GB hard disk and 4GB memory, and 3 sets are configured as . The three configurations are 500GB hard disk and 8GB memory.

(1) HDFS: In the cloud environment, the traditional file system can no longer meet the needs of users for data disaster recovery and data consistency. HDFS is a distributed file system running on general-purpose hardware. It has a lot in common with existing distributed file systems. But at the same time, it is a highly fault-tolerant system suitable for deployment on inexpensive machines. In this design scheme, HDFS is used as the basic file system to provide high-throughput data access, and it can interact with MapReduce, HBase, etc. for data. The number of HDFS backups for each node is set to 3, thus ensuring data integrity when a single node goes down.

(2) MapReduce: Due to the high frequency of power data transmission, the large amount of data, and the large amount of user data, the traditional processing program can no longer meet the demand. MapReduce is a programming model for parallel operations on large-scale data sets (greater than 1TB). In this solution, MapReduce is responsible for processing some data in HDFS and HBase, and writes the data back to HBase regularly. The power cloud platform mainly includes some programs based on MapReduce statistics, SVM predictive model programs and other programs. We configured MapReduce for each node with 1GB of memory and 1 processor. There will be 200 maptasks and 40 reducetasks when the power data starts processing, and the data will be written back to HBase after processing.

The specific processing operations are:

2-1) Statistics of the total power consumption of a single user;

2-2) Classification of user electricity consumption;

2-3) Prediction of user electricity consumption in the next 24 hours;

2-4) Classification of electrical equipment (according to power consumption);

2-5) Some other statistical operations.

(3) HBase: HBase is a distributed, column-oriented open source database, an open source implementation of Google Bigtable, and a distributed storage system for structured data. HBase is a columnar storage database, which belongs to a type of NOSQL database. It is a database suitable for unstructured data storage. In this design scheme, HBase replaces the traditional database, and has functions such as disaster recovery and data consistency, and can interact well with HDFS. The power data in HDFS can be freely handed over to HDFS for operation, and HBase also It can interact with some power data processing programs written based on MapReduce.

(4) Website and APP: The website is written based on JSP and J2EE. The APP includes Android version and IOS version, which are provided for users to access, graphically display some data, and provide some customized services, specifically including the following functions:

4-1) Monthly power consumption reminder;

4-2) Reminder mode setting;

4-3) Recommendations for power-saving electrical appliances;

4-4) Electrical aging prediction.

(5) WebService: A WebService that opens the Rest standard. As the source of our data, the data of the power sensor is uploaded to our cluster through the WebService, and then processed by multiple MapReduce programs. At the same time, WebService also supports the third user to directly upload a large amount of data, or directly obtain data from the power department through WebService. In addition, rest standard service supports the following operations

5-1)PUT

Corresponding to the cloud platform for uploading power data, the specific usage is as follows:

PUT http://aaa.com/user1/device1

5-2) DELETE

Corresponding to the deletion of power data, such as deleting the data of 20150101, the specific usage is as follows:

DELETE http://aaa.com/user1/device1/20150101

5-3) GET

Obtaining power data, such as the total amount of data in a certain day, such as obtaining the data of 20150101, the specific usage is as follows:

GET http://aaa.com/user1/device1/20150101

5-4) POST

You can modify the data in the cloud platform. If you want to modify the data of light1, the specific usage is as follows:

POST http://aaa.com/user1/device1/light1

The specific operation process is as follows:

(1) Data upload:

The data of power sensors is uploaded to our cluster through WebService, and the data collected by users or power departments can also be directly uploaded through WebService. Specifically, it is sent to the actual URL through the POST operation.

(2) Data processing:

After the data is uploaded, some classification and statistical calculations will be performed by default, and some power consumption predictions will be performed later. These steps consist of multiple MapReduce programs. They are respectively responsible for some operations on the data, and will finally be written to HBase.

(3) Data display and management:

The display of data is carried out through the website, and at the same time, by calling WebService to develop an APP, both the website and the APP can display some recent power consumption, speculate on some similar power consumption behaviors, and recommend some relatively power-saving devices. Secondly, users can directly operate their own device data through WebService, such as adding devices, etc. Once the device number is updated, the display data will be updated immediately.

Claims (1)

1. A design scheme of a Hadoop-based power cloud platform is characterized by comprising modules HDFS, MapReduce, HBase, WebService, websites and APP, and specifically comprising the following operation processes:

(1) and (3) data uploading:

the data of the power sensor is uploaded to the cluster through WebService, and the data collected by a user or a power department can also be directly uploaded through WebService;

(2) data processing:

after data are uploaded, classification and statistical operation can be carried out by default, and power consumption can be predicted in the later period, wherein the step is composed of a plurality of MapReduce programs; the data are respectively responsible for data operation and are written into HBase finally;

(3) and (3) displaying and managing data:

data display is carried out through a website, and meanwhile, through calling WebService to develop APP, the website and the APP can carry out display of recent electricity consumption, similar electricity consumption behavior user conjecture and recommend electricity-saving equipment.

Wherein,

the HDFS serves as a basic file system, provides high-throughput data access, and can interact with MapReduce, HBase and the like;

the MapReduce is responsible for processing data in the HDFS and the HBase and writing the data back to the HBase at regular time, and the specific processing operations include:

2-1) counting the total electricity consumption of a single user;

2-2) grading the electricity consumption of the user;

2-3) predicting the electricity consumption of the user in 24 hours in the future;

2-4) classifying the electrical equipment (according to the power consumption condition);

2-5) analyzing the historical electricity consumption of the electrical equipment;

2-6) some other statistical operations;

the HBase replaces a traditional database, has disaster recovery and data consistency functions at the same time, can interact with the HDFS, can freely deliver power data in the HDFS to the HDFS for operation, and can also interact with a power data processing program written based on MapReduce;

the website is written based on JSP and J2EE, APP includes android version and IOS version, provides user access, carries out graphical display of data, provides customized service simultaneously, and it contains specifically:

4-1) reminding the user of electricity consumption every month;

4-2) setting a reminding mode;

4-3) recommending the electric appliance model with electricity saving;

4-4) predicting the aging of the electric appliance;

the Service is used as a data source, the data of the power sensor are uploaded to the cluster through WebService, and then are processed by a plurality of MapReduce programs; meanwhile, WebService also supports a third user to directly upload large batch of data, or directly acquire data from the power department through WebService; service beyond this unexpected Rest standard supports the following operations:

5-1) PUT: uploading cloud platforms corresponding to the electric power data;

5-2) DELETE: deleting corresponding power data;

5-3) GET: acquiring power data;

5-4) POST: and modifying data in the cloud platform.