CN110807032A - HBase-based smart grid acquisition monitoring data storage system and storage method - Google Patents
HBase-based smart grid acquisition monitoring data storage system and storage method Download PDFInfo
- Publication number
- CN110807032A CN110807032A CN201910957072.8A CN201910957072A CN110807032A CN 110807032 A CN110807032 A CN 110807032A CN 201910957072 A CN201910957072 A CN 201910957072A CN 110807032 A CN110807032 A CN 110807032A
- Authority
- CN
- China
- Prior art keywords
- data
- column
- monitoring data
- hbase
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/221—Column-oriented storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/06—Electricity, gas or water supply
Abstract
The invention provides an HBase-based smart power grid acquisition monitoring data storage system and a storage method, wherein a plurality of storage units are arranged, and the data format of data stored in each storage unit comprises a table name, a row key, a column family, a column name and an acquisition detection point sampling value; the row key is composed of equipment codes and data of types of collected monitoring points; the single column family consists of a single character "I"; the column names consist of time of day, minutes, seconds data. The invention transfers and stores partial information of the row keys and the column families, shortens the length of the row keys and the column families, simplifies unnecessary information, reduces redundancy, improves the space utilization rate and improves the loading and accessing performance of the collected monitoring data. Meanwhile, the distribution of row keys (RowKey) is reasonably and fully considered, and the loading performance of the collected monitoring data is improved by adopting a pre-partitioning mode.
Description
Technical Field
The application relates to the technical field of databases, in particular to a smart grid acquisition monitoring data storage system and a smart grid acquisition monitoring data storage method.
Background
The intelligent power grid collected monitoring data is very important data in power grid application, is an important component of four data types of the intelligent power grid, and is an important basis for intelligent power grid electricity utilization information collection, equipment state monitoring, auxiliary decision analysis, offline mining analysis and the like. At present, smart power grids gathers monitoring data storage's key feature: (1) the collected monitoring data volume is large, and the storage period span is long; (2) fixing a data format; (3) the data loading throughput requirement is high; (4) the access mode mainly comprises section query and batch query; (5) the scale of historical data is continuously increased, the richness and interactivity of applications are continuously enhanced, and the access requirement on the historical data is higher and higher.
The prior art scheme is as follows:
the existing storage model for collecting monitoring data of the smart power grid is based on an HBase database. HBase is a distributed, column-oriented open source database, and the technology is derived from the Google paper "Bigtable: a distributed storage system of structured data. Just as Bigtable takes advantage of the distributed data storage provided by the Google File System (File System), HBase provides Bigtable-like capabilities over Hadoop. The HBase is a high-reliability, high-performance, nematic and scalable distributed storage system, and a large-scale structured storage cluster can be built on the cheap PCServer by utilizing the HBase technology. The goal of HBase is to store and process large data, and more specifically, to process large data consisting of thousands of rows and columns using only a common hardware configuration. HBase is different from a general relational database, and is a database suitable for unstructured data storage. HBase utilizes Hadoop HDFS as a file storage system, utilizes Hadoop MapReduce to process mass data, and utilizes Zookeeper as a cooperative service.
The existing storage model for collecting monitoring DATA of the smart grid comprises a row key (RowKey), a column family (ColumnFamily), a column name (ColumnId), a storage structure for collecting sampling values of monitoring points (Value), wherein the structure of a table name is < BT _ PC _ MTTYPE _ yyyyMM >, the structure of the row key is < CC.TG.TIMETAMMP.MT >, the structure of the column family is < DATA >, and the structure of the column name is < MID >, wherein BT is a service type code, PC is a provincial code, MTTYPE is a device type code, yyyMM is a year and month, CC is a city code, TG is a platform area code, MT is a type of collecting monitoring points, MID is a device code, and Value is a sampling Value of collecting monitoring points.
The existing storage model information storage of the intelligent power grid collecting monitoring data based on HBase is redundant, stores more unnecessary information, does not perform reasonable pre-partitioning, is low in space utilization rate and resource utilization rate, cannot adapt to the situation that the scale of system data is continuously increased along with the loading and access performance of the collected monitoring data, and cannot meet the actual requirements of services along with the situation that the richness and interactivity of application are continuously enhanced.
Therefore, a new technical solution is needed to solve the above problems.
Disclosure of Invention
The purpose of the invention is as follows: the invention provides an HBase-based smart grid acquisition and monitoring data storage system and a storage method, and aims to reasonably store information and fully utilize resources to improve the loading and access performance of acquisition and monitoring data.
The technical scheme is as follows: in order to achieve the purpose, the intelligent power grid acquisition monitoring data storage system based on HBase adopts the following technical scheme:
an HBase-based smart power grid acquisition monitoring data storage system comprises a plurality of storage units, wherein the data format of data stored in each storage unit comprises a table name, a row key, a column family, a column name and an acquisition detection point sampling value; the row key is composed of equipment codes and data of types of collected monitoring points; the column family consists of a single character "I"; the column names consist of time of day, minutes, seconds data.
Further, the table name is composed of a service type code, a network province code, a device type code and year and month data.
Furthermore, the storage system is pre-partitioned into a plurality of storage units, and the loading performance of the collected monitoring data is improved.
The HBase-based intelligent power grid acquisition monitoring data storage method adopts the following technical scheme:
an HBase-based smart grid acquisition monitoring data storage method comprises the following steps:
(1) establishing a plurality of storage units, wherein the data format of the stored data in each storage unit is provided with a table name, a row key, a column family, a column name and a sampling value of a collection detection point;
(2) setting the row key to be composed of equipment codes and data of the type of the collected monitoring points; setting a column family to consist of a single character "I"; the column names are set to consist of time of day, minutes, seconds data.
Further, in the step (2), the table name is set to be composed of a service type code, a network province code, a device type code and year and month data.
Furthermore, the loading performance of the collected monitoring data is improved by presetting a plurality of storage units in a pre-partition mode.
Has the advantages that: compared with the prior art, the invention has the beneficial effects that:
1. the data information is reasonably separated, each storage field of HBase is fully utilized, and the loading performance of the collected monitoring data is improved.
2. And a clustering idea is adopted, and a storage unit is established according to the time scale and the service model, so that the access performance of the collected monitoring data is improved.
3. Partial information of the row keys and the column families is transferred and stored, the length of the row keys and the column families is shortened, unnecessary information is simplified, redundancy is reduced, the space utilization rate is improved, and the loading and accessing performance of the collected monitoring data is improved.
4. The distribution of row keys (RowKey) is reasonably and fully considered, and the loading performance of the collected monitoring data is improved by adopting a pre-partitioning mode.
Drawings
Fig. 1 is a schematic structural diagram of an HBase-based smart grid acquisition monitoring data storage system according to the present invention.
Detailed Description
Referring to fig. 1, an embodiment of the present invention provides an HBase-based smart grid collection monitoring data storage system, which includes a plurality of storage units, where a data format of data stored in each storage unit includes a table name (TableName), a row key (RowKey), a column family (columnamily), a column name (ColumnId), and a collection monitoring point sampling Value (Value); the row key is composed of equipment codes and data of types of collected monitoring points; the column family consists of a single character "I"; the column names consist of time of day, minutes, seconds data. The table name is composed of a service type code, a network province code, a device type code and year and month data.
The memory cell model is shown in the following table
The key points of the embodiment are that data information is reasonably separated, partial information of a row key and a column family is transferred and stored, the length of the row key and the column family is shortened, unnecessary information is simplified, redundancy is reduced, the space utilization rate is improved, each storage field of HBase is fully utilized, a clustering idea is adopted to establish a storage unit according to a time scale and a service model, and the loading and access performance of acquired and measured data is improved.
Furthermore, in order to improve the loading performance of data, the system adopts a pre-partitioning mode during storage, if pre-partitioning is not performed during table building, the table has only one region, when the size of one region exceeds a threshold, two regions can be automatically split, and split operation can bring resource consumption, meanwhile, when the number of the regions is too small, the loading pressure of data can be shared on one machine, so that great resource waste can be caused, and two problems can be caused under the conditions of large data volume, large access volume or batch processing program reading and writing: 1. there is a write hot spot problem and 2, performance may be poor. Therefore, the bear fully considers the distribution of row keys, adopts a pre-partitioning mode, avoids unnecessary split operation, can utilize the characteristic of HBase distribution, evenly shares the pressure of data loading to each machine, fully exerts the performance of a plurality of machines and improves the loading efficiency of data.
Implementation effects deployed on hardware:
the effect of the specific implementation mode of the system on hardware will be described below by taking an example that a certain provincial electricity consumption information acquisition system acquires monitoring data, and the new model in the invention is used as a storage model of a historical database HBase.
Server hardware configuration:
server software configuration:
| operating system | Database with a plurality of databases |
| CentOS 6.6 | HBase 1.2.0 |
According to the configuration of the parameters, the scale of the electricity consumption information acquisition measuring point of the region is about 3600 ten thousand, the data generation period is 15 minutes, and the data are divided into pieces by days. Through the storage model designed by the method, the data loading and accessing efficiency is as follows:
| type of operation | Efficiency (ten thousand/second) |
| Loading | 22 |
| Access | 170 (highest efficiency) |
As shown in the above table, the data loading and accessing efficiencies of the method are respectively: 22 ten thousand per second, 170 ten thousand per second (highest efficiency). Wherein, the unit "ten thousand/second" represents: how many tens of thousands of data values are loaded/accessed per second. In this practical scenario, the performance of data loading is improved by approximately thirty percent compared to the 17 ten thousand per second performance of the existing model.
Corresponding to the embodiment of the storage system, the embodiment of the HBase-based intelligent power grid acquisition monitoring data storage method provided by the invention adopts the following technical scheme:
an HBase-based smart grid acquisition monitoring data storage method comprises the following steps:
(1) establishing a plurality of storage units, wherein the data format of the stored data in each storage unit is provided with a table name, a row key, a column family, a column name and a sampling value of a collection detection point;
(2) setting the row key to be composed of equipment codes and data of the type of the collected monitoring points; setting a column family to consist of a single character "I"; setting column names to consist of time of day, minute and second data; the table name is set to be composed of a service type code, a network province code, a device type code and year and month data.
In the same way, in the storage method, the loading performance of the collected monitoring data is improved by presetting a plurality of storage units in a pre-partition mode.
In addition, the present invention has many specific implementations and ways, and the above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be construed as the protection scope of the present invention. All the components not specified in the present embodiment can be realized by the prior art.
Claims (6)
1. An HBase-based smart power grid acquisition monitoring data storage system is characterized by comprising a plurality of storage units, wherein the data format of data stored in each storage unit comprises a table name, a row key, a column family, a column name and an acquisition detection point sampling value; the row key is composed of equipment codes and data of types of collected monitoring points; the column family consists of a single character "I"; the column names consist of time of day, minutes, seconds data.
2. The smart grid collection monitoring data storage system of claim 1, wherein: the table name is composed of a service type code, a network province code, a device type code and year and month data.
3. The smart grid collection monitoring data storage system of claim 1 or 2, wherein: the storage system is pre-partitioned into a plurality of storage units, and the loading performance of the collected monitoring data is improved.
4. An HBase-based smart grid acquisition monitoring data storage method is characterized by comprising the following steps:
(1) establishing a plurality of storage units, wherein the data format of the stored data in each storage unit is provided with a table name, a row key, a column family, a column name and a sampling value of a collection detection point;
(2) setting the row key to be composed of equipment codes and data of the type of the collected monitoring points; setting a column family to consist of a single character "I"; the column names are set to consist of time of day, minutes, seconds data.
5. The smart grid collection monitoring data storage method according to claim 4, wherein: in the step (2), the table name is set to be composed of service type codes, network province codes, equipment type codes and year and month data.
6. The smart grid collection monitoring data storage method according to claim 4 or 5, wherein: the loading performance of the collected monitoring data is improved by presetting a plurality of storage units in a pre-partitioning mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910957072.8A CN110807032A (en) | 2019-10-10 | 2019-10-10 | HBase-based smart grid acquisition monitoring data storage system and storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910957072.8A CN110807032A (en) | 2019-10-10 | 2019-10-10 | HBase-based smart grid acquisition monitoring data storage system and storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110807032A true CN110807032A (en) | 2020-02-18 |
Family
ID=69488113
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910957072.8A Pending CN110807032A (en) | 2019-10-10 | 2019-10-10 | HBase-based smart grid acquisition monitoring data storage system and storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110807032A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140258296A1 (en) * | 2013-03-11 | 2014-09-11 | Dell Products L.P. | System and method for management of network monitoring information |
| CN106557561A (en) * | 2016-11-16 | 2017-04-05 | 贵州大学 | Magnanimity sensing data storage system and method based on HBase |
| CN106844556A (en) * | 2016-12-30 | 2017-06-13 | 江苏瑞中数据股份有限公司 | A kind of intelligent grid time scale measurement date storage method based on HBase |
-
2019
- 2019-10-10 CN CN201910957072.8A patent/CN110807032A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140258296A1 (en) * | 2013-03-11 | 2014-09-11 | Dell Products L.P. | System and method for management of network monitoring information |
| CN106557561A (en) * | 2016-11-16 | 2017-04-05 | 贵州大学 | Magnanimity sensing data storage system and method based on HBase |
| CN106844556A (en) * | 2016-12-30 | 2017-06-13 | 江苏瑞中数据股份有限公司 | A kind of intelligent grid time scale measurement date storage method based on HBase |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105608223A (en) | Hbase database entering method and system for kafka | |
| CN103268336A (en) | Fast data and big data combined data processing method and system | |
| CN103116595B (en) | Implementation method towards the SCADA historical data distributed storage of electrical network | |
| CN104317800A (en) | Hybrid storage system and method for mass intelligent power utilization data | |
| CN107766529B (en) | Mass data storage method for sewage treatment industry | |
| CN105912609A (en) | Data file processing method and device | |
| Ceci et al. | Big data techniques for supporting accurate predictions of energy production from renewable sources | |
| CN105912601A (en) | Partition storage method for distributed real-time memory database of energy management system | |
| CN109669975B (en) | Industrial big data processing system and method | |
| CN104572505A (en) | System and method for ensuring eventual consistency of mass data caches | |
| CN107656971A (en) | A kind of intelligent grid collection Monitoring Data storage method based on Redis | |
| CN104991741B (en) | A kind of situation adaptation power network big data storage method based on key-value model | |
| CN105574074A (en) | Smart grid WAMS-based time-series big data storage method | |
| CN106844556A (en) | A kind of intelligent grid time scale measurement date storage method based on HBase | |
| CN112131214A (en) | Method, system, equipment and storage medium for data writing and data query | |
| CN104035522A (en) | Large database appliance | |
| CN113849499A (en) | Data query method and device, storage medium and electronic device | |
| CN110674080B (en) | Method and system for collecting large-data-volume unstructured file based on NiFi | |
| CN103246695A (en) | Integrating method of HighSoon real-time database and IES600P system | |
| CN108920110A (en) | A kind of parallel processing big data storage system and method calculating mode based on memory | |
| CN110807032A (en) | HBase-based smart grid acquisition monitoring data storage system and storage method | |
| CN104714956A (en) | Comparison method and device for isomerism record sets | |
| CN106951442A (en) | Data interactive method and device between a kind of heterogeneous database | |
| CN116680315A (en) | Data offline processing method and device, electronic equipment and storage medium | |
| CN110825744A (en) | Air quality monitoring big data partition storage method based on cluster environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |