CN104239447A - Power-grid big time series data storage method - Google Patents
Power-grid big time series data storage method Download PDFInfo
- Publication number
- CN104239447A CN104239447A CN201410441649.7A CN201410441649A CN104239447A CN 104239447 A CN104239447 A CN 104239447A CN 201410441649 A CN201410441649 A CN 201410441649A CN 104239447 A CN104239447 A CN 104239447A
- Authority
- CN
- China
- Prior art keywords
- measuring point
- measuring object
- measuring
- label
- data storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/221—Column-oriented storage; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a power-grid big time series data storage method. The information of a batch of measurement points having location relevance to each other in service logic is re-described by selecting an open-source distributed column-oriented database HBase as a storage layer, and combining an SG-CIM model in power grid service; a reasonable index organization mode of a measurement point data storage table is designed, and the physical storage positions of the historical data of the batch of measurement points having location relevance to each other in service logic are adjacent to each other by use of the partitioning and load balancing functions of the HBase, so that the disk seek time can be reduced for querying the historical data of the batch of measurement points, and the query efficiency can be improved; instant query service can be provided for service application.
Description
Technical field
The large data of time series that the present invention relates to location information sensitivity in a kind of electrical network business store, immediate inquiring method, belong to the storage of large data, distributing real-time data bank field.
Background technology
Along with intelligent, informationalized development, the large data management system of sequential, as the important foundation data platform of Large Scale Process industrial enterprise production information, faces increasing challenge.For electric system, be the immense pressure of ultra-large data processing on the one hand.During State Grid Corporation of China SG-ERP builds, magnanimity history/real time data releasing platform construction deepens constantly, its data scale is increasing, wherein measuring point (data collection point) scale is estimated to reach necessarily even more than one hundred million scale, and data storage capacity arrives more than PB byte.High-speed real-time process on the other hand.For WAMS system, need per second number of transactions to be processed can reach 10,000,000, and conventional relational database cannot tackle the challenge of high-speed real-time process like this at all, has higher requirement to the treatment scale of real-time data base, processing speed.
Tradition real-time data base is limited by its traditional software architecture, in data scale, processing power, parallel computation, load balancing, dynamically autonomy etc., cannot meet practical application request.In real-time data base field, introduce large data processing technique and solve the main method that the problems referred to above are the large data storages of current research sequential.But, in the business scenarios such as example power grid accident inverting, electrical network service alarm analysis and power transmission and transforming equipment on-line monitoring, the historical data of a collection of often measuring point that user is concerned about, and this batch of measuring point has position correlation in service logic, ideally these historical datas with a collection of measuring point of position correlation in physical store also should holding position adjacent, the seek time of disk could be reduced when inquiring about in above-mentioned business scenario application like this, improve inquiry velocity, for service application provides immediate inquiring service.Traditional real-time data base due to its architecture design, realize the reasons such as principle, do not accomplish that the historical data of a collection of measuring point service logic with position correlation is also that position is adjacent in physical store.
Through preliminary search, find no the patent entry relevant to content of the present invention temporarily.
Summary of the invention
In order to solve the problem, ensure that historical data service logic with a collection of measuring point of position correlation is that position is adjacent in physical store, for in electrical network service application, the query demand of this batch of measuring point provides immediate inquiring service, the invention provides the large date storage method of a kind of electrical network sequential, its main thought is: select to increase income distributed columnar database HBase as accumulation layer, in conjunction with SG-CIM model in electrical network business, description is re-started to a collection of measuring point information service logic with position correlation, by designing a kind of index organization's mode of measuring point data storage list, utilize subregion and the load-balancing function of HBase, the position of historical data in physical store making a collection of measuring point service logic with position correlation is adjacent.The present invention specifically comprises the steps:
(1) the Business Logic measuring point based on SG-CIM model describes
Based on SG-CIM model, description is re-started to a collection of measuring point service logic with position correlation, by the association between measuring object and measuring point, form the hierarchical relationship of measuring point; In this description, SG-CIM model class is similar to a tree falling to grow, and leaf node is measuring point, and the hierachy number between non-leaf nodes is increased according to actual needs by user or reduces; Father node from the root node of this tree to leaf node the path of process for describing measuring object; Leaf node is for representing the measurement item of this measuring object, and namely the combination of measuring object and measurement item is equivalent to the measuring point in traditional real-time data base; Article one, data record is made up of measuring object mark, timestamp, measured value, label, and label is described by one or more key-value pair.
(2) index organization of measuring point data storage list is set up
Selection increases income distributed columnar database HBase as accumulation layer, and index organization's mode of measuring point data storage list directly has influence on query performance.
The indexing model of batch query measuring point data storage list is designed to: measuring object mark+reference time+label, and wherein measuring object mark adopts based on SG-CIM model the redescribing of a collection of measuring point logic business with position correlation; Reference time selected is that to decide this reference time according to the data acquiring frequency of measuring object to be stored be whole hour/all day; Row are with the side-play amount of the timestamp of image data record relative to reference time.
The indexing model of section inquiry measuring point data storage list is designed to: timestamp+measuring object mark+label, wherein timestamp is the timestamp collecting every bar data record.
Index due to HBase table sorts by lexicographic order, then adopt in this programme based on SG-CIM model to the description of a collection of measuring point service logic with position correlation, the position of the index that the data of this batch of measuring point can be made to be recorded in measuring point data storage list is adjacent.
Because HBase carries out cutting for different region is to complete subregion with the scope of row to storage list, simultaneously in conjunction with the automatic load balancing of HBase self in units of region, the historgraphic data recording that this batch of measuring point can be made to collect also is adjacent on actual physical storage position, time like this for the batch query of this batch of measuring point and section inquiry, the seek time of disk can be reduced, for service application provides immediate inquiring service.
Further, the label in step (1) only can have a measurement item describing this measuring object, but can have the attribute description information of multiple non-measured item.
By adopting technique scheme, the inventive method can ensure that the historical data of a collection of measuring point service logic with position correlation is also adjacent in physical storage locations in the large data of electrical network sequential store, disk seek time can be reduced when like this historical data of this batch of measuring point being inquired about, improve search efficiency, for service application provides immediate inquiring service.
Accompanying drawing explanation
The description schematic diagram of Fig. 1 to be the embodiment of the present invention based on SG-CIM model to community be ammeter in service logic unit.
Fig. 2 is sortord and the subregion schematic diagram of embodiment of the present invention measuring point data storage list index.
Embodiment
Below in conjunction with drawings and Examples, the present invention is described in further detail.
The present embodiment is to be described the power information collection of certain community.Suppose that cell name is lvsejiayuan, this community have A, B, C ... Y totally 25, there are 1,2,3,4,5,6,7,8 totally 8 unit in Mei Dong building, every unit has 01,02 ..., 15 floors, each floor has 01,02 ..., 19,20 resident families.Suppose that there is one piece of ammeter (measuring object) in a resident resident family by forward active energy (direction=fd), reverse active energy (direction=bd), forward is gained merit sharp electric flux (direction=fd type=shark), forward is gained merit peak electric flux (direction=fd type=peak), forward is gained merit ordinary telegram energy (direction=fd type=shoulder), forward is gained merit paddy electric flux (direction=fd type=offpeak), oppositely meritorious sharp electric energy (direction=bd type=shark), oppositely meritorious peak electric flux (direction=bd type=peak), oppositely meritorious ordinary telegram energy (direction=bd type=shoulder), oppositely meritorious paddy electric flux (direction=bd type=offpeak) totally 10 measurement items.
(1) description is re-started to measuring object title
As shown in Figure 1, according to SG-CIM model, description is re-started to a collection of measuring point service logic with position correlation, the all ammeters referred in this community are gathered in example in this power information, service logic belongs to this community, in conjunction with service logic, all ammeters in this community are redescribed, the hierarchical relationship set up measuring object and measure between item, ammeter as certain family resident family is designated: community. building. and unit. ammeter is numbered, and measures the forward active energy that item is ammeter.According to the description of this programme, following form can be described as to the ammeter (measuring object) in community:
lvsejiayuan.A.unit1.0101
lvsejiayuan.A.unit1.0102
lvsejiayuan.A.unit1.0103
……
lvsejiayuan.Y.unit8.1517
lvsejiayuan.Y.unit8.1518
lvsejiayuan.Y.unit8.1519
lvsejiayuan.Y.unit8.1520
(2) history data store table index is set up
The sortord of measuring point data storage list index and subregion are as shown in Figure 2.Suppose to collect partial data sometime as follows:
lvsejiayuan.A.unit1.0101140220180223303direction＝fd?type＝shark
lvsejiayuan.A.unit1.010114022018021751direction＝fd?type＝peak
lvsejiayuan.A.unit1.0101140220180220858direction＝fd?type＝shoulder
lvsejiayuan.A.unit1.0101140220180228723direction＝fd?type＝offpeak
lvsejiayuan.A.unit1.0101140220180214444direction＝fd
lvsejiayuan.A.unit1.0101140220180212166direction＝bd?type＝shark
lvsejiayuan.A.unit1.0101140220180230560direction＝bd?type＝peak
lvsejiayuan.A.unit1.010114022018022254direction＝bd?type＝shoulder
lvsejiayuan.A.unit1.0101140220180229230direction＝bd?type＝offpeak
lvsejiayuan.A.unit1.0101140220180227249direction＝bd
……
A. the index that the above-mentioned data collected are concrete in batch query measuring point data storage list is for being respectively:
lvsejiayuan.A.unit1.01011402200000direction＝bd
lvsejiayuan.A.unit1.01011402200000direction＝bd?type＝offpeak
lvsejiayuan.A.unit1.01011402200000direction＝bd?type＝peak
lvsejiayuan.A.unit1.01011402200000direction＝bd?type＝shark
lvsejiayuan.A.unit1.01011402200000direction＝bd?type＝shoulder
lvsejiayuan.A.unit1.01011402200000direction＝fd
lvsejiayuan.A.unit1.01011402200000direction＝fd?type＝offpeak
lvsejiayuan.A.unit1.01011402200000direction＝fd?type＝peak
lvsejiayuan.A.unit1.01011402200000direction＝fd?type＝shark
lvsejiayuan.A.unit1.01011402200000direction＝fd?type＝shoulder
Under in batch query measuring point data storage list, every bar data record is stored in the row that this recording indexes is expert at, this is classified as the side-play amount of this data record relative to its index reference time.
B. the above-mentioned data collected concrete index in section inquiry measuring point data storage list is respectively:
1402201802lvsejiayuan.A.unit1.0101direction＝bd
1402201802lvsejiayuan.A.unit1.0101direction＝bdtype＝offpeak
1402201802lvsejiayuan.A.unit1.0101direction＝bdtype＝peak
1402201802lvsejiayuan.A.unit1.0101direction＝bdtype＝shark
1402201802lvsejiayuan.A.unit1.0101direction＝bdtype＝shoulder
1402201802lvsejiayuan.A.unit1.0101direction＝fd
1402201802lvsejiayuan.A.unit1.0101direction＝fdtype＝offpeak
1402201802lvsejiayuan.A.unit1.0101direction＝fdtype＝peak
1402201802lvsejiayuan.A.unit1.0101direction＝fdtype＝shark
1402201802lvsejiayuan.A.unit1.0101direction＝fdtype＝shoulder
Under in section inquiry measuring point data storage list, every bar data record storage can simply directly be stored in the row of the row at this recording indexes place, also the number that a field shows this row store data record can be increased in the index of this table, first Hash (hash) is carried out to the measuring object in every bar data record, then the line number delivery that new field in the index represents is stored, under finally this data record value being stored in row corresponding to gained modulus to every bar record.
Technical characteristic involved in above-mentioned embodiment, just can combine mutually as long as do not form conflict to each other.The invention is not restricted to above-described embodiment, all technical schemes adopting equivalent replacement or equivalence replacement to be formed all belong to the scope of protection of present invention.
Claims (2)
1. the large date storage method of electrical network sequential, is characterized in that, comprise the steps:
(1) the Business Logic measuring point based on SG-CIM model describes
Based on SG-CIM model, description is re-started to a collection of measuring point service logic with position correlation, by the association between measuring object and measuring point, form the hierarchical relationship of measuring point; In this description, SG-CIM model is tree, and wherein leaf node is measuring point, and the hierachy number between non-leaf nodes is increased according to actual needs by user or reduces; Father node from the root node of this tree to leaf node the path of process for describing measuring object; Leaf node is for representing the measurement item of this measuring object; Article one, data record is made up of measuring object mark, timestamp, measured value, label, and wherein label is described by key-value pair;
(2) index organization of measuring point data storage list is set up
Selection increases income distributed columnar database HBase as accumulation layer;
The indexing model of batch query measuring point data storage list is designed to: measuring object mark+reference time+label, and wherein measuring object mark adopts based on SG-CIM model the redescribing of a collection of measuring point logic business with position correlation; Reference time selected is that to decide this reference time according to the data acquiring frequency of measuring object to be stored be whole hour/all day; Row are with the side-play amount of the timestamp of image data record relative to reference time;
The indexing model of section inquiry measuring point data storage list is designed to: timestamp+measuring object mark+label, wherein timestamp is the timestamp collecting every bar data record.
2. method according to claim 1, is characterized in that the label in described step (1) is described by key-value pair, and this label only has a measurement item describing described measuring object.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410441649.7A CN104239447A (en) | 2014-09-01 | 2014-09-01 | Power-grid big time series data storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410441649.7A CN104239447A (en) | 2014-09-01 | 2014-09-01 | Power-grid big time series data storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104239447A true CN104239447A (en) | 2014-12-24 |
Family
ID=52227506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410441649.7A Pending CN104239447A (en) | 2014-09-01 | 2014-09-01 | Power-grid big time series data storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104239447A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105574204A (en) * | 2016-01-08 | 2016-05-11 | 国网冀北电力有限公司 | Searching method and system for distributed power grid regulation and operation data |
| CN106250414A (en) * | 2016-07-22 | 2016-12-21 | 北京赛博智通信息技术有限责任公司 | A kind of data-storage system based on measuring point object and method |
| CN106250381A (en) * | 2015-06-04 | 2016-12-21 | 微软技术许可有限责任公司 | The row sequence optimized for input/output in list data |
| CN109613312A (en) * | 2018-12-18 | 2019-04-12 | 宁波三星智能电气有限公司 | A kind of ammeter exchange method based on menu |
| CN112015733A (en) * | 2020-08-04 | 2020-12-01 | 国家电网有限公司客户服务中心 | Method for storing and rapidly inquiring mass data of electric power customer service operation and distribution service |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1283486A2 (en) * | 2001-08-06 | 2003-02-12 | Addamark Technologies, Inc. | Storage of row-column data |
| CN101930481A (en) * | 2009-06-22 | 2010-12-29 | 华东电网有限公司 | Method used for generating CIM model describing power grid change in designated time slot and system thereof |
| CN102567495A (en) * | 2011-12-22 | 2012-07-11 | 国网信息通信有限公司 | Mass information storage system and implementation method |
| CN102855271A (en) * | 2012-07-05 | 2013-01-02 | 中国电力科学研究院 | Storage and traceable management method for multi-version power grid model |
| CN102890722A (en) * | 2012-10-25 | 2013-01-23 | 国家电网公司 | Indexing method applied to time sequence historical database |
| CN103116595A (en) * | 2012-10-24 | 2013-05-22 | 中国电力科学研究院 | Realizing method of supervisory control and data acquisition (SCADA) history data distribution-type storage facing power grid |
| WO2013165532A1 (en) * | 2012-05-04 | 2013-11-07 | Battelle Memorial Institute | Method and system for managing power grid data |
| CN103390045A (en) * | 2013-07-19 | 2013-11-13 | 北京京东尚科信息技术有限公司 | Time sequence storage method and time sequence storage device for monitoring system |
| CN103488727A (en) * | 2013-09-16 | 2014-01-01 | 河海大学 | Two-dimensional time-series data storage and query method based on periodic logs |
| CN103559160A (en) * | 2013-11-20 | 2014-02-05 | 国家电网公司 | SG-CIM-standard-based construction method for semantic information interaction interface of intelligent power distribution system |
-
2014
- 2014-09-01 CN CN201410441649.7A patent/CN104239447A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1283486A2 (en) * | 2001-08-06 | 2003-02-12 | Addamark Technologies, Inc. | Storage of row-column data |
| CN101930481A (en) * | 2009-06-22 | 2010-12-29 | 华东电网有限公司 | Method used for generating CIM model describing power grid change in designated time slot and system thereof |
| CN102567495A (en) * | 2011-12-22 | 2012-07-11 | 国网信息通信有限公司 | Mass information storage system and implementation method |
| WO2013165532A1 (en) * | 2012-05-04 | 2013-11-07 | Battelle Memorial Institute | Method and system for managing power grid data |
| CN102855271A (en) * | 2012-07-05 | 2013-01-02 | 中国电力科学研究院 | Storage and traceable management method for multi-version power grid model |
| CN103116595A (en) * | 2012-10-24 | 2013-05-22 | 中国电力科学研究院 | Realizing method of supervisory control and data acquisition (SCADA) history data distribution-type storage facing power grid |
| CN102890722A (en) * | 2012-10-25 | 2013-01-23 | 国家电网公司 | Indexing method applied to time sequence historical database |
| CN103390045A (en) * | 2013-07-19 | 2013-11-13 | 北京京东尚科信息技术有限公司 | Time sequence storage method and time sequence storage device for monitoring system |
| CN103488727A (en) * | 2013-09-16 | 2014-01-01 | 河海大学 | Two-dimensional time-series data storage and query method based on periodic logs |
| CN103559160A (en) * | 2013-11-20 | 2014-02-05 | 国家电网公司 | SG-CIM-standard-based construction method for semantic information interaction interface of intelligent power distribution system |
Non-Patent Citations (1)
| Title |
|---|
| 王喜珍: ""基于树形结构的地震数据库系统设计方法"", 《地震学报》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106250381A (en) * | 2015-06-04 | 2016-12-21 | 微软技术许可有限责任公司 | The row sequence optimized for input/output in list data |
| CN106250381B (en) * | 2015-06-04 | 2020-11-17 | 微软技术许可有限责任公司 | System and method for determining column layout of tabular storage |
| US11514011B2 (en) | 2015-06-04 | 2022-11-29 | Microsoft Technology Licensing, Llc | Column ordering for input/output optimization in tabular data |
| CN105574204A (en) * | 2016-01-08 | 2016-05-11 | 国网冀北电力有限公司 | Searching method and system for distributed power grid regulation and operation data |
| CN105574204B (en) * | 2016-01-08 | 2020-07-24 | 国网冀北电力有限公司 | Retrieval method and system for distributed power grid regulation and control operation data |
| CN106250414A (en) * | 2016-07-22 | 2016-12-21 | 北京赛博智通信息技术有限责任公司 | A kind of data-storage system based on measuring point object and method |
| CN106250414B (en) * | 2016-07-22 | 2020-01-03 | 北京赛博智通信息技术有限责任公司 | Data storage system and method based on measuring point object |
| CN109613312A (en) * | 2018-12-18 | 2019-04-12 | 宁波三星智能电气有限公司 | A kind of ammeter exchange method based on menu |
| CN112015733A (en) * | 2020-08-04 | 2020-12-01 | 国家电网有限公司客户服务中心 | Method for storing and rapidly inquiring mass data of electric power customer service operation and distribution service |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105354641B (en) | Goods picking path optimization method and goods picking path optimization device | |
| CN104239447A (en) | Power-grid big time series data storage method | |
| CN108255981B (en) | Method for storing and searching index minute freezing of timestamp sequence numbers of continuous time periods | |
| Wang et al. | Energy-efficient reverse skyline query processing over wireless sensor networks | |
| CN104317800A (en) | Hybrid storage system and method for mass intelligent power utilization data | |
| CN106648446A (en) | Time series data storage method and apparatus, and electronic device | |
| CN109582667A (en) | A kind of multiple database mixing storage method and system based on power regulation big data | |
| CN104182435B (en) | The information retrieval system and method marked based on shortage of data | |
| CN107784098A (en) | Real-time data warehouse platform | |
| CN102567497B (en) | Inquiring method of best matching with fuzzy trajectory problems | |
| CN103500173A (en) | Method for inquiring rail transit monitoring data | |
| CN106528787A (en) | Mass data multi-dimensional analysis-based query method and device | |
| US10133779B2 (en) | Query hint management for a database management system | |
| CN105022763B (en) | Realize the method and system of data query | |
| CN109772753B (en) | Power battery sorting system and method | |
| CN105099798A (en) | Online master station operation monitoring and evaluation method based on index system | |
| CN104268298A (en) | Method for creating database index and inquiring data | |
| CN104090934A (en) | Standard service platform distributed parallel computing database and retrieval method thereof | |
| CN104346458A (en) | Data storage method and device | |
| CN104933042B (en) | Database table optimization of collection technology based on big data quantity | |
| CN105989140A (en) | Data block processing method and equipment | |
| CN102508866B (en) | Digital addressing-based method for structured storage and rapid processing of command relation tree | |
| CN103198157A (en) | Compressed storage and processing method for telluric electricity field data | |
| CN107818106B (en) | Big data offline calculation data quality verification method and device | |
| CN105574074A (en) | Smart grid WAMS-based time-series big data storage method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20141224 |