CN103593486A - Method for storing and reading mass data of power quality - Google Patents
Method for storing and reading mass data of power quality Download PDFInfo
- Publication number
- CN103593486A CN103593486A CN201310656313.8A CN201310656313A CN103593486A CN 103593486 A CN103593486 A CN 103593486A CN 201310656313 A CN201310656313 A CN 201310656313A CN 103593486 A CN103593486 A CN 103593486A
- Authority
- CN
- China
- Prior art keywords
- data
- quality
- power supply
- key
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a method for storing and reading mass data of power quality. The method includes the steps that firstly, the mass data of power quality are stored in a key value pair mode; secondly, key value pairs are stored in files in a serialization mode; thirdly, during real-time data read-in, a data area of a monitoring point is positioned according to the ID of equipment, and a corresponding storage area of monitoring data is positioned through an indicator; fourthly, corresponding data in a record slot are read according to an indicator category code of data to be acquired. Because each data file is split into data blocks to be stored through the storage mode of the key value pairs and a data block serves as the minimum unit as for data access in a database, system access speed is improved, and multiple-index and multiple-granularity characteristics and application requirements of the data of power quality are fully taken into consideration in terms of data file management and access modes.
Description
Technical field
The invention belongs to electric system, specifically relate to a kind of quality of power supply mass data storage and read method.
Background technology
Along with electric energy quality monitoring system development, monitoring point scale constantly expands, and Monitoring Data capacity is increasing, and existing relevant database cannot meet the increased requirement of quantity and concurrency.The real-time data base of main flow, by data memory-resident, can realize the fast access of mass data at present.For electric energy quality monitoring data characteristics, set up the data model and the data processing method that are applicable to real-time data base, will improve mass data processing and the high speed Concurrency Access ability of electric energy quality monitoring system.
Summary of the invention
For the deficiencies in the prior art, the invention provides a kind of quality of power supply mass data storage and read method, the present invention is applicable to power quality data storage and the read method of real time database management, power quality data storage mode based on key-value pair, be applicable to data file management and the memory management pattern of quality of power supply real-time data base, be applicable to magnanimity power quality data access management mode.
The object of the invention is to adopt following technical proposals to realize:
Mass data storage and a read method, its improvements are, described method comprises
(1) quality of power supply mass data is adopted to the storage of key-value pair form;
(2) key-value pair is stored hereof in the mode of serializing;
(3) real time data is write fashionablely, locates this data of monitoring point district, by the respective storage areas of pointer position monitor data according to device id;
(4) according to the index classification code that will obtain data, read the corresponding data in track.
Preferably, described step (1) comprises in key-value pair data, and the corresponding value of each key, using power quality monitoring device ID as key; In the data field of measured value value of being put into of this monitoring of equipment index.
Further, the measured value of described monitoring of equipment index comprises voltage deviation, frequency departure, three-phase imbalance, voltage fluctuation and flicker, harmonic wave and a harmonic wave.
Further, in described each value district, the measured value of monitoring is analyzed and encodes by index classification, by calculating side-play amount rapidly locating piece.
Preferably, described step (3) pointer comprises and take 200 milliseconds, 3 seconds, 1 minute and 10 minutes four memory blocks that are data granularity.
Preferably, described step (3) comprises that described real time data data writing is with time stamp, and the difference of the moment point of described time stamp and data field stem is less than 10 minutes, directly by the mistiming, is obtained the skew of data field, can be directly by data write memory, the converging operationJu Hecaozuo that log-on data makes progress simultaneously;
Described real time data data writing is with time stamp, and the difference of the moment point of described time stamp and data field stem is more than or equal to 10 minutes, starts background thread ，Jiang Ci segment data district writing in files.
Preferably, described step (4) comprises that the storing time intervals of data divided by 2 hours, certain the track position on the unit respective file head that each is divided out, and the data block in file is directly pointed in described groove position.
Further, described file is the form of catalogue per year, reads the file in a plurality of catalogues for the period of straddling over year.
Compared with the prior art, beneficial effect of the present invention is:
The present invention is directed to electric energy quality monitoring system, proposed data model, storage administration and mass data access mode based on key-value pair.
In the present invention, key-value pair storage mode cuts into data block by each data file and stores, and to data access in database, is all to using data block as least unit, has improved system access speed.
In the present invention, data file management and access mode have taken into full account the feature of power quality data many indexs, many granularities, and power quality data application demand.
The present invention is applied to electric energy quality monitoring system, can realize storage and the high speed Concurrency Access of large-scale data.
Accompanying drawing explanation
Fig. 1 is key-value pair storage mode schematic diagram in a kind of quality of power supply mass data storage provided by the invention and read method.
Fig. 2 is file way to manage schematic diagram in a kind of quality of power supply mass data storage provided by the invention and read method.
Fig. 3 is storage administration mode schematic diagram in a kind of quality of power supply mass data storage provided by the invention and read method.
Fig. 4 is quality of power supply historical data reading manner schematic diagram in a kind of quality of power supply mass data storage provided by the invention and read method.
Embodiment
Below in conjunction with accompanying drawing, the specific embodiment of the present invention is described in further detail.
1, as shown in Figure 1, the present invention adopts the storage of key-value pair (key-value) form by power quality data, utilizes key-value pair that each data of monitoring point is encapsulated as to index node, by the form of tree or list, organizes index node, improves data query speed.
Wherein, in key-value pair data, the corresponding value (value) of each key (key).Key is ID, using power quality monitoring device ID as key; The measured value of all monitoring indexes of this equipment (as, voltage deviation, frequency departure, three-phase imbalance, voltage fluctuation and flicker, harmonic wave, a harmonic wave etc.) all in the data field of the value of being put into.In Yi Gezhi district, Monitoring Data is analyzed and encodes by index classification, by calculating side-play amount rapidly locating piece.
Because the operation in monoblock region can only be carried out to value in the real-time storehouse of main flow, the data centralization of all indexs in monitoring point is put into a key-value pair, EMS memory occupation is relatively large, for fear of reading continually monoblock value region, in internal memory, set up the buffer zone of currency, when only district on duty is updated to afterbody, just start the write operation to database.
2, key-value pair is stored hereof in the mode of serializing.Utilize internal memory and universal document system, by calculating fast the method for side-play amount, meet application demand.
As shown in Figure 2, file management mode, comprises file directory, file structure, data structure.File is organization directory per year, and the annual data in each monitoring point are stored according to data granularity classification, and data granularity can be divided into 200 milliseconds, 3 seconds, 1 minute, 10 minutes and 2 hours, and specific design is according to practical situations.The inside of each file, arranged 1 annual data for unit divides ，Liang hour data district by 2 hours according to time sequencing.In two hour data districts, comprise this data field type codes (200 milliseconds, 3 seconds, 1 minute, 10 minutes or 2 hours), data length and each data constantly.Each data constantly, comprises the measured value of each power quality index and the quality of value.
As shown in Figure 3, memory management adopts binary tree mode to manage, and by the lock mechanism of operating system, controls the Concurrency Access to this binary tree, by Thread Pool Technology, completes the operation that in tree, data are write to file.In binary tree, in each node, usining device id as key assignments, is that 200 milliseconds, 3 seconds, 1 minute and 10 minutes granularity datas are divided independently storage area.
3, the data in internal memory tree data structure storage current ten minutes, using the ID of monitoring equipment as key assignments, by pointer, locate the data field of this monitoring point, comprise four pointers, point to respectively and take 200 milliseconds, 3 seconds, 1 minute and 10 minutes four memory blocks that are data granularity.
Wherein, when real time data, write fashionablely, first find the corresponding memory node of monitoring equipment, then find corresponding data field according to data granularity.The data that real time data writes are with time stamp, if the difference of the moment point of this time stamp and data field stem is less than 10 minutes, directly by the mistiming, obtained the skew of data field, can be directly by data write memory, and the converging operationJu Hecaozuo that log-on data makes progress simultaneously.
For example: when writing the data in 15 200ms cycles, start converging operationJu Hecaozuo, generate the data in 3 second cycle; Similarly, then by 3 number of seconds according to cycle polymerization in 1 minute, the like.
Wherein, when real time data, write fashionablely, the data that real time data writes are with time stamp, if the difference of the moment point of this time stamp and data field stem is more than or equal to 10 minutes, start background thread ，Jiang Ci segment data district writing in files.200 milliseconds, 3 seconds, 1 minute, 10 minutes and the storage of 2 hour data file sectional lists, for ease of directly writing data in these 5 file pointers, deposit in the pointer of opening after these 5 files in the memory node of this monitoring equipment.File header be take 2 hours each tracks that are unit for 1 year for recording, and in groove, numerical value is for recording the skew of this unit data field constantly.Find corresponding groove position according to the start time of this batch data, by the afterbody of data writing in files, the skew of this segment data is recorded in corresponding groove position simultaneously.
4, being read as of historical data:
The storing time intervals of historical data was divided by 2 hours, certain the track position on the unit respective file head that each is divided out, and the data block in file is directly pointed in these groove positions.According to the index classification code that will obtain data, can read soon the corresponding data in track data field pointed, as shown in Figure 4 again.File is the form of catalogue per year, is equivalent to read the file in a plurality of catalogues for the period of straddling over year.
The present invention adopts service-oriented technology to provide service interface for client software, so that the secondary development of application program, and the expansion of market demand.Interface used can be encapsulated as two kinds, C bank interface and Web service interface, and C bank interface adopts CORBA service bus, and Web service interface adopts Apache or Web Logical etc.
For guaranteeing that device id has the whole network uniqueness, the URI of IEC61970CIM can be mapped as to device id, and adopt unique mapping ruler.The coding of all kinds of power quality indexs also should have the whole network uniqueness.Adopt service-oriented mode, the user who is the whole network by access service interface provides unified coding.
Finally should be noted that: above embodiment is only in order to illustrate that technical scheme of the present invention is not intended to limit, although the present invention is had been described in detail with reference to above-described embodiment, those of ordinary skill in the field are to be understood that: still can modify or be equal to replacement the specific embodiment of the present invention, and do not depart from any modification of spirit and scope of the invention or be equal to replacement, it all should be encompassed in the middle of claim scope of the present invention.
Claims (8)
1. quality of power supply mass data storage and a read method, is characterized in that, described method comprises
(1) quality of power supply mass data is adopted to the storage of key-value pair form;
(2) key-value pair is stored hereof in the mode of serializing;
(3) real time data is write fashionablely, locates this data of monitoring point district, by the respective storage areas of pointer position monitor data according to device id;
(4) according to the index classification code that will obtain data, read the corresponding data in track.
2. a kind of quality of power supply mass data storage as claimed in claim 1 and read method, is characterized in that, described step (1) comprises in key-value pair data, and the corresponding value of each key, using power quality monitoring device ID as key; In the data field of measured value value of being put into of this monitoring of equipment index.
3. a kind of quality of power supply mass data storage as claimed in claim 2 and read method, is characterized in that, the measured value of described monitoring of equipment index comprises voltage deviation, frequency departure, three-phase imbalance, voltage fluctuation and flicker, harmonic wave and a harmonic wave.
4. a kind of quality of power supply mass data storage as claimed in claim 2 and read method, is characterized in that, in described each value district, the measured value of monitoring is analyzed and encodes by index classification, by calculating side-play amount rapidly locating piece.
5. a kind of quality of power supply mass data storage as claimed in claim 1 and read method, is characterized in that, described step (3) pointer comprises take 200 milliseconds, 3 seconds, 1 minute and 10 minutes four memory blocks that are data granularity.
6. a kind of quality of power supply mass data storage as claimed in claim 1 and read method, it is characterized in that, described step (3) comprises that described real time data data writing is with time stamp, the difference of the moment point of described time stamp and data field stem is less than 10 minutes, directly by the mistiming, obtained the skew of data field, can be directly by data write memory, the converging operationJu Hecaozuo that log-on data makes progress simultaneously;
Described real time data data writing is with time stamp, and the difference of the moment point of described time stamp and data field stem is more than or equal to 10 minutes, starts background thread ，Jiang Ci segment data district writing in files.
7. a kind of quality of power supply mass data storage as claimed in claim 1 and read method, it is characterized in that, described step (4) comprises that the storing time intervals of data divided by 2 hours, certain track position on the unit respective file head that each is divided out, the data block in file is directly pointed in described groove position.
8. a kind of quality of power supply mass data storage as claimed in claim 7 and read method, is characterized in that, described file is the form of catalogue per year, reads the file in a plurality of catalogues for the period of straddling over year.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310656313.8A CN103593486B (en) | 2013-12-06 | 2013-12-06 | A kind of quality of power supply mass data storage and read method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310656313.8A CN103593486B (en) | 2013-12-06 | 2013-12-06 | A kind of quality of power supply mass data storage and read method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103593486A true CN103593486A (en) | 2014-02-19 |
| CN103593486B CN103593486B (en) | 2017-11-21 |
Family
ID=50083627
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310656313.8A Active CN103593486B (en) | 2013-12-06 | 2013-12-06 | A kind of quality of power supply mass data storage and read method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103593486B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105606856A (en) * | 2015-12-21 | 2016-05-25 | 南昌市科陆智能电网科技有限公司 | Method for raising power meter system real-time response speed, system and power meter |
| CN106095828A (en) * | 2016-06-01 | 2016-11-09 | 积成电子股份有限公司 | A kind of curve data storage scheme based on power information acquisition terminal |
| CN107834703A (en) * | 2017-11-21 | 2018-03-23 | 武汉精伦电气有限公司 | A kind of intelligent grid power distribution room monitoring management system and method |
| CN109799947A (en) * | 2017-11-16 | 2019-05-24 | 浙江宇视科技有限公司 | Distributed storage method and device |
| CN110046172A (en) * | 2018-12-29 | 2019-07-23 | 阿里巴巴集团控股有限公司 | In line computation data processing method and system |
| CN111858478A (en) * | 2020-07-24 | 2020-10-30 | 深圳市航天泰瑞捷电子有限公司 | Multi-dimensional data processing method and device for electricity consumption data information |
| CN112905684A (en) * | 2021-02-22 | 2021-06-04 | 杭州海兴泽科信息技术有限公司 | Electric power data storage method and device and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080235355A1 (en) * | 2004-10-20 | 2008-09-25 | Electro Industries/Gauge Tech. | Intelligent Electronic Device for Receiving and Sending Data at High Speeds Over a Network |
| CN102158550A (en) * | 2011-03-03 | 2011-08-17 | 河南电力试验研究院 | IEC61850-based power quality transient data transmission method |
| CN102779180A (en) * | 2012-06-29 | 2012-11-14 | 华为技术有限公司 | Operation processing method of data storage system and data storage system |
| US8364677B2 (en) * | 2008-08-01 | 2013-01-29 | International Business Machines Corporation | Method and apparatus for generating partitioning keys for a range-partitioned database |
| CN103294710A (en) * | 2012-02-28 | 2013-09-11 | 北京新媒传信科技有限公司 | Data access method and device |
-
2013
- 2013-12-06 CN CN201310656313.8A patent/CN103593486B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080235355A1 (en) * | 2004-10-20 | 2008-09-25 | Electro Industries/Gauge Tech. | Intelligent Electronic Device for Receiving and Sending Data at High Speeds Over a Network |
| US8364677B2 (en) * | 2008-08-01 | 2013-01-29 | International Business Machines Corporation | Method and apparatus for generating partitioning keys for a range-partitioned database |
| CN102158550A (en) * | 2011-03-03 | 2011-08-17 | 河南电力试验研究院 | IEC61850-based power quality transient data transmission method |
| CN103294710A (en) * | 2012-02-28 | 2013-09-11 | 北京新媒传信科技有限公司 | Data access method and device |
| CN102779180A (en) * | 2012-06-29 | 2012-11-14 | 华为技术有限公司 | Operation processing method of data storage system and data storage system |
Non-Patent Citations (3)
| Title |
|---|
| 李巍: "《Ext—扩展文件系统的研究 》", 《信息系统工程》 * |
| 李曙光: "《ETL优化技术在构建电能质量数据中心中的研究实现》", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 闫湖等: "《基于分布式键值对存储技术的EMS数据库平台》", 《电网技术》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105606856A (en) * | 2015-12-21 | 2016-05-25 | 南昌市科陆智能电网科技有限公司 | Method for raising power meter system real-time response speed, system and power meter |
| CN106095828A (en) * | 2016-06-01 | 2016-11-09 | 积成电子股份有限公司 | A kind of curve data storage scheme based on power information acquisition terminal |
| CN106095828B (en) * | 2016-06-01 | 2019-03-29 | 积成电子股份有限公司 | A kind of curve data storage scheme based on power information acquisition terminal |
| CN109799947A (en) * | 2017-11-16 | 2019-05-24 | 浙江宇视科技有限公司 | Distributed storage method and device |
| CN107834703A (en) * | 2017-11-21 | 2018-03-23 | 武汉精伦电气有限公司 | A kind of intelligent grid power distribution room monitoring management system and method |
| CN110046172A (en) * | 2018-12-29 | 2019-07-23 | 阿里巴巴集团控股有限公司 | In line computation data processing method and system |
| CN110046172B (en) * | 2018-12-29 | 2023-07-11 | 创新先进技术有限公司 | Online computing data processing method and system |
| CN111858478A (en) * | 2020-07-24 | 2020-10-30 | 深圳市航天泰瑞捷电子有限公司 | Multi-dimensional data processing method and device for electricity consumption data information |
| CN112905684A (en) * | 2021-02-22 | 2021-06-04 | 杭州海兴泽科信息技术有限公司 | Electric power data storage method and device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103593486B (en) | 2017-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103593486A (en) | Method for storing and reading mass data of power quality | |
| CA2871313C (en) | Method and system for managing power grid data | |
| CN105069703B (en) | A kind of electrical network mass data management method | |
| CN108197181B (en) | Compression storage method of time sequence data, electronic equipment and storage medium | |
| CN107943831B (en) | HBase-based power grid historical data centralized storage method | |
| CN106503276A (en) | A kind of method and apparatus of the time series databases for real-time monitoring system | |
| Prasad et al. | Smart meter data analytics using OpenTSDB and Hadoop | |
| CN104391903A (en) | Distributed storage and parallel calculation-based power grid data quality detection method | |
| CN104035956A (en) | Time-series data storage method based on distributive column storage | |
| CN108108517A (en) | A kind of Electric Power Network Planning intelligence aided analysis method based on big data | |
| Bach et al. | Power grid time series data analysis with pig on a hadoop cluster compared to multi core systems | |
| CN106095828A (en) | A kind of curve data storage scheme based on power information acquisition terminal | |
| CN105868073A (en) | Data center energy saving strategy implementation method based on time series analysis | |
| Kalavri et al. | Block sampling: Efficient accurate online aggregation in mapreduce | |
| CN105574074A (en) | Smart grid WAMS-based time-series big data storage method | |
| Shah et al. | Performance Study of Time Series Databases | |
| CN110134690B (en) | Method and system for rapidly storing oversized data of Oracle database | |
| CN104731779A (en) | Real-time file system data organization and management method facing real-time databases | |
| CN107238764B (en) | The electric energy quality monitoring method and device of electric system | |
| Wang et al. | Smart grid time series big data processing system | |
| CN107357915A (en) | A kind of date storage method and system | |
| CN110019343A (en) | A kind of new energy meteorological data management method and system | |
| CN112540987A (en) | Big data management system of distribution and utilization electricity based on data mart | |
| Pan et al. | Multi-dimensional Index Construction of Electric Power Multi-source Measurement Data considering Spatio-temporal Correlation | |
| CN110928855A (en) | Method and system for establishing power grid measurement database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20160121 Address after: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Applicant after: State Grid Corporation of China Applicant after: State Grid Smart Grid Institute Applicant after: Jiangsu Electric Power Company Address before: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Applicant before: State Grid Corporation of China Applicant before: State Grid Smart Grid Institute |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |