CN107656971A - A kind of intelligent grid collection Monitoring Data storage method based on Redis - Google Patents
A kind of intelligent grid collection Monitoring Data storage method based on Redis Download PDFInfo
- Publication number
- CN107656971A CN107656971A CN201710781859.4A CN201710781859A CN107656971A CN 107656971 A CN107656971 A CN 107656971A CN 201710781859 A CN201710781859 A CN 201710781859A CN 107656971 A CN107656971 A CN 107656971A
- Authority
- CN
- China
- Prior art keywords
- data
- intelligent grid
- redis
- collection monitoring
- monitoring data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 26
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000013500 data storage Methods 0.000 title claims description 9
- 238000013461 design Methods 0.000 claims description 4
- 238000013499 data model Methods 0.000 claims description 2
- 238000007906 compression Methods 0.000 description 6
- 230000006835 compression Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 2
- 239000004744 fabric Substances 0.000 description 2
- 238000013467 fragmentation Methods 0.000 description 2
- 238000006062 fragmentation reaction Methods 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 230000009183 running Effects 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention relates to the date storage method of intelligent grid, more particularly to a kind of storage method of the collection Monitoring Data based on Redis in intelligent grid.Intelligent grid gathers important component of the Monitoring Data as the Large data types of intelligent grid four, big with data volume, storage cycle span is long, data format is fixed, data loading throughput demands are high, access module is based on section inquiry and batch query, the features such as high to stability requirement, historical data scale constantly increases.For features above, the present invention proposes a kind of intelligent grid collection Monitoring Data distribution Key Value storage methods based on Redis, loading and the access performance of collection Monitoring Data can be substantially improved, while ensure the reliability of system.
Description
Technical field
The present invention relates to the date storage method of intelligent grid, is adopted more particularly to a kind of intelligent grid based on Redis
The storage method of monitor set data.
Technical background
Intelligent grid collection Monitoring Data is highly important data in power grid application, as the big data class of intelligent grid four
The important component of type, it is smart grid electricity usage information gathering, equipment condition monitoring, decision Analysis, offline excavation point
The important foundation of analysis etc..Currently, the main feature of intelligent grid collection Monitoring Data storage：(1) it is big to gather Monitoring Data amount, deposits
Store up cycle span length；(2) data format is fixed；(3) data loading throughput demands are high；(4) access module with section inquiry and
Based on batch query；(5) it is high for data storage stability requirement；(6) historical data scale constantly increases, and application is enriched
Property and it is interactive constantly enhancing, to the requirements for access more and more higher of historical data.
The problem above occurred during Monitoring Data is gathered for intelligent grid, the present invention proposes one kind and is based on
Redis intelligent grid collection Monitoring Data distribution Key-Value storage methods, can be substantially improved collection Monitoring Data
Loading and access performance, while ensure the reliability of system.
Redis be one increase income write using ANSI C languages, support network, can based on internal memory also can persistence day
Will type, Key-Value databases, and the API of multilingual is provided.It supports that the value types of storage are relatively more, including
String (character string), list (chained list), set (set), zset (ordered set) and hash (Hash type).These data class
Type all supports push/pop, add/remove and takes common factor union and difference set and more rich operation, and these operations are all
Atomicity.Redis supports the sequence of various different modes.As memcached, for guaranteed efficiency, data are all slow
Exist in internal memory.Difference is that the data of renewal periodically can be write disk or modification operation write-in is added by redis
Log file, and it is synchronous to realize master-slave (principal and subordinate) on this basis.
The content of the invention
A kind of intelligent grid based on Redis gathers Monitoring Data storage method：(1) according to the markers and industry of measuring point
Business modelling memory cell；(2) according to collection monitoring point scale, data acquiring frequency, design data burst mechanism；(3) adopt
Database purchase file is compressed with LZF algorithms；(4) using RDB skill upgradings data loading reliability.The program is main
Including following key technology point：
(1) Monitoring Data storage model is gathered
Key-Value storage organization of this method based on Redis, the Key structures of design section data are<YMDHMS>,
ValueList structures are<CC+DT+Cid+MT、Value>, the Key structures for designing batch data are<CC+DT+Cid+MT>,
ValueList structures are< YMDHMS、Value>, wherein CC is districts and cities' coding, and DT is that transformer station and electric network model information encode
Combination, YMDHMS are date Hour Minute Second, and MT is collection monitoring vertex type, and MID is device coding, and Value is collection monitoring point
Sampled value.
Data model：
Specific storage model is as shown in Figure 1.
(2) adaptive stripping strategy
This method establishes data fragmentation strategy by way of being hashed to storage model Key, and the strategy can be by number
According to being distributed on the Redis examples of some main frames, and then extend the storage capacity and computing capability of cluster.When data load,
Hash algorithm is carried out to Key first, is assigned to according to hash values on different machines, passes through hash value control datas burst point
Cloth.
(3) data file LZF compression algorithms
LZF compression algorithms use hybrid coding, the compression for the RDB files of Redis examples.It was proved that LZF
For compression algorithm when processing data file constantly increases, compression factor, compression is time-consuming, decompresses time-consuming equal quite stable.
(4) RDB technologies
In order to ensure the reliability of data loading, this method collection RDB technologies, i.e., number is generated in specified time interval
According to the time point snapshot of collection.The loading procedure of data record：First number two is changed by configuring specified time span and Key values
The threshold value of individual triggering persistence snapshot, in Redis example runnings, according to the time span of setting and Key values change number
Triggering generation snapshot document, when appearance power-off, network disconnection and collapse etc. are abnormal in cluster running, rescan snapshot
File is realized the part data of write-in data storage file are reloaded not successfully in internal memory.Because RDB snapshots are to pass through
Fork produces subprocess generation, therefore host process does not have the I/O operation of correlation, and data-handling efficiency is to be guaranteed
's.
(5) implementation result being deployed on hardware
The specific reality of this method is described into the example that Monitoring Data is gathered with certain province power information acquisition system below
Mode is applied, and above-mentioned distributed model is realized using Key-Value databases Redis.
Server hardware configures：CPU-E5-2665 (32 core), 2.4GHz；Internal memory -128GB；Disk -5*900G, wherein 1
Block is as system partitioning, remaining 4 pieces independent partitioned storage data (JBOD patterns)；Quantity -3.
Server software configures：Operating system-CentOS 6.5；Database-Redis-2.8.17.
According to the configuration of above parameter, power information collection measuring point scale in this area's is about 36,000,000, data produce week
Phase is 15 minutes, data are daily burst.The storage model designed by this method, data loading and the efficiency accessed are as follows：
Ten thousand/second of data loading efficiency -17；Ten thousand/second of access efficiency -39.
As it appears from the above, the loading of this method data is respectively with access efficiency：170000/second, 390,000/second.Wherein, unit
" ten thousand/second " represents：The how many ten thousand data values of loading/access in each second.Under the actual scene, the demand of data loading is big
In 40,000/second (36,000,000 measuring points divided by 15 minutes).Therefore, this method efficiently meets this area's power information acquisition system
Gather Monitoring Data application demand.
Beneficial effect
1. a kind of Key-Value storage models of design intelligent grid collection Monitoring Data, using Clustering, according to when
Mark and business model establish memory cell, it is ensured that low volume data connected reference, while data compression is carried out, magnetic disc i/o is reduced, is carried
Rise data access performance；
2. according to collection Monitoring Data scale, frequency acquisition, adaptive data fragmentation strategy is established, it is ensured that point of data
Cloth accesses, and lifts data access performance；
3. use RDB technologies, it is ensured that when system exception collapses, data can be lifted by the fast quick-recovery data of snapshot document
Reliability.
Brief description of the drawings
Fig. 1 is a kind of collection Monitoring Data storage model figure provided by the invention.
Claims (1)
- A kind of 1. intelligent grid collection Monitoring Data storage method based on Redis, it is characterised in that：This storage method is based on Redis Key-Value storage organizations, the Key structures of design section data are<YMDHMS>, ValueList structures are<CC+ DT+Cid+MT、Value>, the Key structures for designing batch data are<CC+DT+Cid+MT>, ValueList structures are< YMDHMS、Value>, wherein CC is districts and cities' coding, and DT is transformer station and electric network model information coded combination, YMDHMS are days Day Hour Minute Second, MT are collection monitoring vertex type, and MID is device coding, and Value is collection monitoring point sampled value.Data model is compiled Code table is as shown in specification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710781859.4A CN107656971A (en) | 2017-09-02 | 2017-09-02 | A kind of intelligent grid collection Monitoring Data storage method based on Redis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710781859.4A CN107656971A (en) | 2017-09-02 | 2017-09-02 | A kind of intelligent grid collection Monitoring Data storage method based on Redis |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107656971A true CN107656971A (en) | 2018-02-02 |
Family
ID=61128275
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710781859.4A Pending CN107656971A (en) | 2017-09-02 | 2017-09-02 | A kind of intelligent grid collection Monitoring Data storage method based on Redis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107656971A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109032511A (en) * | 2018-07-09 | 2018-12-18 | 武汉斗鱼网络科技有限公司 | Data storage method, server and storage medium |
| CN109270565A (en) * | 2018-09-04 | 2019-01-25 | 广东翼卡车联网服务有限公司 | A kind of processing unit of vehicle GPS big data |
| CN111212145A (en) * | 2020-01-09 | 2020-05-29 | 国网福建省电力有限公司 | Redis cluster for power supply service command system |
| CN112235361A (en) * | 2020-09-28 | 2021-01-15 | 青海绿能数据有限公司 | Photovoltaic power plant data switching platform |
| CN113407541A (en) * | 2021-06-23 | 2021-09-17 | 中移（杭州）信息技术有限公司 | Data acquisition method, data acquisition equipment, storage medium and device |
| CN113590379A (en) * | 2021-06-25 | 2021-11-02 | 国电南瑞科技股份有限公司 | Redis data persistence method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105589791A (en) * | 2015-12-28 | 2016-05-18 | 江苏省电力公司信息通信分公司 | Method for application system log monitoring management in cloud computing environment |
| CN105956006A (en) * | 2016-04-21 | 2016-09-21 | 贵州电网有限责任公司信息中心 | Structured query system for key value pair data storage |
| CN106411578A (en) * | 2016-09-12 | 2017-02-15 | 国网山东省电力公司电力科学研究院 | Website monitoring system and method applicable to power industry |
| CN106597968A (en) * | 2016-12-24 | 2017-04-26 | 大连尚能科技发展有限公司 | Converter high-speed real-time monitoring system and method based on Redis |
| CN106649711A (en) * | 2016-12-21 | 2017-05-10 | 国电南瑞南京控制系统有限公司 | Distributed type preposed acquisition system based on Redis real-time database |
| CN106777141A (en) * | 2016-12-19 | 2017-05-31 | 国网山东省电力公司电力科学研究院 | A kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method |
| CN106844556A (en) * | 2016-12-30 | 2017-06-13 | 江苏瑞中数据股份有限公司 | A kind of intelligent grid time scale measurement date storage method based on HBase |
-
2017
- 2017-09-02 CN CN201710781859.4A patent/CN107656971A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105589791A (en) * | 2015-12-28 | 2016-05-18 | 江苏省电力公司信息通信分公司 | Method for application system log monitoring management in cloud computing environment |
| CN105956006A (en) * | 2016-04-21 | 2016-09-21 | 贵州电网有限责任公司信息中心 | Structured query system for key value pair data storage |
| CN106411578A (en) * | 2016-09-12 | 2017-02-15 | 国网山东省电力公司电力科学研究院 | Website monitoring system and method applicable to power industry |
| CN106777141A (en) * | 2016-12-19 | 2017-05-31 | 国网山东省电力公司电力科学研究院 | A kind of acquisition for merging multi-source heterogeneous electric network data and distributed storage method |
| CN106649711A (en) * | 2016-12-21 | 2017-05-10 | 国电南瑞南京控制系统有限公司 | Distributed type preposed acquisition system based on Redis real-time database |
| CN106597968A (en) * | 2016-12-24 | 2017-04-26 | 大连尚能科技发展有限公司 | Converter high-speed real-time monitoring system and method based on Redis |
| CN106844556A (en) * | 2016-12-30 | 2017-06-13 | 江苏瑞中数据股份有限公司 | A kind of intelligent grid time scale measurement date storage method based on HBase |
Non-Patent Citations (1)
| Title |
|---|
| （美）格雷（GRAY，J）: "《 事务处理：概念与技术》", 北京：机械工业出版社, pages: 956 - 957 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109032511A (en) * | 2018-07-09 | 2018-12-18 | 武汉斗鱼网络科技有限公司 | Data storage method, server and storage medium |
| CN109032511B (en) * | 2018-07-09 | 2022-05-13 | 武汉斗鱼网络科技有限公司 | Data storage method, server and storage medium |
| CN109270565A (en) * | 2018-09-04 | 2019-01-25 | 广东翼卡车联网服务有限公司 | A kind of processing unit of vehicle GPS big data |
| CN109270565B (en) * | 2018-09-04 | 2023-03-07 | 广东翼卡车联网服务有限公司 | Processing device for vehicle-mounted GPS big data |
| CN111212145A (en) * | 2020-01-09 | 2020-05-29 | 国网福建省电力有限公司 | Redis cluster for power supply service command system |
| CN112235361A (en) * | 2020-09-28 | 2021-01-15 | 青海绿能数据有限公司 | Photovoltaic power plant data switching platform |
| CN112235361B (en) * | 2020-09-28 | 2022-12-27 | 青海绿能数据有限公司 | Photovoltaic power plant data switching platform |
| CN113407541A (en) * | 2021-06-23 | 2021-09-17 | 中移（杭州）信息技术有限公司 | Data acquisition method, data acquisition equipment, storage medium and device |
| CN113407541B (en) * | 2021-06-23 | 2022-10-18 | 中移（杭州）信息技术有限公司 | Data acquisition method, data acquisition equipment, storage medium and device |
| CN113590379A (en) * | 2021-06-25 | 2021-11-02 | 国电南瑞科技股份有限公司 | Redis data persistence method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107656971A (en) | A kind of intelligent grid collection Monitoring Data storage method based on Redis | |
| Tarasov et al. | Generating realistic datasets for deduplication analysis | |
| EP2263180B1 (en) | Indexing large-scale gps tracks | |
| US8578112B2 (en) | Data management system and data management method | |
| CN103312544B (en) | A kind of control terminal reports the method, apparatus and system of journal file | |
| CN103116661A (en) | Data processing method of database | |
| CN106611046A (en) | Big data technology-based space data storage processing middleware framework | |
| Alsberg | Space and time savings through large data base compression and dynamic restructuring | |
| CN104317800A (en) | Hybrid storage system and method for mass intelligent power utilization data | |
| CN106021031B (en) | A kind of the deletion data reconstruction method and device of BTRFS file system | |
| JP2016532199A (en) | Generation of multi-column index of relational database by data bit interleaving for selectivity | |
| CN110609813B (en) | Data storage system and method | |
| WO2013027230A1 (en) | Storage apparatus and duplicate data detection method | |
| CN103098035A (en) | Storage system | |
| CN103955530A (en) | Data reconstruction and optimization method of on-line repeating data deletion system | |
| CN105512283A (en) | Data quality management and control method and device | |
| CN102013056A (en) | Configurable power report management system and operating method thereof | |
| CN112783901B (en) | Internet of things time sequence big data processing method based on Internet of things middleware | |
| WO2023093607A1 (en) | Offline data fuzzy search method and apparatus, device and medium | |
| CN105144157A (en) | System and method for compressing data in database | |
| CN108647118A (en) | Copy abnormal restoring method, device and computer equipment based on storage cluster | |
| CN105787090A (en) | Index building method and system of OLAP system of electric data | |
| CN103942311A (en) | Multiplexing method of monitoring system variables | |
| CN103678041A (en) | Incremental backup method and system | |
| CN109684517A (en) | A kind of historical data storage method, reading/writing method, storage device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180202 |
|
| RJ01 | Rejection of invention patent application after publication |