CN105117177A - Big data management method based on single-hard-disk multi-partition mode - Google Patents
Big data management method based on single-hard-disk multi-partition mode Download PDFInfo
- Publication number
- CN105117177A CN105117177A CN201510581298.4A CN201510581298A CN105117177A CN 105117177 A CN105117177 A CN 105117177A CN 201510581298 A CN201510581298 A CN 201510581298A CN 105117177 A CN105117177 A CN 105117177A
- Authority
- CN
- China
- Prior art keywords
- data
- subregion
- partition
- hard disk
- day
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a big data management method based on a single-hard-disk multi-partition mode. The method sequentially includes the steps that 1, a hard disk connected with a fault recorder of an intelligent substation is divided into n partitions with a fixed capacity according to the capacity of the hard disk and needs, and the n partitions are numbered according to a certain numbering rule; 2, the minimum residual space value of each partition is set, wherein the minimum residual space value is larger than the sum of quantities of data needing to be collected every day; 3, received data are stored in the partitions, and when data are stored in the ith partition, the residual capacity of the (i+1)th partition is detected and corresponding processing is performed so as to guarantee enough partition space for data storage on the second day; 4, when needing to be stored, the received data are stored in a circulating mode according to the first partition-last partition-first partition rule. Through partition division, the data are read and written and wiped in different partitions, so that the reliability and security of data storage in the single hard disk are improved.
Description
Technical field
The present invention relates to the large data processing field of intelligent substation in electric system, be specifically related to a kind of large data efficient based on single hard disk multi partition and store and secure erase method.
Background technology
In recent years, under the support energetically of national policy, generating, transmission of electricity, electricity consumption are developed rapidly, and incident grid operating conditions also becomes increasingly complex.If can not carry out effective storage and analysis to the various failure messages that may occur in operation of power networks, the quality of power supply of electrical network will can not be guaranteed.
In order to tackle the impact of these faults on stabilization of power grids performance and security performance, each relevant colleges and universities of China, enterprise have carried out the large data analysis research work in operation of power networks in succession, way conventional at present adopts set of device that these data are kept on hard disk, local analysis or long-range linking parsing.In general, under single hard disk, directly preserve data, make write, reading and erase operation under same subregion, easily occur that data security is not high, disk fragments is many, and free space such as to tail off at the shortcoming; The capacity of current hard disk is increasing, adopts many hard-disc storage can produce engineering cost again high, the problem of the wasting of resources.Therefore develop large data managing method safe and reliable under single hard disk to be very important.
Summary of the invention
The large data efficient that the object of this invention is to provide based on single hard disk multi partition stores and secure erase method, easily occurs in order to solve when single hard disk stores in prior art the problem that data security is not high.
For achieving the above object, the solution of the present invention comprises:
Based on a large data managing method for single hard disk multi partition, it is characterized in that, described management method in turn includes the following steps:
Step 1: by the hard disk be connected with intelligent substation fault wave recording device according to its capacity and n the subregion needing to be divided into fixed capacity, these subregions are numbered according to certain coding rule;
Step 2: the least residue spatial value that each subregion is set, this least residue spatial value is greater than the summation needing image data amount every day;
Step 3: be stored in disk partition by the data received, when data storage operations is at i-th subregion, detects the residual capacity of the i-th+1 subregion simultaneously, and makes respective handling, to ensure that the second day data storage has enough partition spaces;
Step 4: when storing the data received, press from first subregion to last subregion again to the rule of first subregion, circulation realizes the storage of data.
Wherein, coding rule certain in step 1 refers to numbers from small to large according to natural order from natural number 1, and wherein a kind of numbering is 1, and 2,3 ... m, m are natural numbers.
Further, the respective handling in step 3 refers to when the i-th+1 subregion residual capacity is less than set least residue spatial value, carries out format process to it.
Further, the time interval stored of circulating in step 4 is one day, and every day only does data at a subregion and stores.
A kind of large data managing method based on single hard disk multi partition provided by the invention, makes the read-write of data and erasing not under same subregion, improves data store under single hard disk reliability, security.
The storage of data is only stored up at a partitioned storage every day, is conducive to searching and playback of historical data like this.
Meanwhile, adopt the mode of format to wipe to data partition, effectively decrease the problem that disk fragments tails off, free space diminishes.
Accompanying drawing explanation
Fig. 1 is the large data managing method schematic diagram based on single hard disk multi partition.
Embodiment
Below in conjunction with accompanying drawing, the present invention will be further described in detail.
Embodiment
The large data managing method of one provided by the invention, in turn includes the following steps:
Step 1: by the hard disk be connected with intelligent substation fault wave recording device according to its capacity and n the subregion needing to be divided into fixed capacity, these subregions are numbered according to certain coding rule;
Step 2: the least residue spatial value that each subregion is set, this least residue spatial value is greater than the summation needing image data amount every day;
Step 3: be stored in disk partition by the data received, when data storage operations is at i-th subregion, detects the residual capacity of the i-th+1 subregion simultaneously, and makes respective handling, to ensure that the second day data storage has enough partition spaces.
Step 4: when storing the data received, press from first subregion to last subregion again to the rule of first subregion, circulation realizes the storage of data.
Below in conjunction with accompanying drawing, the specific embodiment of the present invention is further described in detail:
First, the hard disk be connected is divided into n subregion of fixed capacity according to its capacity, ensures that the capacity of each subregion is greater than the data volume that will gather every day with intelligent substation fault wave recording device.For a certain intelligent substation, intelligent substation 1000Hz sample frequency per second under default configuration, every frame data have the switching value on 192 tunnel analog quantitys and 512 tunnels, then the data total amount produced in a day is: (192 × 2+ (512+15)/16 × 2) Byte/ frame × 1000Hz/ second × 3600/ hour × 24/ day ≈ 36GB.Therefore, the least residue spatial value of each subregion is set as 50G, can ensure the integrality of every day data.At this, the hard disk be connected with intelligent substation fault wave recording device is 1T, and now it is divided into 7 subregions being used for storing intelligent substation data, each subregion is 100G, and remaining space is stayed and used it for anything else.At this, 7 disk partition are numbered 1,2,3,4,5,6,7, alternatively, also can adopt other coding rule, easy to use.
After the hard disk be connected with intelligent substation fault wave recording device receives data bus cycle data, hew out a core buffer, the size of this core buffer is the full-size received in packet, and in order to preserve the data of each cycle, and then forwarding is stored in corresponding subregion.Then the partition number write_num that will store and the partition number detec_num that will detect is set.There is formula:
write_num＝write_num％n；
detec_num＝(write_num+1)％n；
Wherein n is the number of the subregion for receiving intelligent substation data, n=7, write_num=1, detec_num=2 here.
Store data often through one day, then write_num adds 1.Detected by disk and judge partition number is whether the subregion capacity of detec_num is write and expired, every day is detected once, if this subregion residual capacity is less than set least residue spatial value, certain process is done in Ze Duigai district, its residual capacity is remained be greater than the least residue spatial value of setting, meet the storage of data; Here that format process is directly done to this subregion.Therefore, data storage operations only need toward partition number value be write_num subregion on write data, and do not need to consider whether this subregion has enough capacity.
Meanwhile, in each subregion, every day, number was folder name preservation data file by the hour, the time interval that each file is preserved sets according to the data volume that will receive every day, here, each file size is about 20MB, therefore arranges time of file and is spaced apart 10 seconds.
According to above-mentioned steps, data being stored at first day is numbered in the subregion of 1, disk detection is simultaneously numbered residual capacity in the subregion of 2 and whether is greater than least residue spatial value, if not, then processing to the subregion being numbered 2 space increasing this subregion, is to be numbered the partition format of 2 at this.Pass a disk partition every day like this backward, the data of later circulate again to the subregion being numbered 7 storage from the subregion being numbered 1, the storage intelligent substation that so circulates.
Be presented above the embodiment that the present invention is concrete, but the present invention is not limited to described embodiment.Under the thinking that the present invention provides; the mode easily expected to those skilled in the art is adopted to convert the technological means in above-described embodiment, replace, revise; and the effect played goal of the invention that is substantially identical with the relevant art means in the present invention, that realize is also substantially identical; the technical scheme of such formation is carried out fine setting to above-described embodiment and is formed, and this technical scheme still falls within the scope of protection of the present invention.
Claims (5)
1. based on a large data managing method for single hard disk multi partition, it is characterized in that, described management method in turn includes the following steps:
Step 1: by the hard disk be connected with intelligent substation fault wave recording device according to its capacity and n the subregion needing to be divided into fixed capacity, these subregions are numbered according to certain coding rule;
Step 2: the least residue spatial value that each subregion is set, this least residue spatial value is greater than the summation needing image data amount every day;
Step 3: be stored in disk partition by the data received, when data storage operations is at i-th subregion, detects the residual capacity of the i-th+1 subregion simultaneously, and makes respective handling, to ensure that the second day data storage has enough partition spaces;
Step 4: when storing the data received, press from first subregion to last subregion again to the rule of first subregion, circulation realizes the storage of data.
2. a kind of large data managing method based on single hard disk multi partition according to claim 1, is characterized in that, coding rule certain in described step 1 refers to numbers from small to large according to natural order from natural number 1.
3. a kind of large data managing method based on single hard disk multi partition according to claim 2, it is characterized in that, described numbering is 1,2,3 ... m, m are natural numbers.
4. a kind of large data managing method based on single hard disk multi partition according to claim 1, it is characterized in that, respective handling in described step 3 refers to when the i-th+1 subregion residual capacity is less than set least residue spatial value, carries out format process to it.
5. a kind of large data managing method based on single hard disk multi partition according to claim 1, is characterized in that, the time interval stored of circulating in described step 4 is one day, and every day only does data at a subregion and stores.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510581298.4A CN105117177B (en) | 2015-09-14 | 2015-09-14 | A kind of big data management method based on single hard disk multi partition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510581298.4A CN105117177B (en) | 2015-09-14 | 2015-09-14 | A kind of big data management method based on single hard disk multi partition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105117177A true CN105117177A (en) | 2015-12-02 |
| CN105117177B CN105117177B (en) | 2018-08-10 |
Family
ID=54665180
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510581298.4A Expired - Fee Related CN105117177B (en) | 2015-09-14 | 2015-09-14 | A kind of big data management method based on single hard disk multi partition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105117177B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108616583A (en) * | 2018-04-16 | 2018-10-02 | 牛路 | A kind of memory allocation method based on computer cloud |
| CN112346663A (en) * | 2020-11-26 | 2021-02-09 | 上海天旦网络科技发展有限公司 | Multi-partition rotation data writing and clearing method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102662854A (en) * | 2012-03-27 | 2012-09-12 | 许继集团有限公司 | Storage method based on high-capacity multiple hard disks |
| CN103064633A (en) * | 2012-12-13 | 2013-04-24 | 广东威创视讯科技股份有限公司 | Data storage method and device |
| CN104268088A (en) * | 2014-09-15 | 2015-01-07 | 郑州天迈科技股份有限公司 | Vehicle DVR (Digital Video Recorder) hard disk data storage method |
| CN104615381A (en) * | 2015-01-18 | 2015-05-13 | 浙江宇视科技有限公司 | Redundancy array of independent disks of video monitoring system |
-
2015
- 2015-09-14 CN CN201510581298.4A patent/CN105117177B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102662854A (en) * | 2012-03-27 | 2012-09-12 | 许继集团有限公司 | Storage method based on high-capacity multiple hard disks |
| CN103064633A (en) * | 2012-12-13 | 2013-04-24 | 广东威创视讯科技股份有限公司 | Data storage method and device |
| CN104268088A (en) * | 2014-09-15 | 2015-01-07 | 郑州天迈科技股份有限公司 | Vehicle DVR (Digital Video Recorder) hard disk data storage method |
| CN104615381A (en) * | 2015-01-18 | 2015-05-13 | 浙江宇视科技有限公司 | Redundancy array of independent disks of video monitoring system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108616583A (en) * | 2018-04-16 | 2018-10-02 | 牛路 | A kind of memory allocation method based on computer cloud |
| CN108616583B (en) * | 2018-04-16 | 2020-12-15 | 亿享不道张家口信息科技有限公司 | Storage space allocation method based on computer cloud |
| CN112346663A (en) * | 2020-11-26 | 2021-02-09 | 上海天旦网络科技发展有限公司 | Multi-partition rotation data writing and clearing method and system |
| CN112346663B (en) * | 2020-11-26 | 2022-10-11 | 上海天旦网络科技发展有限公司 | Multi-partition rotation data writing and clearing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105117177B (en) | 2018-08-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20240143169A1 (en) | Method for Improving Message Storage Efficiency of Network Chip, Device, and Storage medium | |
| CN104215878B (en) | A kind of distribution line fault wave recording device and its failure wave-recording method | |
| CN101937376B (en) | A kind of data managing method and data storage device | |
| CN105468298B (en) | A kind of key assignments storage method based on log-structured merging tree | |
| CN107256197B (en) | A kind of date storage method, storage device and intelligent terminal | |
| CN103226965B (en) | Based on the audio/video data access method of time bitmap | |
| CN103019957B (en) | A kind of power failure record ripple storage means and device | |
| CN101710270B (en) | High-speed mass memory based on flash memory and chip data management method | |
| CN107491267B (en) | High-speed image data storage device based on L VDS interface | |
| CN106201778B (en) | Information processing method and storage equipment | |
| CN107291405A (en) | A kind of NorFlash data managing method and device | |
| CN106951488A (en) | A kind of log recording method and device | |
| DE102011056139A1 (en) | User device for performing a data retention operation, and storage device and data retention method | |
| CN105100716A (en) | Safe memory cell used for network video monitoring and system thereof | |
| US20110106323A1 (en) | Method and apparatus for processing power system topology structure information | |
| CN104317770A (en) | Data storage structure and data access method for multiple core processing system | |
| CN105117177A (en) | Big data management method based on single-hard-disk multi-partition mode | |
| CN110868458B (en) | Communication method and device for stable control system | |
| US10324959B2 (en) | Garbage collection in storage system | |
| CN102937956A (en) | Method and device for storing real-time messages in intelligent substation | |
| CN102789422B (en) | The method for writing data of solid state hard disc | |
| CN102629235A (en) | Method for increasing read-write speed of double data rate (DDR) memory | |
| CN105138278B (en) | A kind of the naked of intelligent substation network message takes inventory method for storing | |
| CN103365782A (en) | Memory management method | |
| CN108153491B (en) | Storage method and architecture capable of closing part of servers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180810 Termination date: 20190914 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |