CN105117177B - A kind of big data management method based on single hard disk multi partition - Google Patents
A kind of big data management method based on single hard disk multi partition Download PDFInfo
- Publication number
- CN105117177B CN105117177B CN201510581298.4A CN201510581298A CN105117177B CN 105117177 B CN105117177 B CN 105117177B CN 201510581298 A CN201510581298 A CN 201510581298A CN 105117177 B CN105117177 B CN 105117177B
- Authority
- CN
- China
- Prior art keywords
- subregion
- data
- hard disk
- storage
- management method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000005192 partition Methods 0.000 title claims abstract description 23
- 238000000034 method Methods 0.000 title claims abstract description 13
- 238000013523 data management Methods 0.000 title claims abstract description 11
- 238000003860 storage Methods 0.000 claims abstract description 21
- 238000013500 data storage Methods 0.000 claims abstract description 7
- 238000007726 management method Methods 0.000 claims description 2
- 238000001514 detection method Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000005611 electricity Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 230000018199 S phase Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010248 power generation Methods 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a kind of big data management methods based on single hard disk multi partition, in turn include the following steps：1, the hard disk being connected with intelligent substation fault wave recording device according to its capacity and is needed into n subregion for being divided into fixed capacity, and be numbered according to certain coding rule；2, the least residue spatial value of each subregion is set, which is more than the summation for needing gathered data amount daily；3, by the data received storage to subregion, when data storage operations are in i-th of subregion, the residual capacity of i+1 subregion is detected, and make respective handling, to ensure that the storage of the second day data has enough partition spaces；4, when storing the data received, again to the rule of first subregion, storage is recycled by from first subregion to a last subregion.The read-write for making data and erasing are divided by subregion not under the same subregion, improve the reliability, safety that data store under single hard disk.
Description
Technical field
The present invention relates to intelligent substation big data processing fields in electric system, and in particular to one kind is more based on single hard disk
The big data efficient storage of subregion and safe method for deleting.
Background technology
In recent years, under the support energetically of national policy, power generation, transmission of electricity, electricity consumption are developed rapidly, incident
Grid operating conditions also become increasingly complex.If cannot be carried out to the various fault messages being likely to occur in operation of power networks effective
Storage and analysis, the power quality of power grid will cannot be guaranteed.
In order to cope with influence of these failures to stabilization of power grids performance and security performance, each related colleges and universities of China, enterprise's phase
After having carried out to the big data analysis research work in operation of power networks, currently used way is that these are counted using set of device
According to being stored on hard disk, local analysis or long-range linking parsing.In general, data are directly preserved under single hard disk, make write-in,
It reads and erasing operation is under the same subregion, be susceptible to that Information Security is not high, and disk fragments are more, free space tails off
Disadvantage；The capacity of hard disk is increasing at present, engineering cost height is will produce again using more hard-disc storages, the problem of the wasting of resources.
Therefore developing safe and reliable big data management method under single hard disk is very important.
Invention content
The object of the present invention is to provide based on single hard disk multi partition big data efficient storage and safe method for deleting, to
To be susceptible to Information Security not high when solving the problems, such as single hard disk storage in the prior art.
To achieve the above object, the solution of the present invention includes：
A kind of big data management method based on single hard disk multi partition, which is characterized in that the management method includes successively
Following steps：
Step 1：The hard disk being connected with intelligent substation fault wave recording device is divided into fixed appearance according to its capacity and needs
These subregions are numbered n subregion of amount according to certain coding rule；
Step 2：The least residue spatial value of each subregion is set, which, which is more than, needs to acquire number daily
According to the summation of amount；
Step 3：The data received are stored into disk partition, when data storage operations are in i-th of subregion, simultaneously
The residual capacity of i+1 subregion is detected, and makes respective handling, to ensure that the storage of the second day data has enough subregions empty
Between；
Step 4：When storing the data received, by from first subregion to a last subregion again to first subregion
Rule, cycle realize the storage of data.
Wherein, coding rule certain in step 1 refers to that the sequence since natural number 1 according to natural number is compiled from small to large
Number, one of which number is 1,2,3 ..., and m, m are natural numbers.
Further, the respective handling in step 3 refers to surplus less than set minimum in i+1 subregion residual capacity
When complementary space value, processing is formatted to it.
Further, the time interval for storage being recycled in step 4 is one day, only does data storage in a subregion daily.
A kind of big data management method based on single hard disk multi partition provided by the invention so that the read-write and erasing of data
Not under the same subregion, the reliability, safety that data store under single hard disk are improved.
The storage of data is only stored up in a partitioned storage daily, is conducive to the lookup and playback of historical data in this way.
Meanwhile data partition is wiped by the way of formatting, it is effectively reduced disk fragments and tails off, can be used
The problem of space becomes smaller.
Description of the drawings
Fig. 1 is the big data management method schematic diagram based on single hard disk multi partition.
Specific implementation mode
The present invention will be further described in detail below in conjunction with the accompanying drawings.
Embodiment
A kind of big data management method provided by the invention, in turn includes the following steps：
Step 1：The hard disk being connected with intelligent substation fault wave recording device is divided into fixed appearance according to its capacity and needs
These subregions are numbered n subregion of amount according to certain coding rule；
Step 2：The least residue spatial value of each subregion is set, which, which is more than, needs to acquire number daily
According to the summation of amount；
Step 3：The data received are stored into disk partition, when data storage operations are in i-th of subregion, simultaneously
The residual capacity of i+1 subregion is detected, and makes respective handling, to ensure that the storage of the second day data has enough subregions empty
Between.
Step 4：When storing the data received, by from first subregion to a last subregion again to first subregion
Rule, cycle realize the storage of data.
The specific implementation mode of the present invention is further described in detail below in conjunction with attached drawing：
First, the hard disk being connected with intelligent substation fault wave recording device is divided into n of fixed capacity according to its capacity
Subregion, it is ensured that the capacity of each subregion is more than the daily data volume to be acquired.By taking a certain intelligent substation as an example, acquiescence is matched
Lower intelligent substation 1000Hz sample frequencys per second are set, there is the switching value of 192 tunnel analog quantitys and 512 tunnels per frame data, then exist
The total amount of data generated in one day is：(192 × 2+ (512+15)/16 × 2) Byte/ frames × 1000Hz/ seconds × 3600/ hour ×
24/ day ≈ 36GB.Therefore, the least residue spatial value of each subregion is set as 50G, you can ensures the integrality per day data.
Here, the hard disk being connected with intelligent substation fault wave recording device is 1T, it is now divided into 7 for storing intelligent substation number
According to subregion, each subregion be 100G, remaining space, which is stayed, to use it for anything else.Here, be 1,2,3,4,5 by 7 disk partition numbers,
6,7, alternatively, other coding rules can also be used, it is easy to use.
After the hard disk being connected with intelligent substation fault wave recording device receives data/address bus cycle data, one is hewed out
Core buffer, the size of the core buffer are the full-size received in data packet, to preserve the number of each cycle
According to, and then be forwarded in corresponding subregion.Then the setting partition number write_num to be stored and the partition number to be detected
detec_num.There is formula：
Write_num=write_num%n；
Detec_num=(write_num+1) %n；
Wherein n is the number of the subregion for receiving intelligent substation data, herein n=7, write_num=1,
Detec_num=2.
It stores data often to pass through one day, then write_num adds 1.Judge partition number for detec_num's by disk detection
It is full whether subregion capacity is write, and detection is primary daily, if the subregion residual capacity is less than set least residue spatial value,
Certain processing is done to the area, so that its residual capacity is remained more than the least residue spatial value of setting, meets the storage of data；
It is that formatting processing is directly done to the subregion herein.Therefore, data storage operations only need to be write_num's toward partition number value
Data are write on subregion, without considering whether the subregion has enough capacity.
Meanwhile in each subregion, number is that folder name preserves data file by the hour daily, each file preserve when
Between be spaced and set according to the data volume to be received daily, herein, each file size is about 20MB, therefore file is arranged and protects
It is 10 seconds to deposit time interval.
It according to above-mentioned steps, was stored data at first day in the subregion that number is 1, while disk detection number is 2
Subregion in residual capacity whether be more than least residue spatial value, if it is not, then to number be 2 subregion handled with increase
The space of the subregion is that will number the partition format for being 2 herein.A disk partition is so elapsed backward daily, to number
For cycle is stored since the subregion that number is 1 again after 7 subregion, the data of such cycle storage intelligent substation.
Specific embodiment of the present invention is presented above, but the present invention is not limited to described embodiment.At this
It invents under the thinking provided, to the technology hand in above-described embodiment by the way of being readily apparent that those skilled in the art
Duan Jinhang transformation, replace, modification, and play the role of with the present invention in relevant art means it is essentially identical, realize hair
Improving eyesight it is also essentially identical, the technical solution formed in this way is finely adjusted above-described embodiment to be formed, this technical solution
It still falls in protection scope of the present invention.
Claims (3)
1. a kind of big data management method based on single hard disk multi partition, which is characterized in that the management method includes such as successively
Lower step：
Step 1：The hard disk being connected with intelligent substation fault wave recording device according to its capacity and is needed into the n for being divided into fixed capacity
These subregions are numbered a subregion according to certain coding rule；
Step 2：The least residue spatial value of each subregion is set, which is more than needs gathered data amount daily
Summation；
Step 3：By in the data received storage to disk partition, when data storage operations are in i-th of subregion, detect simultaneously
The residual capacity of i+1 subregion, and respective handling is made, to ensure that the storage of the second day data has enough partition spaces；
Step 4：When storing the data received, press from first subregion to a last subregion again to the rule of first subregion
Then, cycle realizes the storage of data；
Respective handling in the step 3 refers to being less than set least residue spatial value in i+1 subregion residual capacity
When, processing is formatted to it；
The time interval that storage is recycled in the step 4 is one day, only does data storage in a subregion daily.
2. a kind of big data management method based on single hard disk multi partition according to claim 1, which is characterized in that described
Certain coding rule refers to that the sequence since natural number 1 according to natural number is numbered from small to large in step 1.
3. a kind of big data management method based on single hard disk multi partition according to claim 2, which is characterized in that described
Number is 1,2,3 ..., and m, m are natural numbers.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510581298.4A CN105117177B (en) | 2015-09-14 | 2015-09-14 | A kind of big data management method based on single hard disk multi partition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510581298.4A CN105117177B (en) | 2015-09-14 | 2015-09-14 | A kind of big data management method based on single hard disk multi partition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105117177A CN105117177A (en) | 2015-12-02 |
| CN105117177B true CN105117177B (en) | 2018-08-10 |
Family
ID=54665180
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510581298.4A Expired - Fee Related CN105117177B (en) | 2015-09-14 | 2015-09-14 | A kind of big data management method based on single hard disk multi partition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105117177B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108616583B (en) * | 2018-04-16 | 2020-12-15 | 亿享不道张家口信息科技有限公司 | Storage space allocation method based on computer cloud |
| CN112346663B (en) * | 2020-11-26 | 2022-10-11 | 上海天旦网络科技发展有限公司 | Multi-partition rotation data writing and clearing method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102662854A (en) * | 2012-03-27 | 2012-09-12 | 许继集团有限公司 | Storage method based on high-capacity multiple hard disks |
| CN103064633A (en) * | 2012-12-13 | 2013-04-24 | 广东威创视讯科技股份有限公司 | Data storage method and device |
| CN104268088A (en) * | 2014-09-15 | 2015-01-07 | 郑州天迈科技股份有限公司 | Vehicle DVR (Digital Video Recorder) hard disk data storage method |
| CN104615381A (en) * | 2015-01-18 | 2015-05-13 | 浙江宇视科技有限公司 | Redundancy array of independent disks of video monitoring system |
-
2015
- 2015-09-14 CN CN201510581298.4A patent/CN105117177B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102662854A (en) * | 2012-03-27 | 2012-09-12 | 许继集团有限公司 | Storage method based on high-capacity multiple hard disks |
| CN103064633A (en) * | 2012-12-13 | 2013-04-24 | 广东威创视讯科技股份有限公司 | Data storage method and device |
| CN104268088A (en) * | 2014-09-15 | 2015-01-07 | 郑州天迈科技股份有限公司 | Vehicle DVR (Digital Video Recorder) hard disk data storage method |
| CN104615381A (en) * | 2015-01-18 | 2015-05-13 | 浙江宇视科技有限公司 | Redundancy array of independent disks of video monitoring system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105117177A (en) | 2015-12-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110806986B (en) | Method, equipment and storage medium for improving message storage efficiency of network chip | |
| CN108733344A (en) | Data read-write method, device and circle queue | |
| CN104731569B (en) | A kind of data processing method and relevant device | |
| CN101944119B (en) | Real-time event management method for intelligent electronic equipment | |
| CN105677742A (en) | Method and apparatus for storing files | |
| CN103019957B (en) | A kind of power failure record ripple storage means and device | |
| CN105676023A (en) | Recording method and system of fault recording data | |
| CN104699422A (en) | Determination method and determination device of cache data | |
| CN105653474A (en) | Coarse-grained dynamic reconfigurable processor-oriented configuration cache controller | |
| CN105117177B (en) | A kind of big data management method based on single hard disk multi partition | |
| CN104317770A (en) | Data storage structure and data access method for multiple core processing system | |
| CN103309626A (en) | Method for realizing multi-read-write port memorizer of network chip and corresponding memorizer | |
| CN102937956A (en) | Method and device for storing real-time messages in intelligent substation | |
| CN102629235A (en) | Method for increasing read-write speed of double data rate (DDR) memory | |
| CN103902471B (en) | Data buffer storage treating method and apparatus | |
| CN105138278B (en) | A kind of the naked of intelligent substation network message takes inventory method for storing | |
| CN107704567A (en) | A kind of analytic method of binary file, device, equipment and storage medium | |
| CN102737171B (en) | Method and device for storing virus file and storage equipment | |
| CN103095595A (en) | Network data management method based on one-way parallel multiple chain lists and system thereof | |
| CN107332725B (en) | Method for rapidly analyzing PCAP message | |
| CN105488047A (en) | Metadata read-write method and device | |
| CN106775480A (en) | A kind of real-time monitoring system data recording method based on LabVIEW | |
| CN103049221A (en) | Method and device for processing disk array cache memory flash | |
| CN104808108A (en) | Large data volume high-speed communication wave recording system and method based on electric system | |
| CN102193868B (en) | Data stack storage circuit and microcontroller |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180810 Termination date: 20190914 |