CN104571929B - A kind of read-write towards disk balances high-speed data acquisition queue storage method - Google Patents
A kind of read-write towards disk balances high-speed data acquisition queue storage method Download PDFInfo
- Publication number
- CN104571929B CN104571929B CN201310468279.1A CN201310468279A CN104571929B CN 104571929 B CN104571929 B CN 104571929B CN 201310468279 A CN201310468279 A CN 201310468279A CN 104571929 B CN104571929 B CN 104571929B
- Authority
- CN
- China
- Prior art keywords
- data
- disk
- metadata
- write
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
A kind of read-write towards disk balances high-speed data acquisition queue storage method, and it is related to data acquisition technology field, and its storage method is：In disk storage system, each magnetic track is divided into two halves, and the high-speed processing apparatus H of a certain capacity is added in conventional computer system；Magnetic head is fixed and does elevator motion；Metadata is written in H first with data, when X% of the data accumulation to half track capacity, starts write-in disk；After the metadata in H reaches to a certain degree, after can arranging, it is written on a magnetic track of disk, while discharging the space in H.Each magnetic track can be divided into two parts by it, and so writing can be with the tracking bandwidth of shared disk with reading.Magnetic head does periodicity elevator motion.In each magnetic track, write with reading to carry out as far as possible simultaneously.Current metadata information is dynamically retained according to acquisition speed and magnetic head movement velocity in high-speed processing apparatus.
Description
Technical field：
The present invention relates to data acquisition technology field, and in particular to a kind of read-write towards disk balances high-speed data acquisition
Queue storage method.
Background technology：
In the Internet, applications, data acquisition is a kind of universal application model.Data acquisition is usually from various
Sensor on by physical quantity by network transmission to server, being then written to file system or data by server
In storehouse." data acquisition " refers in network application, particularly in Internet of Things application, by the data of various sensor,
Pass through network transmission to server（Usually one computer）On, it is then written in the file system of server（Or data
In storehouse, write into Databasce is also equivalent to write file system）.The file system of server, is all to build to cut in temperature extensively at present
This special disk（Referred to as：Hard disk）On.
With the extensive use of Internet technology, particularly in the Internet of Things application of current widespread deployment, data volume is got over
Come bigger, the data of system acquisition must be synchronously written disk（It so just can ensure that data are not lost）, give server file system
Regiment commander carrys out huge load pressure.
In order to solve this problem, it is common practice to use the higher server of performance instead, or use expensive non-temperature
Chester storage disk.But, the physical arrangement of disk is limited to, high-grade server still uses same physical arrangement
Disk, still can not avoid " small documents synchronous write " problem.High-grade server has aobvious in all many-sided performances such as CPU and internal memory
Write and improve, but the limited performance of disk, in the rotary speed and seeking speed of disk, its performance can not be significantly improved.But use
Expensive FLASH disks, are limited to its cost performance well below Winchester disc, can not also widely use.
The major way of storage gathered data still uses disk on the server at present.And disk is limited to its machinery knot
Structure, when a large amount of gathered datas need to be synchronously written into disk, " the small data synchronous write " that is faced with huge is asked
Topic, that is to say, that a large amount of small datas（Small data refers to the usual capacity of data of collection less, and such as capacity is usually that several crossed joints are arrived
Hundreds of bytes）Disk is write simultaneously（Write-in refers to simultaneously：Data, which must be done as quickly as possible in, is written to disk, such ability
Ensure that gathered data is not lost）, disk will be caused to be absorbed in and frequently sought（Tracking refers to that the magnetic head of disk is write to reach needs
Enter the magnetic track of data, and the plenty of time that " coming and going ceaselessly " wastes）In, so that the performance of serious reduction disk, makes entirely to adopt
The performance of collecting system is drastically reduced.
The disk of current computer（That is Winchester disc）Structure all be using magnetic head tracking, disk rotation work
Mode.Mechanical structure and efficiency are limited to, the maximum speed of disk rotation is about 10,000 turns per minute（This limited speed is in machine
Tool structure, over nearly 20 years, basically can not be significantly improved）.This speed has reached the limit of Machine Design, it is difficult to
There is great raising.The speed of magnetic head tracking is about per magnetic track 10ms or so.This limited speed is in the electricity used on disk
The operation of motivation and the factor of machinery, it is also difficult to improve.
RAID is by University of California-Berkeley（University of California-Berkeley）
, the article delivered in 1987：“A Case for Redundant Arrays of Inexpensive Disks”.In article,
This vocabulary of RAID has been spoken of, and has defined RAID 5 levels.Bai Keli university research purposes are that to react CPU at that time quick
Performance.CPU efficiency about grows up 30~50% every year, and Hard Magnetic machine can only be into being about 7%.RAID can first solve disk failures
The problem of speed is lost, but the raising to performance is but very limited.And in RAID system, the performance of small data synchronous write
Bottleneck is still present, and very serious.For example, in RAID5, small data synchronous write will significantly reduce RAID5 performance.
The problem of existing disk face is in the technical scheme of small data synchronous write：
（1）Gathered data is broken generally into " metadata " and " data " two parts,（" metadata " can be understood as the rope of data
Draw, the ID of such as data, and " data " are only real physical values.）This two parts all must in " small data synchronous write " problem
It must be written on disk, could complete once " small data synchronous write ".If metadata is written on disk respectively with data
Different magnetic tracks, then computer must just seek different magnetic tracks, Ran Houxuan respectively when one " small data " is write
Suitable position is gone to, the write-in of metadata or data could be completed.This causes once " small data synchronous write ", can include two
Secondary tracking time delay, two rotational delay times.
（2）The tracking delay of disk is as the tracking distance of disk increases and increases.Therefore, it is necessary to reduce the tracking of disk
Distance, could reduce tracking delay.And in traditional data collecting system, it is not effective to ensure that " metadata " and " data "
Between track distances it is close.The content of the invention：
High-speed data acquisition queue storage method, its energy are balanced it is an object of the invention to provide a kind of read-write towards disk
Each magnetic track is divided into two parts, a part is when writing, and another part can be used for reading.So write with
Reading can be with the tracking bandwidth of shared disk.Magnetic head does periodicity elevator motion.In each magnetic track, write with reading as far as possible while entering
OK.Current metadata information is dynamically retained according to acquisition speed and magnetic head movement velocity in high-speed processing apparatus.This
Sample need not configure substantial amounts of high-speed processing apparatus.
In order to solve the problems existing in background technology, the present invention is to use following technical scheme：Its storage method is：
(1), in disk storage system, each magnetic track is divided into two halves, each using half sector, half be used for data write when
Wait, half is used for data read-out in addition.The high-speed processing apparatus H of a certain capacity is added in conventional computer system；
(2), magnetic head is fixed and does elevator motion；
(3), metadata and data are written in H first, and when X% of the data accumulation to half track capacity, startup is write
Enter disk, at this moment, if magnetic head is read, write read track writes area, if magnetic head is write, finds next empty
Area is write, it is on the schedule；X calculation：1-(（The road number * magnetic heads tracking average time averagely sought）* gathered data reaches speed
Degree)/(H track capacitys * 0.5)/2.
(4) when, not writing plan, when magnetic head carries out elevator motion, data are constantly read, and cleaned
Work, when having write-in plan, while reading other 50% reading data, is cleaned；
(5), after the metadata in H reaches to a certain degree, after can arranging, it is written on a magnetic track of disk, together
When release H in space, when from disk read metadata when, it is necessary to in H metadata carry out " merging ".
The present invention concrete operation method be：
(A) the high-speed processing apparatus H of a certain capacity, described high speed storing, are added in conventional computer system
Equipment H can also typically use now widely used FLASH disks, but be not limited only to right and wrong Winchester disc
FLASH disks；
(B) each cylinder of the disk on former computer, is divided into a subregion, disk has N cylinder, then disk
N number of subregion P is just divided into, meanwhile, corresponding that high-speed processing apparatus H is divided into N number of subregion Q, each P correspond to a Q；
(C), when collection be input to up to when, data syn-chronization wiring method is as follows：Gathered data reach server when
Wait, metadata and data are written in H respectively, after metadata is written in H with data, that is, completes data syn-chronization and write behaviour
Make；When the X% of the data accumulation in H to track capacity, start these data being written to disk, at this moment, if magnetic head is just
Read in certain cylinder, then write area W by what these data were written to the reading cylinder, if magnetic head is just write in certain cylinder, find magnetic head
Next empty cylinder that for can write of the direction of motion, write operation is on the schedule；This period to be written is being waited, is being continued to
Gathered data, continue to be stored in H, X numerical value is dynamic adjustment, method is as follows：X=1-(（The road number * magnetic averagely sought
Head tracking average time）* gathered data arrival rate)/(H track capacitys * 0.5)/2, in upper formula, divided by 2 purpose is
The data of sudden arrival are stored in order to ensure leaving enough spaces；
When gathered data is written to H, the 50% of cylinder is only written to, once write full 50%, remaining data
The other cylinder of write-in；
If the metadata capacity in H has exceeded the 20% of H capacity, it can reach the time earlier wherein metadata
After 10% arranges, it is written on disk, and discharge this segment space on H；
(D), the scheme that speed is read out from system：
If necessary to read data, metadata is searched in H first, if finding metadata, according to metadata token
Position, go to read information in disk；
If not finding metadata in H, go in the meta-data region stored in disk to search metadata；If disk
In meta-data region in also do not find metadata, then the data are not in systems；
When disk does not write plan, when magnetic head carries out elevator motion, data are constantly read, line number of going forward side by side
According to cleaning, when having write-in plan, if the cylinder has the data having been written into, while reading other 50% reading
According to being cleaned, cleaning is once complete, and just metadata of the change in H, represents that the data have been cleared by open system, such as
Fruit does not find the metadata of the data being eliminated in H, then or new metadata is written in H, in future from magnetic
When reading metadata just on disk, it is necessary to and the current metadata " merging " in H, when merging, with the member in H
Data are high priority, can cover the metadata read from disk.
The invention has the advantages that：
1st, each magnetic track is divided into two parts, and a part is when writing, and another part can be used for reading.
So writing can be with the tracking bandwidth of shared disk with reading.
2nd, magnetic head does periodicity elevator motion.In each magnetic track, write with reading to carry out as far as possible simultaneously.
3rd, current first number is retained dynamically according to acquisition speed and magnetic head movement velocity in high-speed processing apparatus H
It is believed that breath.Substantial amounts of high-speed processing apparatus H need not so be configured.
Embodiment：
Present embodiment uses following technical scheme：Its storage method is：(1), in disk storage system, each
Magnetic track is divided into two halves, each using the sector of half, when half is write for data, and half is read for data in addition
Go out.The high-speed processing apparatus H of a certain capacity is added in conventional computer system；
(2), magnetic head is fixed and does elevator motion；
(3), metadata and data are written in H first, and when X% of the data accumulation to half track capacity, startup is write
Enter disk, at this moment, if magnetic head is read, write read track writes area, if magnetic head is write, finds next empty
Area is write, it is on the schedule；X calculation：1-(（The road number * magnetic heads tracking average time averagely sought）* gathered data reaches speed
Degree)/(H track capacitys * 0.5)/2.
(4) when, not writing plan, when magnetic head carries out elevator motion, data are constantly read, and cleaned
Work, when having write-in plan, while reading other 50% reading data, is cleaned；
(5), after the metadata in H reaches to a certain degree, after can arranging, it is written on a magnetic track of disk, together
When release H in space, when from disk read metadata when, it is necessary to in H metadata carry out " merging ".
The concrete operation method of present embodiment is：
(A) the high-speed processing apparatus H of a certain capacity, described high speed storing, are added in conventional computer system
Equipment H can also typically use now widely used FLASH disks, but be not limited only to right and wrong Winchester disc
FLASH disks；
(B) each cylinder of the disk on former computer, is divided into a subregion, disk has N cylinder, then disk
N number of subregion P is just divided into, meanwhile, corresponding that high-speed processing apparatus H is divided into N number of subregion Q, each P correspond to a Q；
(C), when collection be input to up to when, data syn-chronization wiring method is as follows：Gathered data reach server when
Wait, metadata and data are written in H respectively, after metadata is written in H with data, that is, completes data syn-chronization and write behaviour
Make；When the X% of the data accumulation in H to track capacity, start these data being written to disk, at this moment, if magnetic head is just
Read in certain cylinder, then write area W by what these data were written to the reading cylinder, if magnetic head is just write in certain cylinder, find magnetic head
Next empty cylinder that for can write of the direction of motion, write operation is on the schedule；This period to be written is being waited, is being continued to
Gathered data, continue to be stored in H, X numerical value is dynamic adjustment, method is as follows：X=1-(（The road number * magnetic averagely sought
Head tracking average time）* gathered data arrival rate)/(H track capacitys * 0.5)/2, in upper formula, divided by 2 purpose is
The data of sudden arrival are stored in order to ensure leaving enough spaces；
When gathered data is written to H, the 50% of cylinder is only written to, once write full 50%, remaining data
The other cylinder of write-in；
If the metadata capacity in H has exceeded the 20% of H capacity, it can reach the time earlier wherein metadata
After 10% arranges, it is written on disk, and discharge this segment space on H；
(D), the scheme that speed is read out from system：
If necessary to read data, metadata is searched in H first, if finding metadata, according to metadata token
Position, go to read information in disk；
If not finding metadata in H, go in the meta-data region stored in disk to search metadata；If disk
In meta-data region in also do not find metadata, then the data are not in systems；
When disk does not write plan, when magnetic head carries out elevator motion, data are constantly read, line number of going forward side by side
According to cleaning, when having write-in plan, if the cylinder has the data having been written into, while reading other 50% reading
According to being cleaned, cleaning is once complete, and just metadata of the change in H, represents that the data have been cleared by open system, such as
Fruit does not find the metadata of the data being eliminated in H, then or new metadata is written in H, in future from magnetic
When reading metadata just on disk, it is necessary to and the current metadata " merging " in H, when merging, with the member in H
Data are high priority, can cover the metadata read from disk.
Each magnetic track can be divided into two parts by present embodiment, and a part is when writing, and in addition one
Individual part can be used for reading.So writing can be with the tracking bandwidth of shared disk with reading.Magnetic head does periodicity elevator motion.Each
Individual magnetic track, writes with reading to carry out as far as possible simultaneously.Speed is dynamically moved according to acquisition speed and magnetic head in high-speed processing apparatus H
Degree, retains current metadata information.Substantial amounts of high-speed processing apparatus H need not so be configured.
Claims (2)
1. a kind of read-write towards disk balances high-speed data acquisition queue storage method, it is characterised in that its storage method
For：(1), in disk storage system, each magnetic track is divided into two halves, each using the sector of half, is write in half for data
When, half is used for data read-out in addition, and the high-speed processing apparatus of a certain capacity is added in conventional computer system
H；
(2), magnetic head is fixed and does elevator motion；
(3), metadata and data are written in H first, when X% of the data accumulation to half track capacity, start write-in
Disk, at this moment, if magnetic head is read, write read track writes area, if magnetic head is write, finds next empty write
Area, it is on the schedule；
(4) when, not writing plan, when magnetic head carries out elevator motion, data are constantly read, and carry out scavenger
Make, when having write-in plan, while reading other 50% reading data, cleaned；
(5), after the metadata in H reaches to a certain degree, after can arranging, it is written on a magnetic track of disk, releases simultaneously
The space in H is put, it is necessary to carry out " merging " with the metadata in H when metadata is read from disk.
2. a kind of read-write towards disk balances high-speed data acquisition queue storage method, it is characterised in that its concrete operations side
Method is：
(A) the high-speed processing apparatus H, described high-speed processing apparatus H of a certain capacity, are added in conventional computer system
Now widely used FLASH disks can also typically be used, but be not limited only to FLASH with right and wrong Winchester disc
Disk；
(B) each cylinder of the disk on former computer, is divided into a subregion, disk has N number of cylinder, then disk is just drawn
It is divided into N number of subregion P, meanwhile, corresponding that high-speed processing apparatus H is divided into N number of subregion Q, each P correspond to a Q；
(C), when collection be input to up to when, data syn-chronization wiring method is as follows：When gathered data reaches server, point
Metadata and data are not written in H, after metadata is written in H with data, that is, data syn-chronization write operation is completed；Work as H
In data accumulation to track capacity X% when, start these data being written to disk, at this moment, if magnetic head is just at certain
Cylinder is read, then writes area W by what these data were written to the reading cylinder, if magnetic head is just write in certain cylinder, finds magnetic head motion
Next empty cylinder that for can write in direction, write operation is on the schedule；This period to be written is being waited, what is continued to adopts
Collect data, continue to be stored in H, X numerical value is dynamic adjustment；
When gathered data is written to H, the 50% of cylinder is only written to, once writing full 50%, remaining data is write
Enter other cylinder；
If the metadata capacity in H has exceeded the 20% of H capacity, wherein metadata can be reached to the time earlier 10%
After arrangement, it is written on disk, and discharges this segment space on H；
(D), the scheme that speed is read out from system：
If necessary to read data, metadata is searched in H first, if finding metadata, according to the position of metadata token
Put, go to read information in disk；
If not finding metadata in H, go in the meta-data region stored in disk to search metadata；If in disk
Also metadata is not found in meta-data region, then the data are not in systems；
When disk does not write plan, when magnetic head carries out elevator motion, data are constantly read, and it is clear to carry out data
Work is washed, when having write-in plan, if the cylinder has the data having been written into, while other 50% reading data are read,
Cleaned, cleaning is once complete, and just metadata of the change in H, represents that the data have been cleared by open system, if
The metadata of the data being eliminated is not found in H, then or new metadata is written in H, in future from disk
When reading metadata just, it is necessary to and the current metadata " merging " in H, when merging, with the metadata in H
For high priority, the metadata read from disk can be covered.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310468279.1A CN104571929B (en) | 2013-10-09 | 2013-10-09 | A kind of read-write towards disk balances high-speed data acquisition queue storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310468279.1A CN104571929B (en) | 2013-10-09 | 2013-10-09 | A kind of read-write towards disk balances high-speed data acquisition queue storage method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104571929A CN104571929A (en) | 2015-04-29 |
| CN104571929B true CN104571929B (en) | 2017-10-13 |
Family
ID=53088124
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310468279.1A Expired - Fee Related CN104571929B (en) | 2013-10-09 | 2013-10-09 | A kind of read-write towards disk balances high-speed data acquisition queue storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104571929B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6689242B2 (en) * | 2017-09-19 | 2020-04-28 | 株式会社東芝 | Magnetic disk device and magnetic head control method |
| CN110007868B (en) * | 2019-04-12 | 2022-07-22 | 苏州浪潮智能科技有限公司 | SSD disc metadata storage method, device, controller and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1564138A (en) * | 2004-03-26 | 2005-01-12 | 清华大学 | Fast synchronous and high performance journal device and synchronous writing operation method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050165856A1 (en) * | 2004-01-27 | 2005-07-28 | International Business Machines Corporation | System and method for autonomic performance enhancement of storage media |
-
2013
- 2013-10-09 CN CN201310468279.1A patent/CN104571929B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1564138A (en) * | 2004-03-26 | 2005-01-12 | 清华大学 | Fast synchronous and high performance journal device and synchronous writing operation method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104571929A (en) | 2015-04-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2361404B1 (en) | Method and system for queuing transfers of multiple non-contiguous address ranges with a single command | |
| US10303600B2 (en) | Method and storage device for collecting garbage data | |
| US9229653B2 (en) | Write spike performance enhancement in hybrid storage systems | |
| JP3347015B2 (en) | Adaptive localization method and apparatus for frequently accessed and randomly addressed data | |
| US8654472B2 (en) | Implementing enhanced fragmented stream handling in a shingled disk drive | |
| US9019643B2 (en) | Method and apparatus to reduce access time in a data storage device using coded seeking | |
| CN104461387B (en) | It is a kind of to improve method of the solid state hard disc to the reading performance of non-mapping area | |
| US8874875B2 (en) | ICC-NCQ command scheduling for shingle-written magnetic recording (SMR) Drives | |
| WO2012159863A1 (en) | Storage adapter performance optimization | |
| CN103080896A (en) | Reordering access to reduce total seek time on tape media | |
| CN1862476A (en) | Super large capacity virtual magnetic disk storage system | |
| CN103702057A (en) | Block storage algorithm applicable to multiple paths of concurrent-written stream media data | |
| US10095439B2 (en) | Tiered storage system, storage controller and data location estimation method | |
| CN105224473A (en) | The update method that a kind of solid state hard disc is data cached and device | |
| CN104571929B (en) | A kind of read-write towards disk balances high-speed data acquisition queue storage method | |
| CN103916459A (en) | Big data filing and storing system | |
| US8719235B2 (en) | Controlling tape layout for de-duplication | |
| CN107506146A (en) | A kind of data-storage system | |
| CN109144908A (en) | A kind of data-storage system and method based on cascade Expander | |
| CN102609486A (en) | Data reading/writing acceleration method of Linux file system | |
| CN101661378B (en) | Disk array 1 system and reading method for improving reading efficiency | |
| CN102779017A (en) | Control method of data caching area in solid state disc | |
| CN105404471B (en) | A kind of distribution cloud storage cache layer implementation method | |
| CN109683815A (en) | A kind of double control disk array bedding storage method | |
| KR100405110B1 (en) | Magnetic disk device and disk access method therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20171013 Termination date: 20211009 |