CN105511811A - Method and system for raising throughput of file system - Google Patents
Method and system for raising throughput of file system Download PDFInfo
- Publication number
- CN105511811A CN105511811A CN201510890610.8A CN201510890610A CN105511811A CN 105511811 A CN105511811 A CN 105511811A CN 201510890610 A CN201510890610 A CN 201510890610A CN 105511811 A CN105511811 A CN 105511811A
- Authority
- CN
- China
- Prior art keywords
- write request
- file system
- file
- block
- place
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0613—Improving I/O performance in relation to throughput
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and system for increasing the throughput of a file system. The method comprises the steps of responding to a file writing request, pre-distributing a corresponding number of blocks in the file system according to the system digit number for storing data corresponding to the writing request, writing the data which correspond to the writing request and are stored in the blocks into a page cache memory, and synchronizing the data which correspond to the writing request and located in the page cache memory to storage hardware. According to the method and system for raising the throughput of the file system, the blocks are pre-distributed for the writing request, block distribution efficiency is improved, caching efficiency of the page cache memory is driven to be improved, read-write efficiency is further improved, and thereby the throughput of the file system is effectively raised.
Description
Technical field
The present invention relates to Computer Storage field, particularly relate to a kind of method and system improving file system handling capacity.
Background technology
File system is method and the data structure of the file of operating system on clear and definite memory device or subregion, be responsible for user and set up file, stored in, read, amendment, dump file, the access of control documents, cancel file etc. when user does not re-use, in general file system performance has sequential write, order is read, the performance index such as random write, random write.The performance index of a concrete file system are relevant with concrete hardware configuration, and the different various performance index of the mode that file system realizes when hardware configuration is identical are also different.
Along with developing rapidly of network application, network information data amount is increasing, and the handling capacity of file system is not high by the performance of the whole file system of impact.
Summary of the invention
In view of this, fundamental purpose of the present invention is to provide a kind of method and system improving file system handling capacity, effectively can improve the handling capacity of file system.
For achieving the above object, the invention provides a kind of method improving file system handling capacity, comprising:
Response file write request, stores data corresponding to described write request according to system figure place at the block of file system predistribution respective amount;
By data corresponding for the described write request stored in described piece write page cache memory;
By data syn-chronization corresponding for the described write request in described page cache memory to storage hardware.
Preferably, response file write request, also comprises according to the data that system figure place stores described write request corresponding at the block of file system predistribution respective amount:
Receive the described file write request of the client initiation sent through virtual file system.
Preferably, response file write request, comprises according to the data that system figure place stores described write request corresponding at the block of file system predistribution respective amount:
Response file write request, certainty annuity figure place;
For the block of file system predistribution and described system figure place equal number;
By data corresponding for described write request stored in described piece.
Present invention also offers a kind of system improving file system handling capacity, comprising:
Block forward allocator module, for response file write request, stores data corresponding to described write request according to system figure place at the block of file system predistribution respective amount;
Cache module, the data write page cache memory that the described write request for storing in described piece is corresponding;
Memory module, for by data syn-chronization corresponding for the described write request in described page cache memory to storage hardware.
Preferably, the system of described raising file system handling capacity also comprises:
Write request acquisition module, for receiving the described file write request of the client initiation sent through virtual file system.
Preferably, described piece of forward allocator module comprises:
System figure place determination submodule, for response file write request, certainty annuity figure place;
Block distribution sub module, for the block for file system predistribution and described system figure place equal number; Data sub module stored, for by data corresponding for described write request stored in described piece.
Apply a kind of method and system improving file system handling capacity provided by the invention, response file write request, data corresponding to described write request are stored at the block of file system predistribution respective amount according to system figure place, by data corresponding for the described write request stored in described piece write page cache memory, by data syn-chronization corresponding for the described write request in described page cache memory to storage hardware, for write request predistribution block, improve the allocative efficiency of block, drive the buffer efficiency improving page cache memory, further raising read-write efficiency, thus effectively improve the handling capacity of file system.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only embodiments of the invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to the accompanying drawing provided.
Fig. 1 is a kind of process flow diagram improving the embodiment of the method for file system handling capacity of the present invention;
To be that the present invention is a kind of improve the embodiment of the method client of file system handling capacity and the interaction concept schematic diagram of file system to Fig. 2;
To be that the present invention is a kind of improve the embodiment of the method file system of file system handling capacity and the structural principle schematic diagram of memory device to Fig. 3;
Fig. 4 is a kind of structural representation improving the system embodiment of file system handling capacity of the present invention;
Fig. 5 is a kind of detailed construction schematic diagram improving the system embodiment of file system handling capacity of the present invention;
Fig. 6 is a kind of another structural representation improving the system embodiment of file system handling capacity of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
The rear end storage of file system is generally all many disks, has physical disk, has disk array etc.Mention the readwrite performance of whole file system, finally all will be attributed to the performance of single physical disk.Single physical disk performance feature has: the performance of order IO is good, because do not need the seek time of disk; Random IO poor performance, fritter read-write IOPS per second is also with regard to about 100; I/O request length is larger, and handling capacity is higher; Limited by the physical arrangement of disk, the IO performance of disk sheet outer ring is better than inner ring.Disk array is the encapsulation to many disks, although performance characteristics is so obvious unlike single disk, general trend is still consistent.No matter be single disk or disk permutation, we are called storage, store and mainly contain two performance index, handling capacity and IOPS.Generally, IOPS refers to the IOPS of fritter IO, and handling capacity is measured with bulk IO, and wherein, handling capacity refers to the data total amount storing process per second, and IOPS refers to the total number of I/O request storing process per second.
Generally still have certain association, the better performances of storage between handling capacity and IOPS, IOPS is higher, and handling capacity is just higher, and vice versa.IOPS is higher, and the response time of I/O request is shorter.To be handling capacity or IOPS be in addition all, and IO more has order, higher, otherwise more random, lower.
Because order bulk writes the highest write performance determining file system, improved the highest write performance of file system by optimization order bulk write performance.
When client initiates a file write request, the vfs layer first through operating system sends to file system process, finally replies result to client.First the NAS service routine being sent to server end by network is needed for client NAS system, then could send to file system through vfs layer.
Can the utilization rate of storage just can be made to reach 100% during the read-write of order bulk, at this moment just can give play to the maximum throughput of hardware store.The utilization rate stored is 100%, refers to and is stored in any time all in work, do not have the idle time.As reached this target, need the read-write IO number of store queue to continue to be greater than 1, the utilization factor so stored just can reach 100%.The IO number of the queuing stored continues to be greater than 1, and that is file system sends lasting I/O request.Between file system and memory device, there is page cache memory PageCache, introducing the object of cache layer, is to improve IO efficiency.The efficiency how playing PageCache has just become the key improving handling capacity.
The invention provides a kind of method improving file system handling capacity, Fig. 1 shows the process flow diagram that the present invention improves the embodiment of the method for file system handling capacity, comprising:
Step S101: response file write request, stores data corresponding to described write request according to system figure place at the block of file system predistribution respective amount;
First the described file write request of the client initiation sent through Virtual File System vfs layer is received, the file write request of customer in response end, certainty annuity figure place, for the block of file system predistribution and described system figure place equal number, by data corresponding for described write request stored in described piece, the block block size that file system is general is 512 bytes, in block allocation table, usually represent whether a block dispenses with a bit.As the data of single treatment 64 for the system of 64, when each write request arrives, predistribution 64 block are to write request, so each preallocated space is 32k, although a part of space can be wasted like this, but improve the allocative efficiency of block, decrease file system fragmentation, client and file system mutual as shown in Figure 2.
Step S102: data corresponding for the described write request stored in described piece are write page cache memory;
By the data write page cache memory PageCache in block.
Step S103: by data syn-chronization corresponding for the described write request in described page cache memory to storage hardware.
By data syn-chronization corresponding for write request in PageCache to storage hardware, file system and memory device are as shown in Figure 3.
A kind of method improving file system handling capacity that application the present embodiment provides, response file write request, data corresponding to described write request are stored at the block of file system predistribution respective amount according to system figure place, by data corresponding for the described write request stored in described piece write page cache memory, by data syn-chronization corresponding for the described write request in described page cache memory to storage hardware, for write request predistribution block, improve the allocative efficiency of block, drive the buffer efficiency improving page cache memory, further raising read-write efficiency, thus effectively improve the handling capacity of file system.
Present invention also offers a kind of system improving file system handling capacity, Fig. 4 shows the structural representation that the present invention improves the system of file system handling capacity, comprising:
Block forward allocator module 101, for response file write request, stores data corresponding to described write request according to system figure place at the block of file system predistribution respective amount;
Cache module 102, is connected with described piece of forward allocator module 101, the data write page cache memory that the described write request for storing in described piece is corresponding;
Memory module 103, is connected with described cache module 102, for by data syn-chronization corresponding for the described write request in described page cache memory to storage hardware.
As shown in Figure 5, wherein, described piece of forward allocator module 101 can comprise:
System figure place determination submodule 201, for response file write request, certainty annuity figure place;
Block distribution sub module 202, is connected with described system figure place determination submodule 201, for the block for file system predistribution and described system figure place equal number;
Data sub module stored 203, is connected with described piece of distribution sub module 202, for by data corresponding for described write request stored in described piece.
As shown in Figure 6, corresponding to Fig. 4, the system that the present embodiment improves file system handling capacity also can comprise:
Write request acquisition module 100, is connected with described piece of forward allocator module 101, for receiving the described file write request of the client initiation sent through virtual file system.
A kind of system improving file system handling capacity that application the present embodiment provides, response file write request, data corresponding to described write request are stored at the block of file system predistribution respective amount according to system figure place, by data corresponding for the described write request stored in described piece write page cache memory, by data syn-chronization corresponding for the described write request in described page cache memory to storage hardware, for write request predistribution block, improve the allocative efficiency of block, drive the buffer efficiency improving page cache memory, further raising read-write efficiency, thus effectively improve the handling capacity of file system.
It should be noted that, each embodiment in this instructions all adopts the mode of going forward one by one to describe, and what each embodiment stressed is the difference with other embodiments, between each embodiment identical similar part mutually see.For system class embodiment, due to itself and embodiment of the method basic simlarity, so description is fairly simple, relevant part illustrates see the part of embodiment of the method.
Finally, also it should be noted that, in this article, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment comprising described key element and also there is other identical element.
Be described in detail method and system provided by the present invention above, apply specific case herein and set forth principle of the present invention and embodiment, the explanation of above embodiment just understands method of the present invention and core concept thereof for helping; Meanwhile, for one of ordinary skill in the art, according to thought of the present invention, all will change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention.
Claims (6)
1. improve a method for file system handling capacity, it is characterized in that, comprising:
Response file write request, stores data corresponding to described write request according to system figure place at the block of file system predistribution respective amount;
By data corresponding for the described write request stored in described piece write page cache memory;
By data syn-chronization corresponding for the described write request in described page cache memory to storage hardware.
2. the method for raising file system handling capacity according to claim 1, is characterized in that, response file write request, also comprises according to the data that system figure place stores described write request corresponding at the block of file system predistribution respective amount:
Receive the described file write request of the client initiation sent through virtual file system.
3. the method for raising file system handling capacity according to claim 1, is characterized in that, response file write request, comprises according to the data that system figure place stores described write request corresponding at the block of file system predistribution respective amount:
Response file write request, certainty annuity figure place;
For the block of file system predistribution and described system figure place equal number;
By data corresponding for described write request stored in described piece.
4. improve a system for file system handling capacity, it is characterized in that, comprising:
Block forward allocator module, for response file write request, stores data corresponding to described write request according to system figure place at the block of file system predistribution respective amount;
Cache module, the data write page cache memory that the described write request for storing in described piece is corresponding;
Memory module, for by data syn-chronization corresponding for the described write request in described page cache memory to storage hardware.
5. the system of raising file system handling capacity according to claim 1, is characterized in that, also comprise:
Write request acquisition module, for receiving the described file write request of the client initiation sent through virtual file system.
6. the system of raising file system handling capacity according to claim 1, is characterized in that, described piece of forward allocator module comprises:
System figure place determination submodule, for response file write request, certainty annuity figure place;
Block distribution sub module, for the block for file system predistribution and described system figure place equal number;
Data sub module stored, for by data corresponding for described write request stored in described piece.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510890610.8A CN105511811B (en) | 2015-12-07 | 2015-12-07 | A kind of method and system improving file system handling capacity |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510890610.8A CN105511811B (en) | 2015-12-07 | 2015-12-07 | A kind of method and system improving file system handling capacity |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105511811A true CN105511811A (en) | 2016-04-20 |
| CN105511811B CN105511811B (en) | 2018-09-07 |
Family
ID=55719831
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510890610.8A Active CN105511811B (en) | 2015-12-07 | 2015-12-07 | A kind of method and system improving file system handling capacity |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105511811B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107870843A (en) * | 2016-12-30 | 2018-04-03 | 平安科技（深圳）有限公司 | The method and device of nas server performance monitoring |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20010002480A1 (en) * | 1997-09-30 | 2001-05-31 | Lsi Logic Corporation | Method and apparatus for providing centralized intelligent cache between multiple data controlling elements |
| US6675264B2 (en) * | 2001-05-07 | 2004-01-06 | International Business Machines Corporation | Method and apparatus for improving write performance in a cluster-based file system |
| CN1553325A (en) * | 2003-06-05 | 2004-12-08 | 联想(北京)有限公司 | Method for transparent extending document system function |
| CN101256526A (en) * | 2008-03-10 | 2008-09-03 | 清华大学 | Method for implementing document condition compatibility maintenance in inspection point fault-tolerant technique |
| CN101789976A (en) * | 2010-01-27 | 2010-07-28 | 上海交通大学 | Embedded network storage system and method thereof |
| CN102567442A (en) * | 2011-10-25 | 2012-07-11 | 曙光信息产业(北京)有限公司 | Method for synchronizing metadata and disks in distributed file system |
| US8495250B2 (en) * | 2009-12-16 | 2013-07-23 | International Business Machines Corporation | Asynchronous file operations in a scalable multi-node file system cache for a remote cluster file system |
| CN103902474A (en) * | 2014-04-11 | 2014-07-02 | 华中科技大学 | Mixed storage system and method for supporting solid-state disk cache dynamic distribution |
| CN104317529A (en) * | 2014-10-21 | 2015-01-28 | 浪潮电子信息产业股份有限公司 | SW (Shenwei) CPU (central processing unit) based storage system software architecture |
-
2015
- 2015-12-07 CN CN201510890610.8A patent/CN105511811B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20010002480A1 (en) * | 1997-09-30 | 2001-05-31 | Lsi Logic Corporation | Method and apparatus for providing centralized intelligent cache between multiple data controlling elements |
| US6675264B2 (en) * | 2001-05-07 | 2004-01-06 | International Business Machines Corporation | Method and apparatus for improving write performance in a cluster-based file system |
| CN1553325A (en) * | 2003-06-05 | 2004-12-08 | 联想(北京)有限公司 | Method for transparent extending document system function |
| CN101256526A (en) * | 2008-03-10 | 2008-09-03 | 清华大学 | Method for implementing document condition compatibility maintenance in inspection point fault-tolerant technique |
| US8495250B2 (en) * | 2009-12-16 | 2013-07-23 | International Business Machines Corporation | Asynchronous file operations in a scalable multi-node file system cache for a remote cluster file system |
| CN101789976A (en) * | 2010-01-27 | 2010-07-28 | 上海交通大学 | Embedded network storage system and method thereof |
| CN102567442A (en) * | 2011-10-25 | 2012-07-11 | 曙光信息产业(北京)有限公司 | Method for synchronizing metadata and disks in distributed file system |
| CN103902474A (en) * | 2014-04-11 | 2014-07-02 | 华中科技大学 | Mixed storage system and method for supporting solid-state disk cache dynamic distribution |
| CN104317529A (en) * | 2014-10-21 | 2015-01-28 | 浪潮电子信息产业股份有限公司 | SW (Shenwei) CPU (central processing unit) based storage system software architecture |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107870843A (en) * | 2016-12-30 | 2018-04-03 | 平安科技（深圳）有限公司 | The method and device of nas server performance monitoring |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105511811B (en) | 2018-09-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11372544B2 (en) | Write type based crediting for block level write throttling to control impact to read input/output operations | |
| US9747318B2 (en) | Retrieving data in a storage system using thin provisioning | |
| US8996535B1 (en) | Extent hashing technique for distributed storage architecture | |
| US7882304B2 (en) | System and method for efficient updates of sequential block storage | |
| CN105573681B (en) | Method and system for establishing RAID in SSD | |
| US11593272B2 (en) | Method, apparatus and computer program product for managing data access | |
| CN104317742A (en) | Automatic thin-provisioning method for optimizing space management | |
| US20150113026A1 (en) | Rollover strategies in a n-bit dictionary compressed column store | |
| US11093410B2 (en) | Cache management method, storage system and computer program product | |
| US9158695B2 (en) | System for dynamically adaptive caching | |
| CN103116552A (en) | Method and device for distributing storage space in distributed type storage system | |
| CN107092442A (en) | Storage system resources distribution method and device | |
| CN104765575A (en) | Information storage processing method | |
| US10795597B2 (en) | Thinly provisioned disk drives with zone provisioning and compression in relation to zone granularity | |
| CN104765574A (en) | Data cloud storage method | |
| CN103514249A (en) | Method and system for automatic data reduction and storage device | |
| CN104536903A (en) | Mixed storage method and system for conducting classified storage according to data attributes | |
| CN103473266A (en) | Solid state disk and method for deleting repeating data thereof | |
| US11226778B2 (en) | Method, apparatus and computer program product for managing metadata migration | |
| CN107844436B (en) | Organization management method, system and storage system for dirty data in cache | |
| CN104778100A (en) | Safe data backup method | |
| CN109213423A (en) | Concurrent I/O command is handled without lock based on address barrier | |
| US11119703B2 (en) | Utilizing a set of virtual storage units distributed across physical storage units | |
| US11099740B2 (en) | Method, apparatus and computer program product for managing storage device | |
| CN115543187A (en) | Data processing method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |