CN102567548A - Streaming data pre-reading method for network file system - Google Patents
Streaming data pre-reading method for network file system Download PDFInfo
- Publication number
- CN102567548A CN102567548A CN2012100396813A CN201210039681A CN102567548A CN 102567548 A CN102567548 A CN 102567548A CN 2012100396813 A CN2012100396813 A CN 2012100396813A CN 201210039681 A CN201210039681 A CN 201210039681A CN 102567548 A CN102567548 A CN 102567548A
- Authority
- CN
- China
- Prior art keywords
- read
- buffer
- length
- read request
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a streaming data pre-reading method for a network file system. The method comprises the following steps that: 1, initialization is performed; 2, a client receives a reading request from a user; 3, the client transmits a current pre-reading request to a server; 4, when receiving a new reading request, the client firstly judges whether the buffer has the data needed for the reading request, if so, the client directly returns some existing data to the user, and packages the other data, which is not in the buffer, into a new reading request to request for executing the step 4 again; if not, a step 5 is executed; 5, the client judges whether the range of the request is in the range of the current pre-reading request, if so, the client firstly transmits a next pre-reading request, and then collects the currently pre-read data, stores the currently pre-read data in the buffer, returns back the data needed for the user, setting the next pre-reading as the current pre-reading, and returning back to the step 4. In comparison with the prior art, the method provided by the invention has the advantages of improving the throughput of data reading, and the like.
Description
Technical field
The present invention relates to a kind of computer network correlation technique, especially relate to a kind of stream data pre-head method that is used for NFS.
Background technology
NFS is that a network on the file system is abstract, and it allows a computing machine (client) to come to be positioned at the file on the remote computer (server) through access to netwoks with the similar mode of local file system.
Client is initiated read-write requests, its service routine through providing on the network call remote computer.The IO of NFS request not only comprised data on network transmission course but also comprised the disk I process of server end.And network and disk influence IO bottleneck of performance factor often.How to excavate the performance potential of network and disk to greatest extent, always be very active subject for a long time.File is looked ahead, claims again to read in advance, and one of most important optimisation technique just, and become the function of an indispensability of modern operating system.And for NFS, that uses for reference local file system reads thought in advance, reads strategy in advance in customer end adopted, is a kind of effective IO strategy equally.
Effective utilization and the data consistency problem of cache paid attention in present existing research; Like Collective Prefetching for Parallel I/O Systems []; NFS Tricks and Benchmarking Traps []; But for hiding the Network Transmission delay in the read procedure in advance how, the raising network file system performance often lacks further consideration.Through to research of NFS code and packet capturing analysis, find that the disk I of reading in advance to fail Network Transmission and server end of its client walks abreast.These preparatory read requests are asynchronous with respect to client; That is to say after client is sent preparatory read request; Need not to block and wait for that these data return next step the work of control continuation that just can continue the process that obtains, wait when needing these wherein some data really and collect data again.But per twice but is synchronous process between reading in advance, and promptly client must be waited for and receive fully and just can give preparatory read request once after the current preparatory read data.Client receive preparatory read data during this period of time in, server end is not done any IO operation.
Summary of the invention
The object of the invention is exactly for the defective that overcomes above-mentioned prior art existence a kind of stream data pre-head method that is used for NFS that improves the handling capacity of read data to be provided.
The object of the invention can be realized through following technical scheme:
A kind of stream data pre-head method that is used for NFS is characterized in that, may further comprise the steps:
The first step, initial work: client be provided with one be used for data that buffer memory reads from server end to come read buffer zone buffer;
Second step when client is received the read request that the user sends, judged whether to receiving read request first, and if yes, the user end to server end is transmitted read request, and will directly return to the user from the data that server end receives up; If do not carry out for the 4th step;
The 3rd step, the current preparatory read request that user end to server sends;
The 4th step; Client judges whether buffer exists the needed data of this read request earlier when new read request is received; If yes, directly return to the user to the partial data that exists, and the remaining data that does not have among the buffer is packaged into new read request carried out for the 4th step again; If, do not carry out for the 5th step.
In the 5th step, client judges whether the scope of this read request is included in the current preparatory read request scope, if yes; Then send next time read request in advance earlier; Collect the current data of reading in advance again, deposit buffer in, and return the required data of user; Currently read in advance read in advance to be changed to next time again, and returned for the 4th step; If, do not return for second step.
Described buffer comprises data side-play amount buffer.start and the length buffer.length hereof that is cushioned, and this length buffer.length is less than the big or small BUF_SIZE of buffer, and wherein BUF_SIZE can be provided with as required.
The client state of receiving read request first in described second step is: buffer is for sky and do not send out read request in advance before.
Read request in described second step and the 4th step comprises file handle read.fd, numbering read.no, side-play amount read.offset, the length read.length of request and is used to collect the user buffer address read.buf of data.
Current preparatory read request in described the 3rd step comprises file handle currentPrefetch.fd, skew currentPrefetch.offset and the length currentPretch.length of preparatory read request; The file handle of described preparatory read request is consistent with the file handle of read request; Be currentPrefetch.fd=read.fd; The skew of described preparatory read request is the length of the skew+current read request of current read request; Be currentPrefetch.offset=read.offset+read.length, currentPretch.length is for read the default value PREFETCH_SIZE of size in advance.
Buffer in described the 4th step exist the needed data of this read request be the data that comprise more than or equal to buffer of the side-play amount of read request hereof side-play amount and add the data length hereof that buffer comprises, i.e. read.offset＞=buffer.start and read.offset＜=buffer.start+buffer.length smaller or equal to the data side-play amount hereof that buffer comprises;
The described user's data that returns to is for being side-play amount with read.offset in the represented file of buffer, and (buffer.start+buffer.length-read.offset read.length) is the data of length to min.
Describedly the remaining data that does not have among the buffer be packaged into new read request be specially:
In the time of read.offset+read.length＞buffer.start+buffer.length, except that returning to user's data, the read request of remaining packing data Cheng Xin
read.fd＝read.fd；read.no＝read.no+1；
read.length＝read.offset+read.length-(buffer.start+buffer.length)；
read.offset＝buffer.start+buffer.length。
Read request scope in described the 5th step is included in preparatory read request scope and is:
The side-play amount of read request equals the length of side-play amount+read request of side-play amount and read request of preparatory read request smaller or equal to the side-play amount of read request in advance+the in advance length of read request, i.e. read.offset=currentPrefetch.offset and read.offset+read.length＜=currentPrefetch.offset+currentPrefetch.length.
The file handle that preparatory next time read request of described the 5th step is this new preparatory read request and the file handle of current preparatory read request are consistent; Request number is current preparatory read request number+1; Skew is the length of the skew+current preparatory read request of current preparatory read request; Length is the default length of preparatory read request, promptly
nextPrefetch.fd＝currentPrefetch.fd；nextPrefetch.no＝currentPrefetch.no+1；
nextPrefetch.offset＝currentPrefetch.offset+currentPrefetch.length；
nextPrefetch.length＝PREFETCH_SIZE。
The data that deposit among the buffer in described the 5th step are meant the data that current preparatory read request reads, i.e. buffer.start=currentPrefetch.offset; Buffer.length=currentprefetch.length.
Compared with prior art; The present invention has to existing NFS read in advance to fail in the strategy shortcoming that walks abreast Network Transmission and server end disk I; Send preparatory read request and the order that receives preparatory read data through changing; Make Network Transmission and disk I walk abreast, improved the handling capacity of read data like this, be specially:
1) Interleave (interlocking) Network Transmission and disk visit realizes postponing to hide;
When client need receive the data of current preparatory read request; If it just receives data immediately and then sends the preparatory read request of next time; The IO of server end does not just get into smooth when its receives data so, and server must wait for that client harvests the preparatory read request that data send next time then fully and just can carry out the disk I operation.Advantage of the present invention is when client need receive the data of current preparatory read request; Earlier send to server end to preparatory read request next time; Can give client the data transmission of current preparatory read request through Network Transmission so on the one hand; Server can carry out the IO operation owing to early received preparatory read request next time on the one hand, has realized fully postponing to hide.
2) a plurality of requests are sent successively, eliminate the magnetic head interference phenomenon that a plurality of requests cause;
In order to improve performance, the network file system server end often adopts the mode of multi-threaded I, but has also brought some problems.Be regarded as random request to the order request of client such as meeting, influenced server end read in advance the strategy and the disk tracking.Because our preparatory read request is not to send to server end together but send successively, has reduced this sequential read request to greatest extent and has been regarded as the situation of read request at random, has effectively utilized the IO characteristic of disk.
Description of drawings
Fig. 1 is a process flow diagram of the present invention;
Fig. 2 is the existing pre-head method figure that taken time;
Fig. 3 is the pre-head method of the present invention figure that taken time.
Embodiment
Below in conjunction with accompanying drawing and specific embodiment the present invention is elaborated.
Embodiment
Client in the present embodiment is a (SuSE) Linux OS, and the NFSv3 CLIENT PROGRAM is installed, and service end also is a (SuSE) Linux OS, and the NFSv3 server program is installed.
Concrete steps shown in 1,
The first step, initial work: be provided with one and read buffer zone buffer and be used for buffer memory and read the data of coming from server end, and the content of buffer all is initialized as 0, the size of buffer is 1MB.The content side-play amount hereof of institute's buffer memory is buffer.offset=0, and length is buffer.length=0.Default read big or small PREFETCH_SIZE=1MB in advance.
In second step, client is received the read request first time that application is sent, such as
read.fd＝1；read.no＝1；read.offset＝0；read.length＝1MB，
Because read.offset+read.length=0+1MB=1MB＞buffer.length=0; Explain and do not use required data in the buffer; And do not send out read request (currentPreftch=NULL) in advance before yet; So just transmit read request, and coming directly to return to application on the Data Receiving to server end.
The 3rd step, send for the first time read request in advance, be 1MB such as the current length of read request in advance, then
currentPrefetch.fd＝read.fd＝1；currentPrefetch.no＝1；
currentPrefetch.offset＝read.offset+read.length＝0+1MB＝1MB；
currentPrefetch.length＝PREFETCH_SIZE＝1MB
The 4th step when new read request arrives, was read.fd=1 such as this read request; Read.no=2; Read.offset=1MB, read.length=512KB.Earlier inspection buffer sees the data that whether need because read.offset+read.length=1MB+512KB=1.5MB＞buffer.length=0, explain do not have in the buffer need data.Carry out next step.
In the 5th step, whether the scope of inspection read request is included in the current request scope of reading in advance, because
Read.offset=1MB=currentPrefetch.offset and
Read.length=512KB＜=currentPrefetch.length=1MB explains in scope, sends preparatory read request nextPrefetch.fd=1 next time earlier; NextPrefetch.no=currentPrefetch.no+1=1+1=2; NextPrefetch.offset=currentPrefetch.offset+currentPrefet ch.length=1MB+1MB=2MB, nextPrefetch.length=PREFETCH_SIZE=1MB.
On the current Data Receiving of reading in advance, deposit buffer in, then again
Buffer.start=currentPrefetch.offset=1MB; Buffer.length=currentPrefetch.length=1MB, and a required data of application is returned.Currently read in advance read in advance to be changed to next time again; Be currentPrefetch.no=nextPrefetch, currentPrefetch.fd=1 then, currentPrefetch.no=2; CurrentPrefetch.offset=2MB, currentPrefetch.length=1MB.
The 6th step when new read request arrives, was read.fd=1 such as this read request; Read.no=3; Read.offset=1.5MB, read.length=1MB.
Inspection buffer sees the data that whether need earlier; Because read.offset=1.5MB＞=buffer.start=1MB and read.offset＜=buffer.start+buffer.length=1MB+1MB=2MB; The data that buffer has to be needed are described, are directly returned, and because
Read.offset+read.length=1.5MB+1MB=1.5MB＞buffer.start+buffer.length=2MB need be the read request read.fd=read.fd=1 of unnecessary packing data Cheng Xin; Read.no=read.no+1=4;
read.length＝read.offset+read.length-(buffer.start+buffer.length)＝1.5MB+1MB-(1MB+1MB)＝512KB；
read.offset＝buffer.start+buffer.length＝1MB+1MB＝2MB。Again carry out this step.
In the 7th step, this new read request is read.fd=1, read.no=4, read.offset=2MB, read.length=512KB.Inspection buffer sees the data that whether need earlier, because
Read.offset+read.length=2MB+512KB=2.5MB＞buffer.start+buffer.length=1MB+1MB=2MB explains the data that do not have needs in the buffer.Carry out next step.
In the 8th step, whether the scope of inspection read request is included in the current request scope of reading in advance, because
Read.offset=2MB=currentPrefetch.offset and
Read.length=512KB＜=currentPrefetch.length=1MB explains in scope, sends preparatory read request next time earlier
nextPrefetch.fd＝currentPrefetch.fd＝1；
nextPrefetch.no＝currentPrefetch.no+1＝2+1＝3；
nextPrefetch.offset＝currentPrefetch.offset+currentPrefetch.length＝2MB+1MB＝3MB，nextPrefetch.length＝PREFETCH_SIZE＝1MB。
On the current Data Receiving of reading in advance, deposit buffer in again,
Buffer.start=currentPrefetch.offset=2MB then; Buffer.length=currentPrefetch.length=1MB, and a required data of application is returned.Currently read in advance read in advance to be changed to next time again; Be currentPrefetch.no=nextPrefetch, currentPrefetch.fd=1 then, currentPrefetch.no=3; CurrentPrefetch.offset=3MB, currentPrefetch.length=1MB.
Carry out down according to execution in step successively.
Fig. 2 and Fig. 3 are the advantages that the present invention compares original pre-head method.Wherein horizontal ordinate is a time shaft.
Can find out that from Fig. 2, Fig. 3 suppose that the time of the preparatory read request of transmission is T1, the time that server end is received preparatory read data is T3 (network latency), the time that server end is done disk I is T2.Wherein T1＜＜T2, T1＜＜T3, can ignore.
Then according to traditional pre-head method, the time of need reading in advance for n time is n* (T1+T2+T3).
And according to method of the present invention need the time be 2*T1+T2+T3+ (n-1) * max (T2, T3).
Claims (10)
1. a stream data pre-head method that is used for NFS is characterized in that, may further comprise the steps:
The first step, initial work: client be provided with one be used for data that buffer memory reads from server end to come read buffer zone buffer;
Second step when client is received the read request that the user sends, judged whether to receiving read request first, and if yes, the user end to server end is transmitted read request, and will directly return to the user from the data that server end receives up; If do not carry out for the 4th step;
The 3rd step, the current preparatory read request that user end to server sends;
The 4th step; Client judges whether buffer exists the needed data of this read request earlier when new read request is received; If yes, directly return to the user to the partial data that exists, and the remaining data that does not have among the buffer is packaged into new read request carried out for the 4th step again; If, do not carry out for the 5th step;
In the 5th step, client judges whether the scope of this read request is included in the current preparatory read request scope, if yes; Then send next time read request in advance earlier; Collect the current data of reading in advance again, deposit buffer in, and return the required data of user; Currently read in advance read in advance to be changed to next time again, and returned for the 4th step; If, do not return for second step.
2. a kind of stream data pre-head method that is used for NFS according to claim 1; It is characterized in that; Described buffer comprises data side-play amount buffer.start and the length buffer.length hereof that is cushioned; This length buffer.length is less than the big or small BUF_SIZE of buffer, and wherein BUF_SIZE can be provided with as required.
3. a kind of stream data pre-head method that is used for NFS according to claim 1 is characterized in that, the client state of receiving read request first in described second step is: buffer is for sky and do not send out read request in advance before.
4. a kind of stream data pre-head method that is used for NFS according to claim 2; It is characterized in that the read request in described second step and the 4th step comprises file handle read.fd, numbering read.no, side-play amount read.offset, the length read.length of request and is used to collect the user buffer address read.buf of data.
5. a kind of stream data pre-head method that is used for NFS according to claim 4; It is characterized in that; Current preparatory read request in described the 3rd step comprises file handle currentPrefetch.fd, skew currentPrefetch.offset and the length currentPretch.length of preparatory read request; The file handle of described preparatory read request is consistent with the file handle of read request; Be currentPrefetch.fd=read.fd; The skew of described preparatory read request is the length of the skew+current read request of current read request, i.e. currentPrefetch.offset=read.offset+read.length, and currentPretch.length is for read the default value PREFETCH_SIZE of size in advance.
6. a kind of stream data pre-head method that is used for NFS according to claim 5; It is characterized in that; Buffer in described the 4th step exist the needed data of this read request be the data that comprise more than or equal to buffer of the side-play amount of read request hereof side-play amount and add the data length hereof that buffer comprises, i.e. read.offset＞=buffer.start and read.offset＜=buffer.start+buffer.length smaller or equal to the data side-play amount hereof that buffer comprises;
The described user's data that returns to is for being side-play amount with read.offset in the represented file of buffer, and (buffer.start+buffer.length-read.offset read.length) is the data of length to min.
7. a kind of stream data pre-head method that is used for NFS according to claim 6 is characterized in that, describedly the remaining data that does not have among the buffer is packaged into new read request is specially:
In the time of read.offset+read.length＞buffer.start+buffer.length, except that returning to user's data, the read request of remaining packing data Cheng Xin
read.fd＝read.fd；read.no＝read.no+1；
read.length＝read.offset+read.length-(buffer.start+buffer.length)；
read.offset＝buffer.start+buffer.length。
8. a kind of stream data pre-head method that is used for NFS according to claim 7 is characterized in that, the read request scope in described the 5th step is included in preparatory read request scope and is:
The side-play amount of read request equals the length of side-play amount+read request of side-play amount and read request of preparatory read request smaller or equal to the side-play amount of read request in advance+the in advance length of read request, i.e. read.offset=currentPrefetch.offset and read.offset+read.length＜=currentPrefetch.offset+currentPrefetch.length.
9. a kind of stream data pre-head method that is used for NFS according to claim 8; It is characterized in that; Described the 5th step next time in advance read request be that the file handle of file handle and current preparatory read request of this new preparatory read request is consistent, to ask number be current preparatory read request number+1, skew is the length of the skew+current preparatory read request of current preparatory read request; Length is the default length of preparatory read request, promptly
nextPrefetch.fd＝currentPrefetch.fd；nextPrefetch.no＝currentPrefetch.no+1；
nextPrefetch.offset＝currentPrefetch.offset+currentPrefetch.length；
nextPrefetch.length＝PREFETCH_SIZE。
10. a kind of stream data pre-head method that is used for NFS according to claim 9; It is characterized in that; The data that deposit among the buffer in described the 5th step are meant the data that current preparatory read request reads, i.e. buffer.start=currentPrefetch.offset; Buffer.length=currentprefetch.length.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012100396813A CN102567548A (en) | 2012-02-21 | 2012-02-21 | Streaming data pre-reading method for network file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012100396813A CN102567548A (en) | 2012-02-21 | 2012-02-21 | Streaming data pre-reading method for network file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102567548A true CN102567548A (en) | 2012-07-11 |
Family
ID=46412946
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012100396813A Pending CN102567548A (en) | 2012-02-21 | 2012-02-21 | Streaming data pre-reading method for network file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102567548A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105487987A (en) * | 2015-11-20 | 2016-04-13 | 深圳市迪菲特科技股份有限公司 | Method and device for processing concurrent sequential reading IO (Input/Output) |
| CN105653684A (en) * | 2015-12-29 | 2016-06-08 | 曙光云计算技术有限公司 | Pre-reading method and device of distributed file system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5623699A (en) * | 1994-12-06 | 1997-04-22 | Thunderwave, Inc. | Read only linear stream based cache system |
| CN101382955A (en) * | 2008-09-28 | 2009-03-11 | 中国科学院计算技术研究所 | File reading method in cluster file system and system |
-
2012
- 2012-02-21 CN CN2012100396813A patent/CN102567548A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5623699A (en) * | 1994-12-06 | 1997-04-22 | Thunderwave, Inc. | Read only linear stream based cache system |
| CN101382955A (en) * | 2008-09-28 | 2009-03-11 | 中国科学院计算技术研究所 | File reading method in cluster file system and system |
Non-Patent Citations (1)
| Title |
|---|
| 闫鹤，李小勇，胡鹏，刘海涛: "分布式文件系统的流式数据预读", 《计算机研究与发展》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105487987A (en) * | 2015-11-20 | 2016-04-13 | 深圳市迪菲特科技股份有限公司 | Method and device for processing concurrent sequential reading IO (Input/Output) |
| CN105487987B (en) * | 2015-11-20 | 2018-09-11 | 深圳市迪菲特科技股份有限公司 | A kind of concurrent sequence of processing reads the method and device of IO |
| CN105653684A (en) * | 2015-12-29 | 2016-06-08 | 曙光云计算技术有限公司 | Pre-reading method and device of distributed file system |
| CN105653684B (en) * | 2015-12-29 | 2020-03-03 | 曙光云计算集团有限公司 | Pre-reading method and device of distributed file system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Guo et al. | Clio: A hardware-software co-designed disaggregated memory system | |
| Han et al. | {MegaPipe}: A New Programming Interface for Scalable Network {I/O} | |
| US10764202B2 (en) | Container-based mobile code offloading support system in cloud environment and offloading method thereof | |
| WO2020078314A1 (en) | Method and apparatus for accelerating cold-starting of application, and terminal | |
| WO2013044829A1 (en) | Data readahead method and device for non-uniform memory access | |
| CN103795781A (en) | Distributed cache model based on file prediction | |
| CN111182008B (en) | Establishing socket connections in user space | |
| US11132221B2 (en) | Method, apparatus, and computer-readable medium for dynamic binding of tasks in a data exchange | |
| CN110196681A (en) | Data in magnetic disk write-in control method and device, the electronic equipment of business write operation | |
| CN106257456A (en) | The method of data base's stability, Apparatus and system is improved under high concurrent request | |
| CN108090003A (en) | A kind of method, the system of the promotion WEB server performance based on zero-copy | |
| Goldenberg et al. | Zero copy sockets direct protocol over infiniband-preliminary implementation and performance analysis | |
| Hruby et al. | On Sockets and System Calls: Minimizing Context Switches for the Socket {API} | |
| Trivedi et al. | A case for RDMA in clouds: turning supercomputer networking into commodity | |
| Banerjee et al. | NUMA aware I/O in virtualized systems | |
| Nordal et al. | Paravirtualizing tcp | |
| CN102567548A (en) | Streaming data pre-reading method for network file system | |
| CN101826031A (en) | Implementation method for capturing PCM (Pulse Code Modulation) stream based on Linux system | |
| CN102291298B (en) | Efficient computer network communication method oriented to long message | |
| Trahay et al. | A multithreaded communication engine for multicore architectures | |
| Svärd et al. | The Noble Art of Live VM Migration-Principles and Performance of precopy, postcopy and hybrid migration of demanding workloads | |
| CN102223418A (en) | Distributed Cache system based on dynamic pipeline network server and working method thereof | |
| Li et al. | Improving spark performance with zero-copy buffer management and RDMA | |
| WO2023010879A1 (en) | Memory management method and apparatus, and computer device | |
| Han et al. | Command queue-aware host I/O stack for mobile flash storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20120711 |