CN105740413A - File movement method by FUSE on Linux platform - Google Patents
File movement method by FUSE on Linux platform Download PDFInfo
- Publication number
- CN105740413A CN105740413A CN201610064269.5A CN201610064269A CN105740413A CN 105740413 A CN105740413 A CN 105740413A CN 201610064269 A CN201610064269 A CN 201610064269A CN 105740413 A CN105740413 A CN 105740413A
- Authority
- CN
- China
- Prior art keywords
- fuse
- user
- file
- space
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
Abstract
The invention relates to a file movement method by FUSE on a Linux platform. The file movement method specifically comprises the following steps: using FUSE to obtain a page pointer of a user space read-write process in a kernel space; through the obtained page pointer, sharing a memory between the user space read-write process and a corresponding kernel space daemon process; and in the daemon process, changing the page pointer of the user space read-write process into the page pointer where a file which needs to move is positioned. The page pointer of the user space is obtained in the kernel space, and an effect that file movement is realized without copying data in the memory is realized through a way that memory data is shared between a read process and a write process.
Description
Technical field
The present invention relates to the file mobile field in user's space file system (FUSE, FilesysteminUserspace) under Linux platform, be specifically related to a kind of Linux platform uses the file moving method of the internal storage data zero-copy of FUSE.
Background technology
Linux server, owing to having good stability and opening, is supported various file system, thus is used increasingly extensive in the Internet.In Linux server, there is user's space file system (FUSE, FilesysteminUserspace) function.When user uses FUSE, during by application program to underlying device transmission data, it is necessary to repeatedly copying data at user's space and kernel spacing, this can consume memory bandwidth and CPU bandwidth, having a strong impact on the execution efficiency of system, final result is to cause that readwrite performance declines.
Additionally, along with multimedia development, in Linux server, the demand of mobile big file is on the increase.When response is to the mobile demand of big file, use adaptive direct IO accelerated method can improve the performance that order is read and write big file.But adaptive direct IO accelerated method also can make the read-write efficiency dramatic drop-off of random small documents, destroy the overall read-write of server harmonious, it is impossible to meet the business demand moving big file and small documents fast and efficiently simultaneously.
Summary of the invention
The technical problem to be solved in the present invention is in that, drawbacks described above for prior art, the file providing a kind of shared drive data moves scheme, constantly switch between user space and kernel space when overcoming prior art, cause CPU bandwidth and memory bandwidth exorbitant expenditure, system execution efficiency is low, causes the defect that readwrite performance declines, and completely without the support of hardware.
The technical solution adopted for the present invention to solve the technical problems is: use the file moving method of FUSE in a kind of Linux platform, described Linux platform includes user's space and kernel spacing, described Linux platform uses the file moving method of FUSE include: use FUSE to obtain the page pointer of user's space read-write process in kernel spacing；By acquired page pointer, read and write shared drive between process and corresponding kernel spacing finger daemon at described user's space；In described finger daemon, the page pointer that described user's space is read and write process is rewritten as the page pointer at the file place needing movement, and the file to realize memory headroom zero-copy moves.
Described Linux platform uses the further improvement of the file moving method of FUSE, the described process comprising the steps: user application unlatching read-write FUSE in kernel spacing acquisition user's space page pointer process, after being called by system, enters kernel spacing；In kernel spacing, FUSE obtains the page pointer that the parameter that passes over of user's space is corresponding with data buffer zone, and is delivered in the request queue of FUSE after described page pointer is packaged into request；The service object of FUSE opens and reads and writes, with user application place, the finger daemon that process is corresponding, obtains that generated by described read-write process from kernel spacing and has added request pending described FUSE queue；Described finger daemon gets required corresponding requests by queue waiting mechanism.
Described Linux platform uses the further improvement of the file moving method of FUSE, uses " get_user_pages " operation to obtain the page pointer of described user's space in kernel state, the page pointer in user space processes is write in described FUSE queue.
Described Linux platform uses the further improvement of the file moving method of FUSE, the step of described shared drive includes: kernel state call " mmap " operation carry out internal storage data share, and directly data to be moved are filled to the internal storage data relief area that described read-write process is shared with described finger daemon according to the purpose of described request, it is achieved the file of internal storage data zero-copy moves.
Using the further improvement of the file moving method of FUSE in described Linux platform, the step of described shared drive also includes: data head is also carried out data sharing, with the further data volume reducing copy.
Using the further improvement of the file moving method of FUSE in described Linux platform, the step of described shared drive also includes: described finger daemon completes in request after execution, returns to the read-write process in described user's space by performing result.
Described Linux platform uses the further improvement of the file moving method of FUSE, uses " mmap " operation to realize internal memory between described consumer process and finger daemon and share.
Described Linux platform uses the further improvement of the file moving method of FUSE, utilizes " remap_pfn_range " operation or page faults to realize described " mmap " operation.
Using the further improvement of the file moving method of FUSE in described Linux platform, the maximum data length of FUSE request is arranged by FUSE, is defaulted as 128KB.
The beneficial effects of the present invention is, by opening two processes, transmit page pointer between the two processes, thus shared drive data, reach the effect of internal storage data zero-copy and mobile file.
The beneficial aspects of the present invention also resides in, it is not necessary to hardware supported DMA, it becomes possible to realizes the mobile file of zero memory copy, makes hardware cost reduce.
Other user file systems such as the present invention also can service at such as exfat, androidsdcard, distributed file system moosefs and application use.
Accompanying drawing explanation
Below in conjunction with drawings and Examples, the invention will be further described, in accompanying drawing:
Fig. 1 be ordinary file perform twice at data copy when moving realize block diagram；
Fig. 2 is the flow chart of file moving method one embodiment using FUSE in Linux platform of the present invention.
Fig. 3 is the refinement flow chart of file moving method one embodiment using FUSE in Linux platform of the present invention；
Fig. 4 be Linux platform of the present invention uses FUSE file moving method one embodiment realize block diagram.
Detailed description of the invention
In conjunction with accompanying drawing, presently preferred embodiments of the present invention is elaborated.
FUSE is that (SuSE) Linux OS allows user to realize the module of user-defined file system, realizes user-defined file system for user and provides one group of application programming interfaces (API).
Fig. 2 is the flow chart of file moving method one embodiment using FUSE in Linux platform of the present invention.FUSE is used to obtain the page pointer of user's space read-write process in kernel spacing；By acquired page pointer, read and write shared drive between process and corresponding kernel spacing finger daemon at described user's space；In described finger daemon, the page pointer that described user's space is read and write process is rewritten as the page pointer at the file place needing movement, and the file to realize memory headroom zero-copy moves.
In the present embodiment, described kernel spacing obtain user's space page pointer process can be: user application open read-write FUSE process, after being called by system enter kernel spacing；In kernel spacing, FUSE obtains the page pointer that the parameter that passes over of user's space is corresponding with data buffer zone, and is delivered in the request queue of FUSE after described page pointer is packaged into request；The service object of FUSE opens and reads and writes, with user application place, the finger daemon that process is corresponding, obtains that generated by described read-write process from kernel spacing and has added request pending described FUSE queue；Described finger daemon gets required corresponding requests by queue waiting mechanism.
In the present embodiment, the step of shared drive can be: kernel state call " mmap " operation carry out internal storage data share, and directly data to be moved are filled to the internal storage data relief area that described read-write process is shared with described finger daemon according to the purpose of described request, it is achieved the file of internal storage data zero-copy moves.In other embodiments, in order to reduce the data volume of copy further, data head is also carried out data sharing.Described finger daemon completes in request after execution, returns to the read-write process in described user's space by performing result.
Fig. 3 is the refinement flow chart of file moving method one embodiment using FUSE in Linux platform of the present invention；Fig. 4 be Linux platform of the present invention uses FUSE file moving method one embodiment realize block diagram.Describe how the present invention realizes the effect of internal storage data zero-copy and mobile file below in conjunction with Fig. 3 and Fig. 4.As shown in Figure 4, when the mobile file of needs, first with FUSE carry space catalogue for object, open a consumer process A, start read-write operation, call entrance kernel spacing by system.After entering kernel spacing, FUSE uses " get_user_pages " operation to obtain the page pointer corresponding to user's space data buffering.After obtaining pointer, FUSE is packaged into request according to including the concrete operations such as parameter and page pointer, adds in request queue.
Opening another consumer process B, described process B is the service object of FUSE.Process B obtains the request in process A addition request queue by reading FUSE equipment.
After process B obtains request, FUSE equipment calls mmap is operated, libfuse process is called fuse_dev interface, use remap_pfn_range operation or the page pointer recorded in the request transmitted is shared in user's space by page faults.So, data, according to request purpose, are directly filled in shared buffering by process B, it is achieved that sharing the data being stored in internal memory in process B and process A.Complete share after, process B will perform result write FUSE equipment, FUSE equipment according to receive request execution result, execution result is returned to process A.Process A, after receiving execution result, terminates read-write operation.
Fig. 1 be in conventional methods where file perform twice at when moving data copy realize block diagram, according to Fig. 1, when using traditional method that file is moved, need first to be copied in kernel spacing by the file data of user's space, again file data is copied in another user's space by kernel spacing, completes a file and move and needed twice memory copying.
And in the present invention, utilize " get_user_pages " operation that the memory pages pointer of the read-write process of user's space passes to the finger daemon of kernel spacing, it is to avoid by user's space to kernel spacing copy number evidence；Subsequently, utilize mmap to operate, between read-write process and finger daemon, achieve internal memory share, it is to avoid copied data by kernel spacing to another user's space.
Measuring through reality, the amplitude of sequential write performance boost is: 20%～30%, and when application operating packet more hour, after optimization, the ratio of performance boost is more big.So, the space that random write speed promotes will be bigger.One common U disk of actual test, after optimization, random write performance improves nearly 1 times.The random read-write experimental data being storage medium with common U disk:
Additionally it is also tested under Android system and uses apk to operate the behavior pattern of ntfs subregion, use " little Bai file manager " to copy a 256MB file in an external ntfs zoned format USB flash disk, use primary FUSE, consuming time be 2 points 0 second, and use the present invention, consuming time it is only 1 point 18 seconds.Visible, when the present invention is applied in actual product, the lifting of performance is become apparent from.
When encapsulating FUSE request, it is necessary to consider the data length of request bag.If FUSE single request data length is too big, powered-off fault is easily lost too much data, and too little can affect FUSE performance.Accordingly, it is determined that the data length of FUSE request is extremely important.It is as follows that request data length changes the test result on read or write speed impact:
The order read-write experimental data being storage medium with 4TB Seagate portable hard drive:
Considering data safety and read or write speed and promote demand, the maximum data length of single request is given tacit consent at present and is controlled at 128K.
When described process A and described process B works simultaneously, it is necessary to use queue waiting mechanism to realize data syn-chronization.Queue waiting mechanism is provided as a kind of standard resource by current linux kernel, the present embodiment directly employs this queue waiting mechanism.
It should be appreciated that above example is only in order to illustrate technical scheme, it is not intended to limit, it will be understood by those skilled in the art that the technical scheme described in above-described embodiment can be modified, or wherein portion of techniques feature is carried out equivalent replacement；And all such modifications and replacement, all should belong to the protection domain of claims of the present invention.
Claims (9)
1. a Linux platform uses the file moving method of FUSE, it is characterised in that including:
FUSE is used to obtain the page pointer of user's space read-write process in kernel spacing；
By acquired page pointer, read and write shared drive between process and corresponding kernel spacing finger daemon at described user's space；
In described finger daemon, the page pointer that described user's space is read and write process is rewritten as the page pointer at the file place needing movement, and the file to realize memory headroom zero-copy moves.
2. Linux platform according to claim 1 uses the file moving method of FUSE, it is characterised in that described kernel spacing obtain user's space page pointer process comprise the steps:
User application opens the process of read-write FUSE, enters kernel spacing by system after being called；
In kernel spacing, FUSE obtains the page pointer that the parameter that passes over of user's space is corresponding with data buffer zone, and is delivered in the request queue of FUSE after described page pointer is packaged into request；
The service object of FUSE opens and reads and writes, with user application place, the finger daemon that process is corresponding, obtains that generated by described read-write process from kernel spacing and has added request pending described FUSE queue；
Described finger daemon gets required corresponding requests by queue waiting mechanism.
3. Linux platform according to claim 2 uses the file moving method of FUSE, it is characterized in that, use " get_user_pages " operation to obtain the page pointer of described user's space in kernel state, the page pointer in user space processes is write in described FUSE queue.
4. Linux platform according to claim 2 uses the file moving method of FUSE, it is characterised in that the step of described shared drive includes:
Kernel state call " mmap " operation carry out internal storage data share, and directly data to be moved are filled to the internal storage data relief area that described read-write process is shared with described finger daemon according to the purpose of described request, it is achieved the file of internal storage data zero-copy moves.
5. Linux platform according to claim 4 uses the file moving method of FUSE, it is characterised in that the step of described shared drive also includes: data head is also carried out data sharing, with the further data volume reducing copy.
6. Linux platform according to claim 4 uses the file moving method of FUSE, it is characterised in that the step of described shared drive also includes:
Described finger daemon completes in request after execution, returns to the read-write process in described user's space by performing result.
7. Linux platform according to claim 1 uses the file moving method of FUSE, it is characterised in that use " mmap " operation to realize internal memory between described consumer process and finger daemon and share.
8. Linux platform according to claim 7 uses the file moving method of FUSE, it is characterised in that utilize " remap_pfn_range " operation or page faults to realize described " mmap " operation.
9. Linux platform according to claim 1 uses the file moving method of FUSE, it is characterised in that the maximum data length of FUSE request is arranged by FUSE, is defaulted as 128KB.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610064269.5A CN105740413A (en) | 2016-01-29 | 2016-01-29 | File movement method by FUSE on Linux platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610064269.5A CN105740413A (en) | 2016-01-29 | 2016-01-29 | File movement method by FUSE on Linux platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105740413A true CN105740413A (en) | 2016-07-06 |
Family
ID=56247126
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610064269.5A Pending CN105740413A (en) | 2016-01-29 | 2016-01-29 | File movement method by FUSE on Linux platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105740413A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107239236A (en) * | 2017-06-27 | 2017-10-10 | 北京小米移动软件有限公司 | Method for writing data and device |
| CN109308170A (en) * | 2018-09-11 | 2019-02-05 | 北京北信源信息安全技术有限公司 | A kind of data processing method and device |
| CN109800190A (en) * | 2019-01-22 | 2019-05-24 | 星辰天合（北京）数据科技有限公司 | Memory accelerated processing method and device, storage medium and processor |
| CN110334069A (en) * | 2019-07-10 | 2019-10-15 | 中国民航信息网络股份有限公司 | Data sharing method and relevant apparatus between multi-process |
| CN110941595A (en) * | 2019-11-19 | 2020-03-31 | 北京奇艺世纪科技有限公司 | File system access method and device |
| CN114398318A (en) * | 2022-03-25 | 2022-04-26 | 广东统信软件有限公司 | File operation method of user space file system and user space file system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103945456A (en) * | 2014-05-12 | 2014-07-23 | 武汉邮电科学研究院 | LTE (long term evolution) base station user plane efficient UDP (user datagram protocol) data transmission optimization method based on Linux system |
| CN104301442A (en) * | 2014-11-17 | 2015-01-21 | 浪潮电子信息产业股份有限公司 | Method for achieving client of access object storage cluster based on fuse |
-
2016
- 2016-01-29 CN CN201610064269.5A patent/CN105740413A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103945456A (en) * | 2014-05-12 | 2014-07-23 | 武汉邮电科学研究院 | LTE (long term evolution) base station user plane efficient UDP (user datagram protocol) data transmission optimization method based on Linux system |
| CN104301442A (en) * | 2014-11-17 | 2015-01-21 | 浪潮电子信息产业股份有限公司 | Method for achieving client of access object storage cluster based on fuse |
Non-Patent Citations (2)
| Title |
|---|
| 曹婧华等: "分布式消息队列的设计与实现", 《河南科技大学学报:自然科学版》 * |
| 陈浩: "Linux 下用户态和内核态内存共享的实现", 《电脑编程技巧与维护》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107239236A (en) * | 2017-06-27 | 2017-10-10 | 北京小米移动软件有限公司 | Method for writing data and device |
| CN109308170A (en) * | 2018-09-11 | 2019-02-05 | 北京北信源信息安全技术有限公司 | A kind of data processing method and device |
| CN109800190A (en) * | 2019-01-22 | 2019-05-24 | 星辰天合（北京）数据科技有限公司 | Memory accelerated processing method and device, storage medium and processor |
| CN110334069A (en) * | 2019-07-10 | 2019-10-15 | 中国民航信息网络股份有限公司 | Data sharing method and relevant apparatus between multi-process |
| CN110334069B (en) * | 2019-07-10 | 2022-02-01 | 中国民航信息网络股份有限公司 | Data sharing method among multiple processes and related device |
| CN110941595A (en) * | 2019-11-19 | 2020-03-31 | 北京奇艺世纪科技有限公司 | File system access method and device |
| CN110941595B (en) * | 2019-11-19 | 2023-08-01 | 北京奇艺世纪科技有限公司 | File system access method and device |
| CN114398318A (en) * | 2022-03-25 | 2022-04-26 | 广东统信软件有限公司 | File operation method of user space file system and user space file system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105740413A (en) | File movement method by FUSE on Linux platform | |
| US10394580B2 (en) | Dynamic addition and removal of operating system components | |
| US20120017209A1 (en) | Optimizing a file system interface in a virtualized computing environment | |
| US10228993B2 (en) | Data dump for a memory in a data processing system | |
| US11243899B2 (en) | Forced detaching of applications from DMA-capable PCI mapped devices | |
| US20170270056A1 (en) | Main memory including hardware accelerator and method of operating the same | |
| US8572338B1 (en) | Systems and methods for creating space-saving snapshots | |
| US10983702B2 (en) | Reducing data storage system I/O bandwidth via read-once point in time copy | |
| US20140082275A1 (en) | Server, host and method for reading base image through storage area network | |
| US9207946B2 (en) | Auto-cloudifying applications via runtime modifications | |
| CN114691300A (en) | Hot migration method of virtual machine instance | |
| US20200371827A1 (en) | Method, Apparatus, Device and Medium for Processing Data | |
| US10467190B2 (en) | Tracking access pattern of inodes and pre-fetching inodes | |
| US10025607B2 (en) | Optimizing a file system interface in a virtualized computing environment | |
| US9405709B1 (en) | Systems and methods for performing copy-on-write operations | |
| US10901914B2 (en) | Method for writing multiple copies into storage device, and storage device | |
| JP2016515258A (en) | File aggregation for optimized file operation | |
| US10372347B2 (en) | Selectively limiting throughput of test objects that share system resources with production objects | |
| US20220155987A1 (en) | Performance of Dispersed Location-Based Deduplication | |
| US20170094502A1 (en) | Management method, management device and terminal for contacts in terminal | |
| US20180089086A1 (en) | Tracking access pattern of inodes and pre-fetching inodes | |
| JP2005018643A (en) | Access system to shared disk device on storage area network, and client therefor | |
| US11726885B2 (en) | Efficient method and system of intelligent deploying file level restore agent in multicloud | |
| US7974953B1 (en) | System and method for deletion of writeable PPIS | |
| KR101434794B1 (en) | The method and system for defending program hacking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160706 |
|
| RJ01 | Rejection of invention patent application after publication |