CN102063326A - System for testing file system capacity based on virtualization and method thereof - Google Patents
System for testing file system capacity based on virtualization and method thereof Download PDFInfo
- Publication number
- CN102063326A CN102063326A CN2010106205064A CN201010620506A CN102063326A CN 102063326 A CN102063326 A CN 102063326A CN 2010106205064 A CN2010106205064 A CN 2010106205064A CN 201010620506 A CN201010620506 A CN 201010620506A CN 102063326 A CN102063326 A CN 102063326A
- Authority
- CN
- China
- Prior art keywords
- module
- store module
- file system
- mapping
- virtual store
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention relates to a system for testing file system capacity based on virtualization and a method thereof. The system comprises a capacity testing managing module, a virtual storage module, a physical storage module and a mapping managing module, wherein the capacity testing managing module is used for establishing the virtual storage module for testing the file system capacity under the tested file system, and managing the file system to create, read and delete files on the virtual storage module; the virtual storage module is used for virtually storing the file written in by the capacity testing managing module; the physical storage module is used for actually storing the file written in the virtual storage module by the capacity testing managing module; and the mapping managing module is used for dynamically managing the mapping from the virtual storage module to the physical storage module in an operation system block device drive layer, and directly mapping the file operation of the virtual storage module into the file operation to the storage unit of the physical storage module so as to guarantee that the virtualized virtual storage module can be normally read and written, thereby effectively testing the file system capacity.
Description
Technical field
The present invention relates to a kind of test macro and method, specifically is a kind of test macro and method based on virtualized file system capacity.
Background technology
In order to store growing magnanimity information data, domestic and international research mechanism has proposed file system multiple unit or distributed, for example ext3 (full name is Third extended file system), Lustre (being developed by Linux and Clusters, is the brand-new file system that designs for solution mass memory problem) etc.The max cap. that these file system can be supported has nothing in common with each other, and for example ext3 is maximum supports 8TB (full name is TeraByte, terabyte), and Lustre supports 64PB (full name PetaByte) space in theory at least.Though the development of disk array has at present surpassed other restriction of TB level for a long time, will reach the restriction that the PB rank still has various factorss such as technology and cost generally speaking.Therefore when capacity that the test file system supports, need a kind of practicable method of testing, the key of this method is to want to provide to test required high capacity storage space.
According to the modern operating system framework, carry out virtually at the operating system bottom, making up virtual memory facilities is possible as hard disk etc.The virtual instrument of some similar memory devices has been arranged at present, as DiskSim and Device mapper etc.DiskSim comprises abundant assembly, various actions that can the emulation storage subsystem.But can not actual storage and reading of data by the virtual memory device that comes out of DiskSim, can't be used in the file system volume test.Another one instrument Device mapper can fictionalize the memory device of random capacity in theory, but the virtual store module is merely able to the normal read-write of part.This be since Device mapper in design just simply, the part block with the virtual store module has been mapped to physical storage devices one to one, cause having only mapped in the virtual store module block (physical block, equal 512B) can normally read and write, therefore the capacity of the capacity of actual memory device and file system to be measured must be in full accord, and jumbo memory device like this will be provided, all unrealistic on the technical and cost.。Therefore, above-mentioned tool and method all can't satisfy the requirement of file system volume test.
Summary of the invention
For this reason, to be solved by this invention is the technical matters that the existing virtual instrument of memory device can't fictionalize the memory device that can normally read and write, and a kind of test macro and method based on virtualized file system capacity of carrying out the normal read write data is provided.
For solving the problems of the technologies described above, the technical solution used in the present invention is as follows:
A kind of test macro based on virtualized file system capacity comprises:
The volume test administration module, under tested described file system, create out the virtual store module that is used for described file system volume test, and manage described file system comprises establishment, read-write and deletion on described virtual store module file operation;
The virtual store module has the max cap. that described file system can be supported, is used for the file that the described volume test administration module of virtual store writes;
The physical store module is used for the file that the described volume test administration module of actual storage writes described virtual store module;
The mapping management module, the described virtual store module of dynamic management will be mapped directly to the file operation to the storage unit of described physical store module to the file operation of described virtual store module to the mapping of described physical store module in operating system block device Drive Layer; Limit the capacity that all file sizes that write described virtual store module are less than or equal to described physical store module.
Described mapping management module realizes described dynamic management by a mapping table; Described mapping table stores the mapping relations of the storage unit of the storage unit of described virtual store module and described physical store module, and described mapping management module realizes inquiry, distribution and the recovery of mapping object space by the described mapping table of Dynamic Maintenance.
The storage unit of described virtual store module is mapped to the storage unit of described physical store module, the size of mapping block by described volume test administration module according to test needs adjustment.
Described mapping management module is by increasing or remove entries, dynamic assignment or dynamically reclaim described physical memory cell.
Simultaneously, provide a kind of method of testing virtualized file system capacity, comprise the steps,
1. create the virtual store module, it has the max cap. that described file system can be supported, is used for the file that the described file system of virtual store is created;
2. the physical store module is set, and it has predetermined volumes, is used for the file that the described file system of actual storage is created;
3. the volume test administration module is managed described file system comprises establishment, read-write and deletion to described virtual store module file operation;
4. by dynamically mapping, will directly be mapped as file operation to the file operation that described virtual store module is carried out to described physical store module.
Described mapping management module realizes described dynamic management by a mapping table; Described mapping table stores the mapping relations of the storage unit of the storage unit of described virtual store module and described physical store module, and described mapping management module realizes inquiry, distribution and the recovery of mapping object space by the described mapping table of Dynamic Maintenance.
The storage unit of described virtual store module is mapped to the storage unit of described physical store module, the size of map unit by described volume test administration module according to test needs adjustment.
Described mapping management module is by increasing or remove entries, dynamic assignment or dynamically reclaim described physical memory cell.
Create and reading and writing of files in, described volume test administration module specify a plurality of described files to take respectively at least the virtual store module storage space begin place, middle locating and end.
Technique scheme of the present invention has the following advantages compared to existing technology:
Test macro based on virtualized file system capacity of the present invention, comprise: the volume test administration module, under tested described file system, create out the virtual store module that is used for described file system volume test, manage described file system comprises establishment, read-write and deletion to described virtual store module file operation; The virtual store module has the max cap. that described file system can be supported, is used for the file that the described volume test administration module of virtual store writes; The physical store module is used for the file that the described volume test administration module of actual storage writes described virtual store module; The mapping management module, the described virtual store module of dynamic management will be mapped directly to the file operation to the storage unit of described physical store module to the file operation of described virtual store module to the mapping of described physical store module in operating system block device Drive Layer; Limit the capacity that all file sizes that write described virtual store module are less than or equal to described physical store module; And then pass through of the mapping of the realization of the mapping mode as required virtual store module of dynamic and intelligent to the physical store module, the virtual store module that assurance fictionalizes can normally be read and write, and is convenient to test the capacity of jumbo file system.
Description of drawings
For the easier quilt of content of the present invention is clearly understood, below according to a particular embodiment of the invention and in conjunction with the accompanying drawings, the present invention is further detailed explanation, wherein
Fig. 1 be in the one embodiment of the invention virtual store module to the mapping hierarchy chart of physical store module in operating system;
Fig. 2 is the volume test environment configurations figure of middle distributed file system Lustre embodiment illustrated in fig. 1;
Fig. 3 is a data flow synoptic diagram in the volume test process of Lustre shown in Fig. 2;
Fig. 4 is the synoptic diagram of mapping relations as required of dynamic and intelligent among the present invention;
Fig. 5 is a mapping relations corresponding mapping table shown in Fig. 4.
Embodiment
As virtual store module in the one embodiment of the invention as shown in Figure 1 to the mapping layer aggregated(particle) structure of physical store module in operating system, file system volume test administration module moves at user's space, can on designated equipment, carry out operations such as document creation, read-write and deletion according to POSIX (abbreviation of Portable Operating System Interface of Unix, expression portable operating system interface) file system access interface.Provide POSIX normative document access interface function by Virtual File System VFS (full name is Virtual FileSystem) in the kernel spacing.Under the VFS file system to be tested such as Journal File System ext3, distributed file system Lustre etc.Under file system to be measured, be the block device Drive Layer, the virtual store module is operated in this layer, and provides the dynamic and intelligent mapping of virtual store module to the actual physical storage module by this layer.Block device is physical store module drive and physical store module under driving.
In the block device Drive Layer, take mapping as required (the Intelligent on-demand dynamic mapping) mode of dynamic and intelligent from the virtual store module to the mapping of actual physical storage module.Virtual in this way described virtual store module of coming out can be read and write to a certain extent normally.The mapping mode as required of dynamic and intelligent mainly is meant mapping table of operating system Dynamic Maintenance, has write down the concrete mapping relations between the storage unit of the storage unit of virtual store module and physical store module in this table.And, have only and when the user applies data space, just carry out mapped amount of physical memory distribution, when user's deleted file, reclaim amount of physical memory.By inquiry, increase and the deletion mapping item, when the test and management module is passed through file system read-write virtual store module, read write command and data stream etc. is redirected to the physical store module.The virtual store module that adopts above-mentioned technology to realize is readable writing, and has guaranteed the demand of file system volume test, and adopted this mapping method that the virtual memory facilities that can fictionalize random capacity in theory is used for the file system volume test.
File system volume test administration module is created, is read and write on the virtual store module and operation such as deletion, because the effect of mapping table, in fact data storage is on the physical store module.Because file system only needs a spot of physics storage space of depositing to carry out readwrite tests in concrete test, therefore the virtual store module can have enough big storage space, and the physical store module of actual storage data only need satisfy file system formatization, and the required a little space of operation such as creates, writes.
Therefore, adopt above-mentioned technology, in the file system volume test of reality, can fictionalize the storage space of random capacity in theory, and the capacity of virtual store module and mapping block size be adjustable parameter, greatly strengthened the dirigibility of test.Though the physical store module capacity is less, but owing to take the mapping mode as required of dynamic and intelligent, as long as the data total amount that test procedure writes to the virtual store module is no more than the active volume of physical store module at every turn, the virtual store module is just as normal readable the writing of common physical store module.Because above-mentioned mapping process all occurs in the operating system nucleus driving, in file system, what be directly used in the storage data is jumbo virtual store module, and normal readable writing, and has so just reached the purpose of volume test.
The Lustre distributed file system is a kind of distributed file system of increasing income of present widespread use, supports the storage space of 64PB at least in theory.Fig. 2 is the typical test environment of a Lustre distributed file system, and this test environment is made up of the Ethernet (GE/10GE) or the fiber optic network (FC) of client (clientl-clientx), meta data server (MDS), object storage server (OSD) and connection the said equipment.The last runs file system volume test of client administration module is equivalent among Fig. 1 at user's space visit Lustre file system and managing virtual memory device.MDS provides the metadata service, and OSD provides physical storage device.On object storage server OSD, by virtual method shown in Figure 1, fictionalize the virtual memory facilities of enough capacity, and adopt the mapping as required of dynamic and intelligent, file system volume test administration module is mapped to the actual physical memory module to the read-write of virtual store module.
Need in a plurality of file operations in memory allocated space create, write and format etc., representative capacity positions such as the beginning of the storage space that described volume test administration module specifies a plurality of described files to take respectively to comprise the virtual store module located, middle place and end, to take at least described virtual store module storage space begin place and end.By the selection of above-mentioned particular memory space, to effective covering of volume space, and then guarantee the correct, reliable of test result when guaranteeing the file system volume test.As file operation success as described in a plurality of, judge that then tested file system can support the max cap. of nominal.
Fig. 3 is illustrated in the flow direction of data in the actual file system volume test.File system volume test administration module carries out POSIX file operation (as create, read and write etc.) at user's space.The direct control of mentioned order and data is to liking the virtual store module, and still after bottom was through the conversion of unit maps table, actual operand was the physical store module.By mapping table read write command and data stream etc. is redirected to the physical store module, the virtual store module of creating out like this is readable writing, and has guaranteed the demand of file system volume test.
Fig. 4 and Fig. 5 represent the synoptic diagram of mapping mode as required of the dynamic and intelligent from the virtual store module to the physical store module.The mapping as required of dynamic and intelligent is embodied in following several aspect: at first, the elementary cell of mapping can be the physical block of memory module, also can be a map unit that defines according to testing requirement, is map unit of 4k, 1M, 2M etc. such as capacity.Secondly, have only and when the user applies data space, just carry out mapped amount of physical memory distribution, when user's deleted file, reclaim amount of physical memory.The 3rd, the unit records of the physical store module of the storage unit of mapped virtual store module and conduct mapping target is in dynamic mapping table, and the process of above-mentioned distribution and recovery mapping space just is reduced to the increase and the delete procedure of mapping table list item.The mapping mode as required of the dynamic and intelligent by has as shown in Figure 4 guaranteed that limited physical capacity can be fully utilized, and can fictionalize the virtual store module of random capacity in theory.In this case, described physical store module only need have the required capacity of test and get final product, and the capacity of general described physical store module is no more than 1% of max cap. that file system can support.
Adopt above-mentioned method of testing, the virtual memory facilities that can fictionalize enough capacity is used for the volume test of Lustre file system.Because in concrete file system volume test, can not also need not accomplish all tests of covering of having living space, and only need small quantity of memory space to carry out readwrite tests, therefore as long as the data total amount that writes to the virtual store module of test procedure is no more than the active volume of mapped physical store module, the virtual store module is just as normal readable the writing of physical store module.Because above-mentioned mapping process all occurs in the operating system block device Drive Layer, in the Lustre file system, what be directly used in the storage data is jumbo virtual store module, and normal readable writing, and has so just reached the purpose of Lustre file system volume test.
Obviously, the foregoing description only is for example clearly is described, and is not the qualification to embodiment.For those of ordinary skill in the field, can also make other changes in different forms on the basis of the above description.Here need not also can't give exhaustive to all embodiments.And conspicuous variation of being extended out thus or change still are among the protection domain of the invention.
Claims (9)
1. the test macro based on virtualized file system capacity is characterized in that, comprising:
The volume test administration module, under tested described file system, create out the virtual store module that is used for described file system volume test, and manage the file operation that comprises establishment, read-write and deletion that described file system is carried out on described virtual store module;
The virtual store module has the max cap. that described file system can be supported, is used for the file that the described volume test administration module of virtual store writes;
The physical store module is used for the file that the described volume test administration module of actual storage writes described virtual store module;
The mapping management module, the described virtual store module of dynamic management will be mapped directly to the file operation to described physical store module to the file operation of described virtual store module to the mapping of described physical store module in operating system block device Drive Layer; Limit the capacity that all file sizes that write described virtual store module are less than or equal to described physical store module.
2. test macro according to claim 1 is characterized in that: described mapping management module realizes described dynamic management by a mapping table; Described mapping table stores the mapping relations of the storage unit of the storage unit of described virtual store module and described physical store module, and described mapping management module realizes inquiry, distribution and the recovery of mapped physical space by the described mapping table of Dynamic Maintenance.
3. test macro according to claim 2 is characterized in that: the storage unit of described virtual store module is mapped to the storage unit of described physical store module, the size of map unit by described volume test administration module according to test needs adjustment.
4. test macro according to claim 3 is characterized in that: described mapping management module is by increasing or remove entries, dynamic assignment or dynamically reclaim described physical memory cell.
5. the method for testing based on virtualized file system capacity is used the arbitrary described system of claim 1-4, it is characterized in that: comprise the steps,
1. create the virtual store module, it has the max cap. that described file system can be supported, is used for the file that the described file system of virtual store is created;
2. the physical store module is set, and it has predetermined volumes, is used for the file that the described file system of actual storage is created;
3. the volume test administration module is managed the file operation that comprises establishment, read-write and deletion that described file system carries out on described virtual store module, described volume test administration module specify a plurality of described files to take respectively at least the virtual store module storage space begin place and end;
4. by dynamically mapping, will directly be mapped as file operation to the file operation that described virtual store module is carried out to described physical store module;
5. file operation success, then the file system of judgement test can be supported the described max cap. of nominal.
6. method of testing according to claim 5 is characterized in that: described mapping management module realizes described dynamic management by a mapping table; Described mapping table stores the mapping relations of the storage unit of the storage unit of described virtual store module and described physical store module, and described mapping management module realizes inquiry, distribution and the recovery of mapping object space by the described mapping table of Dynamic Maintenance.
7. method of testing according to claim 6 is characterized in that: the storage unit of described virtual store module is mapped to described physical store module stores unit, the size of map unit by described volume test administration module according to test needs adjustment.
8. method of testing according to claim 7 is characterized in that: described mapping management module is by increasing or remove entries, dynamic assignment or dynamically reclaim described physical memory cell.
9. method of testing according to claim 8, it is characterized in that: create and write in the operating process of a plurality of files comprising, described volume test administration module specify a plurality of described files to take respectively at least the virtual store module storage space begin place, middle place and end.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010620506 CN102063326B (en) | 2010-12-31 | 2010-12-31 | System for testing file system capacity based on virtualization and method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010620506 CN102063326B (en) | 2010-12-31 | 2010-12-31 | System for testing file system capacity based on virtualization and method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102063326A true CN102063326A (en) | 2011-05-18 |
| CN102063326B CN102063326B (en) | 2013-01-02 |
Family
ID=43998612
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010620506 Expired - Fee Related CN102063326B (en) | 2010-12-31 | 2010-12-31 | System for testing file system capacity based on virtualization and method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102063326B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102215129A (en) * | 2011-06-15 | 2011-10-12 | 中兴通讯股份有限公司 | Service module extraposition method, device and system |
| CN102291466A (en) * | 2011-09-05 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Method for optimizing cluster storage network resource configuration |
| CN105701409A (en) * | 2015-12-29 | 2016-06-22 | 北京金山安全软件有限公司 | Method and device for realizing file virtualization |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1963782A (en) * | 2006-11-28 | 2007-05-16 | 北京中星微电子有限公司 | Method and system for testing embeded file system |
| CN101201776A (en) * | 2006-12-14 | 2008-06-18 | 英业达股份有限公司 | Method for integrating and testing platform of multiple operating systems |
| CN101706787A (en) * | 2009-11-20 | 2010-05-12 | 深圳市中兴移动通信有限公司 | File system management method for logic memory |
-
2010
- 2010-12-31 CN CN 201010620506 patent/CN102063326B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1963782A (en) * | 2006-11-28 | 2007-05-16 | 北京中星微电子有限公司 | Method and system for testing embeded file system |
| CN101201776A (en) * | 2006-12-14 | 2008-06-18 | 英业达股份有限公司 | Method for integrating and testing platform of multiple operating systems |
| CN101706787A (en) * | 2009-11-20 | 2010-05-12 | 深圳市中兴移动通信有限公司 | File system management method for logic memory |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102215129A (en) * | 2011-06-15 | 2011-10-12 | 中兴通讯股份有限公司 | Service module extraposition method, device and system |
| CN102291466A (en) * | 2011-09-05 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Method for optimizing cluster storage network resource configuration |
| CN102291466B (en) * | 2011-09-05 | 2014-02-26 | 浪潮电子信息产业股份有限公司 | Method for optimizing cluster storage network resource configuration |
| CN105701409A (en) * | 2015-12-29 | 2016-06-22 | 北京金山安全软件有限公司 | Method and device for realizing file virtualization |
| CN105701409B (en) * | 2015-12-29 | 2019-05-17 | 珠海豹趣科技有限公司 | A kind of method and device for realizing file virtualization |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102063326B (en) | 2013-01-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8504797B2 (en) | Method and apparatus for managing thin provisioning volume by using file storage system | |
| US10521131B2 (en) | Storage apparatus and storage control apparatus | |
| CN102405460B (en) | Virtual storage system and operation method thereof | |
| US8909887B1 (en) | Selective defragmentation based on IO hot spots | |
| US9311015B2 (en) | Storage system capable of managing a plurality of snapshot families and method of operating thereof | |
| US8793466B2 (en) | Efficient data object storage and retrieval | |
| US8473698B2 (en) | Converting LUNs into files or files into LUNs in real | |
| CN102023809B (en) | Storage system, method for reading data from storage system and method for writing data to storage system | |
| US8793290B1 (en) | Metadata management for pools of storage disks | |
| CN103514249B (en) | A kind of automatic data reduction and system and storage device | |
| US20140351515A1 (en) | Providing data attributes to a storage manager to use to select a storage tier to use for a data set management operation | |
| US20130124785A1 (en) | Data deleting method and apparatus | |
| US20140351505A1 (en) | Providing data attributes to a storage manager to use to select a storage tier to use for a data set management operation | |
| US20090265519A1 (en) | Method and system for power aware i/o scheduling | |
| US8621134B2 (en) | Storage tiering with minimal use of DRAM memory for header overhead | |
| JP2006293981A (en) | Database storing method, and database storing system | |
| US20140019706A1 (en) | System and method of logical object management | |
| US20140181455A1 (en) | Category based space allocation for multiple storage devices | |
| CN102982182B (en) | Data storage planning method and device | |
| Appuswamy et al. | Integrating flash-based SSDs into the storage stack | |
| CN108399050B (en) | Data processing method and device | |
| CN104536903A (en) | Mixed storage method and system for conducting classified storage according to data attributes | |
| US20200133543A1 (en) | Locality-aware, memory-efficient, time-efficient hot data identification using count-min-sketch for flash or streaming applications | |
| CN110147203A (en) | A kind of file management method, device, electronic equipment and storage medium | |
| CN102063326B (en) | System for testing file system capacity based on virtualization and method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C17 | Cessation of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130102 Termination date: 20131231 |