CN102063326B - System for testing file system capacity based on virtualization and method thereof - Google Patents
System for testing file system capacity based on virtualization and method thereof Download PDFInfo
- Publication number
- CN102063326B CN102063326B CN 201010620506 CN201010620506A CN102063326B CN 102063326 B CN102063326 B CN 102063326B CN 201010620506 CN201010620506 CN 201010620506 CN 201010620506 A CN201010620506 A CN 201010620506A CN 102063326 B CN102063326 B CN 102063326B
- Authority
- CN
- China
- Prior art keywords
- module
- store module
- file system
- mapping
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a system for testing file system capacity based on virtualization and a method thereof. The system comprises a capacity testing managing module, a virtual storage module, a physical storage module and a mapping managing module, wherein the capacity testing managing module is used for establishing the virtual storage module for testing the file system capacity under the tested file system, and managing the file system to create, read and delete files on the virtual storage module; the virtual storage module is used for virtually storing the file written in by the capacity testing managing module; the physical storage module is used for actually storing the file written in the virtual storage module by the capacity testing managing module; and the mapping managing module is used for dynamically managing the mapping from the virtual storage module to the physical storage module in an operation system block device drive layer, and directly mapping the file operation of the virtual storage module into the file operation to the storage unit of the physical storage module so as to guarantee that the virtualized virtual storage module can be normally read and written, thereby effectively testing the file system capacity.
Description
Technical field
The present invention relates to a kind of test macro and method, specifically a kind of test macro and method based on virtualized file system capacity.
Background technology
In order to store growing magnanimity information data, research institution both domestic and external has proposed file system multiple unit or distributed, for example ext3 (full name is Third extended file system), Lustre (being developed by Linux and Clusters, is the brand-new file system that designs for solution mass memory problem) etc.The max cap. that these file system can be supported is different, and for example ext3 is maximum supports 8TB (full name is TeraByte, terabyte), and Lustre supports 64PB (full name PetaByte) space in theory at least.Although the at present development of disk array has surpassed other restriction of TB level for a long time, will reach the restriction that the PB rank still has the various factorss such as technology and cost generally speaking.Therefore when capacity that the test file system supports, a kind of practicable method of testing need to be arranged, the key of the method is to want to provide to test required large capacity storage space.
According to the modern operating system framework, carry out virtually at the operating system bottom, making up virtual memory facilities is possible such as hard disk etc.The virtual instrument of some similar memory devices has been arranged at present, such as DiskSim and Device mapper etc.DiskSim comprises abundant assembly, various actions that can the emulation storage subsystem.But by storage and the reading out data that the virtual memory device out of DiskSim can not be actual, can't be used in the file system capacity test.Another one instrument Device mapper can fictionalize the memory device of random capacity in theory, but the virtual store module is merely able to the normal read-write of part.This be since Device mapper in design just simply, the part block with the virtual store module has been mapped to physical storage devices one to one, cause only having mapped in the virtual store module block (physical block, equal 512B) can normally read and write, therefore the capacity of the capacity of actual memory device and file system to be measured must be in full accord, and jumbo memory device like this will be provided, all unrealistic on the technical and cost.。Therefore, above-mentioned tool and method all can't satisfy the requirement of file system capacity test.
Summary of the invention
For this reason, to be solved by this invention is the technical matters that the existing virtual instrument of memory device can't fictionalize the memory device that can normally read and write, and a kind of test macro and method based on virtualized file system capacity that normally reads and writes data is provided.
For solving the problems of the technologies described above, the technical solution used in the present invention is as follows:
A kind of test macro based on virtualized file system capacity comprises:
The volume test administration module, under tested described file system, create out the virtual store module for described file system capacity test, and manage described file system comprises establishment, read-write and deletion in described virtual store module file operation;
The virtual store module has the max cap. that described file system can be supported, is used for the file that the described volume test administration module of virtual store writes;
The physical store module is used for the file that the described volume test administration module of actual storage writes described virtual store module;
The mapping management module, the described virtual store module of dynamic management will be mapped directly to the file operation of described virtual store module the file operation to the storage unit of described physical store module to the mapping of described physical store module in operating system Block Device Driver layer; Limit the capacity that all file sizes that write described virtual store module are less than or equal to described physical store module.
Described mapping management module realizes described dynamic management by a mapping table; Described mapping table stores the mapping relations of the storage unit of the storage unit of described virtual store module and described physical store module, and described mapping management module realizes inquiry, distribution and the recovery of mapping object space by the described mapping table of Dynamic Maintenance.
The storage unit of described virtual store module is mapped to the storage unit of described physical store module, the mapping block size by described volume test administration module according to test needs adjustment.
Described mapping management module is by increasing or remove entries, dynamic assignment or dynamically reclaim described physical memory cell.
Simultaneously, provide a kind of method of file system capacity of test fictitious, comprise the steps,
1. create the virtual store module, it has the max cap. that described file system can be supported, is used for the file that the described file system of virtual store creates;
2. the physical store module is set, and it has predetermined volumes, is used for the file that the described file system of actual storage creates;
3. the volume test administration module is managed described file system comprises establishment, read-write and deletion to described virtual store module file operation;
4. by dynamic mapping, will directly be mapped as to the file operation that described virtual store module is carried out the file operation to described physical store module.
Described mapping management module realizes described dynamic management by a mapping table; Described mapping table stores the mapping relations of the storage unit of the storage unit of described virtual store module and described physical store module, and described mapping management module realizes inquiry, distribution and the recovery of mapping object space by the described mapping table of Dynamic Maintenance.
The storage unit of described virtual store module is mapped to the storage unit of described physical store module, the size of map unit by described volume test administration module according to test needs adjustment.
Described mapping management module is by increasing or remove entries, dynamic assignment or dynamically reclaim described physical memory cell.
Create and reading and writing of files in, described volume test administration module specify a plurality of described files to take respectively at least the virtual store module storage space begin place, middle locating and end.
Technique scheme of the present invention has the following advantages compared to existing technology:
Test macro based on virtualized file system capacity of the present invention, comprise: the volume test administration module, under tested described file system, create out the virtual store module for described file system capacity test, manage described file system comprises establishment, read-write and deletion to described virtual store module file operation; The virtual store module has the max cap. that described file system can be supported, is used for the file that the described volume test administration module of virtual store writes; The physical store module is used for the file that the described volume test administration module of actual storage writes described virtual store module; The mapping management module, the described virtual store module of dynamic management will be mapped directly to the file operation of described virtual store module the file operation to the storage unit of described physical store module to the mapping of described physical store module in operating system Block Device Driver layer; Limit the capacity that all file sizes that write described virtual store module are less than or equal to described physical store module; And then pass through the as required mapping mode realization virtual store module of dynamic and intelligent to the mapping of physical store module, the virtual store module that assurance fictionalizes can normally be read and write, and is convenient to test the capacity of jumbo file system.
Description of drawings
For content of the present invention is more likely to be clearly understood, the below according to a particular embodiment of the invention and by reference to the accompanying drawings, the present invention is further detailed explanation, wherein
Fig. 1 be in the one embodiment of the invention virtual store module to the mapping hierarchy chart of physical store module in operating system;
Fig. 2 is the volume test environment configurations figure of middle distributed file system Lustre embodiment illustrated in fig. 1;
Fig. 3 is data flow schematic diagram in the volume test process of Lustre shown in Fig. 2;
Fig. 4 is the as required mapping relations schematic diagram of dynamic and intelligent among the present invention;
Fig. 5 is mapping tables corresponding to mapping relations shown in Fig. 4.
Embodiment
As virtual store module in the one embodiment of the invention to the mapping layer aggregated(particle) structure of physical store module in operating system as shown in Figure 1, file system capacity test and management module is moved at user's space, can on designated equipment, carry out the operations such as document creation, read-write and deletion according to POSIX (abbreviation of Portable Operating System Interface of Unix, expression portable operating system interface) file system access interface.Provide POSIX normative document access interface function by Virtual File System VFS (full name is Virtual FileSystem) in the kernel spacing.File system to be tested such as Journal File System ext3, distributed file system Lustre etc. under the VFS.Under file system to be measured, be the Block Device Driver layer, the virtual store module is operated in this layer, and provides the dynamic and intelligent mapping of virtual store module to the actual physical storage module by this layer.Physical store module drive and physical store module under the Block Device Driver.
In the Block Device Driver layer, take as required mapping (the Intelligent on-demand dynamic mapping) mode of dynamic and intelligent from the virtual store module to the mapping of actual physical storage module.Virtual described virtual store module out can be read and write to a certain extent normally in this way.The as required mapping mode of dynamic and intelligent mainly refers to mapping table of operating system Dynamic Maintenance, has recorded the concrete mapping relations between the storage unit of the storage unit of virtual store module and physical store module in this table.And, only have and when user's request for data storage space, just carry out mapped amount of physical memory distribution, when user's deleted file, reclaim amount of physical memory.By inquiry, increase and the deletion mapping item, when the test and management module is passed through file system read-write virtual store module, read write command and data stream etc. is redirected to the physical store module.The virtual store module that adopts above-mentioned technology to realize is readable writing, and has guaranteed the demand of file system capacity test, and has adopted this mapping method, and the virtual memory facilities that can fictionalize in theory random capacity is used for the file system capacity test.
File system capacity test and management module creates, reads and writes and the operation such as deletion in the virtual store module, because the effect of mapping table, in fact data are stored on the physical store module.Because file system only needs a small amount of physics storage space of depositing to carry out readwrite tests in concrete test, therefore the virtual store module can have enough large storage space, and the physical store module of actual storage data only need to satisfy file system format, and the required a small amount of space of operation such as creates, writes.
Therefore, adopt above-mentioned technology, in the test of the file system capacity of reality, can fictionalize in theory the storage space of random capacity, and the capacity of virtual store module and mapping block size be adjustable parameter, greatly strengthened the dirigibility of test.Although the physical store module capacity is less, but owing to taking the as required mapping mode of dynamic and intelligent, as long as the data total amount that test procedure writes to the virtual store module is no more than the active volume of physical store module at every turn, the virtual store module is normal readable writing just as common physical store module.Because above-mentioned mapping process all occurs in the operating system nucleus driving, in file system, what be directly used in the storage data is jumbo virtual store module, and normal readable writing, and has so just reached the purpose of volume test.
The Lustre distributed file system is a kind of distributed file system of increasing income of present widespread use, supports in theory at least storage space of 64PB.Fig. 2 is the typical test environment of a Lustre distributed file system, and this test environment is comprised of Ethernet (GE/10GE) or the fiber optic network (FC) of client (clientl-clientx), meta data server (MDS), object storage server (OSD) and connection the said equipment.The upper runs file system volume test of client administration module is equivalent among Fig. 1 at user's space access Lustre file system and managing virtual memory device.MDS provides Metadata Service, and OSD provides physical storage device.On object storage server OSD, by virtual method shown in Figure 1, fictionalize the virtual memory facilities of enough capacity, and adopt the as required mapping of dynamic and intelligent, file system capacity test and management module is mapped to the physical store module of reality to the read-write of virtual store module.
Need in a plurality of file operations in memory allocated space create, write and format etc., the representative capacity positions such as the beginning of the storage space that described volume test administration module specifies a plurality of described files to take respectively to comprise the virtual store module located, middle place and end, to take at least described virtual store module storage space begin place and end.By the selection of above-mentioned particular memory space, to effective covering of volume space, and then guarantee the correct, reliable of test result when guaranteeing the file system capacity test.Such as file operation success as described in a plurality of, judge that then tested file system can support the max cap. of nominal.
Fig. 3 is illustrated in the flow direction of data in the actual file system capacity test.File system capacity test and management module is carried out POSIX file operation (such as create, read and write etc.) at user's space.The direct control of mentioned order and data is to liking the virtual store module, and still after bottom was through the conversion of unit maps table, actual operand was the physical store module.By mapping table read write command and data stream etc. is redirected to the physical store module, the virtual store module that creates out like this is readable writing, and has guaranteed the demand of file system capacity test.
Fig. 4 and Fig. 5 represent the as required mapping mode schematic diagram of the dynamic and intelligent from the virtual store module to the physical store module.The as required mapping of dynamic and intelligent is embodied in following several aspect: at first, the elementary cell of mapping can be the physical block of memory module, also can be a map unit that defines according to testing requirement, is map unit of 4k, 1M, 2M etc. such as capacity.Secondly, only have and when user's request for data storage space, just carry out mapped amount of physical memory distribution, when user's deleted file, reclaim amount of physical memory.The 3rd, the unit records of the physical store module of the storage unit of mapped virtual store module and conduct mapping target is in dynamic mapping table, and the process of above-mentioned distribution and recovery mapping space just is reduced to increase and the delete procedure of mapping table list item.The as required mapping mode of the dynamic and intelligent by has as shown in Figure 4 guaranteed that limited physical capacity can be fully utilized, and can fictionalize in theory the virtual store module of random capacity.In this case, described physical store module only need to have the required capacity of test and get final product, and the capacity of general described physical store module is no more than 1% of max cap. that file system can support.
Adopt above-mentioned method of testing, the virtual memory facilities that can fictionalize enough capacity is used for the test of Lustre file system capacity.Because in concrete file system capacity test, can not also need not accomplish all tests of covering of having living space, and only need a small amount of storage space to carry out readwrite tests, therefore as long as the data total amount that writes to the virtual store module of test procedure is no more than the active volume of mapped physical store module, the virtual store module is normal readable writing just as the physical store module.Because above-mentioned mapping process all occurs in the operating system Block Device Driver layer, in the Lustre file system, what be directly used in the storage data is jumbo virtual store module, and normal readable writing, and has so just reached the purpose of Lustre file system capacity test.
Obviously, above-described embodiment only is for example clearly is described, and is not the restriction to embodiment.For those of ordinary skill in the field, can also make other changes in different forms on the basis of the above description.Here need not also can't give all embodiments exhaustive.And the apparent variation of being extended out thus or change still are among the protection domain of the invention.
Claims (9)
1. the test macro based on virtualized file system capacity is characterized in that, comprising:
The volume test administration module, under tested described file system, create out the virtual store module for described file system capacity test, and manage the file operation that comprises establishment, read-write and deletion that described file system is carried out in described virtual store module;
The virtual store module has the max cap. that described file system can be supported, is used for the file that the described volume test administration module of virtual store writes;
The physical store module is used for the file that the described volume test administration module of actual storage writes described virtual store module;
The mapping management module, the described virtual store module of dynamic management will be mapped directly to the file operation of described virtual store module the file operation to described physical store module to the mapping of described physical store module in operating system Block Device Driver layer; Limit the capacity that all file sizes that write described virtual store module are less than or equal to described physical store module.
2. test macro according to claim 1, it is characterized in that: described mapping management module realizes described dynamic management by a mapping table; Described mapping table stores the mapping relations of the storage unit of the storage unit of described virtual store module and described physical store module, and described mapping management module realizes inquiry, distribution and the recovery of mapped physical space by the described mapping table of Dynamic Maintenance.
3. test macro according to claim 2, it is characterized in that: the storage unit of described virtual store module is mapped to the storage unit of described physical store module, the size of map unit by described volume test administration module according to the adjustment of test needs.
4. test macro according to claim 3 is characterized in that: described mapping management module is by increasing or remove entries, dynamic assignment or dynamically reclaim described physical memory cell.
5. method of testing based on virtualized file system capacity, right to use requires the arbitrary described system of 1-4, it is characterized in that: comprise the steps,
1. create the virtual store module, it has the max cap. that described file system can be supported, is used for the file that the described file system of virtual store creates;
2. the physical store module is set, and it has predetermined volumes, is used for the file that the described file system of actual storage creates;
3. the volume test administration module manage described file system described virtual store module carry out comprise establishment, the read-write and the deletion file operation, described volume test administration module specify a plurality of described files to take respectively at least the virtual store module storage space begin the place and end;
4. by dynamic mapping, will directly be mapped as to the file operation that described virtual store module is carried out the file operation to described physical store module;
5. file operation success, then the file system of judgement test can be supported the described max cap. of nominal.
6. method of testing according to claim 5, it is characterized in that: described mapping management module realizes described dynamic management by a mapping table; Described mapping table stores the mapping relations of the storage unit of the storage unit of described virtual store module and described physical store module, and described mapping management module realizes inquiry, distribution and the recovery of mapping object space by the described mapping table of Dynamic Maintenance.
7. method of testing according to claim 6, it is characterized in that: the storage unit of described virtual store module is mapped to described physical store module stores unit, the size of map unit by described volume test administration module according to the adjustment of test needs.
8. method of testing according to claim 7 is characterized in that: described mapping management module is by increasing or remove entries, dynamic assignment or dynamically reclaim described physical memory cell.
9. method of testing according to claim 8, it is characterized in that: create and write in the operating process of a plurality of files comprising, described volume test administration module specify a plurality of described files to take respectively at least the virtual store module storage space begin place, middle place and end.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010620506 CN102063326B (en) | 2010-12-31 | 2010-12-31 | System for testing file system capacity based on virtualization and method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010620506 CN102063326B (en) | 2010-12-31 | 2010-12-31 | System for testing file system capacity based on virtualization and method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102063326A CN102063326A (en) | 2011-05-18 |
| CN102063326B true CN102063326B (en) | 2013-01-02 |
Family
ID=43998612
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010620506 Expired - Fee Related CN102063326B (en) | 2010-12-31 | 2010-12-31 | System for testing file system capacity based on virtualization and method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102063326B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102215129B (en) * | 2011-06-15 | 2017-10-10 | 中兴通讯股份有限公司 | The external method of business module, apparatus and system |
| CN102291466B (en) * | 2011-09-05 | 2014-02-26 | 浪潮电子信息产业股份有限公司 | Method for optimizing cluster storage network resource configuration |
| CN105701409B (en) * | 2015-12-29 | 2019-05-17 | 珠海豹趣科技有限公司 | A kind of method and device for realizing file virtualization |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1963782A (en) * | 2006-11-28 | 2007-05-16 | 北京中星微电子有限公司 | Method and system for testing embeded file system |
| CN101201776A (en) * | 2006-12-14 | 2008-06-18 | 英业达股份有限公司 | Method for integrating and testing platform of multiple operating systems |
| CN101706787A (en) * | 2009-11-20 | 2010-05-12 | 深圳市中兴移动通信有限公司 | File system management method for logic memory |
-
2010
- 2010-12-31 CN CN 201010620506 patent/CN102063326B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1963782A (en) * | 2006-11-28 | 2007-05-16 | 北京中星微电子有限公司 | Method and system for testing embeded file system |
| CN101201776A (en) * | 2006-12-14 | 2008-06-18 | 英业达股份有限公司 | Method for integrating and testing platform of multiple operating systems |
| CN101706787A (en) * | 2009-11-20 | 2010-05-12 | 深圳市中兴移动通信有限公司 | File system management method for logic memory |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102063326A (en) | 2011-05-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8504797B2 (en) | Method and apparatus for managing thin provisioning volume by using file storage system | |
| US10521131B2 (en) | Storage apparatus and storage control apparatus | |
| US8521949B2 (en) | Data deleting method and apparatus | |
| US8793290B1 (en) | Metadata management for pools of storage disks | |
| CN102023809B (en) | Storage system, method for reading data from storage system and method for writing data to storage system | |
| US8095728B2 (en) | Method and system for power aware I/O scheduling | |
| US20140351515A1 (en) | Providing data attributes to a storage manager to use to select a storage tier to use for a data set management operation | |
| US20140351505A1 (en) | Providing data attributes to a storage manager to use to select a storage tier to use for a data set management operation | |
| CN103514249B (en) | A kind of automatic data reduction and system and storage device | |
| US20140173226A1 (en) | Logical object deletion | |
| US20060212495A1 (en) | Method and system for storing data into a database | |
| US8621134B2 (en) | Storage tiering with minimal use of DRAM memory for header overhead | |
| US20140019706A1 (en) | System and method of logical object management | |
| US20150193473A1 (en) | Database Storage System based on Optical Disk and Method Using the System | |
| CN109324758A (en) | Data migration method, device and storage equipment | |
| CN108399050B (en) | Data processing method and device | |
| US20140181455A1 (en) | Category based space allocation for multiple storage devices | |
| US11144224B2 (en) | Locality-aware, memory-efficient, time-efficient hot data identification using count-min-sketch for flash or streaming applications | |
| CN102063326B (en) | System for testing file system capacity based on virtualization and method thereof | |
| CN107346209B (en) | Multi-disk aggregation type data storage system and implementation method and application method thereof | |
| CN103942301A (en) | Distributed file system oriented to access and application of multiple data types | |
| US7480684B2 (en) | Method and system for object allocation using fill counts | |
| US7424574B1 (en) | Method and apparatus for dynamic striping | |
| US10089125B2 (en) | Virtual machines accessing file data, object data, and block data | |
| US11513702B2 (en) | Placement of metadata on data storage drives in a first storage enclosure of a data storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C17 | Cessation of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130102 Termination date: 20131231 |