CN105224607B - A kind of Virtual File System design method for simulating cloud storage equipment - Google Patents
A kind of Virtual File System design method for simulating cloud storage equipment Download PDFInfo
- Publication number
- CN105224607B CN105224607B CN201510561398.0A CN201510561398A CN105224607B CN 105224607 B CN105224607 B CN 105224607B CN 201510561398 A CN201510561398 A CN 201510561398A CN 105224607 B CN105224607 B CN 105224607B
- Authority
- CN
- China
- Prior art keywords
- size
- file
- block
- index node
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 16
- 238000013461 design Methods 0.000 title description 2
- 238000013500 data storage Methods 0.000 claims abstract description 6
- 238000004088 simulation Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/188—Virtual file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention devises a kind of implementation method of Virtual File System for simulating cloud storage equipment, based on the ext3 file system of Linux, extend the physical structure of its data storage, according to the parameter being passed to when file mount, calculate relevant configuration threshold value, configuring it in threshold range as common file system when starting write-in, metadata information is only written when being more than configuration threshold values, and it is not written into data information, to simulate huge memory capacity using limited physical equipment.
Description
Technical field
It is the present invention relates to computer cluster NAS field of storage, in particular to a kind of to simulate big rule using limited storage device
The method of the equipment of mould cloud storage cluster.
Background technique
With the arrival of Internet era, whole world all trades and professions all produce a large amount of data daily, including individual's
Data also produce volatile growth, for these data of collection management, some big Internet enterprises or other generations
The company of mass data all establishes public or privately owned data center, and traditional memory module is no longer satisfied data
Volatile growth, and each enterprise establishes a data center will generate the great wasting of resources, and cloud storage technology is at this
It comes into being under a background.
Cloud storage is in new concept cloud computing conceptive extension and developed, and is that the emerging network of one kind is deposited
Storage technology refers to through functions such as cluster application, network technology or distributed file systems, by various inhomogeneities a large amount of in network
The storage equipment of type gathers collaborative work by application software, common externally to provide data storage and business access function
One system.The storage vendor of relatively mainstream has all put into huge manpower and material resources in the field at present.
Storage cluster environment is inevitably needed to go to be tested and be debugged during research and development, but to true
Cluster is built in real physical equipment area, will be a very big expense, and if cluster is very huge, entire debugging exploitation
Period will be very long.
Summary of the invention
In order to solve the above-mentioned technical problem, the present invention is transformed upgrading on the basis of linux ext3 file system,
Its physical storage structure on block device is planned again, make its start write-in when configuration threshold range in it is general
Logical file system is the same, metadata information is only written when being more than configuration threshold values, and be not written into data information.Due to one
A Inode data take up space as 256 bytes, thus can the equipment internal simulation of very little again go out the equipment of super large.
In order to solve the above-mentioned technical problems, the present invention provides a kind of Virtual File System realizations for simulating cloud storage equipment
Method, comprising: the physical structure of growth data storage；According to the parameter being passed to when file mount, the size of extended area is calculated
And configuration threshold value；Compare the file and configure the size of threshold value, it is real without storing that metadata is only written according to comparison result
Border data.
Further, described to be extended to based on the ext3 file system of linux, the addition extension index section in block group
Bit map/bitmap EIB and extension index node table EIT.
Further, the EIB and EIT is located at after index node bitmap and index node table.
Further, the parameter is the percentage P of quantity of documents upper limit N to be simulated or actual physics block device.
Further, if in file system option be passed to parameter be quantity of documents upper limit N to be simulated, by with
Lower formula calculates the size of EIB and EIT:
Size (EIT)=N*size (inode)/size (block)；
Size (EIB)=N/8/size (block)；
If the parameter being passed in file system option is actual physics block device percentage P, pass through following formula meter
Calculate the size of EIB and EIT:
Extent_itsize=size (conventional bar) * P；
Size (EIT)=Extent_itsize/size (block)；
Size (EIB)=Extent_itsize/size (inode)/8/size (block).
Further, the configuration threshold value is size (conventional bar)-size (EIB)-size (EIT).
Further, when file is more than configuration threshold size, the inode write-in extension index node of the file of write-in
Table only records metadata without storing real data.
Further, Custom Attributes is written in inode to determine whether being virtual file.
Detailed description of the invention
Attached drawing is used to provide to further understand technical solution of the present invention, and constitutes part of specification, with this
The embodiment of application technical solution for explaining the present invention together, does not constitute the limitation to technical solution of the present invention.
Fig. 1 shows the Virtual File System block device physics of the simulation cloud storage equipment of an embodiment according to the present invention
Topology layout.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with attached drawing to the present invention
Embodiment be described in detail.It should be noted that in the absence of conflict, in the embodiment and embodiment in the application
Feature can mutual any combination.
The present invention devises a kind of a kind of implementation method of Virtual File System for simulating cloud storage equipment, with Linux's
Based on ext3 file system, the physical structure of its data storage is extended, according to the parameter being passed to when file mount, is calculated related
Configuration threshold value, make its start write-in when configuration threshold range in as common file system, when be more than match
Metadata information is only written when setting threshold values, and is not written into data information, to simulate Pang using limited physical equipment
Big memory capacity.
Disk partition is divided into multiple pieces of groups (Block group), several adjacent blocks by ext3 file system
(Block) a block group is formed, each piece of group size is identical.Structurally, block group is by superblock (Super block), group
Descriptor table (Group descriptor table), bitmap block (Block bitmap), index node bitmap (Inode
Bitmap), index node table (Inode table), multiple data blocks (Data blocks) are constituted.
With reference to attached drawing 1, the present invention based on the ext3 file system of Linux, to its storage organization on disk into
Row transformation, addition extension index node bitmap EIB (extent inode bitmap) and extension index node table in block group
EIT (extent inode table), and EIB and EIT are located at after index node bitmap and index node table.
Being passed to quantity of documents upper limit N to be simulated in file system option when creating file system, (there are upper by N
Limit) or actual physics block device percentage P (0 < P≤1), later according to incoming different parameters calculate EIB size and
The size of EIT, later according to calculated result initialization metadata region.Extended area size calculation is as follows:
If the parameter being passed in file system option is quantity of documents upper limit N to be simulated, pass through following formula meter
Calculate the size of EIB and EIT:
Size (EIT)=N*size (inode)/size (block)；
Size (EIB)=N/8/size (block).
If the parameter being passed in file system option is actual physics block device percentage P, pass through following formula meter
Calculate the size of EIB and EIT:
Extent_itsize=size (conventional bar) * P；
Size (EIT)=Extent_itsize/size (block)；
Size (EIB)=Extent_itsize/size (inode)/8/size (block).
According to the size of above-mentioned extended area calculated, configuration threshold value is calculated, the configuration threshold value is size (common
Block)-size (EIB)-size (EIT).
Data are write under carry file system, when file is more than configuration threshold size, the inode for the file being written later
All write-in extension index node tables, all files only record metadata without storing real data, and write in inode
Enter Custom Attributes to determine whether being virtual file.Otherwise, data are write in the usual way.
It can be by the large-scale cluster of equipment simulating excess of export of low capacity, up to PB grades, to test cloud by above method
The functional completeness and system stability of storage system.But this document system is unable to test reading and writing data performance, but can be with
Assessment test for file system metadata read-write.
In the exemplary embodiment, a kind of non-transitorycomputer readable storage medium including instruction, example are additionally provided
It such as include the memory of instruction, above-metioned instruction can be executed by the processor of a device to complete the above method.Non- face for example, described
When property computer readable storage medium can be ROM, random access memory (RAM), CD-ROM, tape, floppy disk and light data
Store equipment etc..
Although disclosed herein embodiment it is as above, the content only for ease of understanding the present invention and use
Embodiment is not intended to limit the invention.Technical staff in any fields of the present invention is taken off not departing from the present invention
Under the premise of the spirit and scope of dew, any modification and variation, but the present invention can be carried out in the form and details of implementation
Scope of patent protection, still should be subject to the scope of the claims as defined in the appended claims.
Claims (8)
1. a kind of method for the Virtual File System for realizing simulation cloud storage equipment characterized by comprising
The physical structure of growth data storage；
According to the parameter being passed to when file mount, the size and configuration threshold value of extended area are calculated；
Compare the file and configure the size of threshold value, metadata is only written according to comparison result without storing real data；Its
In, metadata is only written according to comparison result without storing real data for the size of the file and configuration threshold value
Refer to: when file is more than configuration threshold size, the index node inode write-in extension index node table of the file of write-in, only
Metadata is recorded without storing real data.
2. the method as described in claim 1, which is characterized in that it is described to be extended to based on the ext3 file system of linux,
The addition extension index node bitmap EIB and extension index node table EIT in block group.
3. method according to claim 2, which is characterized in that the EIB and EIT are located at index node bitmap and index
After node table.
4. the method as described in claim 1, which is characterized in that the parameter is quantity of documents upper limit N to be simulated or reality
The percentage P of border physics block device.
5. method as claimed in claim 4, which is characterized in that if the parameter being passed in file system option is to be simulated
Quantity of documents upper limit N is then calculated by the following formula extension index node bitmap EIB and extends the size of index node table EIT:
Size (EIT)=N*size (inode)/size (block)；
Size (EIB)=N/8/size (block), wherein inode is index node, and block is block；
If the parameter being passed in file system option is actual physics block device percentage P, it is calculated by the following formula EIB
With the size of EIT:
Extent_itsize=size (conventional bar) * P；
Size (EIT)=Extent_itsize/size (block)；
Size (EIB)=Extent_itsize/size (inode)/8/size (block), wherein inode is index node,
Block is block.
6. method according to claim 2, which is characterized in that the configuration threshold value is size (conventional bar)-size (EIB)-
size(EIT)。
7. the method as described in claim 1, which is characterized in that further comprise be written in index node inode it is customized
Attribute determines whether it is virtual file.
8. a kind of device for the Virtual File System for simulating cloud storage equipment characterized by comprising
Physical structure device for growth data storage；
For calculating the size of extended area and configuring the device of threshold value according to the parameter being passed to when file mount；
For the size of the file and configuration threshold value, metadata is only written according to comparison result without storing real data
Device；Wherein, metadata is only written according to comparison result without storing for the size of the file and configuration threshold value
Real data refers to: when file is more than configuration threshold size, the index node inode write-in extension index section of the file of write-in
Point table only records metadata without storing real data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510561398.0A CN105224607B (en) | 2015-09-06 | 2015-09-06 | A kind of Virtual File System design method for simulating cloud storage equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510561398.0A CN105224607B (en) | 2015-09-06 | 2015-09-06 | A kind of Virtual File System design method for simulating cloud storage equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105224607A CN105224607A (en) | 2016-01-06 |
| CN105224607B true CN105224607B (en) | 2019-05-24 |
Family
ID=54993575
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510561398.0A Active CN105224607B (en) | 2015-09-06 | 2015-09-06 | A kind of Virtual File System design method for simulating cloud storage equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105224607B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108228673B (en) * | 2016-12-22 | 2021-09-03 | 上海凯翔信息科技有限公司 | Method and system for rapidly merging files |
| CN107357878A (en) * | 2017-07-06 | 2017-11-17 | 成都睿胜科技有限公司 | Extension type mini-file system and its implementation |
| CN109814946B (en) * | 2018-12-29 | 2022-03-25 | 深圳创新科技术有限公司 | Method and device for realizing distributed storage in XenServer |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102609365A (en) * | 2012-02-15 | 2012-07-25 | 合一网络技术(北京)有限公司 | Virtual disk system and file storage method based on virtual disk system |
| CN102831240A (en) * | 2012-09-05 | 2012-12-19 | 曙光信息产业(北京)有限公司 | Storage method and storage structure of extensible metadata documents |
| CN102855284A (en) * | 2012-08-03 | 2013-01-02 | 北京联创信安科技有限公司 | Method and system for managing data of cluster storage system |
| CN103324446A (en) * | 2013-06-07 | 2013-09-25 | 曙光信息产业(北京)有限公司 | Achieving method of high-performance elastic-capacity virtual machine disk |
| EP1745395B1 (en) * | 2004-04-30 | 2013-12-18 | Network Appliance, Inc. | Extension of a system and method for write allocation within a write anywhere file layout file system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4907605B2 (en) * | 2007-06-25 | 2012-04-04 | ヒューレット−パッカード デベロップメント カンパニー エル．ピー． | Inode allocation and deallocation method and computer program |
-
2015
- 2015-09-06 CN CN201510561398.0A patent/CN105224607B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1745395B1 (en) * | 2004-04-30 | 2013-12-18 | Network Appliance, Inc. | Extension of a system and method for write allocation within a write anywhere file layout file system |
| CN102609365A (en) * | 2012-02-15 | 2012-07-25 | 合一网络技术(北京)有限公司 | Virtual disk system and file storage method based on virtual disk system |
| CN102855284A (en) * | 2012-08-03 | 2013-01-02 | 北京联创信安科技有限公司 | Method and system for managing data of cluster storage system |
| CN102831240A (en) * | 2012-09-05 | 2012-12-19 | 曙光信息产业(北京)有限公司 | Storage method and storage structure of extensible metadata documents |
| CN103324446A (en) * | 2013-06-07 | 2013-09-25 | 曙光信息产业(北京)有限公司 | Achieving method of high-performance elastic-capacity virtual machine disk |
Non-Patent Citations (1)
| Title |
|---|
| 云存储性能评测技术研究;贺秦禄等;《计算机应用研究》;20130515;第30卷(第5期);1357-1360 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105224607A (en) | 2016-01-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102255962B (en) | Distributive storage method, device and system | |
| US10339123B2 (en) | Data management for tenants | |
| US10802927B2 (en) | System and method for checking and characterizing snapshot metadata using snapshot metadata database | |
| CN106611046A (en) | Big data technology-based space data storage processing middleware framework | |
| CN106021031B (en) | A kind of the deletion data reconstruction method and device of BTRFS file system | |
| CN102915278A (en) | Data deduplication method | |
| CN111427847B (en) | Indexing and querying method and system for user-defined metadata | |
| CN105224607B (en) | A kind of Virtual File System design method for simulating cloud storage equipment | |
| CN103812939A (en) | Big data storage system | |
| CN106407355A (en) | Data storage method and device | |
| CN106104463A (en) | System and method for the failsafe operation of storage device | |
| CN108604162A (en) | Optimize the access to creation data | |
| CN104536908B (en) | A kind of magnanimity small records efficient storage management method towards unit | |
| Hashem et al. | An Integrative Modeling of BigData Processing. | |
| CN103942301A (en) | Distributed file system oriented to access and application of multiple data types | |
| Mao et al. | NoSQL based 3D city model management system | |
| CN106558346A (en) | The method and device that a kind of solid-state disk service life based on RAIM frameworks is calculated | |
| CN109947743A (en) | A kind of the NoSQL big data storage method and system of optimization | |
| CN109684517A (en) | A kind of historical data storage method, reading/writing method, storage device and equipment | |
| CN104391947A (en) | Real-time processing method and system of mass GIS (geographic information system) data | |
| CN104636452B (en) | A kind of unified file system management method towards SCM | |
| CN106681893A (en) | Realization method of NAND FLASH simulator | |
| Abdullahi et al. | Big data: performance profiling of meteorological and oceanographic data on hive | |
| CN101751390B (en) | Disk configuration method of object orientation storage device | |
| San-Lucas et al. | Towards a fast multi-tier storage system simulator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |