CN104537050A - Method for batch and rapid establishment of metadata and data of file system - Google Patents

Method for batch and rapid establishment of metadata and data of file system Download PDF

Info

Publication number
CN104537050A
CN104537050A CN201410826066.6A CN201410826066A CN104537050A CN 104537050 A CN104537050 A CN 104537050A CN 201410826066 A CN201410826066 A CN 201410826066A CN 104537050 A CN104537050 A CN 104537050A
Authority
CN
China
Prior art keywords
data
metadata
area
disk
bitmap
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201410826066.6A
Other languages
Chinese (zh)
Other versions
CN104537050B (en
Inventor
曹强
钱璐
谭诗诗
谢长生
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huazhong University of Science and Technology
Original Assignee
Huazhong University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huazhong University of Science and Technology filed Critical Huazhong University of Science and Technology
Priority to CN201410826066.6A priority Critical patent/CN104537050B/en
Publication of CN104537050A publication Critical patent/CN104537050A/en
Application granted granted Critical
Publication of CN104537050B publication Critical patent/CN104537050B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/172Caching, prefetching or hoarding of files
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/164File meta data generation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method for batch and rapid establishment of metadata and data of a file system. According to the method for batch and rapid establishment of the metadata and data of the file system, specifically aiming at a pre-known working set such as catalog copying and decompressing of compressed files, three kinds of metadata, including super blocks, block group descriptors, a data bitmap and an Inode bitmap, in a metadata area are modified firstly on the premise that the reliability of the system file is guaranteed; then, Inode information and data are written into the metadata area and a data area respectively according to the operating sequence. According to the method for batch and rapid establishment of the metadata and data of the file system, frequent random lower cases introduced onto a metadata page during back writing are avoided for the file system; writing delay time is prolonged, a part of IO requests are absorbed and combined by the metadata and the data area to a greater extent, and repeated updating for issuing the same metadata object to a disk is replaced with one-time disk updating; in the metadata and the data area, sequential establishment of Inodes and data blocks is achieved, and head track seeking frequency and positioning frequency are reduced.

Description

A kind of method of batch fast creation file system metadata and data
Technical field
The invention belongs to computer memory system technical field, more specifically, relate to a kind of method of batch fast creation file system metadata and data.
Background technology
File system is the core component of storage system, and primary responsibility is constituent act and data on a storage device.File system is made up of three parts: the interface of file system, the operation of object and the software assembly (object and attribute) of management.
Decades in the past, the research of file system mainly concentrates on lifting file system performance, and the magnetic disk comprising file optimizing system, to make full use of Disk bandwidth and to reduce disk tracking, reduces the I/O operation etc. to disk by lifting memory efficient.But along with the increase of system complexity and the growth of storage data volume, the reliability of balanced file system and performance become the focus of academia and industry member research gradually.
Based in the file system of disk, in order to reach the reliability requirement of system, needing the metadata on internal memory and data to update on disk, ensureing consistance and the persistence of metadata and data.But renewal rewards theory will produce the small letter of a large amount of dispersion frequently, thus affects the performance of file system.From hardware point of view analysis, the little write operation of dispersion can increase the seek time of magnetic head, reduces the speed upgraded; From software respective analysis, internal system is serialized processes for the synchronization mechanism of metadata and data, cannot make full use of on the basis of Disk bandwidth, further limit the speed of renewal rewards theory.
And at large data age, copy and move heap file data to become and operate more and more frequently, this operation relates to the establishment of heap file catalogue and data, a large amount of metadata and data can be produced, according to existing method, in order to ensure consistance and persistence, all catalogues and file creation operation are that serial performs, respective meta-data and data must write disk according to order, thus produce a large amount of random little I/O, seriously limit reproduction speed.
Present stage is divided into both direction for the primary solutions of this difficult problem.A direction is the time suitably increasing write delay, with the amendment of merger metadata and data; Another direction is the research of log-structuredization file system.But write delay method is limited for the lifting of file system performance, and the disk space that the design of log-structuredization file system makes file system must be taken by the expired invalid data of extra garbage reclamation operation cleaning, when disk space utilization rate is higher, performance can decline fast, and this makes log-structuredization file system fail to be promoted in disk file system.This two schemes all can not solve the problem that when copying and move heap file data, the processing time is long very well.
Summary of the invention
For above defect or the Improvement requirement of prior art, the invention provides the method and system of a kind of batch fast creation file system metadata and data, its object is to solve in the file system of existing ext series the technical matters easily causing file system processing speed limited existing for method copying and move heap file data and provide metadata and data serial to create.
For achieving the above object, according to one aspect of the present invention, provide a kind of method of batch fast creation file system metadata and data, comprise the following steps:
(1) in the catalogue copy or compressed file decompression operations of user's execution, by traveling through the number of sub-directory and file under the data set statistics copy or decompress(ion) catalogue predicted in this operation, to obtain the number of Inode node, it equals the two quantity sum N inodes;
(2) in the internal memory of disk file system, metadata area and the data area in continuation address space is built respectively, by the copies of metadata in disk in the metadata area in continuation address space;
(3) in the metadata area created and data area, metadata and data are created successively according to the data set of precognition, until terminate the operation of data set, the metadata in the metadata area of internal memory and data area and data rapid batch are updated in meta-data region corresponding in disk and data field.
Preferably, the metadata of copy comprises superblock, block group descriptor, Inode node bitmap, data bitmap.
Preferably, step (2) comprises following sub-step:
(2-1) the size S of metadata area to be built is calculated metadata_chunk, build in the internal memory of disk file system and there is this size S metadata_chunkaddress space continuous print metadata area;
(2-2) the metadata replication in disk to build meta-data region in;
(2-3) the size S of data area to be built is set data_chunk, in the internal memory of disk file system, build size is S data_chunkaddress space continuous print data area.
Preferably, in step (2-1), metadata area size adopts following formula to calculate:
S metadata _ chunk = ( N inodes * S inode 1024 ) + S super _ block + S inode _ bitmap + S block _ bitmap ( KB )
Wherein, S inodefor single Inode node size, S super_blockthe size of superblock, S inode_bitmapthe size of Inode node bitmap, S block_bitmapit is the size of data bitmap.
Preferably, data bitmap size S block_bitmapfollowing formula is adopted to calculate:
S block _ bitmap = S Disk _ cpapcity 8 * S Disk _ block _ size
Wherein, S disk_capacitythe amount of capacity of disk, S disk_block_sizefor disk block size;
Inode node bitmap size S inode_bitmapfollowing formula is adopted to calculate:
S inode _ bitmap = S block _ bitmap N block _ per _ inode
Wherein N block_per_inodethe data block quantity that in disk, each node takies.
Preferably, step (3) comprises following sub-step:
(3-1) the Inode nodal information in the catalogue copy of user's execution or compressed file decompression operations and corresponding data are write metadata area and internal storage data region respectively, renewal rewards theory is carried out to the superblock copied in metadata area, block group descriptor, data block bitmap, Inode node bitmap three kinds of metadata simultaneously;
(3-2) respectively the metadata in the metadata area of internal memory and data area and data rapid batch are updated in meta-data region corresponding in disk and data field.
Preferably, step (3-1) comprises following sub-step:
(3-1-1) counter i=1 is set;
(3-1-2) i-th Inode nodal information and corresponding data in the catalogue copy of user's execution or compressed file decompression operations are write metadata area and internal storage data region respectively, and renewal rewards theory is carried out to the superblock copied in metadata area, block group descriptor, data block bitmap, Inode node bitmap three kinds of metadata;
(3-1-3) current data region size and data area size S is judged data_chunkbetween ratio whether reach threshold value, if it is proceed to step (3-1-4), otherwise proceed to step (3-1-5);
(3-1-4) in internal memory, a new distribution size is S data_chunkthe data area in continuation address space, data corresponding for remaining Inode nodal information are write in this new data area, simultaneously by the content of former data area with whole region for transmission unit batch writes back to data field corresponding on disk
In;
(3-1-5) i=i+1 is set, and judges whether i equals N inodesif then process terminates, otherwise return step (3-1-2).
Preferably, step (3-2) comprises following sub-step:
(3-2-1) batch data also not writing back disk in the data area of internal memory is write back in the data field of disk;
(3-2-2) superblock in the metadata area of internal memory, block group descriptor, data block bitmap, Inode node bitmap are write back to superblock, block group descriptor, data block bitmap corresponding in disk, in Inode node bit map area;
(3-2-3) the Inode nodal information batch in the metadata area of internal memory is write back to Inode corresponding on disk to show in district.
According to another aspect of the present invention, provide the system of a kind of batch fast creation file system metadata and data, comprising:
First module, for perform user catalogue copy or compressed file decompression operations in, by traveling through the number of sub-directory and file under the data set statistics copy or decompress(ion) catalogue predicted in this operation, to obtain the number of Inode node, it equals the two quantity sum N inodes;
Second module, for building metadata area and the data area in continuation address space in the internal memory of disk file system respectively, by the copies of metadata in disk in the metadata area in continuation address space;
3rd module, for creating metadata and data successively according to the data set of precognition in the metadata area created and data area, until terminate the operation of data set, the metadata in the metadata area of internal memory and data area and data rapid batch are updated in meta-data region corresponding in disk and data field.
In general, the above technical scheme conceived by the present invention compared with prior art, can obtain following beneficial effect:
1, owing to have employed step (3-1), so file system adds the time of write delay, make in metadata area in internal memory and data area, the part I/O request of merger to a greater degree, the repeatedly renewal rewards theory making same metadata object be issued to disk is merged into a disk operating, decrease the synchronization times with disk, avoid the dispersion small letter problem of frequent Timing Belt.
2, simultaneously owing to have employed step (3-2), in metadata area in internal memory and data area, realize Inode nodal information and the order establishment of data block ground, make them can write back to continuation address space corresponding on disk by rapid batch, so decrease the tracking of magnetic head and the number of times of location, the speed copying and move heap file of significant increase file system.
Accompanying drawing explanation
Fig. 1 is application internal memory cut-away view of the present invention;
Fig. 2 is application disk cut-away view of the present invention;
Fig. 3 is the framework organization chart of the method for the present invention's batch fast creation file system metadata and data;
Fig. 4 is the overview flow chart of the method for the present invention's batch fast creation file system metadata and data;
Fig. 5 is the refinement process flow diagram of step in the inventive method (2);
Fig. 6 is the refinement process flow diagram of step in the inventive method (3);
Fig. 7 is the refinement process flow diagram of step in the inventive method (3-1).
Embodiment
In order to make object of the present invention, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein only in order to explain the present invention, be not intended to limit the present invention.In addition, if below in described each embodiment of the present invention involved technical characteristic do not form conflict each other and just can mutually combine.
Basic ideas of the present invention are, a kind of method of batch fast creation file system metadata and data is provided, it is specifically for the data set of precognition, and this data set comprises catalogue and the fileinfo of the establishment of all needs, such as, copy source directory and compressed file set etc.Under the prerequisite ensureing file system reliability, focus in two different regions by internal memory respectively to the creation operation of metadata and data, described region is continuous print space in address in internal memory, for metadata, called after metadata area, for data, called after data area.In operating process, Inode nodal information in the catalogue copy of user's execution or compressed file decompression operations and corresponding data are write metadata area and internal storage data region respectively, simultaneously to the superblock copied in metadata area, block group descriptor, data block bitmap, Inode node bitmap three kinds of metadata carry out renewal rewards theory.Until operated, the metadata finally metadata area in internal memory and data area created respectively and data entirety have write back in meta-data region corresponding to disk and data field, complete and copy and move heap file data.
Batch creates the method for file system metadata and data specifically for a data set for precognition, and this data set comprises catalogue and the fileinfo of the establishment of all needs.Described a kind of batch creates the method for file system metadata and data specifically for specific operation, comprises catalogue copy and compressed file decompression operations etc.
First, some nouns and term in the present invention are made an explanation:
File system metadata: file system metadata refers to the management data structure in space and the metadata of file system file and directory tree structure, comprise the data structure managing whole file system, the data structure of management allocation of space, the attribute of file and pointer, the attribute of catalogue and content.For EXT2, superblock, block group descriptor, data block bitmap, Inode node bitmap, Inode shows, and the directory entry page all belongs to the metadata of file system.Wherein Inode table is the set of Inode nodal information, has both comprised file Inode nodal information, and has also comprised catalogue Inode nodal information; The directory entry page is the set of directory entry, records the information such as the name of All Files and catalogue under this catalogue and No. Inode.
File system data: file system data refers to the content of file.For EXT2, the data of file system are the file contents leaving data field in.
Metadata area: metadata area refers to the one section of memory address continuous print space distributed in internal memory.It is for buffer memory superblock, block group descriptor, Inode node bitmap, data bitmap and copy the Inode nodal information created in catalogue and compressed file decompression operations process.The establishment data centralization of size by predicting of metadata area, the number of catalogue and file calculates.
Data area: data area refers to the one section of memory address continuous print space distributed in internal memory.Its data message for creating in cached copies catalogue and compressed file decompression operations process.The size S of data area data_chunkfor the default value (such as 64MB) preset, it can be arranged according to concrete scene dynamics by user.
In order to accelerating file system operations, alleviate the performance bottleneck that disk at a slow speed brings, system utilizes memory cache partial document system metadata and data.Because inside save as volatile storage, thus the maintenance requirement metadata page of file system metadata is write back to non-volatile memory apparatus in time.
Fig. 1 is internal memory cut-away view of the present invention.As shown in Figure 1, superblock, block group descriptor, data block bitmap, Inode node bitmap, Inode nodal information and data are cached with in internal memory, for concrete catalogue copy and the operation of compressed file decompress(ion), system in Memory Allocation address space continuous print metadata area and data area, can be respectively used to metadata and the data of journal copy and decompression operations.
Fig. 2 is disk cut-away view of the present invention.As shown in Figure 2, superblock, block group descriptor, data block bitmap, Inode node bitmap, Inode table and data block is stored lastingly in disk, in internal memory, the volatile metadata of buffer memory and data need the metadata corresponding with on disk and data content in time to carry out synchronously, therefore the small letter of a large amount of dispersion can be produced, cause frequent tracking and the location of magnetic head, affect the processing speed of file system.
As shown in Figure 4, the method for the present invention's batch fast creation file system metadata and data comprises the following steps:
(1) in the catalogue copy or compressed file decompression operations of user's execution, by traveling through the number of sub-directory and file under the data set statistics copy or decompress(ion) catalogue predicted in this operation, to obtain the number of Inode node, it equals the two quantity sum N inodes;
(2) in the internal memory of disk file system, build metadata area and the data area in continuation address space respectively, by the copies of metadata in disk in the metadata area in continuation address space, the metadata wherein copied comprises superblock, block group descriptor, Inode node bitmap, data bitmap;
As shown in Figure 5, step (2) comprises following sub-step:
(2-1) the size S of metadata area to be built is calculated metadata_chunk, build in the internal memory of disk file system and there is this size S metadata_chunkaddress space continuous print metadata area;
Specifically, metadata area size adopts following formula to calculate:
S metadata _ chunk = ( N inodes * S inode 1024 ) + S super _ block + S inode _ bitmap + S block _ bitmap ( KB )
Wherein, S inodefor single Inode node size, unit is Byte, S super_blockbe the size of superblock, unit is KB, it typically is 1KB, S inode_bitmapbe the size of Inode node bitmap, unit is KB, S block_bitmapbe the size of data bitmap, unit is KB;
Wherein, data bitmap size S block_bitmapfollowing formula is adopted to calculate:
S block _ bitmap = S Disk _ cpapcity 8 * S Disk _ block _ size
Wherein, S disk_capacitybe the amount of capacity of disk, unit is KB, S disk_block_sizefor disk block size, unit is KB;
Inode node bitmap size S inode_bitmapfollowing formula is adopted to calculate:
S inode _ bitmap = S block _ bitmap N block _ per _ inode
Wherein N block_per_inodethe data block quantity that in disk, each node takies;
(2-2) metadata (comprising superblock, block group descriptor, Inode node bitmap and data bitmap) in disk is copied in the meta-data region of structure;
(2-3) the size S of data area to be built is set data_chunk, in the internal memory of disk file system, build size is S data_chunkaddress space continuous print data area; Specifically, the size S of data area data_chunkfor the default value (such as 64MB) preset, it can be arranged (file size of copy or decompression operations is larger, then this value is larger, otherwise then less) according to concrete scene dynamics by user.
(3) in the metadata area created and data area, metadata and data are created successively according to the data set of precognition, until terminate the operation of data set, the metadata in the metadata area of internal memory and data area and data rapid batch are updated in meta-data region corresponding in disk and data field:
As shown in Figure 6, this step comprises following sub-step:
(3-1) the Inode nodal information in the catalogue copy of user's execution or compressed file decompression operations and corresponding data are write metadata area and internal storage data region respectively, renewal rewards theory is carried out to the superblock copied in metadata area, block group descriptor, data block bitmap, Inode node bitmap three kinds of metadata simultaneously, specifically, because a kind of method of batch fast creation file system metadata and data copies or compressed file decompression operations for the catalogue predicting building work collection, so in the process, Inode nodal information and data all belong to the new content created, Inode nodal information and data are sequentially written in metadata area and data area according to what operate successively, wherein Inode node and data are distributed in order, the address space that Inode nodal information and data are corresponding on disk simultaneously is also continuous print, with realize metadata area and data area content batch be written back to disk.
Specifically, as shown in Figure 7, this step comprises following sub-step:
(3-1-1) counter i=1 is set;
(3-1-2) i-th Inode nodal information and corresponding data in the catalogue copy of user's execution or compressed file decompression operations are write metadata area and internal storage data region respectively, and renewal rewards theory is carried out to the superblock copied in metadata area, block group descriptor, data block bitmap, Inode node bitmap three kinds of metadata;
(3-1-3) current data region size and data area size S is judged data_chunkbetween ratio whether reach threshold value, if it is proceed to step (3-1-4), otherwise proceed to step (3-1-5); Specifically, the large I of threshold value is by user's free setting;
(3-1-4) in internal memory, a new distribution size is S data_chunkthe data area in continuation address space, data corresponding for remaining Inode nodal information are write in this new data area, simultaneously by the content of former data area with whole region for transmission unit batch writes back in data field corresponding on disk;
(3-1-5) i=i+1 is set, and judges whether i equals N inodesif then process terminates, otherwise return step (3-1-2);
(3-2) respectively the metadata in the metadata area of internal memory and data area and data rapid batch are updated in meta-data region corresponding in disk and data field.
This step comprises following sub-step:
(3-2-1) batch data also not writing back disk in the data area of internal memory is write back in the data field of disk;
(3-2-2) superblock in the metadata area of internal memory, block group descriptor, data block bitmap, Inode node bitmap are write back to superblock, block group descriptor, data block bitmap corresponding in disk, in Inode node bit map area;
(3-2-3) the Inode nodal information batch in the metadata area of internal memory is write back to Inode corresponding on disk to show in district.
The invention has the advantages that:
Owing to have employed step (3-1), so file system adds the time of write delay, make in metadata area in internal memory and data area, the part I/O request of merger to a greater degree, the repeatedly renewal rewards theory making same metadata object be issued to disk is merged into a disk operating, decrease the synchronization times with disk, avoid the dispersion small letter problem of frequent Timing Belt.Simultaneously, owing to have employed step (3-2), in metadata area in internal memory and data area, realize Inode nodal information and the order establishment of data block ground, make them can write back to continuation address space corresponding on disk by rapid batch, so decrease the tracking of magnetic head and the number of times of location, thus the speed copying and move heap file of significant increase file system.
Those skilled in the art will readily understand; the foregoing is only preferred embodiment of the present invention; not in order to limit the present invention, all any amendments done within the spirit and principles in the present invention, equivalent replacement and improvement etc., all should be included within protection scope of the present invention.

Claims (9)

1. a method for batch fast creation file system metadata and data, is characterized in that, comprise the following steps:
(1) in the catalogue copy or compressed file decompression operations of user's execution, by traveling through the number of sub-directory and file under the data set statistics copy or decompress(ion) catalogue predicted in this operation, to obtain the number of Inode node, it equals the two quantity sum N inodes;
(2) in the internal memory of disk file system, metadata area and the data area in continuation address space is built respectively, by the copies of metadata in disk in the metadata area in continuation address space;
(3) in the metadata area created and data area, metadata and data are created successively according to the data set of precognition, until terminate the operation of data set, the metadata in the metadata area of internal memory and data area and data rapid batch are updated in meta-data region corresponding in disk and data field.
2. method according to claim 1, is characterized in that, the metadata of copy comprises superblock, block group descriptor, Inode node bitmap, data bitmap.
3. method according to claim 1, is characterized in that, step (2) comprises following sub-step:
(2-1) the size S of metadata area to be built is calculated metadata_chunk, build in the internal memory of disk file system and there is this size S metadata_chunkaddress space continuous print metadata area;
(2-2) the metadata replication in disk to build meta-data region in;
(2-3) the size S of data area to be built is set data_chunk, in the internal memory of disk file system, build size is S data_chunkaddress space continuous print data area.
4. method according to claim 3, is characterized in that, in step (2-1), metadata area size adopts following formula to calculate:
S metadata _ chunk = ( N inodes * S inode 1024 ) + S super _ block + S inode _ bitmap + S block _ bitmap ( KB )
Wherein, S inodefor single Inode node size, S super_blockthe size of superblock, S inode_bitmapthe size of Inode node bitmap, S block_bitmapit is the size of data bitmap.
5. method according to claim 4, is characterized in that,
Data bitmap size S block_bitmapfollowing formula is adopted to calculate:
S block _ bitmap = S Disk _ capacity 8 * S Disk _ block _ size
Wherein, S disk_capacitythe amount of capacity of disk, S disk_block_sizefor disk block size;
Inode node bitmap size S inode_bitmapfollowing formula is adopted to calculate:
S inode _ bitmap = S block _ bitmap N block _ per _ inode
Wherein N block_per_inodethe data block quantity that in disk, each node takies.
6. method according to claim 1, is characterized in that, step (3) comprises following sub-step:
(3-1) the Inode nodal information in the catalogue copy of user's execution or compressed file decompression operations and corresponding data are write metadata area and internal storage data region respectively, renewal rewards theory is carried out to the superblock copied in metadata area, block group descriptor, data block bitmap, Inode node bitmap three kinds of metadata simultaneously;
(3-2) respectively the metadata in the metadata area of internal memory and data area and data rapid batch are updated in meta-data region corresponding in disk and data field.
7. method according to claim 6, is characterized in that, step (3-1) comprises following sub-step:
(3-1-1) counter i=1 is set;
(3-1-2) i-th Inode nodal information and corresponding data in the catalogue copy of user's execution or compressed file decompression operations are write metadata area and internal storage data region respectively, and renewal rewards theory is carried out to the superblock copied in metadata area, block group descriptor, data block bitmap, Inode node bitmap three kinds of metadata;
(3-1-3) current data region size and data area size S is judged data_chunkbetween ratio whether reach threshold value, if it is proceed to step (3-1-4), otherwise proceed to step (3-1-5);
(3-1-4) in internal memory, a new distribution size is S data_chunkthe data area in continuation address space, data corresponding for remaining Inode nodal information are write in this new data area, simultaneously by the content of former data area with whole region for transmission unit batch writes back in data field corresponding on disk;
(3-1-5) i=i+1 is set, and judges whether i equals N inodesif then process terminates, otherwise return step (3-1-2).
8. method according to claim 7, is characterized in that, step (3-2) comprises following sub-step:
(3-2-1) batch data also not writing back disk in the data area of internal memory is write back in the data field of disk;
(3-2-2) superblock in the metadata area of internal memory, block group descriptor, data block bitmap, Inode node bitmap are write back to superblock, block group descriptor, data block bitmap corresponding in disk, in Inode node bit map area;
(3-2-3) the Inode nodal information batch in the metadata area of internal memory is write back to Inode corresponding on disk to show in district.
9. a system for batch fast creation file system metadata and data, is characterized in that, comprising:
First module, for perform user catalogue copy or compressed file decompression operations in, by traveling through the number of sub-directory and file under the data set statistics copy or decompress(ion) catalogue predicted in this operation, to obtain the number of Inode node, it equals the two quantity sum N inodes;
Second module, for building metadata area and the data area in continuation address space in the internal memory of disk file system respectively, by the copies of metadata in disk in the metadata area in continuation address space;
3rd module, for creating metadata and data successively according to the data set of precognition in the metadata area created and data area, until terminate the operation of data set, the metadata in the metadata area of internal memory and data area and data rapid batch are updated in meta-data region corresponding in disk and data field.
CN201410826066.6A 2014-12-25 2014-12-25 A kind of method of the quick establishment file system metadata of batch and data Active CN104537050B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201410826066.6A CN104537050B (en) 2014-12-25 2014-12-25 A kind of method of the quick establishment file system metadata of batch and data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201410826066.6A CN104537050B (en) 2014-12-25 2014-12-25 A kind of method of the quick establishment file system metadata of batch and data

Publications (2)

Publication Number Publication Date
CN104537050A true CN104537050A (en) 2015-04-22
CN104537050B CN104537050B (en) 2017-12-15

Family

ID=52852578

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201410826066.6A Active CN104537050B (en) 2014-12-25 2014-12-25 A kind of method of the quick establishment file system metadata of batch and data

Country Status (1)

Country Link
CN (1) CN104537050B (en)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105389232A (en) * 2015-10-28 2016-03-09 武汉噢易云计算有限公司 Valid data analysis method for EXT file system
CN105701146A (en) * 2015-12-28 2016-06-22 小米科技有限责任公司 Directory creating method and apparatus
CN106354890A (en) * 2016-11-22 2017-01-25 中国科学院上海微系统与信息技术研究所 Implementation method of random access file system based on N-ary tree structure
CN106933970A (en) * 2017-02-10 2017-07-07 福州瑞芯微电子股份有限公司 A kind of fast writing data to EXT subregions method and apparatus
CN108121514A (en) * 2017-12-29 2018-06-05 北京奇虎科技有限公司 Metamessage update method, device, computing device and computer storage media
CN109614237A (en) * 2018-12-10 2019-04-12 浪潮(北京)电子信息产业有限公司 A kind of method for releasing resource, device, equipment and medium
CN109977092A (en) * 2019-03-26 2019-07-05 武汉深之度科技有限公司 A kind of method and calculating equipment of duplication file
CN110389863A (en) * 2019-07-26 2019-10-29 苏州浪潮智能科技有限公司 A kind of data reconstruction method, device, equipment and computer readable storage medium
CN111176580A (en) * 2019-12-30 2020-05-19 成都易我科技开发有限责任公司 Method and device for lossless movement of partitioned data and readable storage medium
CN113419670A (en) * 2021-03-31 2021-09-21 阿里巴巴新加坡控股有限公司 Data writing processing method and device and electronic equipment
CN114063935A (en) * 2022-01-17 2022-02-18 阿里云计算有限公司 Method and device for processing data

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109933570B (en) * 2019-03-15 2020-02-07 中山大学 Metadata management method, system and medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103067418A (en) * 2011-10-19 2013-04-24 北大方正集团有限公司 Method, device and system for reconstructing metadata
CN103152395A (en) * 2013-02-05 2013-06-12 北京奇虎科技有限公司 Storage method and device of distributed file system
CN103294786A (en) * 2013-05-17 2013-09-11 华中科技大学 Metadata organization and management method and system of distributed file system
US20130262805A1 (en) * 2005-04-13 2013-10-03 Ling Zheng Method and Apparatus for Identifying and Eliminating Duplicate Data Blocks and Sharing Data Blocks in a Storage System
US20130290284A1 (en) * 2010-05-06 2013-10-31 Go Daddy Operating Company, LLC Verifying and Balancing Server Resources via Stored Usage Data
CN103838853A (en) * 2014-03-17 2014-06-04 华中科技大学 Mixed file system based on different storage media

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130262805A1 (en) * 2005-04-13 2013-10-03 Ling Zheng Method and Apparatus for Identifying and Eliminating Duplicate Data Blocks and Sharing Data Blocks in a Storage System
US20130290284A1 (en) * 2010-05-06 2013-10-31 Go Daddy Operating Company, LLC Verifying and Balancing Server Resources via Stored Usage Data
CN103067418A (en) * 2011-10-19 2013-04-24 北大方正集团有限公司 Method, device and system for reconstructing metadata
CN103152395A (en) * 2013-02-05 2013-06-12 北京奇虎科技有限公司 Storage method and device of distributed file system
CN103294786A (en) * 2013-05-17 2013-09-11 华中科技大学 Metadata organization and management method and system of distributed file system
CN103838853A (en) * 2014-03-17 2014-06-04 华中科技大学 Mixed file system based on different storage media

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LIANG XIAO,ET AL.: "Study and Optimize the Process of Batch Small Files Replication", 《2008 JAPAN-CHINA JOINT WORKSHOP ON FRONTIER OF COMPUTER SCIENCE AND TECHNOLOGY》 *

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105389232B (en) * 2015-10-28 2018-02-09 武汉噢易云计算股份有限公司 EXT file system valid data analysis methods
CN105389232A (en) * 2015-10-28 2016-03-09 武汉噢易云计算有限公司 Valid data analysis method for EXT file system
CN105701146A (en) * 2015-12-28 2016-06-22 小米科技有限责任公司 Directory creating method and apparatus
CN105701146B (en) * 2015-12-28 2019-07-05 小米科技有限责任公司 The method and apparatus to create directory
CN106354890A (en) * 2016-11-22 2017-01-25 中国科学院上海微系统与信息技术研究所 Implementation method of random access file system based on N-ary tree structure
CN106354890B (en) * 2016-11-22 2019-05-21 中国科学院上海微系统与信息技术研究所 A kind of implementation method of the file system of the random access based on N-ary tree construction
CN106933970B (en) * 2017-02-10 2019-11-22 福州瑞芯微电子股份有限公司 A kind of fast writing data to EXT subregion method and apparatus
CN106933970A (en) * 2017-02-10 2017-07-07 福州瑞芯微电子股份有限公司 A kind of fast writing data to EXT subregions method and apparatus
CN108121514A (en) * 2017-12-29 2018-06-05 北京奇虎科技有限公司 Metamessage update method, device, computing device and computer storage media
CN109614237A (en) * 2018-12-10 2019-04-12 浪潮(北京)电子信息产业有限公司 A kind of method for releasing resource, device, equipment and medium
CN109977092A (en) * 2019-03-26 2019-07-05 武汉深之度科技有限公司 A kind of method and calculating equipment of duplication file
CN109977092B (en) * 2019-03-26 2021-06-08 武汉深之度科技有限公司 Method and computing device for copying files
CN110389863A (en) * 2019-07-26 2019-10-29 苏州浪潮智能科技有限公司 A kind of data reconstruction method, device, equipment and computer readable storage medium
CN111176580A (en) * 2019-12-30 2020-05-19 成都易我科技开发有限责任公司 Method and device for lossless movement of partitioned data and readable storage medium
CN111176580B (en) * 2019-12-30 2023-04-07 成都易我科技开发有限责任公司 Method and device for lossless movement of partitioned data and readable storage medium
CN113419670A (en) * 2021-03-31 2021-09-21 阿里巴巴新加坡控股有限公司 Data writing processing method and device and electronic equipment
CN114063935A (en) * 2022-01-17 2022-02-18 阿里云计算有限公司 Method and device for processing data

Also Published As

Publication number Publication date
CN104537050B (en) 2017-12-15

Similar Documents

Publication Publication Date Title
CN104537050A (en) Method for batch and rapid establishment of metadata and data of file system
CN109213772B (en) Data storage method and NVMe storage system
CN104731921B (en) Storage and processing method of the Hadoop distributed file systems for log type small documents
US20170228392A1 (en) Deduplicated file system
CN103179185B (en) Method and system for creating files in cache of distributed file system client
US8694469B2 (en) Cloud synthetic backups
CN101917396B (en) Real-time repetition removal and transmission method for data in network file system
CN101908073B (en) Method for deleting duplicated data in file system in real time
US9047330B2 (en) Index compression in databases
US7844643B2 (en) Storage management system with integrated continuous data protection and remote copy
CN102129435B (en) Data storage service control method and system
US20030004947A1 (en) Method, system, and program for managing files in a file system
CN103020255B (en) Classification storage means and device
CN103544045A (en) HDFS-based virtual machine image storage system and construction method thereof
CN104298681B (en) A kind of date storage method and device
JP2014508362A (en) Backup and restore strategies for data deduplication
CN103577513A (en) Systems and/or methods for caching xml information sets with delayed node instantiation
EP4100845A2 (en) Data management system and method of controlling
CN103793475B (en) A kind of method of Distributed File System Data migration
US20220335028A1 (en) Data access method, a data access control device, and a data access system
CN100504800C (en) Method for snapshot of magnetic disc
CN104572492A (en) Method and apparatus of burning data to FAT (File Allocation Table)32 partition
CN107506466A (en) A kind of small documents storage method and system
US20140195495A1 (en) Packing deduplicated data in a self-contained deduplicated repository
CN103841168A (en) Data copy updating method and metadata server

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant