WO2024254899A1 - 文件系统的元数据管理方法、终端设备及计算机存储介质 - Google Patents
文件系统的元数据管理方法、终端设备及计算机存储介质 Download PDFInfo
- Publication number
- WO2024254899A1 WO2024254899A1 PCT/CN2023/102466 CN2023102466W WO2024254899A1 WO 2024254899 A1 WO2024254899 A1 WO 2024254899A1 CN 2023102466 W CN2023102466 W CN 2023102466W WO 2024254899 A1 WO2024254899 A1 WO 2024254899A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- file
- allocated
- target
- index
- space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/122—File system administration, e.g. details of archiving or snapshots using management policies
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Definitions
- the present application relates to the field of data processing technology, and in particular to a metadata management method for a file system, a terminal device, and a computer-readable storage medium.
- the main purpose of the present application is to provide a metadata management method, apparatus, terminal device and computer-readable storage medium for a file system, aiming to enable the file management system to store file data continuously and adjacently on a storage device when storing file data, thereby reducing the number of times the device accesses the storage device during the process of reading file data, improving the performance of the file management system, and further enabling the file management system to meet the user's low-latency requirements when facing a large number of file creation and deletion scenarios.
- the present application provides a metadata management method for a file system, the metadata management method for a file system comprising the following steps:
- a merge operation is performed on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space;
- the initial key value format is assigned a corresponding target index value according to the index value contained in the target index management space to obtain a target key value format, and the file is stored in a target storage structure based on the target key value format.
- the step of encoding based on the path name to obtain the initial key value format corresponding to the file includes:
- the initial key value format corresponding to the file is obtained by encoding based on the parent directory index value corresponding to the parent directory of the file, the file name and the file attribute information.
- the step of performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the pre-allocated key value format to obtain a target index management space includes:
- a merge operation is performed on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain the target index management space.
- the step of performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain a target index management space includes:
- the pre-allocated index management space is merged into the parent directory index management space to obtain a target index management space.
- the step of performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space based on the parent directory index value further includes:
- a node balancing operation is performed on the pre-allocated index management space and the parent directory index management space based on the number of each allocated node, and the pre-allocated index management space after the balancing operation is performed is determined as a target index management space.
- the method further includes:
- a preset delete key is recursively inserted into the target file directory and each sub-file directory corresponding to the target file directory.
- the method further includes:
- Similar hot data files corresponding to the target file are determined, and the similar hot data files are inserted into a preset log merging storage structure tree.
- the method further includes:
- a merging and storing operation is performed on the file and each of the duplicate files.
- the present application also provides a terminal device, which includes: a memory, a processor, and a metadata management program of a file system stored on the memory and executable on the processor, and the metadata management program of the file system, when executed by the processor, implements the steps of the metadata management method of the file system as described above.
- the present application also provides a computer-readable storage medium, on which a metadata management program of a file system is stored.
- a metadata management program of a file system is stored.
- the steps of the metadata management method of the file system as described above are implemented.
- the metadata management method, apparatus, terminal device and computer medium of the file system obtain the path name corresponding to the file to be stored, and encode based on the path name to obtain the initial key value format corresponding to the file; perform a pre-allocation operation through a preset pre-allocated index management space to assign a corresponding pre-allocated index value to the initial key value format to obtain the pre-allocated key value format; perform a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the pre-allocated key value format to obtain a target index management space; assign a corresponding target index value to the initial key value format according to the index value contained in the target index management space to obtain a target key value format, and store the file in a target storage structure based on the target key value format.
- the file management system in the terminal device first obtains the file to be stored and determines the path name corresponding to the file, thereby encoding the file based on the file name to obtain the initial key value format corresponding to the file. After that, the file management system obtains the preset pre-allocated index management space, and performs a pre-allocation operation on the initial key value format corresponding to the file to be stored based on the pre-allocated index value space, thereby assigning the corresponding pre-allocated index value to the initial key value format to generate the pre-allocated key value format.
- the file management system performs a merge operation on the pre-allocated index management space and the parent directory index management space based on the pre-allocated index value and the parent directory index value contained in the pre-allocated key value format to generate a target index management space.
- the file management system assigns the corresponding target index value to the initial key value format corresponding to the file according to the index value contained in the target index management space to generate the corresponding target key value format.
- the file management system then saves the file in the storage structure corresponding to the storage device according to the target key value format.
- the present application achieves the technical effect of reducing the number of times a device accesses a storage device during the process of reading file data and improving the performance of the file management system by allowing the file management system to store files in continuous locations within a storage structure based on continuous index values contained in an index management space. This enables the file management system to meet users' low-latency requirements when faced with large-scale file creation and deletion scenarios.
- FIG1 is a schematic diagram of the structure of a terminal device in a hardware operating environment according to an embodiment of the present application
- FIG2 is a flowchart of a first embodiment of a metadata management method for a file system of the present application
- FIG. 3 is a schematic diagram of the layout of the index management space involved in an embodiment of the metadata management method of the file system of the present application
- FIG. 4 is a schematic diagram of a system structure involved in an embodiment of a metadata management method for a file system of the present application.
- Figure 1 is a schematic diagram of the terminal device structure of the hardware operating environment involved in the embodiment of the present application.
- Figure 1 is a schematic diagram of the hardware operating environment of the terminal device.
- the terminal device of the embodiment of the present application can be a device that executes the metadata management method of the file system of the present application, and the terminal device can specifically be a mobile terminal, a data storage control terminal, a PC or a portable computer.
- the terminal device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005.
- the communication bus 1002 is used to realize the connection and communication between these components.
- the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface.
- the network interface 1004 may include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface).
- the memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory.
- RAM Random Access Memory
- NVM Non-Volatile Memory
- the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
- FIG. 1 does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.
- the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, and a metadata management program of a file system.
- the network interface 1004 is mainly used for data communication with other devices;
- the user interface 1003 is mainly used for data interaction with the user;
- the processor 1001 and the memory 1005 in the terminal device of the present application can be set in the terminal device, and the terminal device calls the metadata management program of the file system stored in the memory 1005 through the processor 1001, and executes the metadata management method of the file system provided in the embodiment of the present application.
- the local file system for storage devices is the basis of all data storage and management systems.
- mainstream databases are built on file systems, and distributed file systems must rely on local file systems to organize data on devices.
- big data storage systems represented by Hbase and Dynamo are either built on distributed file systems or directly call local file systems to read and write data on storage devices. In this way, the performance of local file systems has a crucial impact on all data storage and management systems.
- the local file system can be roughly divided into two components in terms of structure: name space management and address space management.
- Address space management is mainly responsible for organizing data on storage devices.
- the device manufacturer will abstract the physical address space of the device into a linear logical address space
- the address space management module is responsible for organizing user data and file system metadata in the linear logical address space. Therefore, the main function of name space management is to organize and manage file and directory names, provide users with a file system operation interface, and implement the path resolution process from file name to file information.
- Hierarchical file system refers to a parent-child relationship between files and directories, forming a tree structure, and each node of the tree represents a file or directory.
- This structure allows files and directories in the file system to be easily organized and managed.
- each node has a unique path name that is used to identify the position of the node in the tree structure.
- /usr/local/bin is a path name that represents the bin directory under the local directory under the usr directory under the root directory.
- a flat file system means that all files and directories are at the same level, without a parent-child relationship.
- a hash file system means that a unique value is calculated by a hash function for the file name, and then the file is stored and searched based on this value.
- Hierarchical namespace management Compared with hierarchical management, its path resolution is simpler and does not need to be resolved hierarchically. However, it does not retain the relationship between adjacent files at the same level, and the processing operations involving multiple files are more complicated, such as file renaming and listing directory file information operations.
- the Ext series file systems (such as Ext2, Ext3, and Ext4) use a hierarchical structure to manage the namespace. They define directory-type files to save information about files at the same level.
- the Ext series file system must search for the inode (i.e., index node) of the target directory and parse the directory content to obtain the inode number of the next level directory.
- the Ext system file system generates a large amount of disk I/O (i.e., input/output) during the path search process, not only because of the directory resolution, but also because of its index node prefetching mechanism.
- BtrFS is a new type of copy-on-write (CoW) Linux file system that uses B-tree to store directory entries, so it is very efficient to search for files in a given directory.
- BtrFS inserts a directory entry with a hash value as the key each time a new file is created, and also inserts another directory entry index at the same time.
- the key of this directory entry index uses the sequence number as the key value of the B-tree. The sequence number increases linearly each time a new file is created, so the files read by readdir can be ordered.
- BtrFS is very efficient in searching for files in a given directory
- the depth of the B-tree will become deeper and deeper, and the delay in searching the target file in the B-tree will become larger and larger, resulting in a decrease in file access performance.
- the file system performance is greatly reduced due to the influence of a large number of random lowercases.
- TableFS is a file system based on LSM-Tree, stacked on the lower file system, which ensures that metadata is written to disk in the form of non-overwriting, sorted logs. Due to the write-friendly characteristics of LSM tree, TableFS is very friendly to write-intensive scenarios and can write random lower-write aggregation order to disk. However, LSM-Tree itself is not read-friendly, so when LSM-Tree is directly used as the metadata storage structure, the performance of the file system's read operation is poor. The performance of TableFS's read file operations (such as stat) is much lower than that of common file systems such as ext4.
- the present application proposes a metadata management method, terminal device and computer-readable storage medium for a file system, which obtains the path name corresponding to the file to be stored, and encodes the path name to obtain the initial key value format corresponding to the file; performs a pre-allocation operation through a preset pre-allocated index management space to assign the corresponding pre-allocated index value to the initial key value format to obtain the pre-allocated key value format; performs a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the pre-allocated key value format to obtain the target index management space; assigns the corresponding target index value to the initial key value format according to the index value contained in the target index management space to obtain the target key value format, and stores the file in the target storage structure based on the target key value format.
- the present application achieves the technical effect of reducing the number of times the device accesses the storage device in the process of reading file data and improving the performance of the file management system by allowing the file management system to store the file in a continuous position in the storage structure based on the continuous index values contained in the index management space, thereby enabling the file management system to meet the user's low latency requirements when facing a large number of file creation and deletion scenarios.
- FIG. 2 is a flowchart of a first embodiment of a metadata management method for a file system of the present application.
- the metadata management method of the file system of the present application may include the following steps:
- Step S10 obtaining a path name corresponding to a file to be stored, and encoding based on the path name to obtain an initial key value format corresponding to the file;
- the terminal device when the terminal device is running, it first obtains the file to be stored, and the file management system in the terminal device determines the path name corresponding to the file, and encodes it based on the preset encoding format and the path name to obtain an initial key value format.
- the terminal device when the terminal device is running, it first obtains the file that needs to be stored, and the file management system in the terminal device determines the path name corresponding to the file. After that, the file management system encodes the path name according to the preset key value format to obtain the initial key value format corresponding to the file: ⁇ pinode: filename, stat>.
- the step of "encoding the file to obtain the initial key value format corresponding to the file" in the above step S10 may specifically include:
- Step S101 Determine the file parent directory, file name and file attribute information corresponding to the path name;
- Step S102 Encoding the parent directory index value corresponding to the parent directory of the file, the file name and the file attribute information to obtain an initial key value format corresponding to the file;
- the file management system first determines the parent directory of the file corresponding to the file based on the path name, thereby determining the parent directory index value pinode corresponding to the parent directory of the file. At the same time, the file management system determines the file name filename corresponding to the file. At the same time, the file management system determines the file attribute information stat corresponding to the file. After that, the file management system encodes the obtained parent directory index value pinode, file name filename and file attribute information stat to obtain the initial key value format ⁇ pinode: filename, stat> corresponding to the file.
- Step S20 performing a pre-allocation operation through a preset pre-allocated index management space to allocate a corresponding pre-allocated index value to the initial key value format to obtain a pre-allocated key value format;
- Figures 3 and 4 where Figure 3 is a schematic diagram of the index management space layout involved in an embodiment of the metadata management method of the file system of the present application, and Figure 4 is a schematic diagram of the system structure involved in an embodiment of the metadata management method of the file system of the present application;
- the index management space is a space used to manage index values (i.e., inode_number), wherein the size corresponding to an index value should be set to 64 bits; in this way, the index management space can identify 2 64 files, as shown in Figure 4, since the size of the key value corresponding to a file is 256B, and a 4KiB storage block can accommodate 16 key-value pairs, therefore, the size of the index management space defaults to 16, and a portion of the space will be reserved for subsequent merging.
- index values i.e., inode_number
- index values in the index management space should be similar in lexicographic order, and each index value in the index management space should contain a minimum index value min_inode_number and a maximum index value max_inode_number.
- the file management system can determine an index value in the index management space inode_space and assign the index node inode corresponding to the index value to the newly created file.
- the file management system can create a new index management space inode_space.
- the index management space inode_space is divided into an index table in the memory (i.e., the DRAM in FIG3 ) and a linear table on the storage device (i.e., the SSD in FIG3 ), wherein the table entries of the linear table include the root node inode number (i.e., the root inode in FIG3 ), the maximum inode number max_inode that can be allocated by the index management space inode_space, the total number of reserved inodes step, and the currently allocated number count.
- the total number of reserved inodes step is the reserved space corresponding to the index management space inode_space
- the allocated number count corresponds to the allocated nodes corresponding to the index management space inode_space.
- the file management system obtains a preset pre-allocated index management space for performing a pre-allocation operation, and performs an index value pre-allocation operation based on the pre-allocated index management space to allocate a corresponding pre-allocated index value to the generated initial key value format, and the file management system then writes the pre-allocated index value to the initial key value format to generate a pre-allocated key value format.
- the file management system first obtains a preset pre-allocated index management space inode_space1 for performing a pre-allocation operation, and performs a pre-allocation operation based on the pre-allocated index management space inode_space1 to allocate a corresponding pre-allocated index value cur_inode to the initial key value format ⁇ pinode: filename, stat>, and then the file management system writes the allocated pre-allocated index value cur_inode to the initial key value format to obtain a pre-allocated key value format: ⁇ pinode: file_name, cur_inode...>.
- the keys of sub-files in the same directory can all start with the inode of the parent directory, so that the sub-files in the same directory will be stored consecutively and adjacently on the storage device. Since the storage device generally has a read and write granularity of KB, such as the minimum read granularity of SSD is 4KiB, one I/O can read the metadata information of files arranged adjacently on multiple storage devices, which can greatly improve the performance of file system range queries.
- Step S30 performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the pre-allocated key value format to obtain a target index management space;
- the file management system performs a merge operation on the preallocated index management space corresponding to the preallocated key value format and the parent directory index management space based on the preallocated index value and the parent directory index value contained in the preallocated key value format to obtain the target index management space.
- the file management system performs a merge operation on the parent directory index management space inode_space2 corresponding to the preallocated index value cur_inode and the parent directory index value pinode contained in the preallocated key value format ⁇ pinode: file_name, cur_inode ...> and the preallocated index management space inode_space1 to obtain the target index management space inode_space.
- step S30 may specifically include:
- Step S301 Compare the pre-allocated index value contained in the pre-allocated key value format with the parent directory index value to obtain a first comparison result
- the file management system first determines the preallocated index value and the parent directory index value contained in the preallocated key value format, and compares the preallocated index value with the parent directory index value to obtain the difference between the preallocated index value and the parent directory index value. At the same time, the file management system obtains a first threshold preset by a technician, and compares the difference with the first threshold to obtain a first comparison result. The file management system then determines whether the difference between the preallocated index value and the parent directory index value is too large based on the first comparison result.
- Step S302 If it is determined that the first comparison result is that the difference between the pre-allocated index value and the parent directory index value is greater than a preset first threshold, then based on the parent directory index value, a merge operation is performed on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space;
- the file management system determines that the first comparison result is that the above difference is greater than or equal to the preset first threshold, it determines that the difference between the pre-allocated index value and the parent directory index value is too large, and thus performs a merge operation on the parent directory index management space and the pre-allocated index management space based on the above parent directory index value to obtain the target index management space.
- the file management system first determines the preallocated index value cur_inode and the parent directory index value pinode contained in the preallocated key value format ⁇ pinode: file_name, cur_inode ...>, and compares the preallocated index value cur_inode with the parent directory index value pinode to obtain the difference between the two. At the same time, the file management system obtains a preset first threshold, and compares the difference with the first threshold to obtain a first comparison result.
- the file management system determines that the first comparison result is that the above difference is greater than or equal to the first threshold, it is determined that the difference between the preallocated index value cur_inode and the parent directory index value pinode is too large.
- the file management system then performs a merge operation based on the parent directory index management space inode_space2 corresponding to the parent directory index value pinode in the preallocated key value format ⁇ pinode: file_name, cur_inode ...> and the preallocated index management space inode_space1 corresponding to the preallocated index value cur_inode, thereby obtaining the target index management space inode_space.
- the step of "performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space based on the parent directory index value" in the above step S302 may specifically include:
- Step S3021 determining whether the space allocation value corresponding to the pre-allocated index management space is less than a preset second threshold
- Step S3022 if it is determined that the space allocation value is less than the second threshold, the pre-allocated index management space is merged into the parent directory index management space to obtain a target index management space;
- the file management system determines to perform a merge operation, it first obtains the second threshold preset by the technician, and compares the corresponding space allocation number in the pre-allocated index management space inode_space1 with the second threshold to obtain a second comparison result. Afterwards, if the file management system determines that the second comparison result is that the space allocation number is less than the second threshold, it determines that there are fewer subfiles in the index management space inode_space1. Therefore, there is no need to maintain the locality of the index management space inode_space1.
- the file management system then merges the pre-allocated index management space inode_space1 into the above-mentioned parent directory index management space inode_space2, thereby using the merged parent directory index management space inode_space2 as the target index management space inode_space.
- the step of "performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space based on the parent directory index value" in the above step S302 may also include:
- Step S3023 Determine whether the reserved space corresponding to the parent directory index management space is used up;
- the file management system determines to perform a merge operation, it first determines whether the reserved space included in the parent directory index management space is used up.
- Step S3024 if it is determined that the reserved space is used up, determining the number of allocated nodes corresponding to the pre-allocated index management space and the parent directory index management space respectively;
- the file management system determines that the reserved space has been used up, it determines that it is necessary to perform a split operation on the parent directory index management space, and then respectively detects the pre-allocated index management space and the parent directory index management space to determine the number of allocated nodes corresponding to the pre-allocated index management space and the parent directory index management space.
- Step S3025 performing a node balancing operation on the pre-allocated index management space and the parent directory index management space based on the number of each allocated node, and determining the pre-allocated index management space after performing the balancing operation as the target index management space;
- the file management system performs node balancing operations on the parent directory index management space and the pre-allocated index management space based on the obtained number of allocated nodes, thereby migrating some nodes in the parent directory index management space to the pre-allocated index management space to balance the number of nodes in the two index management spaces.
- the file management system determines the pre-allocated index management space after the balancing operation as the target index management space.
- the file management system when the file management system determines to perform a merge operation, it first determines the working status corresponding to the reserved space contained in the parent directory index management space inode_space2, and determines whether the reserved space is used up based on the working status. Afterwards, if the file management system determines that the reserved space is used up, it determines that it is necessary to perform a split operation of the index management space, and detects the number of allocated nodes corresponding to the pre-allocated index management space inode_space1 and the parent directory index management space inode_space2, respectively, so as to determine the inode node range [x, y] that needs to be migrated.
- the file management system determines the average number of nodes corresponding to the pre-allocated index management space inode_space1 and the parent directory index management space inode_space2 based on the obtained number of allocated nodes, and based on the average number of nodes, migrates some nodes contained in the parent directory index management space inode_space2 to the pre-allocated index management space inode_space1 according to the above migration operation to complete the node balancing operation, thereby determining the balanced pre-allocated index management space inode_space1 as the target index management space and updating the corresponding index table information.
- Step S40 assigning a corresponding target index value to the initial key value format according to the index value contained in the target index management space to obtain a target key value format, and storing the file in a target storage structure based on the target key value format;
- the file management system after completing the merge operation, performs an index allocation operation based on the index value contained in the obtained target index management space, thereby assigning the corresponding target index value to the initial key value format to generate a target key value format, and the file management information then stores the file in the corresponding storage structure according to the target key value format.
- the file management system after completing the merge operation, performs an inode allocation operation based on the target index management space inode_space, thereby allocating the corresponding target index value cur_inode to the initial key value format ⁇ pinode:filename, stat> to obtain the target key value format. Afterwards, the file management system stores the corresponding file in the target storage structure based on the target key value format.
- the terminal device when the terminal device is running, it first obtains the file to be stored, and the file management system in the terminal device determines the path name corresponding to the file, and encodes the file according to the preset encoding format and the path name to obtain the initial key value format. After that, the file management system obtains the preset pre-allocated index management space for performing the pre-allocation operation, and performs the index value pre-allocation operation based on the pre-allocated index management space to allocate the corresponding pre-allocated index value to the generated initial key value format. The file management system then writes the pre-allocated index value into the initial key value format to generate the pre-allocated key value format.
- the file management system After that, the file management system performs a merge operation on the pre-allocated index management space corresponding to the pre-allocated key value format and the above-mentioned parent directory index management space based on the pre-allocated index value and the parent directory index value contained in the pre-allocated key value format to obtain the target index management space. Finally, after completing the merge operation, the file management system performs an index allocation operation based on the index value contained in the obtained target index management space, thereby allocating the corresponding target index value to the initial key value format to generate the target key value format, and the file management information then stores the file in the corresponding storage structure according to the target key value format.
- the present application achieves the technical effect of reducing the number of times the device accesses the storage device in the process of reading file data and improving the performance of the file management system by allowing the file management system to store files in continuous locations in the storage structure based on the continuous index values contained in the index management space, thereby enabling the file management system to meet the user's low latency requirements when facing a large number of file creation and deletion scenarios. Furthermore, based on the first embodiment of the metadata management method of the file system of the present application, a second embodiment of the metadata management method of the file system of the present application is proposed here.
- the metadata management method of the file system of the present application may further include the following steps:
- Step A10 detecting the foreground load when receiving the file deletion instruction, and determining whether the foreground load is greater than or equal to a preset third threshold;
- the file management system when the file management system receives a file deletion instruction, it obtains the third threshold preset by the technician. At the same time, the file management system detects the terminal device based on the file deletion instruction, thereby obtaining the current foreground load of the terminal device, and compares the foreground load with the third threshold to obtain a third comparison result.
- Step A20 if it is determined that the foreground load is greater than or equal to the third threshold, inserting a preset full delete key into the target file directory corresponding to the file deletion instruction;
- the file management system determines that the third comparison result is that the foreground load is greater than or equal to the above-mentioned third threshold, it determines the target file directory to be deleted according to the file deletion instruction, and inserts the preset full delete key value into the target file directory, so that when the file management system performs the merge operation, the full delete key performs the file recovery operation to complete the deletion operation of the target file directory.
- Step A30 if it is determined that the foreground load is less than the third threshold, recursively inserting a preset delete key into the target file directory and each sub-file directory corresponding to the target file directory;
- the file management system determines that the third comparison result is that the foreground load is less than the above-mentioned third threshold, it determines the target file directory to be deleted and the subdirectories under the target file directory according to the file deletion instruction, and the file management system then recursively inserts the delete key under the target file directory and each subdirectory to remove the file directory marked with the delete key during the merge operation to complete the deletion operation.
- a file deletion instruction can be generated through a terminal device, and the terminal device then sends the file deletion instruction to the file management system. Later, when the file management system receives the file deletion instruction, it first obtains the third threshold preset by the technician and determines the target file directory corresponding to the target file to be deleted. At the same time, the file management system detects the foreground load corresponding to the terminal device, and the file management system then compares the foreground load with the third threshold to obtain a third comparison result.
- the file management system determines that the third comparison result is that the foreground load is greater than or equal to the third threshold, it obtains the preset full delete key full_delete_key, and inserts the full delete key full_delete_key into the target file directory, so that the file management system can search the key directory in the storage structure SSTable for the file containing the full delete key full_delete_key, and then when performing the above-mentioned index management space merge operation, the file directory corresponding to the file containing the full delete key full_delete_key is directly deleted;
- the file management system determines that the third comparison result is that the foreground load is less than the third threshold, the target file directory corresponding to the target file to be deleted and the subdirectories under the target file directory are determined according to the file deletion instruction. The file management system then recursively inserts the preset delete key into each of the subdirectories to remove these delete keys and files with the delete key from the storage device during the subsequent merge process.
- the file management system when it receives a file deletion instruction, it obtains a third threshold value preset by a technician. At the same time, the file management system detects the terminal device based on the file deletion instruction to obtain the current foreground load of the terminal device, and compares the foreground load with the third threshold value to obtain a third comparison result. Afterwards, if the file management system determines that the third comparison result is that the foreground load is greater than or equal to the third threshold value, it determines the target file directory to be deleted according to the file deletion instruction, and inserts the preset full delete key value into the target file directory, so that when the file management system performs a merge operation, the full delete key performs a file recovery operation to complete the deletion operation of the target file directory.
- the file management system determines that the third comparison result is that the foreground load is less than the third threshold value, it determines the target file directory to be deleted according to the file deletion instruction, and each subdirectory under the target file directory. The file management system then recursively inserts the delete key under the target file directory and each of the subdirectories to remove the file directory marked with the delete key during the merge operation to complete the deletion operation.
- the present application determines the foreground load situation of the terminal device, and when the foreground load situation is higher than a preset threshold, inserts a full delete key into the target file directory to be deleted, and when the foreground load situation is lower than the threshold, recursively inserts the delete key into the target file directory and the corresponding subdirectories.
- the metadata management method of the file system of the present application may further include the following steps:
- Step B10 upon receiving a file access instruction, determining a target file according to the file access instruction, and detecting an access frequency corresponding to the target file;
- the file management system when receiving a file access instruction, the file management system counts the access frequency of the target file corresponding to the file access instruction through a preset time window.
- Step B20 determining whether the data type corresponding to the target file is hot data based on the access frequency
- the file management system obtains a preset fourth threshold, and compares the obtained access frequency with the fourth threshold to obtain a fourth comparison result, and the file management system further determines whether the data type corresponding to the target file is hot data based on the fourth comparison result.
- Step B30 if it is determined that the data type is the hot data, determining the target index management space corresponding to the target file;
- the file management system determines that the fourth comparison result is that the access frequency is greater than or equal to the fourth threshold, it determines that the data type corresponding to the target file is hot data, and then determines the corresponding target index value based on the above-mentioned target key value format corresponding to the target file, and searches for the target index management space corresponding to the target index value.
- Step B40 determining each similar hot data file corresponding to the target file based on the target index management space, and inserting each similar hot data file into a preset log merging storage structure tree;
- the file management system determines each similar hot data file corresponding to the target metadata based on multiple index values included in the target index management space, and then inserts each similar hot data file into the surface storage structure in the file management system.
- the file management system receives a file access instruction, it opens a time window in the system background, and uses the time window to count the access frequency freq of the target file corresponding to the file access instruction.
- the file management system obtains a preset fourth threshold, and compares the access frequency freq with the fourth threshold to obtain a fourth comparison result.
- the file management system determines that the fourth comparison result is that the access frequency freq is greater than or equal to the fourth threshold, it determines that the data type corresponding to the target file is a hot file filehot, and the file management system further determines the target index value inode hot corresponding to the target file, and determines the target index management space inode_space hot corresponding to the target index value inode hot . Afterwards, the file management system will determine each similar hot data file corresponding to the target file based on the index management space inode_space hot , and insert each similar hot data file into a lower-level SSTable in the file management system, so that users can access each hot data file faster.
- the file management system when the file management system receives a file access instruction, it counts the access frequency of the target file corresponding to the file access instruction through a preset time window. Thereafter, the file management system obtains a preset fourth threshold value, and compares the obtained access frequency with the fourth threshold value to obtain a fourth comparison result. The file management system then determines whether the data type corresponding to the target file is hot data based on the fourth comparison result.
- the file management system determines that the fourth comparison result is that the access frequency is greater than or equal to the fourth threshold value, it determines that the data type corresponding to the target file is hot data, and then determines the corresponding target index value according to the above-mentioned target key value format corresponding to the target file, and searches for the target index management space corresponding to the target index value. Finally, the file management system determines each similar hot data file corresponding to the target metadata based on the multiple index values contained in the target index management space, and then inserts each similar hot data file into the surface storage structure in the file management system.
- the present application adopts a method of determining each similar hot data file corresponding to the hot data based on the target index value corresponding to the hot data when determining that the accessed data is hot data, and then inserting each similar hot data file into the surface storage structure in the system, thereby achieving the purpose of improving the access performance of hot data and reducing the delay generated by users when reading hot data. Furthermore, based on the various embodiments of the metadata management method of the file system of the present application above, a fourth embodiment of the metadata management method of the file system of the present application is proposed here.
- the metadata management method of the file system of the present application may further include the following steps:
- Step C10 determining each duplicate file that is duplicate of the file based on the initial key value format
- the file management system compares the initial key-value formats corresponding to each file to obtain a fifth comparison result, and determines the duplicate files corresponding to the file based on the fifth comparison result.
- Step C20 performing a merge storage operation on the file and each of the duplicate files
- the file management system performs a merge storage operation on the file and the duplicate files corresponding to the file.
- the file management system first compares the initial key-value formats ⁇ pinode: filename, stat> corresponding to each file to obtain a fifth comparison result. Thereafter, the file management system determines the duplicate files with the same initial key-value format based on the fifth comparison results, and merges the files and the duplicate files for storage, so that the memory table in the file management system can store more key-value pairs.
- the present application also provides a terminal device, which has a metadata management program for a file system that can be run on a processor, and when the terminal device executes the metadata management program for the file system, the steps of the metadata management method for the file system described in any of the above embodiments are implemented.
- the specific embodiments of the terminal device of the present application are basically the same as the embodiments of the metadata management method for the file system described above, and will not be described in detail here.
- the present application also provides a computer-readable storage medium, on which a metadata management program for a file system is stored, and when the metadata management program for the file system is executed by a processor, the steps of the metadata management method for the file system described in any of the above embodiments are implemented.
- the specific embodiments of the computer-readable storage medium of the present invention are basically the same as the embodiments of the metadata management method for the file system described above, and are not described in detail here.
- the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) as described above, including a number of instructions for enabling a terminal device (which can be a device that executes the metadata management method of the file system of the present application, and the terminal device can specifically be a mobile terminal, a data storage control terminal, a PC or a portable computer and other terminals) to execute the methods described in each embodiment of the present application.
- a terminal device which can be a device that executes the metadata management method of the file system of the present application, and the terminal device can specifically be a mobile terminal, a data storage control terminal, a PC or a portable computer and other terminals
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请公开了一种文件系统的元数据管理方法、终端设备及计算机可读存储介质，本申请数据文件的管理方法包括：基于需要存储的文件对应的路径名进行编码得到所述文件对应的初始键值格式；为初始键值格式分配对应的预分配索引数值从而得到预分配键值格式；基于预分配键值格式对预分配索引管理空间和预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间；得到目标键值格式，并基于目标键值格式将文件存储在目标存储结构内。
Description
本申请要求于2023年6月12日申请的、申请号为202310695689.3的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
本申请涉及数据处理技术领域，尤其涉及一种文件系统的元数据管理方法、终端设备及计算机可读存储介质。
当前，如MapReduce、Spark等大数据处理框架在面对大量文件创建的场景时，往往需要频繁创建和更新文件信息，如此，文件管理系统在就需要大量读取磁盘文件，从而难以满足用户的低延迟需求，而元数据读写延迟在很多场景下是影响应用性能的关键因素。
因此，如何令文件管理系统在面对存在大量文件创建的场景时，能够满足用户对于读写数据的低延迟需求，也就成了行业内亟需解决的技术问题。
本申请的主要目的在于提供一种文件系统的元数据管理方法、装置、终端设备及计算机可读存储介质，旨在令文件管理系统在存储文件数据时，能够将文件数据存储设备上连续相邻存储，从而减少设备在读取文件数据过程中访问存储设备的次数，提升文件管理系统的性能，进而使得文件管理系统在面对大量文件创建、删除场景时，能够满足用户的低延迟需求。
为实现上述目的，本申请提供一种文件系统的元数据管理方法，所述文件系统的元数据管理方法包括以下步骤：
获取需要存储的文件对应的路径名，并基于所述路径名进行编码得到所述文件对应的初始键值格式；
通过预设的预分配索引管理空间进行预分配操作以为所述初始键值格式分配对应的预分配索引数值从而得到预分配键值格式；
基于所述预分配键值格式对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间；
按照所述目标索引管理空间内包含的索引数值为所述初始键值格式分配对应的目标索引数值以得到目标键值格式，并基于所述目标键值格式将所述文件存储在目标存储结构内。
进一步地，所述基于所述路径名进行编码得到所述文件对应的初始键值格式的步骤，包括：
确定所述路径名对应的文件父目录、文件名及文件属性信息；
基于所述文件父目录对应的父目录索引数值、所述文件名及所述文件属性信息进行编码得到所述文件对应的初始键值格式。
进一步地，所述基于所述预分配键值格式对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间的步骤，包括：
将所述预分配键值格式内包含的所述预分配索引数值和所述父目录索引数值进行比对得到第一比对结果；
若判断到所述第一比对结果为所述预分配索引数值和所述父目录索引数值之间的差值大于预设的第一阈值，则基于所述父目录索引数值对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间。
进一步地，所述基于所述父目录索引数值对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间的步骤，包括：
判断所述预分配索引管理空间对应的空间分配数值是否小于预设的第二阈值；
若判断到所述空间分配数值小于所述第二阈值，则将所述预分配索引管理空间合并入所述父目录索引管理空间以得到目标索引管理空间。
进一步地，所述基于所述父目录索引数值对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间的步骤，还包括：
判断所述父目录索引管理空间对应的预留空间是否被用完；
若判断到所述预留空间被用完，则确定所述预分配索引管理空间和所述父目录索引管理空间各自对应的已分配结点数量；
基于各所述已分配结点数量对所述预分配索引管理空间和所述父目录索引管理空间执行结点平衡操作，并将执行所述平衡操作后的预分配索引管理空间确定为目标索引管理空间。
进一步地，在所述基于所述目标键值格式将所述文件存储在目标存储结构内的步骤之后，所述方法还包括：
在接收到文件删除指令时检测前台负载，并判断所述前台负载是否大于或等于预设的第三阈值；
若判断到所述前台负载大于或等于所述第三阈值，则向所述文件删除指令对应的目标文件目录插入预设的全删除键；
若判断所述前台负载小于所述第三阈值，则向所述目标文件目录和所述目标文件目录对应的各子文件目录递归插入预设的删除键。
进一步地，在所述基于所述目标键值格式将所述文件存储在目标存储结构内的步骤之后，所述方法还包括：
在接收到文件访问指令时，根据所述文件访问指令确定目标文件，并检测所述目标文件对应的访问频率；
基于所述访问频率判断所述目标文件对应的数据类型是否为热数据；
若判断到所述数据类型为所述热数据，则确定所述目标文件对应的所述目标索引管理空间；
基于所述目标索引管理空间确定所述目标文件对应的各相似热数据文件，将各所述相似热数据文件插入到预设的日志合并存储结构树。
进一步地，在所述通过预设的预分配索引管理空间进行预分配操作以为所述初始键值格式分配对应的预分配索引数值从而得到预分配键值格式的步骤之前，所述方法还包括：
基于所述初始键值格式确定与文件重复的各重复文件；
对所述文件和各所述重复文件执行合并存储操作。
此外，为实现上述目的，本申请还提供一种终端设备，所述终端设备包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的文件系统的元数据管理程序，所述文件系统的元数据管理程序被所述处理器执行时实现如上述的文件系统的元数据管理方法的步骤。
此外，为实现上述目的，本申请还提供一种计算机可读存储介质，所述计算机可读存储介质上存储有文件系统的元数据管理程序，所述文件系统的元数据管理程序被处理器执行时实现如上述的文件系统的元数据管理方法的步骤。
本申请实施例提供的文件系统的元数据管理方法、装置、终端设备及计算机介质，通过获取需要存储的文件对应的路径名，并基于所述路径名进行编码得到所述文件对应的初始键值格式；通过预设的预分配索引管理空间进行预分配操作以为所述初始键值格式分配对应的预分配索引数值从而得到预分配键值格式；基于所述预分配键值格式对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间；按照所述目标索引管理空间内包含的索引数值为所述初始键值格式分配对应的目标索引数值以得到目标键值格式，并基于所述目标键值格式将所述文件存储在目标存储结构内。
在本实施例中，终端设备在运行时，由终端设备内的文件管理系统首先获取需要进行存储的文件，并确定文件对应的路径名，从而基于该文件名对文件进行编码从而得到文件对应的初始键值格式，之后，文件管理系统获取预设的预分配索引管理空间，并基于该预分配索引数值空间对需要存储的文件对应的初始键值格式执行预分配操作，从而为初始键值格式分配对应的预分配索引数值以生成预分配键值格式，再之后，文件管理系统基于预分配键值格式内包含的预分配索引数值和父目录索引数值对预分配索引管理空间和父目录索引管理空间执行合并操作生成目标索引管理空间，最后，文件管理系统按照目标索引管理空间内包含的索引数值为文件对应的初始键值格式分配对应的目标索引数值从而生成对应的目标键值格式，文件管理系统进而按照该目标键值格式将文件保存在存储装置对应的存储结构中。
本申请通过令文件管理系统基于索引管理空间内包含的连续的索引数值，将文件存储在存储结构内的连续位置上的方式，达到了减少设备在读取文件数据过程中访问存储设备的次数，提升文件管理系统的性能的技术效果，进而使得文件管理系统在面对大量文件创建、删除场景时，能够满足用户的低延迟需求。
图1是本申请实施例方案涉及的硬件运行环境的终端设备的结构示意图；
图2为本申请文件系统的元数据管理方法第一实施例的流程示意图；
图3为本申请文件系统的元数据管理方法一实施例涉及的索引管理空间布局示意图；
图4为本申请文件系统的元数据管理方法一实施例涉及的系统结构示意图。
本申请目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
参照图1，图1为本申请实施例方案涉及的硬件运行环境的终端设备结构示意图。
需要说明的是，图1即可为终端设备的硬件运行环境的结构示意图。本申请实施例终端设备可以是执行本申请文件系统的元数据管理方法的设备，该终端设备具体可以是移动终端、数据存储控制终端、PC或者便携计算机等终端。
如图1所示，该终端设备可以包括：处理器1001，例如中央处理器（Central Processing Unit，CPU），通信总线1002、用户接口1003，网络接口1004，存储器1005。其中，通信总线1002用于实现这些组件之间的连接通信。用户接口1003可以包括显示屏（Display）、输入单元比如键盘（Keyboard），用户接口1003还可以包括标准的有线接口、无线接口。网络接口1004可以包括标准的有线接口、无线接口（如无线保真（WIreless-FIdelity，WI-FI）接口）。存储器1005可以是高速的随机存取存储器（Random Access Memory，RAM）存储器，也可以是稳定的非易失性存储器（Non-Volatile Memory，NVM），例如磁盘存储器。存储器1005还可以是独立于前述处理器1001的存储装置。
本领域技术人员可以理解，图1中示出的结构并不构成对终端设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
如图1所示，作为一种存储介质的存储器1005中可以包括操作系统、数据存储模块、网络通信模块、用户接口模块以及文件系统的元数据管理程序。
在图1所示的终端设备中，网络接口1004主要用于与其他设备进行数据通信；用户接口1003主要用于与用户进行数据交互；本申请终端设备中的处理器1001、存储器1005可以设置在终端设备中，所述终端设备通过处理器1001调用存储器1005中存储的文件系统的元数据管理程序，并执行本申请实施例提供的文件系统的元数据管理方法。
基于上述的终端设备，提供本申请文件系统的元数据管理方法的各个实施例。
需要说明的是，面向存储设备的本地文件系统是一切数据存储与管理系统的基础。例如，主流的数据库都建立在文件系统之上，分布式文件系统必须依托本地文件系统实现设备上的数据组织，其中，以Hbase、Dynamo为代表的大数据存储系统要么构建在分布式文件系统之上，要么直接调用本地文件系统读写存储设备上的数据。如此，本地文件系统的性能对所有数据存储与管理系统都有着至关重要的影响。
其中，本地文件系统从结构上大致可划分为两个组件：名字空间管理和地址空间管理。地址空间管理则主要负责数据在存储设备上的组织。一般说来，无论存储设备采用什么技术手段实现，设备厂商都会将设备的物理地址空间抽象为线性的逻辑地址空间，而地址空间管理模块则负责将用户数据和文件系统元数据组织在线性的逻辑地址空间上。因此，名字空间管理的主要功能是对文件和目录名称进行组织和管理，为用户提供文件系统的操作接口，实现由文件名到文件信息的路径解析过程。
而名字空间管理的常见分类有：层次型、扁平型、哈希型等。其中，层次型文件系统是指文件和目录之间存在父子关系，形成一棵树状结构，树的每个节点代表一个文件或目录。这种结构使得文件系统中的文件和目录可以被方便地组织和管理。在这种结构中，每个节点都有一个唯一的路径名，用于标识该节点在树形结构中的位置。例如，/usr/local/bin就是一个路径名，表示根目录下的usr目录下的local目录下的bin目录。进行路径解析时，需要从树的根节点出发，自上而下依次访问中间目录文件，最后得到目标文件信息；
同样的，扁平型文件系统是指所有的文件和目录都在同一级别上，没有父子关系。哈希型文件系统是指将文件名通过哈希函数计算得到一个唯一的值，然后根据这个值来存储和查找文件。层次型名字空间管理。与层次式管理相比，它的路径解析更加简单，不需要按层级依次解析。但是它没有保留同一层级相邻文件关系，在涉及多个文件的处理操作比较复杂，比如文件重命名和列出目录文件信息操作。
不同的文件系统采用不同的名字空间管理方法。例如，Ext系列文件系统（比如Ext2、Ext3、Ext4）采用层次式结构来管理名字空间，通过定义目录类型的文件来保存同一层次下文件的信息，在路径查找过程中，对于目标路径的每一层，Ext系列文件系统都必须搜索目标目录的inode（即索引节点）并解析目录内容以获得下一层目录的inode号。如此，Ext系统文件系统在路径查找过程中会产生大量磁盘I/O（即输入/输出），这不仅是因为目录解析，还因为它的索引节点的预取机制。然而，这种机制在解析深的路径时没有效果，因为Ext很难保证磁盘上的元数据的纵向局部性。Ext目录文件存储的文件信息直接线性的排布于存储设备，同一目录的子文件数量太大时，查找一个子文件需要遍历所有文件，将导致查询性能下降严重。Ext3之后对目录文件里面所存的子文件排列进行优化，它采用了Hash和B+树来组织子文件，加速同一层的文件查询性能。然而，在面对写入密集型的工作负载时，受大量随机小写的影响，Ext文件系统的性能将大大下降；除此之外，文件数量多，文件的路径较深时，在进行路径解析方面的性能也将下降，比如文件深度为N，则需要进行N次磁盘的I/O，这大大降低了路径解析的性能。
同样的，BtrFS是一种新型的写时复制（CoW）Linux文件系统，它本身采用B-tree存储目录项，所以在给定目录下搜索文件的效率非常高。为了满足readdir的需求，BtrFS在每次创建新的文件时，除了插入以hash值为Key的目录项外，还同时插入另外一种目录项索引，该目录项索引的Key以sequence number（即序列号）作为B-tree的键值，sequence number在每次创建新文件时线性增加，所以可以使得readdir读出的文件有序。虽然BtrFS给定目录下搜索文件的效率非常高，但是涉及文件数量较多，且文件路径较深的文件时，B树的深度会越来越深，在B树种搜索目标文件的延迟越来越大，从而导致文件访问性能的下降。不仅如此，和Ext系列文件系统类似，在面对写入数据密集型的负载时，受大量随机小写影响，文件系统性能大大下降。
同样的，TableFS是一个基于LSM-Tree的文件系统，堆叠于下层文件系统之上，它确保元数据以非覆盖、排序的日志形式写入磁盘。由于LSM树对写入友好的特点，TableFS对写密集型的场景非常友好，可以将随机小写聚合顺序写入到磁盘。但是LSM-Tree本身对读不友好，所以直接将LSM-Tree作为元数据存储结构时，文件系统的读操作的性能不佳。TableFS的读文件操作（比如stat）性能远低于常见的文件系统比如ext4，原因是其路径解析需要多次读取LSM-Tree，特别是当路径上各文件不在同一个SSTable时，需要多次访问磁盘，造成读取性能低下，延迟较高。综上，目前的各大数据处理框架在面对大量文件创建的场景时，往往需要频繁创建和更新文件信息，如此，文件管理系统在就需要大量读取磁盘文件，从而难以满足用户的低延迟需求，而元数据读写延迟在很多场景下是影响应用性能的关键因素。因此，如何设计一个面向存在大量文件创建的场景时，仍然对写入密集友好、文件路径解析低延迟的本地文件系统，也就成了行业内亟需解决的技术问题。
针对上述技术问题，本申请提出了一种文件系统的元数据管理方法、终端设备及计算机可读存储介质，通过获取需要存储的文件对应的路径名，并基于所述路径名进行编码得到所述文件对应的初始键值格式；通过预设的预分配索引管理空间进行预分配操作以为所述初始键值格式分配对应的预分配索引数值从而得到预分配键值格式；基于所述预分配键值格式对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间；按照所述目标索引管理空间内包含的索引数值为所述初始键值格式分配对应的目标索引数值以得到目标键值格式，并基于所述目标键值格式将所述文件存储在目标存储结构内。如此，本申请通过令文件管理系统基于索引管理空间内包含的连续的索引数值，将文件存储在存储结构内的连续位置上的方式，达到了减少设备在读取文件数据过程中访问存储设备的次数，提升文件管理系统的性能的技术效果，进而使得文件管理系统在面对大量文件创建、删除场景时，能够满足用户的低延迟需求。
请参照图2，图2为本申请文件系统的元数据管理方法第一实施例的流程示意图。
应当理解的是，虽然在流程图中示出了逻辑顺序，但是在某些情况下，本申请文件系统的元数据管理方法当然也可以以不同于此处的顺序执行所示出或描述的步骤。
在本实施例中，本申请文件系统的元数据管理方法，可以包括以下步骤：
步骤S10：获取需要存储的文件对应的路径名，并基于所述路径名进行编码得到所述文件对应的初始键值格式；
在本实施例中，终端设备在运行时，首先获取需要存储的文件，并由终端设备内的文件管理系统确定该文件对应的路径名，并基于该按照预设的编码格式和该路径名进行编码从而得到初始键值格式。
示例性地，例如，终端设备在运行时，首先获取需要进行存储的文件，并由终端设备内的文件管理系统确定该文件对应的路径名，之后，文件管理系统按照预设的键值格式和该路径名进行编码从而得到与文件对应的初始键值格式：<pinode：filename，stat>。
可以理解的是，采用上述的键值格式比起当前技术中将目录或文件的完整路径作为键的格式更容易被接受，因为它执行移动（move）或重命名（rename）操作的开销更低。
进一步地，在一种可行的实施例中，上述步骤S10中“对所述文件进行编码得到所述文件对应的初始键值格式”的步骤，具体可以包括：
步骤S101：确定所述路径名对应的文件父目录、文件名及文件属性信息；
步骤S102：基于所述文件父目录对应的父目录索引数值、所述文件名及所述文件属性信息进行编码得到所述文件对应的初始键值格式；
示例性地，例如，文件管理系统首先根据路径名确定文件对应的文件父目录，从而确定该文件父目录对应的父目录索引数值pinode，同时，文件管理系统确定该文件对应的文件名filename，同时，文件管理系统确定该文件对应的文件属性信息stat，之后，文件管理系统基于获取的父目录索引数值pinode、文件名filename及文件属性信息stat进行编码从而得到文件对应的初始键值格式<pinode：filename，stat>。
步骤S20：通过预设的预分配索引管理空间进行预分配操作以为所述初始键值格式分配对应的预分配索引数值从而得到预分配键值格式；
进一步地，请参照图3和图4，其中，图3为本申请文件系统的元数据管理方法一实施例涉及的索引管理空间布局示意图，图4为本申请文件系统的元数据管理方法一实施例涉及的系统结构示意图；索引管理空间为用于对索引数值（即inode_number）进行管理的空间，其中，一个索引数值对应的大小应设置为64位；如此，索引管理空间可以标识2
64个文件，如图4所示，由于一个文件对应的键值的大小为256B，而一个4KiB的存储块可以容纳16个键值对，因此，该索引管理空间的大小默认为16，且其中会预留一部分空间用于后续的合并。
此外，索引管理空间内的索引数值在字典序上应相近，且该索引管理空间内的各索引数值中应包含一个最小索引数值min_inode_number和一个最大索引数值max_inode_number，如此，当存在新的文件创建时，文件管理系统可以在索引管理空间inode_space内确定一个索引数值，并将该索引数值对应的索引结点inode分配给新创建的文件，而当该索引管理空间inode_space内的索引结点inode已经都被分配完毕，则文件管理系统可以创建一个新的索引管理空间inode_space。
如图3所示，上述索引管理空间inode_space分为内存中（即图3中的DRAM）的索引表和存储设备（即图3中的SSD）上的线性表格，其中，线性表格的表项包含根节点inode号（即图3中的root inode），该索引管理空间inode_space可分配的最大inode号max_inode、预留分配的inode总数step及当前已分配的数量count，可以理解的是，该预留分配的inode总数step即该索引管理空间inode_space对应的预留空间，而该已分配的数量count对应该索引管理空间inode_space对应的已分配结点。
在本实施例中，文件管理系统获取预设的用于执行预分配操作的预分配索引管理空间，并基于该预分配索引管理空间执行索引数值预分配操作从而为生成的初始键值格式分配对应的预分配索引数值，文件管理系统进而将该预分配索引数值写入该初始键值格式以生成预分配键值格式。示例性地，例如，文件管理系统首先获取预设的用于执行预分配操作的预分配索引管理空间inode_space1，并基于该预分配索引管理空间inode_space1执行预分配操作从而为初始键值格式<pinode：filename，stat>分配对应的预分配索引数值cur_inode，之后，文件管理系统将分配的预分配索引数值cur_inode写入该初始键值格式以得到预分配键值格式：<pinode：file_name，cur_inode…>。
可以理解的是，基于预分配键值格式能够令同一目录下子文件的key均以父目录的inode开头，从而令同一目录的子文件会被连续相邻存于存储设备。由于存储设备一般读写粒度为KB级别，比如SSD的最小读粒度为4KiB，一次I/O可以读取多个存储设备上相邻排布的文件元数据信息，如此，可以大大提升文件系统范围查询的性能。
步骤S30：基于所述预分配键值格式对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间；
在本实施例中，文件管理系统基于预分配键值格式内包含的预分配索引数值和父目录索引数值，对该预分配键值格对应的预分配索引管理空间和上述父目录索引管理空间执行合并操作从而得到目标索引管理空间。示例性地，例如，文件管理系统基于该预分配键值格式<pinode：file_name，cur_inode …>内包含的预分配索引数值cur_inode和父目录索引数值pinode对应的父目录索引管理空间inode_space2，和预分配索引管理空间inode_space1执行合并操作从而得到目标索引管理空间inode_space。
进一步地，在一种可行的实施例中，上述步骤S30，具体可以包括：
步骤S301：将所述预分配键值格式内包含的所述预分配索引数值和所述父目录索引数值进行比对得到第一比对结果；
在本实施例中，文件管理系统首先确定预分配键值格式内包含的预分配索引数值和父目录索引数值，并将该预分配索引数值和该父目录索引数值进行比对得到预分配索引数值和父目录索引数值之间产生的差值，同时，文件管理系统获取技术人员预设的第一阈值，并将该差值与该第一阈值进行比对得到第一比对结果，文件管理系统进而基于该第一比对结果判断预分配索引数值和父目录索引数值之间是否相差过大。
步骤S302：若判断到所述第一比对结果为所述预分配索引数值和所述父目录索引数值之间的差值大于预设的第一阈值，则基于所述父目录索引数值对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间；
在本实施例中，文件管理系统若判断到第一比对结果为上述差值大于或等于预设的第一阈值，则确定预分配索引数值和父目录索引数值之间相差过大，从而基于上述父目录索引数值对父目录索引管理空间和预分配索引管理空间执行合并操作得到目标索引管理空间。
示例性地，例如，文件管理系统首先确定预分配键值格式<pinode：file_name，cur_inode …>内包含的预分配索引数值cur_inode和父目录索引数值pinode，并将该预分配索引数值cur_inode和该父目录索引数值pinode进行比对以得到两者之间的差值，同时，文件管理系统获取预设的第一阈值，并将该差值与该第一阈值进行比对得到第一比对结果，之后，文件管理系统若判断到第一比对结果为上述差值大于或等于该第一阈值，则确定预分配索引数值cur_inode和父目录索引数值pinode之间相差过大，文件管理系统进而基于该预分配键值格式<pinode：file_name，cur_inode …>内父目录索引数值pinode对应的父目录索引管理空间inode_space2，和预分配索引数值cur_inode对应的预分配索引管理空间inode_space1执行合并操作，从而得到目标索引管理空间inode_space。
此外，在另一实施例中，文件管理系统还可以计算预分配索引管理空间inode_space1对应的空闲率α=1-count/（i max_inode-root_inode），并将该空闲率与预设的空闲率阈值进行比对，之后，文件管理系统若确定该空闲率高于该空闲率阈值，则确定执行合并操作。
进一步地，在一种可行的实施例中，上述步骤S302中“基于所述父目录索引数值对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间”的步骤，具体可以包括：
步骤S3021：判断所述预分配索引管理空间对应的空间分配数值是否小于预设的第二阈值；
步骤S3022：若判断到所述空间分配数值小于所述第二阈值，则将所述预分配索引管理空间合并入所述父目录索引管理空间以得到目标索引管理空间；
示例性地，例如，当文件管理系统确定执行合并操作时，首先获取技术人员预设的第二阈值，并将预分配索引管理空间inode_space1内对应的空间分配数与该第二阈值进行比对以得到第二比对结果，之后，文件管理系统若判断到第二比对结果为该空间分配数小于该第二阈值，则确定该索引管理空间inode_space1内的字文件较少，因此，不需要维护该索引管理空间inode_space1内的局部性，文件管理系统从而将该预分配索引管理空间inode_space1合并到上述父目录索引管理空间inode_space2内，从而将合并后的父目录索引管理空间inode_space2作为目标索引管理空间inode_space。
进一步地，在一种可行的实施例中，上述步骤S302中“基于所述父目录索引数值对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间”的步骤，还可以包括：
步骤S3023：判断所述父目录索引管理空间对应的预留空间是否被用完；
在本实施例中，当文件管理系统确定执行合并操作时，首先判断父目录索引管理空间内包含的预留空间是否被用完。
步骤S3024：若判断到所述预留空间被用完，则确定所述预分配索引管理空间和所述父目录索引管理空间各自对应的已分配结点数量；
在本实施例中，文件管理系统若判断待该预留空间已经被用完，则确定需要对父目录索引管理空间执行分裂操作，进而对预分配索引管理空间和父目录索引管理空间分别进行检测，从而确定预分配索引管理空间和父目录索引管理空间各自对应的已分配结点数量。
步骤S3025：基于各所述已分配结点数量对所述预分配索引管理空间和所述父目录索引管理空间执行结点平衡操作，并将执行所述平衡操作后的预分配索引管理空间确定为目标索引管理空间；
在本实施例中，文件管理系统基于获取的各已分配结点数量对父目录索引管理空间和预分配索引管理空间执行结点平衡操作，从而将父目录索引管理空间内的部分结点迁移到预分配索引管理空间内，以平衡两个索引管理空间内的结点数量，文件管理系统进而将平衡操作后的预分配索引管理空间确定为目标索引管理空间。
示例性地，例如，文件管理系统在确定执行合并操作时，首先判断父目录索引管理空间inode_space2内包含的预留空间对应的工作状态，并基于该工作状态判断该预留空间是否被用完，之后，文件管理系统若判断到该预留空间被用完，则确定需要执行索引管理空间的分裂操作，并检测预分配索引管理空间inode_space1和父目录索引管理空间inode_space2各自对应的已分配结点数量从而确定需要迁移的inode节点范围[x，y]，同时，文件管理系统确定该迁移操作涉及的存储Level为L
k和L
k-1，从而在文件管理系统内的L
0到L
k层之间搜索满足x:*<= Key<= y:*条件的键值对，并修改键值对的inode为new_inode，如此，文件管理系统基于获取的各已分配结点数量确定预分配索引管理空间inode_space1和父目录索引管理空间inode_space2对应的平均结点数量，并基于该平均结点数量按照上述迁移操作将父目录索引管理空间inode_space2内包含的部分结点迁移到预分配索引管理空间inode_space1以完成结点平衡操作，从而将平衡后的预分配索引管理空间inode_space1确定为目标索引管理空间并更新对应的索引表信息。
步骤S40：按照所述目标索引管理空间内包含的索引数值为所述初始键值格式分配对应的目标索引数值以得到目标键值格式，并基于所述目标键值格式将所述文件存储在目标存储结构内；
在本实施例中，文件管理系统在完成合并操作之后，基于得到的目标索引管理空间内包含的索引数值执行索引分配操作，从而为初始键值格式分配对应的目标索引数值以生成目标键值格式，文件管理信息进而按照该目标键值格式将该文件存储在对应的存储结构内。
示例性地，例如，文件管理系统在完成合并操作之后，基于得到目标索引管理空间inode_space执行inode分配操作，从而为初始键值格式<pinode：filename，stat>分配与其对应的目标索引数值cur_inode，以得到目标键值格式，之后，文件管理系统基于该目标键值格式将对应的文件存储于目标存储结构内。
在本实施例中，终端设备在运行时，首先获取需要存储的文件，并由终端设备内的文件管理系统确定该文件对应的路径名，并基于该按照预设的编码格式和该路径名进行编码从而得到初始键值格式，之后，文件管理系统获取预设的用于执行预分配操作的预分配索引管理空间，并基于该预分配索引管理空间执行索引数值预分配操作从而为生成的初始键值格式分配对应的预分配索引数值，文件管理系统进而将该预分配索引数值写入该初始键值格式以生成预分配键值格式，再之后，文件管理系统基于预分配键值格式内包含的预分配索引数值和父目录索引数值，对该预分配键值格对应的预分配索引管理空间和上述父目录索引管理空间执行合并操作从而得到目标索引管理空间，最后，文件管理系统在完成合并操作之后，基于得到的目标索引管理空间内包含的索引数值执行索引分配操作，从而为初始键值格式分配对应的目标索引数值以生成目标键值格式，文件管理信息进而按照该目标键值格式将该文件存储在对应的存储结构内。
如此，本申请通过令文件管理系统基于索引管理空间内包含的连续的索引数值，将文件存储在存储结构内的连续位置上的方式，达到了减少设备在读取文件数据过程中访问存储设备的次数，提升文件管理系统的性能的技术效果，进而使得文件管理系统在面对大量文件创建、删除场景时，能够满足用户的低延迟需求。进一步地，基于上述本申请文件系统的元数据管理方法的第一实施例，在此提出本申请文件系统的元数据管理方法第二实施例。
在上述步骤S40之后，本申请文件系统的元数据管理方法还可以包括以下步骤：
步骤A10：在接收到文件删除指令时检测前台负载，并判断所述前台负载是否大于或等于预设的第三阈值；
在本实施例中，文件管理系统在接收到文件删除指令时，获取技术人员预设的第三阈值，同时，文件管理系统基于该文件删除指令对终端设备进行检测，从而获取该终端设备当前的前台负载，并将该前台负载与该第三阈值进行比对得到第三比对结果。
步骤A20：若判断到所述前台负载大于或等于所述第三阈值，则向所述文件删除指令对应的目标文件目录插入预设的全删除键；
在本实施例中，文件管理系统若判断到第三比对结果为前台负载大于或等于上述第三阈值，则根据该文件删除指令确定需要删除的目标文件目录，并将预设的全删除键值插入该目标文件目录下，以令文件管理系统在执行合并操作时，由全删除键执行文件回收操作从而完成对目标文件目录的删除操作。
步骤A30：若判断所述前台负载小于所述第三阈值，则向所述目标文件目录和所述目标文件目录对应的各子文件目录递归插入预设的删除键；
在本实施例中，文件管理系统若判断到第三比对结果为前台负载小于上述第三阈值，则根据该文件删除指令确定需要删除的目标文件目录，和该目标文件目录下的各子目录，文件管理系统进而在该目标文件目录和各该子目录下递归插入删除键，以在合并操作过程中将标记有该删除键的文件目录移除从而完成删除操作。
示例性地，例如，当用户需要删除文件时，可通过终端设备生成文件删除指令，终端设备进而将该文件删除指令发送到文件管理系统，之后，当文件管理系统接收到该文件删除指令时，首先获取技术人员预设的第三阈值并确定需要删除的目标文件对应的目标文件目录，同时，文件管理系统检测终端设备对应的前台负载，文件管理系统进而将该前台负载与该第三阈值进行比对得到第三比对结果，再之后，文件管理系统若确定第三比对结果为前台负载大于或等于该第三阈值时，则获取预设的全删除键full_delete_key，并将该全删除键full_delete_key插入该目标文件目录下，以供文件管理系统在存储结构SSTable里搜索key目录包含全删除键full_delete_key的文件，进而在执行上述索引管理空间合并操作时，将包含该全删除键full_delete_key的文件对应的文件目录直接删除；
而若文件管理系统确定第三比对结果为前台负载小于该第三阈值时，则根据该文件删除指令确定需要删除的目标文件对应的目标文件目录，和该目标文件目录下的各子目录，文件管理系统进而将预设的删除键递归插入各该子目录，以在后续的合并过程中将这些删除键和带有该删除键的文件从存储设备中移除。
在本实施例中，文件管理系统在接收到文件删除指令时，获取技术人员预设的第三阈值，同时，文件管理系统基于该文件删除指令对终端设备进行检测，从而获取该终端设备当前的前台负载，并将该前台负载与该第三阈值进行比对得到第三比对结果，之后，文件管理系统若判断到第三比对结果为前台负载大于或等于上述第三阈值，则根据该文件删除指令确定需要删除的目标文件目录，并将预设的全删除键值插入该目标文件目录下，以令文件管理系统在执行合并操作时，由全删除键执行文件回收操作从而完成对目标文件目录的删除操作，最后，文件管理系统若判断到第三比对结果为前台负载小于上述第三阈值，则根据该文件删除指令确定需要删除的目标文件目录，和该目标文件目录下的各子目录，文件管理系统进而在该目标文件目录和各该子目录下递归插入删除键，以在合并操作过程中将标记有该删除键的文件目录移除从而完成删除操作。
如此，本申请通过确定终端设备的前台负载情况，在前台负载情况高于预设的阈值时，向需要删除的目标文件目录内插入全删除键，而在前台负载情况小于阈值时，向目标文件目录和对应的各子目录下递归插入删除键的方式，令文件管理系统能够在执行文件删除操作时根据前台负载情况选择效率更好的文件删除方案，从而达到了快速回收存储设备上的空间，提高文件管理系统的性能。
进一步地，基于上述本申请文件系统的元数据管理方法的第一实施例和/或者第二实施例，在此提出本申请文件系统的元数据管理方法第三实施例。
在上述步骤S40之后，本申请文件系统的元数据管理方法还可以包括以下步骤：
步骤B10：在接收到文件访问指令时，根据所述文件访问指令确定目标文件，并检测所述目标文件对应的访问频率；
在本实施例中，文件管理系统在接收到文件访问指令时，则通过预设的时间窗口统计该文件访问指令对应的目标文件的访问频率。
步骤B20：基于所述访问频率判断所述目标文件对应的数据类型是否为热数据；
在本实施例中，文件管理系统获取预设的第四阈值，并将获取的访问频率与该第四阈值进行比对得到第四比对结果，文件管理系统进而基于该第四比对结果判断该目标文件对应的数据类型是否为热数据。
步骤B30：若判断到所述数据类型为所述热数据，则确定所述目标文件对应的所述目标索引管理空间；
在本实施例中，文件管理系统若确定第四比对结果为访问频率大于或等于该第四阈值，则确定该目标文件对应的数据类型为热数据，进而根据该目标文件对应的上述目标键值格式确定对应的目标索引数值，并查找该目标索引数值对应的目标索引管理空间。
步骤B40：基于所述目标索引管理空间确定所述目标文件对应的各相似热数据文件，将各所述相似热数据文件插入到预设的日志合并存储结构树；
在本实施例中，文件管理系统基于目标索引管理空间内包含的多个索引数值，确定与该目标元数据对应的各相似热数据文件，进而将各该相似热数据文件插入文件管理系统内的表层存储结构中。示例性地，例如，文件管理系统在接收到文件访问指令时，在系统后台开一个时间窗口，并通过该时间窗口统计该文件访问指令对应的目标文件的访问频率freq，同时，文件管理系统获取预设的第四阈值，并将该访问频率freq与该第四阈值进行比对得到第四比对结果，之后，若文件管理系统确定该第四比对结果为该访问频率freq大于或等于该第四阈值，则确定该目标文件对应的数据类型为热文件filehot，文件管理系统进而确定该目标文件对应的目标索引数值inode
hot，并确定该目标索引数值inode
hot对应的目标索引管理空间inode_space
hot，再之后，文件管理系统将基于该索引管理空间inode_space
hot确定该目标文件对应的各相似热数据文件，并将各该相似热数据文件插入到文件管理系统内较低层级的SSTable中，以使得用户能更快的访问各该热数据文件。
在本实施例中，文件管理系统在接收到文件访问指令时，则通过预设的时间窗口统计该文件访问指令对应的目标文件的访问频率，之后，文件管理系统获取预设的第四阈值，并将获取的访问频率与该第四阈值进行比对得到第四比对结果，文件管理系统进而基于该第四比对结果判断该目标文件对应的数据类型是否为热数据，再之后，文件管理系统若确定第四比对结果为访问频率大于或等于该第四阈值，则确定该目标文件对应的数据类型为热数据，进而根据该目标文件对应的上述目标键值格式确定对应的目标索引数值，并查找该目标索引数值对应的目标索引管理空间，最后，文件管理系统基于目标索引管理空间内包含的多个索引数值，确定与该目标元数据对应的各相似热数据文件，进而将各该相似热数据文件插入文件管理系统内的表层存储结构中。
如此，本申请采用在确定被访问的数据为热数据时，基于该热数据对应的目标索引数值确定与该热数据对应的各相似热数据文件，进而将各该相似热数据文件插入系统内的表层存储结构的方式，达到了提升热数据的访问性能，降低用户在读取热数据时产生的延迟的目的。进一步地，基于上述本申请文件系统的元数据管理方法的各实施例，在此提出本申请文件系统的元数据管理方法第四实施例。
在上述步骤S20之前，本申请文件系统的元数据管理方法还可以包括以下步骤：
步骤C10：基于所述初始键值格式确定与文件重复的各重复文件；
在本实施例中，文件管理系统将各文件各自对应的初始键值格式进行比对从而得到第五比对结果，并基于该第五比对结果确定与文件对应的各重复文件。
步骤C20：对所述文件和各所述重复文件执行合并存储操作；
在本实施例中，文件管理系统将文件，和该文件对应的各重复文件执行合并存储操作。
示例性地，例如，文件管理系统首先将基于各文件各自对应的初始键值格式<pinode：filename， stat>分别进行比对得到第五比对结果，之后，文件管理系统基于各该第五比对结果确定初始键值格式一致的各重复文件，并将文件和重复的各重复文件进行合并存储操作，从而令文件管理系统中的内存表可以存储更多的键值对。
此外，本申请还提供一种终端设备，该终端设备上有可在处理器上运行的文件系统的元数据管理程序，所述终端设备执行所述文件系统的元数据管理程序时实现如以上任一项实施例所述的文件系统的元数据管理方法的步骤。本申请终端设备的具体实施例与上述文件系统的元数据管理方法各实施例基本相同，在此不作赘述。
此外，本申请还提供一种计算机可读存储介质，该计算机可读存储介质上存储有文件系统的元数据管理程序，所述文件系统的元数据管理程序被处理器执行时实现如以上任一项实施例所述文件系统的元数据管理方法的步骤。本发计算机可读存储介质的具体实施例与上述文件系统的元数据管理方法各实施例基本相同，在此不作赘述。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者系统不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者系统所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者系统中还存在另外的相同要素。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在如上所述的一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是执行本申请文件系统的元数据管理方法的设备，该终端设备具体可以是移动终端、数据存储控制终端、PC或者便携计算机等终端)执行本申请各个实施例所述的方法。
以上仅为本申请的优选实施例，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本申请的专利保护范围内。
Claims (10)
- 一种文件系统的元数据管理方法，其中，所述文件系统的元数据管理方法包括以下步骤：获取需要存储的文件对应的路径名，并基于所述路径名进行编码得到所述文件对应的初始键值格式；通过预设的预分配索引管理空间进行预分配操作以为所述初始键值格式分配对应的预分配索引数值从而得到预分配键值格式；基于所述预分配键值格式对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间；按照所述目标索引管理空间内包含的索引数值为所述初始键值格式分配对应的目标索引数值以得到目标键值格式，并基于所述目标键值格式将所述文件存储在目标存储结构内。
- 如权利要求1所述的文件系统的元数据管理方法，其中，所述基于所述路径名进行编码得到所述文件对应的初始键值格式的步骤，包括：确定所述路径名对应的文件父目录、文件名及文件属性信息；基于所述文件父目录对应的父目录索引数值、所述文件名及所述文件属性信息进行编码得到所述文件对应的初始键值格式。
- 如权利要求2所述的文件系统的元数据管理方法，其中，所述基于所述预分配键值格式对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间的步骤，包括：将所述预分配键值格式内包含的所述预分配索引数值和所述父目录索引数值进行比对得到第一比对结果；若判断到所述第一比对结果为所述预分配索引数值和所述父目录索引数值之间的差值大于预设的第一阈值，则基于所述父目录索引数值对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间。
- 如权利要求3所述的文件系统的元数据管理方法，其中，所述基于所述父目录索引数值对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间的步骤，包括：判断所述预分配索引管理空间对应的空间分配数值是否小于预设的第二阈值；若判断到所述空间分配数值小于所述第二阈值，则将所述预分配索引管理空间合并入所述父目录索引管理空间以得到目标索引管理空间。
- 如权利要求4所述的文件系统的元数据管理方法，其中，所述基于所述父目录索引数值对所述预分配索引管理空间和所述预分配键值格式对应的父目录索引管理空间执行合并操作得到目标索引管理空间的步骤，还包括：判断所述父目录索引管理空间对应的预留空间是否被用完；若判断到所述预留空间被用完，则确定所述预分配索引管理空间和所述父目录索引管理空间各自对应的已分配结点数量；基于各所述已分配结点数量对所述预分配索引管理空间和所述父目录索引管理空间执行结点平衡操作，并将执行所述平衡操作后的预分配索引管理空间确定为目标索引管理空间。
- 如权利要求1所述的文件系统的元数据管理方法，其中，在所述基于所述目标键值格式将所述文件存储在目标存储结构内的步骤之后，所述方法还包括：在接收到文件删除指令时检测前台负载，并判断所述前台负载是否大于或等于预设的第三阈值；若判断到所述前台负载大于或等于所述第三阈值，则向所述文件删除指令对应的目标文件目录插入预设的全删除键；若判断所述前台负载小于所述第三阈值，则向所述目标文件目录和所述目标文件目录对应的各子文件目录递归插入预设的删除键。
- 如权利要求1所述的文件系统的元数据管理方法，其中，在所述基于所述目标键值格式将所述文件存储在目标存储结构内的步骤之后，所述方法还包括：在接收到文件访问指令时，根据所述文件访问指令确定目标文件，并检测所述目标文件对应的访问频率；基于所述访问频率判断所述目标文件对应的数据类型是否为热数据；若判断到所述数据类型为所述热数据，则确定所述目标文件对应的所述目标索引管理空间；基于所述目标索引管理空间确定所述目标文件对应的各相似热数据文件，将各所述相似热数据文件插入到预设的日志合并存储结构树。
- 如权利要求1所述的文件系统的元数据管理方法，其中，在所述通过预设的预分配索引管理空间进行预分配操作以为所述初始键值格式分配对应的预分配索引数值从而得到预分配键值格式的步骤之前，所述方法还包括：基于所述初始键值格式确定与文件重复的各重复文件；对所述文件和各所述重复文件执行合并存储操作。
- 一种终端设备，其中，所述终端设备包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的文件系统的元数据管理程序，所述文件系统的元数据管理程序被所述处理器执行时实现如权利要求1至8中任一项所述的文件系统的元数据管理方法的步骤。
- 一种计算机可读存储介质，其中，所述计算机可读存储介质上存储有文件系统的元数据管理程序，所述文件系统的元数据管理程序被处理器执行时实现如权利要求1至8中任一项所述的文件系统的元数据管理方法的步骤。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310695689.3 | 2023-06-12 | ||
| CN202310695689.3A CN116737659B (zh) | 2023-06-12 | 2023-06-12 | 文件系统的元数据管理方法、终端设备及计算机存储介质 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2024254899A1 true WO2024254899A1 (zh) | 2024-12-19 |
Family
ID=87902371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2023/102466 Ceased WO2024254899A1 (zh) | 2023-06-12 | 2023-06-26 | 文件系统的元数据管理方法、终端设备及计算机存储介质 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN116737659B (zh) |
| WO (1) | WO2024254899A1 (zh) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN121412193A (zh) * | 2024-07-18 | 2026-01-27 | 腾讯云计算（北京）有限责任公司 | 文件系统元数据管理方法、装置、电子设备和可读介质 |
| CN119669157B (zh) * | 2024-11-27 | 2025-10-24 | 北京百度网讯科技有限公司 | 用于文件系统的操作处理方法、装置、设备和介质 |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7752226B1 (en) * | 2002-12-20 | 2010-07-06 | Symantec Operating Corporation | Reverse pathname lookup by inode identifier |
| CN104731921A (zh) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Hadoop分布式文件系统针对日志型小文件的存储和处理方法 |
| CN109933570A (zh) * | 2019-03-15 | 2019-06-25 | 中山大学 | 一种元数据管理方法、系统及介质 |
| US20200201821A1 (en) * | 2018-12-21 | 2020-06-25 | Vmware, Inc. | Synchronization of index copies in an lsm tree file system |
| CN114238226A (zh) * | 2021-12-22 | 2022-03-25 | 中山大学 | 一种基于simd指令的nvm本地文件管理系统及方法 |
| CN115098466A (zh) * | 2022-07-18 | 2022-09-23 | 重庆紫光华山智安科技有限公司 | 元数据管理方法、装置、存储节点及可读存储介质 |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2059615A1 (en) * | 1991-01-23 | 1992-07-24 | Edward J. Neubauer | Method of selecting and representing data |
| CN103942262A (zh) * | 2014-03-26 | 2014-07-23 | 浪潮电子信息产业股份有限公司 | 一种视频存储系统中元数据的优化分配方法 |
| CN115705313A (zh) * | 2021-08-11 | 2023-02-17 | 腾讯科技（深圳）有限公司 | 一种数据处理方法、装置、设备及计算机可读存储介质 |
| CN115481089A (zh) * | 2022-09-19 | 2022-12-16 | 北京星辰天合科技股份有限公司 | 文件元数据的存储方法及存储装置、电子设备 |
-
2023
- 2023-06-12 CN CN202310695689.3A patent/CN116737659B/zh active Active
- 2023-06-26 WO PCT/CN2023/102466 patent/WO2024254899A1/zh not_active Ceased
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7752226B1 (en) * | 2002-12-20 | 2010-07-06 | Symantec Operating Corporation | Reverse pathname lookup by inode identifier |
| CN104731921A (zh) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | Hadoop分布式文件系统针对日志型小文件的存储和处理方法 |
| US20200201821A1 (en) * | 2018-12-21 | 2020-06-25 | Vmware, Inc. | Synchronization of index copies in an lsm tree file system |
| CN109933570A (zh) * | 2019-03-15 | 2019-06-25 | 中山大学 | 一种元数据管理方法、系统及介质 |
| CN114238226A (zh) * | 2021-12-22 | 2022-03-25 | 中山大学 | 一种基于simd指令的nvm本地文件管理系统及方法 |
| CN115098466A (zh) * | 2022-07-18 | 2022-09-23 | 重庆紫光华山智安科技有限公司 | 元数据管理方法、装置、存储节点及可读存储介质 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116737659A (zh) | 2023-09-12 |
| CN116737659B (zh) | 2026-01-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10754878B2 (en) | Distributed consistent database implementation within an object store | |
| CN109933570B (zh) | 一种元数据管理方法、系统及介质 | |
| US10515052B2 (en) | File system that supports both case sensitive and case insensitive directory lookup | |
| CN106021266B (zh) | 支持动态更新的快速多层索引 | |
| US10346363B2 (en) | Deduplicated file system | |
| US9146930B2 (en) | Method and apparatus for file storage | |
| US12463662B2 (en) | Data compression method and apparatus | |
| US12013814B2 (en) | Managing snapshotting of a dataset using an ordered set of B+ trees | |
| WO2024254899A1 (zh) | 文件系统的元数据管理方法、终端设备及计算机存储介质 | |
| US11132334B2 (en) | Methods and apparatus for filtering dynamically loadable namespaces (DLNs) | |
| US10558636B2 (en) | Index page with latch-free access | |
| WO2023237120A1 (zh) | 一种数据处理系统及装置 | |
| US12253974B2 (en) | Metadata processing method and apparatus, and a computer-readable storage medium | |
| CN115328859A (zh) | 一种元数据访问方法、装置、设备及介质 | |
| US10997126B1 (en) | Methods and apparatus for reorganizing dynamically loadable namespaces (DLNs) | |
| CN115510000B (zh) | 文件合并方法、装置、电子设备、存储介质及程序产品 | |
| Fattahi et al. | A new approach for directory management in GlusterFS | |
| Shu | File Systems | |
| CN121412179A (zh) | 面向微内核操作系统的快速目录解析文件系统 | |
| CN120653710A (zh) | 一种数据系统、方法、装置、设备、介质及程序产品 | |
| WO2026016696A1 (zh) | 文件系统元数据管理方法、装置、电子设备和存储介质 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23941132 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |