CN115563057A

CN115563057A - File system metadata storage method and device based on daos

Info

Publication number: CN115563057A
Application number: CN202211174513.5A
Authority: CN
Inventors: 刘倩; 罗雄; 王英
Original assignee: CLP Cloud Digital Intelligence Technology Co Ltd
Current assignee: CLP Cloud Digital Intelligence Technology Co Ltd
Priority date: 2022-09-26
Filing date: 2022-09-26
Publication date: 2023-01-03

Abstract

The application discloses a daos-based file system metadata storage method and device, wherein the metadata storage method comprises the following steps: when a parent-level directory is created, setting directory fragments and configuring corresponding first dkey for recording fragment values of the directory fragments; utilizing the directory fragments to store the subdirectories under the directory and/or the metadata of the subfiles, and configuring corresponding akey and value for the subdirectories and/or the subfiles, wherein the akey is used for recording the subdirectories and/or the file names of the subfiles, and the value is used for recording the metadata of the subdirectories and/or the subfiles. According to the metadata storage method, the metadata are reorganized, the data organization balance is improved, and the interaction times with the daos cluster are greatly reduced.

Description

Daos-based file system metadata storage method and device

Technical Field

The application relates to the technical field of computers, in particular to a daos-based file system metadata storage method and device.

Background

The current Daos-based file system has DFS, and the metadata organization form of the sub-files/sub-directories under the directory and directory is as shown in fig. 1, and it is assumed that the parent directory is dir1, and the sub-directories dir2, sub-file 1 and sub-file 2 are contained under the parent directory:

it creates a daos object for each directory, then creates a dkey for each subdirectory/subfile under the directory, under which the attribute of each subdirectory/subfile creates an akey, which has the following drawbacks:

when metadata under a certain large directory is enumerated, the efficiency is not high, and when the directory is enumerated, a dkey set of directory objects needs to be enumerated first; and then traversing the dkey set, listing all the akeys under each dkey, then inquiring the values corresponding to all the akeys under the dkey in batch, and organizing the values obtained by inquiry into metadata of the subdirectories/subfiles.

Assuming that there are N sub-directories/sub-files under a directory, and each sub-directory/sub-file has on average M metadata attributes, the number of interactions with the daos cluster required to enumerate the directory is: n +1, when the number of the sub-files/sub-directories is less, the influence cannot be achieved, and when the number of the sub-files/sub-directories reaches one hundred thousand and millions, the query time is long.

When the directory is large, the data organization of directory objects is unbalanced, daos objects organize dkey through a B + tree, each dkey organizes akey through a B + tree, when subfiles/subdirectories under the directory are large, it appears that all B + trees corresponding to dkey are large, while all B + trees corresponding to akey under each dkey are small (usually, the metadata attribute is less than 50 items), so that the feeling of heavy head and light foot is provided, and all B + trees corresponding to akey under each dkey are not fully utilized.

Disclosure of Invention

The embodiment of the application provides a daos-based file system metadata storage method and device, which are used for reorganizing metadata, improving the balance of data organization and greatly reducing the number of interaction with daos clusters.

The embodiment of the application provides a daos-based file system metadata storage method, wherein the file system comprises a parent directory, subdirectories and/or subfiles are arranged under the parent directory, and the metadata storage method comprises the following steps:

when a parent-level directory is created, setting directory fragments and configuring a corresponding first dkey for recording fragment values of the directory fragments;

utilizing the directory fragments to store subdirectories and/or metadata of subfiles under the directory, and configuring corresponding akey and value for the subdirectories and/or the subfiles, wherein the akey is used for recording the subdirectories and/or file names of the subfiles, and the value is used for recording the metadata of the subdirectories and/or the subfiles.

Optionally, under the condition that the number of the sub-directories and/or the number of the sub-files under the directory fragment reach a first preset threshold, splitting the directory fragment to form a sub-directory fragment;

and merging the subdirectory fragments under the condition that the sub files under the split subdirectory fragments and/or the sum of the numbers of the subdirectories is lower than a second preset threshold value.

Optionally, in the case of splitting the directory fragment, recalculating the child directory, and/or calculating a hash value of the child file, and migrating part of the child directory, and/or the metadata of the child file to the child directory fragment.

Optionally, the method further includes configuring a second dkey for storing the fragmentation information of the parent directory.

Optionally, the value is specifically used to record metadata of the sub-directory and/or a collection of multiple attributes of the subfile.

Optionally, the akey and the value are both stored in the form of SingleValue.

An embodiment of the application further provides a computer device comprising a processor and a memory, the memory storing a computer program, the computer program when executed by the processor implementing the steps of the daos-based file system metadata storage method according to any one of claims 1 to 6.

Embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the aforementioned daos-based file system metadata storage method.

According to the metadata storage method, the metadata are reorganized, the data organization balance is improved, and the number of interaction with the daos cluster is greatly reduced.

The above description is only an overview of the technical solutions of the present application, and the present application may be implemented in accordance with the content of the description so as to make the technical means of the present application more clearly understood, and the detailed description of the present application will be given below in order to make the above and other objects, features, and advantages of the present application more clearly understood.

Drawings

Various additional advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 is an example of a metadata organizational architecture for a prior art Daos-based file system;

fig. 2 is an example of a metadata organization architecture of a Daos-based file system according to an embodiment of the present application.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

when a parent-level directory is created, setting a directory fragment and configuring a corresponding first dkey for recording a fragment value of the directory fragment;

As shown in fig. 2, in this embodiment, for a daos object of a directory, a directory fragment is set for a parent directory, for example dir1 in fig. 2, and a fragment value of a directory fragment stored by dkey, for example, fragment 0, fragment 1, fragment 2, and. In some embodiments, the value is specifically used to record metadata of a collection of multiple attributes of the subfile, that is, the value stores a collection of multiple attribute values into a data packet.

In some embodiments, when the number of sub-directories and/or sub-files under the directory fragment reaches a first preset threshold, the directory fragment is split to form a sub-directory fragment. Specifically, when the parent directory is created, only one directory fragment (e.g., fragment 0) is defined, and all the subfiles/subdirectory metadata in the parent directory are stored in the directory fragment. When the number of sub-directories/sub-files under a directory fragment reaches a threshold (e.g., 10000), the directory fragment is divided into 2 sub-directory fragments.

In some embodiments, in the case of splitting the directory fragment, the child directory is recalculated, and/or the hash value of the subfile is recalculated, and portions of the child directory, and/or the metadata of the child file is migrated to the child directory fragment. Specifically, a hash value of the subdirectory/subfile is calculated, and a part of metadata is moved to a new fragment. Specifically, the hash value of the subdirectory/subfile is calculated by performing hash operation according to the file name and measuring the number of fragments, and the obtained value is the fragment where the file is located. Over time, if the subdirectory/subfile of a subdirectory slice again reaches the threshold, the splitting continues.

And merging the subdirectory fragments under the condition that the sub files under the split subdirectory fragments and/or the sum of the numbers of the subdirectories is lower than a second preset threshold value. Specifically, for example, sub-fragments A1 and A2 are split under the parent fragment a, and when the sum of the numbers of sub-files/sub-directories under the sub-fragments is lower than a second preset threshold, the fragments A1 and A2 are merged.

In some embodiments, configuring a second dkey for storing sharding information of the parent directory is further included. In the daos object of the directory, a special dkey is configured for storing the slice information of the directory. The slice information referred to in this example may be the number of slices, e.g., frag-info in fig. 2 may record the number of slices for dir 1.

In some embodiments, the akey and the value are both stored in the form of SingleValue.

Assuming that there are N sub-directories/sub-files under a directory, and each sub-directory/sub-file has on average M metadata attributes, the number of interactions with the daos cluster required to enumerate the directory is: the number of fragments +1, assuming that metadata for 1 ten thousand sub-directories/sub-files, 1 hundred million sub-directories/sub-files are stored under each fragment. By reorganizing the metadata, the number of interactive IO times is 10001, and the number of interactive times of DFS is (1 hundred million + 1), so that the method reduces about 1 ten thousand times and greatly reduces the number of interactive times.

Assuming that 1 million sub-directories/sub-files exist under a directory and 1 million sub-directories/sub-files exist under each fragment, the number of dkey of the directory is 1 ten thousand, the number of akey under each dkey is 1 ten thousand, and compared with the case that the number of dkey of the DFS is 1 million, the number of akey under each dkey of the metadata storage method is about 20, and the data organization is more balanced.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a component of' 8230; \8230;" does not exclude the presence of another like element in a process, method, article, or apparatus that comprises the element.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.

While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the present embodiments are not limited to those precise embodiments, which are intended to be illustrative rather than restrictive, and that various changes and modifications may be effected therein by one skilled in the art without departing from the scope of the appended claims.

Claims

1. A daos-based file system metadata storage method, wherein the file system includes a parent directory having sub-directories and/or sub-files thereunder, the metadata storage method comprising:

when a parent-level directory is created, setting directory fragments and configuring corresponding first dkey for recording fragment values of the directory fragments;

2. The daos-based file system metadata storage method according to claim 1, wherein the directory fragments are split to form subdirectory fragments when the number of subdirectories under the directory fragments and/or the number of subfiles reaches a first preset threshold;

3. The daos-based file system metadata storage method of claim 2, wherein in case of splitting the directory fragments, the sub-directories, and/or the sub-files, are re-computed for hash values and parts of the sub-directories, and/or the sub-files' metadata are migrated to sub-directory fragments.

4. The daos-based file system metadata storage method of claim 3, further comprising configuring a second dkey for storing sharding information of the parent directory.

5. The daos-based file system metadata storage method of claim 1, wherein the value is specifically used for recording one metadata of the sub-directory and/or an assembly of a plurality of attributes of the sub-file.

6. The daos-based file system metadata storage method of claim 1, wherein the akey and the value are both stored in a SingleValue form.

7. A computer device comprising a processor and a memory, the memory storing a computer program which, when executed by the processor, performs the steps of the daos-based file system metadata storage method of any one of claims 1 to 6.

8. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, carries out the steps of the daos-based file system metadata storage method according to any one of claims 1 to 6.