CN109684282B

CN109684282B - Method and device for constructing metadata cache

Info

Publication number: CN109684282B
Application number: CN201811405321.4A
Authority: CN
Inventors: 赵辉; 李志然
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2018-11-23
Filing date: 2018-11-23
Publication date: 2021-01-01
Anticipated expiration: 2038-11-23
Also published as: CN109684282A

Abstract

A method and a device for constructing a metadata cache comprise the following steps: and the MDS receives the metadata request, if the cache node of the first directory does not exist in the tree cache structure and the MDS is the home MDS, the cache node of the first directory can be created, the first metadata file stored in the OSD is read according to the identifier of the first directory, and the content of the first metadata file is stored in the cache node corresponding to the first directory. Because the content of the first metadata file comprises the metadata information of the subitem of the first directory, through the processing, the cache structure meeting the metadata processing requirement can be constructed without sequentially reading the metadata file of each directory on the path from the root directory of the subtree to the first directory from the OSD, thereby effectively reducing the times of reading the disc from the OSD, shortening the processing time of the metadata request and improving the metadata processing efficiency.

Description

Method and device for constructing metadata cache

Technical Field

The invention relates to the technical field of computers, in particular to a method and a device for constructing metadata cache.

Background

In a distributed file system, a metadata server (MDS) generally manages metadata caches in a dynamic sub-tree partition manner, as shown in fig. 1, a directory tree of a whole user is divided into a plurality of dynamic sub-tree partitions, and each MDS is responsible for managing metadata caches of some dynamic sub-trees, processing metadata operations under the dynamic sub-trees, and maintaining consistency between the metadata caches and metadata files stored in an Object Storage Device (OSD).

Because the MDS manages the metadata cache by using the tree-shaped cache structure, when the MDS processes the metadata information of a certain file or directory, the MDS needs to construct the cache structures of all directories on the path from the subtree root directory of the dynamic subtree where the file or directory is located to the file or directory.

In the prior art, when an MDS receives a metadata request for processing metadata of a certain file or directory, but a metadata cache of a parent directory of the file or directory does not exist in the MDS, a discovery (discover) process is triggered, a path from a subtree root directory of a dynamic subtree where the file or directory is located to a current file or directory is first queried, and then a lookup (lookup) operation is performed layer by layer from the subtree root directory. And in each lookup operation, searching metadata information of the child directory in a metadata cache of the parent directory, if the metadata information of the child directory does not exist, reading the content of the metadata file of the parent directory from the OSD according to the storage position of the metadata file of the parent directory in the OSD, and storing the content into the metadata cache of the parent directory. It can be seen that in the prior art, a lookup operation needs to be performed once when a first-level metadata cache is constructed, which makes the longer the path from the root directory of the sub-tree to the file or directory to be operated, the more times of lookup in the discover process, and the longer the time consumption of the discover process. Especially, under the condition that the MDS just manages the dynamic subtree originally managed by the faulty MDS, because the cache structure of the dynamic subtree needs to be rebuilt, the discover process triggered by the metadata requests of the first times after the management almost needs to read the metadata file in the OSD every lookup operation, and the time consumption of the discover process is obviously increased, so that the efficiency of metadata processing is poor.

Disclosure of Invention

The application provides a method and a device for constructing metadata cache, which are used for solving the technical problems that in the prior art, a discover process consumes a long time and metadata processing efficiency is poor.

In a first aspect, an embodiment of the present application provides a method for constructing a metadata cache, where the method includes: a metadata server receives a metadata request sent by client proxy equipment, wherein the metadata request comprises a first identifier and a second identifier, the first identifier is an identifier of a first directory, and the second identifier is an identifier of a subitem of the first directory; if the cache node corresponding to the first directory does not exist in the tree cache structure stored by the metadata server, and the metadata server is a home metadata server of the dynamic subtree where the subitem is located, the metadata server creates the cache node corresponding to the first directory in the tree cache structure; and the metadata server reads a first metadata file stored in a storage device according to the first identifier, and stores the content of the first metadata file to a cache node corresponding to the first directory, wherein the first metadata file is the metadata file of the first directory, and the content of the first metadata file comprises metadata information of the sub-item.

In this embodiment of the application, when the metadata server receives a metadata request for a subentry of the first directory, if a cache node of the first directory does not exist in a tree cache structure of the metadata server and the metadata server is a home metadata server of a dynamic subtree where the subentry of the first directory is located, the metadata server may create a cache node corresponding to the first directory in the tree cache structure, read a first metadata file stored in the storage device according to an identifier of the first directory, and store the content of the first metadata file into the cache node corresponding to the first directory. In the embodiment of the present application, a cache node corresponding to a first directory is created in a tree cache structure, and the content of a first metadata file is stored in the created cache node corresponding to the first directory, so that a cache structure supporting metadata operation on the subentry of the first directory can be constructed, and it is not necessary to sequentially read and store metadata files of each directory on a directory identification path from a subtree root directory to the first directory from top to bottom from a storage device.

In one possible design, the metadata server creates a cache node corresponding to the first directory in the tree cache structure, including: the metadata server acquires a directory identification path from a subtree root directory of the dynamic subtree where the subitem is located to the first directory; the metadata server sequentially executes the following processing on each directory in the directory identification path according to the sequence from the root directory of the sub-tree to the first directory until the creation of the cache node corresponding to the first directory is completed: aiming at any directory in the directory identification path, if the tree-shaped cache structure does not have a cache node corresponding to the directory, creating a cache node corresponding to the directory in the tree-shaped cache structure;

after the metadata server creates a cache node corresponding to the first directory in the tree cache structure, the metadata server further includes: and the metadata server creates a cache node corresponding to the sub-item in the tree cache structure.

Because a tree-shaped cache structure is adopted, a cache node of a next level directory must be created based on the created cache node of the previous level directory. Therefore, in the embodiment of the present application, constructing a cache node corresponding to the first directory in the tree cache structure or constructing a cache structure of the first directory means that starting from the subtree root directory, a subtree root directory is sequentially created from top to bottom to a cache node corresponding to each directory on the directory identification path of the first directory. Therefore, the metadata server can be supported to process the metadata information in the cache node corresponding to the first directory.

In one possible design, after the metadata server stores the content of the first metadata file to the cache node corresponding to the first directory, the method further includes: the metadata server sets a self-untrusted mark and a sub-item trusted mark for a cache node corresponding to the first directory, sets a self-trusted mark and a sub-item untrusted mark for a cache node corresponding to the sub-item, and sets a self-untrusted mark and a sub-item untrusted mark for cache nodes corresponding to other directories newly created in a directory identification path from the subtree root directory to the first directory.

In this embodiment of the present application, in the process of creating a cache node corresponding to a first directory, a metadata server only creates a basic cache structure that supports processing of metadata information of a sub-entry of the first directory, that is, although a cache node corresponding to each directory on a directory identification path from a subtree root directory to the first directory is created, only the content of the first metadata file is obtained from a storage device and stored, and for cache nodes corresponding to other directories on the directory identification path, the content of the metadata file is obtained and stored after a delay. It can be understood that, for a cache node corresponding to any directory in the tree cache structure, if the metadata server has received a metadata request for a sub-entry of the directory or processed metadata information of the sub-entry of the directory, the cache node holds the content of the metadata file of the directory, and if the metadata server has not received a metadata request for a sub-entry of the directory or processed metadata information of a sub-entry of the directory, the cache node only holds an identifier of the directory, and cannot support the metadata server to process the metadata information of the sub-entry of the directory. Therefore, in order to effectively identify whether each cache node in the tree cache structure holds the content of the metadata file of the directory, in this embodiment of the application, the metadata server may set a corresponding state identifier for each cache node, where the state identifier may include four types, that is, "a sub-item trusted identifier", "a sub-item untrusted identifier", "a self trusted identifier", and "a self untrusted identifier".

In one possible design, after the metadata server receives the metadata request sent by the client agent device, the method further includes: if a cache node corresponding to the first directory exists in a tree-shaped cache structure stored by the metadata server and a subentry untrusted tag is set on the cache node corresponding to the first directory, the metadata server reads a first metadata file stored in the storage device according to the first identifier and stores the content of the first metadata file to the cache node corresponding to the first directory; and the metadata server clears the sub-item non-credible mark of the cache node corresponding to the first directory and sets a sub-item credible mark for the cache node corresponding to the first directory.

In this embodiment of the application, if a cache node corresponding to the first directory exists in the tree cache structure, but the cache node corresponding to the first directory is provided with the sub-entry untrusted tag, it indicates that only the first identifier is stored in the cache node corresponding to the first directory, and therefore, the metadata server may read the first metadata file stored in the storage device according to the first identifier, and store the content of the first metadata file in the cache node corresponding to the first directory, so as to subsequently process the metadata information of the sub-entry of the first directory.

In one possible design, before the metadata server reads the first metadata file stored in the storage device, the method further includes: if the tree cache structure stored by the metadata server does not have the cache node corresponding to the subentry, the metadata server creates the cache node corresponding to the subentry in the tree cache structure;

after the metadata server stores the content of the first metadata file to the cache node corresponding to the first directory, the method further includes: and the metadata server sets a self credible mark and a subitem incredible mark for the cache node corresponding to the subitem.

In this embodiment of the application, when a cache node corresponding to a first directory exists in a tree-shaped cache structure and a cache node corresponding to the first directory is provided with a sub-entry untrusted tag, a cache node corresponding to a currently requested sub-entry may exist in the tree-shaped cache structure, or a cache node corresponding to the sub-entry may not exist in the tree-shaped cache structure.

In one possible design, before the metadata server creates a cache node corresponding to the first directory in the tree cache structure, the metadata server further includes: the metadata server acquires a birth record of the first directory stored in a birth record server, wherein the birth record of the first directory comprises protection level information of the first directory;

the metadata server reads a first metadata file stored in a storage device, and the method comprises the following steps: and the metadata server determines the storage position of the first metadata file in the storage device according to the first identifier and the protection level information of the first directory, and reads the first metadata file from the storage position.

In this embodiment of the present application, a storage location of the first metadata file in the storage device is related to the protection level information of the first directory, and the protection level information of the first directory is located in the birth record of the first directory, so that the metadata server may first obtain the birth record of the first directory stored in the birth record server to obtain the protection level information of the first directory, further determine, according to the first identifier and the protection level information of the first directory, a storage location of the first metadata file in the storage device, and read the first metadata file from the storage location.

In one possible design, the metadata request further includes a file operation command;

after the metadata server stores the content of the first metadata file to the cache node corresponding to the first directory, the method further includes: and the metadata server processes the metadata information of the sub-items stored in the cache nodes of the first directory according to the file operation command.

after the metadata server receives the metadata request sent by the client agent device, the method further comprises: and if the cache node corresponding to the first directory exists in the tree-shaped cache structure stored by the metadata server and the cache node corresponding to the first directory is provided with a subitem credible mark, the metadata server directly processes the metadata information of the subitem stored in the cache node of the first directory according to the file operation command.

In this embodiment of the application, if a cache node corresponding to the first directory exists in the tree cache structure, and the cache node corresponding to the first directory is provided with a subitem trusted flag, it indicates that the cache node corresponding to the first directory stores the content of the first metadata file, and therefore, the metadata server directly processes the metadata information of the subitem in the cache node corresponding to the first directory.

In one possible design, before the processing, by the metadata server, the metadata information of the child item stored in the cache node of the first directory, the method further includes: if the cache node corresponding to the sub-item does not exist in the tree-shaped cache structure stored by the metadata server, the metadata server creates the cache node corresponding to the sub-item in the tree-shaped cache structure, and sets a self-credible mark and a sub-item non-credible mark for the cache node corresponding to the sub-item.

In the embodiment of the present application, when a cache node corresponding to a first directory exists in a tree-shaped cache structure and a cache node corresponding to the first directory is provided with a sub-entry trusted tag, a cache node corresponding to a currently requested sub-entry may still not exist in the tree-shaped cache structure, so that a metadata server may create a cache node corresponding to a sub-entry and set a corresponding state tag when a cache node corresponding to the sub-entry does not exist in the tree-shaped cache structure.

In one possible design, after the processing, by the metadata server, the metadata information of the child item stored in the cache node of the first directory, the method further includes: and the metadata server sends the processed metadata information of the sub-items to the storage device.

In this embodiment, after processing the metadata information of the sub-item stored in the cache node of the first directory, the metadata server may send the processed metadata information of the sub-item to the object storage device, so that the object storage device updates the content of the first metadata file.

In a second aspect, an embodiment of the present application provides an apparatus for constructing a metadata cache, where the apparatus has a function of implementing the metadata server in the first aspect or any possible design of the first aspect, where the function may be implemented by hardware, or may be implemented by hardware executing corresponding software, where the hardware or software includes one or more modules corresponding to the above functions.

In one possible design, the apparatus includes a processing module and a transceiver module in a structure, and the processing module is configured to support the apparatus to perform corresponding functions in the first aspect or any one of the designs of the first aspect. The transceiver module is used for supporting the communication between the device and other communication equipment (such as a birth record server and an object storage device). The apparatus may also include a memory module, coupled to the processing module, that stores program instructions and data necessary for the apparatus to function. As an example, the processing module may be a processor, the communication module may be a transceiver, and the storage module may be a memory.

In a third aspect, an embodiment of the present application provides a chip, which is coupled to a memory and configured to read and execute a software program stored in the memory, so as to implement the method in any one of the possible designs of the first aspect or the method in any one of the possible designs of the second aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, which stores computer-readable instructions, and when the computer-readable instructions are read and executed by a computer, the computer-readable instructions cause the computer to perform the method in any one of the possible designs of the first aspect.

In a fifth aspect, the present application provides a computer program product, which when read and executed by a computer, causes the computer to perform the method of any one of the possible designs of the first aspect.

Drawings

FIG. 1 is a diagram illustrating a structure of a dynamic subtree partition in an embodiment of the present application;

FIG. 2 is a system architecture diagram of a metadata processing system provided by an embodiment of the present application;

FIG. 3 is a diagram illustrating a structure of a metadata file stored in an OSD according to an embodiment of the present application;

fig. 4 is a schematic diagram of an interaction relationship between network elements of various types in a metadata processing system according to an embodiment of the present application;

fig. 5 is a schematic flowchart of a method for constructing a metadata cache according to an embodiment of the present application;

FIG. 6a is a diagram illustrating a structure of a dynamic subtree according to an embodiment of the present application;

fig. 6b is a schematic structural diagram of a tree cache structure provided in the embodiment of the present application;

fig. 7 is a schematic structural diagram of an apparatus for constructing a metadata cache according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of another apparatus for constructing a metadata cache according to an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application are described in detail below with reference to the accompanying drawings. It is noted that the terminology used in the description of the embodiments section of the present application is for the purpose of describing particular embodiments of the present application only and is not intended to be limiting of the present application.

Fig. 2 is a system architecture diagram of a metadata processing system according to an embodiment of the present disclosure, and as shown in fig. 2, the metadata processing system includes an NAS cluster including a plurality of Client Agent (CA) modules, an MDS cluster including a plurality of MDSs, a plurality of data subfields including OSDs, and a birth record server (birth server). The following is a detailed description of each type of network element in the metadata processing system.

1) CA module

The CA module can communicate with the client and receive and respond to the file access request sent by the client, and meanwhile, the CA module is also an upper layer service interface of the MDS and can communicate with the MDS and send a metadata request to the MDS. Specifically, the CA module may perform semantic parsing on a file operation word in a file access request sent by the client, convert the file operation word into a file operation command inside the file system, and send the file operation command to the MDS by being carried in the metadata request. It should be noted that the file operation word referred in the embodiment of the present application may be any one or more of new create (create), read (read), write (write), and create sub-directory (mkdir), and may also be other types of file operation words, which is not limited in the present application. It should be understood that the CA module in the embodiment of the present application may be deployed on one physical node independently, or may be deployed on the same physical node as the MDS, but its processing logic is independent from the MDS.

2)MDS

The MDS is responsible for managing metadata caches of one or more dynamic subtrees in a directory tree of the file system, and provides metadata service processing to the outside, including updating the content of the metadata caches, receiving a metadata request sent by the CA module, performing metadata operations on metadata according to a file operation command, and the like. For example, the directory tree in fig. 1 is divided into 4 non-overlapping dynamic subtree partitions (which may be simply referred to as dynamic subtrees), wherein the MDS1 is responsible for managing the metadata cache of the dynamic subtree 1, the MDS2 is responsible for managing the metadata cache of the dynamic subtree 2, the MDS3 is responsible for managing the metadata cache of the dynamic subtree 3, and the MDS4 is responsible for managing the metadata cache of the dynamic subtree 4.

In the embodiment of the present application, the parent node in the dynamic subtree may represent a directory in the file system, and the leaf node in the dynamic subtree may represent a directory or a file in the file system. In order to implement cache management of metadata, the MDS stores a tree cache structure of a dynamic subtree managed by the MDS, and the tree cache structure may be regarded as a cache of the dynamic subtree and is gradually created in the MDS along with gradual access of the CA module to the metadata, and may be implemented by a data structure similar to a binary tree or other types of data structures, which is not specifically limited in this application. It should be understood that, when the MDS is responsible for managing multiple dynamic subtrees, the MDS also correspondingly stores the tree cache structures of the multiple dynamic subtrees, but in the embodiment of the present application, the method for constructing the metadata cache is described by taking the tree cache structure of one of the dynamic subtrees managed by the MDS as an example, and details are not described below.

Specifically, the tree cache structure includes at least one cache node, i.e., a cache node corresponding to a subtree root directory of the dynamic subtree. The cache nodes in the tree cache structure are used for storing metadata caches corresponding to directories or files, and the metadata caches are caches of contents of metadata files (also called disk objects) of the directories or files in the MDS. In the embodiment of the application, the metadata file of the directory is persistently stored in the object storage device, and the metadata operation is processed in the metadata cache of the MDS and then sent to the object storage device for storage after the processing is finished. When the metadata operation is needed, if the metadata cache of the directory does not exist in the metadata server, the metadata server needs to read the content of the metadata file of the directory from the object storage device, store the content in the MDS and process the content, namely, the metadata of the directory is loaded into the cache of the MDS from the object storage device. It should be noted that, in the embodiment of the present application, the content of the metadata cache may include all the content of the metadata file, or may include only part of the content of the metadata file, such as an identifier of a directory. However, it should be noted that the metadata cache including only the identifier of the directory cannot support the MDS to perform the metadata operation, and the metadata server still needs to read the metadata from the object storage device and store the metadata in the cache of the MDS when performing the metadata operation.

It should be noted that, in the embodiment of the present application, the structure of the dynamic subtree may be dynamically adjusted according to the requirement of the system, and is not fixed. When the number of all sub-items under the dynamic sub-tree is large or the access quantity is too hot, the dynamic sub-tree can be split into a plurality of small dynamic sub-trees. Dynamic subtrees with overheated visitors can also be managed by migrating from a busy MDS to a relatively idle MDS. However, one dynamic subtree can only belong to one MDS at a time, and cannot belong to two or more MDSs at the same time, otherwise, metadata downloading conflict can be caused. In the embodiment of the application, "belonging" means: if a dynamic subtree belongs to an MDS, all metadata processing under the dynamic subtree is processed on the belonging MDS, and other MDSs in the MDS cluster cannot process metadata operations under the dynamic subtree.

3)MDS leader

One of the MDS clusters can also be used as an MDS leader in the MDS cluster, and the MDS leader has functions of managing dynamic subtrees such as splitting and migration besides functions of general MDSs, and is responsible for maintaining the corresponding relationship between each dynamic subtree of the file system and the corresponding attributed MDS, and managing and downloading global objects such as Fsblock. In the embodiment of the application, the global object at least comprises an fsblock object and a globalsubtree object, wherein the fsblock object records the name of a file system and metadata information of a system root directory, the globalsubtree object records the corresponding relationship between the root directory of each dynamic subtree in the file system and the attributive MDS, and the corresponding relationship is downloaded to an OSD layer through the globalsubtree object. It should be noted that each global object in the embodiment of the present application has a global identifier, and the OSD layer stores the disk objects corresponding to the two global objects.

For example, when a common MDS fails, an MDS leader may designate other normal MDSs to take over the dynamic subtree of the failed MDS. And after the takeover MDS receives a takeover notice sent by the MDS leader, reading all subtree root information originally managed by the faulty MDS from the MDS leader, and creating a metadata cache corresponding to a subtree root directory of the dynamic subtrees in the memory according to the subtree root information. With the access of the service, the take-over MDS gradually reads from the OSD and creates a metadata cache of the sub-items under the root directory of the sub-tree. In a possible design, when a certain MDS fails, multiple dynamic subtrees managed by the failed MDS can be respectively taken over by different normal MDSs, so that the number of the dynamic subtrees to be taken over by the MDS can be effectively reduced, the situation that the service volume of the taken over MDS is increased greatly due to the fact that all the dynamic subtrees managed by the failed MDS are taken over by one MDS is avoided, and the influence on the system performance when the MDS fails is reduced.

When the MDS leader fails, other MDSs in the MDS cluster pass through a certain election rule, for example, an MDS with a smaller ID can be selected, and one MDS is selected as a successor MDS leader from normal MDSs. It should be understood that there is a management process in each MDS in the MDS cluster, and the MDSs can interact with each other through heartbeat packets, and when an MDS leader fails, other normal MDSs can negotiate who acts as a new MDS leader in this way. The successor MDS leader not only needs to take over the dynamic subtree managed by the predecessor MDS leader, but also needs to take over the global object management function of the MDS leader. When the successor MDS leader learns that the MDS leader is upgraded to the leader, firstly, the leader takeover process is started, the fsblock object and the globalsubtree object are read from the OSD and stored in the cache structure, and the leader takeover is successful. And then, the MDS leader searches the dynamic subtrees managed by the predecessor MDS leader in the saved globalsubtree object, and creates metadata caches corresponding to subtree root directories of the dynamic subtrees in the memory. With service access, the metadata cache of the sub-entries under the root directory of the sub-tree is gradually read and created from the OSD.

4)birth server

The birth server is used for recording parent-child relationship information between a parent directory and child directories in the file system. Specifically, a birth (birth) record may be used to represent a parent-child relationship between a child directory and its parent directory, for example, a child directory "subdir 1" is included under the directory "dir 1", and the content of the birth record of the child directory "subdir 1" is: { subfid, parentfid, protectlevel, namehash }, where subfid refers to the identifier of the subdirectory "subdir 1", parentfid refers to the identifier of the directory "dir 1", protectlevel refers to the protection level information of the subdirectory "subdirector 1", and namehash refers to a 4-byte hash value obtained by a hash algorithm for this directory name "subdir 1".

In this embodiment, the birth server may include a database constructed by using the identifiers of the subdirectories as key values, and configured to store the birth records in the file system, and provide operations such as adding, deleting, modifying, and checking the birth records. The birth server can be single-node or distributed, and can be deployed in an independent physical node or the same physical node with the MDS according to the total number of birth records, but the processing logic is independent from the MDS. Therefore, all the bert records on the path from any directory to the system root directory can be obtained by inquiring the bert server for multiple times, and further the identifiers of all the superior directories of the any directory are obtained.

It should be noted that the directories, files, or global objects mentioned above in the embodiments of the present application all use a uniform naming mode, and the system may assign a global identifier fid to each file, directory, or global object as a unique identifier in the file system. In one possible design, the identification may be an 8-byte number.

5)OSD

The entire OSD layer is a persistent storage layer providing reliable object storage functionality. The data storage device comprises a plurality of data sub-fields, each data sub-field comprises one or more OSD nodes, and each OSD node (namely an object storage device OSD) is an intelligent device which is provided with a storage medium, a processor, a memory, a network system and the like and is responsible for managing locally stored objects.

In the embodiment of the application, all the OSD nodes in the internal OSD layer can be communicated through messages. The MDS shares the OSD storage layer and provides a uniform object storage interface for the external OSD layer. Thus, it can be appreciated that all MDSs in a MDS cluster share a persistent storage layer, each MDS can see all OSD nodes, and any OSD node in the OSD layer can be accessed.

Specifically, the OSD is used to store metadata files (also referred to as disk objects) of directories or files in the file system, the metadata file of each directory stores metadata information of a direct subdirectory and/or a subfile of the directory, and the metadata information of a sub-item of the subdirectory is stored in the metadata file of the subdirectory itself.

In one possible design, metadata information of a directory or a file is stored by using a structure of a directory entry (dentry) and an index node (inode), the dentry is used for storing information such as a directory name (or a file name), the inode is used for storing basic attributes of the directory or the file, permission attributes and pointers of metadata caches pointing to sub-entries, wherein the basic attributes refer to information such as a file size, an access time and a file type of the file, and the permission attributes refer to access permission management (ACL) permissions, user groups and the like.

Fig. 3 illustrates contents of metadata files of directories in the OSD, and as shown in fig. 3, a metadata file of a directory includes a head header for storing contents such as the number of sub-items and modification time of the latest sub-item, and contents of dentry and inode of its sub-items. As can be seen from fig. 1, the metadata file of the system root directory stores metadata contents of its child entries, user and home, including the dentry and inode of the directory user and the dentry and inode of the directory home. The metadata file of the directory user stores the metadata contents of the sub-entries file, test and file1, including the dentry and inode of the file, the dentry and inode of the test and the dentry and inode of the file 1. The metadata contents of sub items lib and src, dentry and inode of lib, and dentry and inode of src are stored in the metadata file of the directory home.

It should be noted that, in the embodiment of the present application, the metadata file of the directory is stored in the OSD in a mirror image manner. The number of images is related to the protection level information of the directory, and the protection level information is recorded in the birthday record of the directory, and the higher the protection level information is, the greater the number of images is. For example, protection level +2 corresponds to 3 disk locations in the OSD, protection level +3 corresponds to 4 disk locations in the OSD, and protection level +4 corresponds to 5 disk locations in the OSD. When the MDS downloads the metadata of a certain directory to the OSD for storage, several disk positions on the NLUN can be selected for writing through a Hash algorithm according to the identification of the directory and the protection level information of the directory, and each disk position stores one current piece of metadata.

Fig. 4 exemplarily shows an interaction relationship among the CA module, MDS, birthserver, MDS leader, and OSD in the metadata processing system provided in the present application. As shown in FIG. 4, the CA module is configured to send metadata requests to the MDS and receive metadata request responses sent by the MDS. The MDS is used for receiving the metadata request, constructing a cache structure required by processing the metadata request, processing the metadata request and storing the content of the metadata cache to the OSD. The Birthserver is used for providing operations of adding, deleting, modifying, checking and the like of the birth records, supporting the MDS to construct a cache structure, and the MDS leader is used for managing and recording the corresponding relation between the subtree root directory and the attributive MDS and storing the subtree root directory and the attributive MDS in the OSD.

Based on the above system architecture, fig. 5 exemplarily illustrates a method for constructing a metadata cache provided by the present application, and as shown in fig. 5, the method includes the following steps:

step S501: the metadata server receives a metadata request sent by the client agent device, wherein the metadata request comprises a first identifier and a second identifier, the first identifier is an identifier of the first directory, and the second identifier is an identifier of a sub-item of the first directory.

Step S502: and if the cache node corresponding to the first directory does not exist in the tree-shaped cache structure stored by the metadata server and the metadata server is the home metadata server of the dynamic subtree where the subitem is located, the metadata server creates the cache node corresponding to the first directory in the tree-shaped cache structure.

Step S503: and the metadata server reads a first metadata file stored in the storage device according to the first identifier, and stores the content of the first metadata file to a cache node corresponding to the first directory, wherein the first metadata file is the metadata file of the first directory, and the content of the first metadata file is the metadata information of the sub-item.

In this embodiment of the application, if a metadata request sent by a client agent device includes a first identifier of a first directory and a second identifier of a sub-item of the first directory, it indicates that the client agent device requests a metadata server to process metadata information of the sub-item having the second identifier in the first directory. After the metadata server receives the metadata request, under the condition that a cache node of a first directory does not exist in a currently stored tree-shaped cache structure and the metadata server is a home metadata server of a dynamic sub-tree where a sub-item with a second identifier is located under the first directory, the metadata server can create a cache node corresponding to the first directory in the tree-shaped cache structure, read a metadata file, namely a first metadata file, of the first directory stored in the storage device according to the identifier of the first directory, and store the content of the first metadata file into the cache node corresponding to the first directory. Because the first metadata file comprises the metadata information of the subitem of the first directory, the metadata server creates a cache node corresponding to the first directory in the tree cache structure, and stores the content of the first metadata file into the created cache node, and then the metadata server constructs a cache structure capable of supporting processing the metadata information of the subitem with the second identifier, without sequentially reading the metadata file of each directory on the path from the root directory of the sub-tree to the first directory from top to bottom according to the directory hierarchy, thereby effectively reducing the number of times of reading the disc from the storage device in the discover process, shortening the processing time of the metadata request, and accelerating the process of returning the process to the main service processing.

Specifically, in step S501, the first directory is a directory in the file system, and its identifier is referred to as a first identifier. The first directory may include one or more sub-items, and the second identifier is an identifier of a sub-item under the first directory, where the sub-item may be a file in the file system or a directory, and for simplicity of description, the sub-item is referred to as a sub-item to be operated. The metadata server may be any MDS in the MDS cluster in the metadata processing system, and the client agent device may be a CA module in the metadata processing system.

In this embodiment, the client proxy device may carry the first identifier and the second identifier to send a metadata request to the metadata server, where the metadata request may further include a file operation command, which is used to instruct the metadata server to process the metadata information of the currently requested sub-item with the second identifier according to the file operation command. The file operation command related to the metadata information processing may be a file operation command corresponding to any one of the file operation words described above, or may be another file operation command, and is not particularly limited.

In step S502, since the metadata information of the sub-item to be operated is located in the first metadata file, after receiving the metadata request, the metadata server may search for a cache node corresponding to the first directory in the currently stored tree cache structure to determine whether a metadata cache of the first directory exists.

If the cache node corresponding to the first directory does not exist in the current tree cache structure, the metadata server can trigger a discover process to construct a metadata cache of the first directory and the sub-items to be operated. Specifically, the metadata server may start with the first directory, and sequentially obtain, by the birth record server, birth records of all directories on a path from the first directory to the system root directory of the file system. The metadata server may send the first identification to a birth record server to obtain a birth record for the first directory. The birth record server can query a birth record with a key value as the first identifier from the birth records stored in the database according to the first identifier, and send the queried birth record to the metadata server, wherein the birth record is the birth record of the first directory, and the birth record comprises the first identifier, the identifier of the parent directory of the first directory, the protection level information of the first directory, and the like.

Since the birth record of each directory includes the identity of the parent directory of the directory, the metadata server may obtain the identity of the parent directory of the first directory upon receiving the birth record of the first directory. Assume that the parent directory of the first directory is a third directory and the identifier of the third directory is a third identifier. Subsequently, the metadata server may send the third identifier to the birth record server to obtain the birth record of the third directory, so that the identifier of the parent directory of the third directory can be obtained from the queried birth record of the third directory. And in the same way, the metadata server sends the identifier of the parent directory obtained from the birth record of the child directory to the birth record server every time, obtains the birth record of the parent directory, obtains the identifier of the parent directory, and finally, the parent directory in a certain piece of the inquired birth record is the system root directory.

By obtaining all the birth records on the path from the first directory to the system root directory of the file system, the metadata server can obtain the identifications of all the directories on the path from the first directory to the system root directory. Furthermore, the metadata server can sequentially splice the identifiers of the directories by using a "/" symbol according to the sequence from the system root directory to the first directory to obtain a directory identifier path from the system root directory to the first directory, and finally splice the identifier of the sub-item to be operated at the end of the directory identifier path by using the "/" symbol to obtain a directory identifier path from the system root directory to the sub-item to be operated.

Subsequently, the metadata server may send a directory identification path from the system root directory to the child item to be operated to a subtree home management server (i.e., an MDS leader in the MDS cluster) to query an identification of a subtree root directory of the dynamic subtree where the child item to be operated is located and a home metadata server of the dynamic subtree. Then, the MDS leader may send the identifier of the subtree root directory of the dynamic subtree in which the child item to be operated is located and the identifier of the home metadata server corresponding to the dynamic subtree to the metadata server.

If the metadata server is not the home metadata server of the dynamic sub-tree in which the sub-item to be operated is located, the metadata server may send a failure response message to the client agent device, thereby ending the current metadata processing procedure. In this embodiment of the present application, the failure response message includes an identifier of the home metadata server of the sub-item to be operated, where the identifier may be used for the client agent device to resend the metadata request to the home metadata server of the dynamic sub-tree in which the sub-item to be operated is located after receiving the failure response message, and the home metadata server processes the metadata request.

If the metadata server is a home metadata server of a dynamic sub-tree in which a sub-item to be operated is located, the metadata server may create a cache node corresponding to a first directory in a current tree cache structure, and store a first identifier in the cache node corresponding to the first directory. In one possible design, the creating, by the metadata server, a cache node corresponding to the first directory in the tree cache structure specifically includes: firstly, a metadata server determines a directory identification path from a subtree root directory to a first directory according to an identifier of the subtree root directory of a dynamic subtree where a sub item to be operated is sent by a subtree home management server; then, the metadata server traverses each directory on the directory identification path according to the sequence from the root directory of the sub-tree to the first directory, and executes the following processing for each directory on the directory identification path: and searching a cache node corresponding to the currently processed directory in the current tree cache structure, if the cache node corresponding to the directory is determined not to be included in the tree cache structure, creating the cache node corresponding to the directory in the tree cache structure, and storing the identifier of the directory in the cache node.

Therefore, all the directories on the directory identification path from the subtree root directory to the first directory are sequentially traversed, the tree-shaped cache structure comprises cache nodes corresponding to all the directories on the path from the subtree root directory to the first directory, and the cache structure of all the directories on the path from the subtree root directory to the first target is constructed.

Further, as the sub-entry of the first directory, when the cache node of the first directory is not included in the tree cache structure, the cache node of the sub-entry to be operated is not included in the tree cache structure. In a possible design, after the metadata server creates the cache node corresponding to the first directory in the tree cache structure, on the basis of the created cache node of the first directory, a cache node of a sub-item to be operated may be created below the cache node of the first directory, and the second identifier may be stored in the cache node.

It should be noted that, in this process, when the metadata server creates a corresponding cache node for a certain directory, the content of the metadata file of the directory is not read from the storage device and is stored in the cache node, so that only the identifier of the directory is temporarily stored in the newly created cache node, and the content of the metadata file of the directory is not stored, so that the current tree cache structure cannot support the metadata server to process the metadata information of the sub-item to be operated. More specifically, in this embodiment of the present application, creating a corresponding cache node for a directory may be to create a corresponding entry and an inode for the directory, where the entry is used to store an identifier of the directory, and the inode is used to store metadata information of a sub-item in the directory. In the case that the metadata file is not read from the storage device, the inode part of the newly created cache node is normally empty, and the metadata server cannot be supported to process the metadata information of the sub-item of the directory.

In step S503, the storage device may be an object storage device OSD in the metadata processing system, and the metadata server may read a first metadata file stored in the object storage device according to the first identifier, and store the content of the first metadata file in a cache node corresponding to the first directory. In the embodiment of the present application, the content of the first metadata file includes metadata information of each sub-item in the first directory, and the metadata information of each sub-item may include a dentry structure and an inode structure as shown in fig. 3. It is understood that the first metadata file includes metadata information of the sub-item to be operated, and metadata information of other sub-items except the sub-item to be operated under the first directory. After the metadata server stores the content of the first metadata file in the cache node of the first directory in the tree cache structure, the cache node of the first directory stores the metadata information of the sub-item to be operated, and the metadata server can be supported to process the metadata information of the sub-item to be operated, so that the discover process is ended.

In one possible design, the metadata server may determine a storage location of the first metadata file in the object storage device according to the first identifier and protection level information of the first directory obtained from the birth record of the first directory, further read the first metadata file from the storage device, and store the content of the first metadata file in a cache node corresponding to the first directory. Specifically, the metadata server may obtain a storage location of the first metadata file in the object storage device through a hash algorithm according to the first identifier and the protection level information of the first directory. In an actual application scenario, the first metadata file may have a plurality of storage locations in the object storage device according to the protection level information of the first directory, and each storage location stores one identical piece of the first metadata file. Therefore, the storage location of the first metadata file in the object storage device determined by the metadata server in this step may be any storage location of the first metadata file in a plurality of storage locations of the object storage device.

After step S503, the metadata server may process the metadata information of the sub-item to be operated, which is stored in the cache node corresponding to the first directory in the tree cache structure, according to the file operation command in the metadata request. In a possible design, the metadata request may further include metadata content set by the client proxy device for the sub-item to be operated according to the file operation command, for example, the metadata content set for the sub-item to be operated may be attribute information such as file size, access time, access right, and the like set for the sub-item to be operated. In this way, the metadata server can modify the metadata information of the sub-item to be operated according to the file operation command and the metadata content set for the sub-item to be operated.

After step S503, the metadata server may further send metadata information of each sub-entry of the first directory saved in the cache node corresponding to the first directory to the object storage device, so that the object storage device updates the stored first metadata file. In a possible design, the metadata server may further send only the metadata information of the to-be-operated sub-item stored in the cache node corresponding to the first directory to the object storage device, so that the object storage device replaces the original metadata information of the to-be-operated sub-item in the metadata file of the first directory with the processed metadata information of the to-be-operated sub-item. In addition, the metadata server may also send a processing success response to the client proxy device to notify the client proxy device that the processing of the requested metadata information has been completed.

Because the protection level information of each directory is recorded in the birth record of the directory, the metadata server can obtain the storage position of the metadata file of the directory in the object storage device according to the identifier of the directory and the protection level information obtained from the birth record of the directory, so that the content of the metadata file of the directory is stored in the corresponding cache node in the tree cache structure without being limited by whether the cache node of the parent directory of the directory stores the metadata information of the directory or not. Even if the cache node corresponding to the parent directory of the directory in the current tree cache structure only stores the identifier of the parent directory and does not store any metadata content of the parent directory, the metadata server is not influenced to acquire the storage position of the metadata file of the directory in the object storage device, and the content of the metadata file of the directory is read from the object storage device and is stored in the corresponding cache node.

Thus, considering that the tree cache structure requires the metadata server to construct a cache structure of all directories on a path from the subtree root directory where the sub-item to be operated is located to the first directory before the metadata information is operated. In the embodiment of the application, the metadata server can only construct a basic cache structure supporting the metadata server to perform metadata operation when the metadata server receives the metadata request but the currently stored tree cache structure does not have a cache node corresponding to the first directory, so that the response time of the metadata request is effectively shortened, and the metadata processing efficiency is improved. In the embodiment of the present application, specifically, constructing a basic cache structure supporting a metadata server to perform metadata operations means, creating a cache node corresponding to each directory on a path from a subtree root directory of a dynamic subtree of a sub-item to be operated to a first directory in a tree cache structure (unless the cache node of the directory already exists), acquiring the content of the first metadata file only aiming at the first directory in the path and saving the content to the cache node corresponding to the first directory, and aiming at other directories on the path, which are different from the first directory, the metadata server does not read the content of the metadata file in the object storage device and stores the content, only the identifiers of the directories are stored in the created cache nodes, and when a metadata request requesting to operate the metadata information of the sub-items under the directories is subsequently received, the required metadata content is read from the object storage device. Therefore, the times of reading the metadata from the object storage device by the discover process can be effectively reduced, and the time consumption of the discover process is shortened.

By adopting the technical scheme provided by the embodiment of the application, in the tree cache structure stored in the metadata server, the cache nodes corresponding to the subtree root directory store the contents of the metadata files of the subtree root directory, while the cache nodes corresponding to other directories of the dynamic subtree may store the contents of the metadata files of the directory or only store the identifiers of the directories. For example, if the metadata server processes the metadata information of the sub-item of the directory after creating the cache node, the cache node may store the content of the metadata file of the directory, if the metadata server does not process the metadata information of the sub-item of the directory after creating the cache node, or although the metadata server processes the metadata information of the sub-item of the directory, the content of the metadata file of the directory stored in the cache node is eliminated in the cache due to factors such as long time, low heat, and the like, and only the identifier of the directory may be stored in the cache node.

Therefore, in order to effectively identify whether the metadata server has read the content of the metadata file of the directory from the object storage device, or whether the cache node holds the content of the metadata file of the directory, in this embodiment of the application, the metadata server may further set a status flag for each cache node in the tree cache structure, where the status flag may include four types, that is, "sub-entry trusted flag", "sub-entry untrusted flag", "self trusted flag", and "self untrusted flag".

Specifically, after a cache node corresponding to a first directory is created in the tree-shaped cache structure, the metadata server may set a "self-untrusted tag" and a "subentry trusted tag" for the cache node corresponding to the first directory, set a "self-untrusted tag" and a "subentry untrusted tag" for the cache node corresponding to a subentry to be operated, and set a "self-untrusted tag" and a "subentry untrusted tag" for a cache node corresponding to another directory newly created in a path from a subtree root directory to the first directory.

In this embodiment of the present application, a directory is provided with a "sub-entry trust flag" to indicate that a metadata server has read a metadata file of the directory from an object storage device, and a cache node corresponding to the directory in a current tree cache structure stores content of the metadata file of the directory, that is, stores metadata information of each sub-entry under the directory, and accordingly, a cache node corresponding to each sub-entry under the directory is provided with a "self-trust flag". And a directory is provided with a "sub-entry untrusted sign" to indicate that the metadata server does not need to read the content of the metadata file of the directory from the object storage device for a while, only the identifier of the directory is temporarily stored in the cache node corresponding to the directory of the current tree-shaped cache structure, and the content of the metadata file of the directory is not stored, and accordingly, the cache node corresponding to each sub-entry of the directory is provided with a "self untrusted sign".

Therefore, if the metadata server receives a metadata request sent by the client-side proxy device, and a cache node corresponding to the first directory does not exist in the tree-shaped cache structure currently stored by the metadata server, the metadata server may, after creating a cache node of each directory on a directory identification path from a subtree root directory where a sub-item to be operated is located to the first directory, obtain only the content of the metadata file of the first directory from the object storage device and store the content of the metadata file to the cache node corresponding to the first directory, set a "sub-item trusted flag" for the cache node corresponding to the first directory, set a "sub-item non-trusted flag" for other cache nodes corresponding to directories on the directory identification path that do not temporarily need to obtain the content of the metadata file from the object storage device, and wait for a foreground service request to process the metadata information of the sub-items of the directories, and then the content of the corresponding metadata file is obtained from the object storage equipment and stored. Therefore, the metadata cache contents of other directories except the first directory on the directory identification path are stored to the cache node after being delayed, so that the metadata processing system can quickly respond to foreground request processing, the overlarge fluctuation of performance reduction of taking over the MDS when the MDS fails is reduced, and the limitation on the directory depth of the file system can be solved.

In one possible design, the metadata server may also set "child trust flag" and "self trust flag" only for the cache nodes in the tree cache structure. In this way, when the "child item credibility flag" is not set in the cache node of a certain directory, it may indicate that the cache node is untrusted for the child item, and the metadata server does not need to read the content of the metadata file of the directory and save the content to the cache node for a while; when the "self-trusted flag" is not set in the cache node of a directory, it may indicate that the cache node is not trusted by itself, and the metadata server temporarily does not need to read the content of the metadata file of the parent directory of the directory and save the content to the cache node of the parent directory.

In view of this, in step S502, if after the metadata server receives the metadata request sent by the client agent device, a cache node corresponding to the first directory exists in the tree cache structure currently stored by the metadata server, and the cache node corresponding to the first directory is provided with the sub-entry untrusted flag, the metadata server may also need to read the first metadata file stored in the storage device according to the first identifier, and store the content of the first metadata file in the cache node corresponding to the first directory. Specifically, the metadata server may determine a storage location of the first metadata file in the object storage device according to the first identifier and the protection level information of the first directory, obtain the content of the first metadata file from the storage location, and store the content in the cache node corresponding to the first directory; subsequently, the metadata server can also clear the sub-item untrusted mark of the cache node corresponding to the first directory and set the sub-item trusted mark; and further, according to a file operation command in the metadata request, processing the metadata information of the sub-item to be operated, which is stored in the cache node of the first directory.

If the metadata server receives a metadata request sent by the client-side proxy device, a cache node corresponding to the first directory exists in a tree-shaped cache structure currently stored by the metadata server, and the cache node corresponding to the first directory is provided with a sub-item credible mark, the metadata server can directly process metadata information of a sub-item to be operated, which is stored in the cache node of the first directory, according to a file operation command in the metadata request.

It should be noted that, after the metadata server receives the metadata request sent by the client proxy device, if a cache node corresponding to the first directory exists in the tree cache structure currently stored by the metadata server, if the cache node corresponding to the first directory is provided with the subitem untrusted flag, a cache node corresponding to a subitem to be operated may or may not exist in the current tree cache structure. Therefore, before determining the storage location of the first metadata file in the object storage device, if it is determined that there is no cache node corresponding to the sub-item to be operated in the tree cache structure, the metadata server may create a cache node corresponding to the sub-item, and set its own trusted flag and sub-item untrusted flag for the cache node corresponding to the sub-item.

Similarly, after the metadata server receives the metadata request sent by the client-side proxy device, if the cache node corresponding to the first directory exists in the tree-shaped cache structure currently stored by the metadata server, if the cache node corresponding to the first directory is provided with the sub-item trusted tag, the cache node corresponding to the sub-item to be operated may not exist in the current tree-shaped cache structure. Therefore, before processing the metadata information of the sub-item to be operated and stored in the cache node of the first directory, the metadata server may create a cache node corresponding to the sub-item in the tree cache structure, and set a self-trusted flag and a sub-item-untrusted flag for the cache node corresponding to the sub-item.

The metadata processing method provided by the embodiment of the present application is described in detail below by a specific embodiment.

Fig. 6a exemplarily shows a dynamic subtree in this embodiment, as shown in fig. 6a, the system root directory is 400, the subtree root directory of the dynamic subtree is a directory 2100, and three subdirectories are respectively a directory 2200, a directory 2201 and a directory 2202 under the directory 2100, where three subdirectories are respectively a directory 2300, a directory 2301 and a directory 2302 under the directory 2200, and two subdirectories are respectively a directory 2400 and a directory 2401 under the directory 2300.

Figure 6b illustrates a tree cache structure maintained in MDS1 corresponding to the dynamic subtree shown in figure 6 a. Assume that this dynamic sub-tree in figure 6a belongs to MDS2 management, but MDS1 takes over this sub-tree due to MDS2 failing. Before the MDS1 receives the metadata request sent by the client agent device, the MDS1 just takes over the dynamic subtree, and the tree cache structure stored in the MDS1 only includes the cache node corresponding to the subtree root directory 2100.

First, the MDS1 receives a metadata request sent by the CA module, which includes the identities of the directory 2300 and the directory 2400.

Subsequently, MDS1 looks up the cache node corresponding to directory 2300 in the tree cache structure shown in fig. 6 b. And if finding that the cache node corresponding to the directory 2300 does not exist in the tree cache structure, triggering a discover process. The MDS1 sends the identifier of the directory 2300 to the birthday server, and obtains the birth record of the directory 2300, wherein the birthday record of the directory 2300 includes the identifier of the directory 2300, the identifier of the directory 2200, the protection level of the directory 2300, and the like. Then, MDS1 sends the identifier of directory 2200 to the birthday server, and obtains the birthday record of directory 2200, where the birthday record of directory 2200 includes the identifier of directory 2200, the identifier of directory 2100, the protection level of directory 2200, and so on. In analogy, the MDS1 obtains the birth records of all directories from the directory 2300 to the system root directory 400, and obtains the directory identification path from the directory 2300 to the system root directory 400 according to the identifications of the directories: "400/2100/2200/2300". The metadata server splices the identifier of the directory 2400 in the directory identifier path from the directory 2300 to the system root directory, and obtains the directory identifier path from the directory 2400 to the system root directory 400: "400/2100/2200/2300/2400".

Further, the MDS1 sends the directory identification path from the directory 2400 to the system root directory 400 to the MDS leader, and obtains the subtree root directory and the home MDS of the dynamic subtree in which the directory 2400 is located. Accordingly, the MDS leader returns the identification of directory 2100 and the identification of MDS1 to MDS 1. After the MDS1 determines that it is the home MDS of the dynamic subtree in which the directory 2400 is located, the cache node of the directory 2200, the cache node of the directory 2300, and the cache node of the directory 2400 are sequentially created in the tree cache structure according to the order from top to bottom. Each cache node created here includes two parts, namely, a dentry for storing the identifier of the directory and an inode for storing the metadata information of each sub-item of the directory. However, since the metadata content is not read from the OSD when the three cache nodes are created, the inode portion of the cache nodes is also actually metadata information including the sub-items.

Furthermore, the MDS1 may determine a storage location of the metadata file of the directory 2300 in the OSD according to the identifier of the directory 2300 and the protection level obtained from the birthday record of the directory 2300, obtain the content of the metadata file of the directory 2300 from the storage location, and store the content in the cache node of the directory 2300, where the metadata file of the directory 2300 includes metadata information of the directory 2400 and the directory 2401. Thus, the MDS1 may set the child item trusted flag and the self-untrusted flag for the cache node of the directory 2300, set the child item untrusted flag and the self-trusted flag for the directory 2400, and set the child item untrusted flag and the self-untrusted flag for the cache node of the directory 2200, so that the discover process is ended.

Finally, the MDS1 may process the metadata information of 2400 stored in the cache node of the directory 2300 according to the file operation command in the metadata request, and send the metadata content stored in the cache node of the directory 2300 to the OSD for storage after the processing is completed.

In addition, when the MDS1 subsequently receives a metadata request requesting to process the metadata of the child items of the directory 2300, the MDS1 can directly operate in the cache node of the directory 2300 without triggering the discover flow. When the MDS1 subsequently receives a metadata request requesting to process the metadata of the sub-entry of the directory 2200, the MDS1 may read the content of the metadata file of the directory 2200 from the OSD and store the content into the cache node of the directory 2200 according to the untrusted flag of the sub-entry set by the directory 2200.

Because the protection level information of the directory 2300 exists in the birthday record of the directory 2300, when creating a cache node of each directory on the path from the subtree root directory 2100 to the directory 2300, the MDS1 can read the content of the metadata file of the directory 2300 in the OSD directly according to the identifier of the directory 2300 and the protection level information of the directory 2300, instead of reading the content of the metadata file of the directory 2300 only after reading the content of the metadata file of the directory 2200 first. Therefore, when the metadata request is processed, the MDS1 can only read the metadata content from the object storage device once, so that the disk reading times in the discover flow can be effectively reduced, the response speed of metadata processing is improved, and the problems of performance reduction of taking over the MDS caused by taking over a dynamic subtree, MDS processing overtime caused by over-deep directory and the like are avoided.

Based on the same inventive concept, an embodiment of the present application further provides a device for constructing a metadata cache, please refer to fig. 7, which is a schematic structural diagram of the device for constructing a metadata cache according to the embodiment of the present application. As shown in fig. 7, the apparatus includes: a transceiver module 710 and a processing module 720.

A transceiving module 710, configured to perform an operation of receiving a metadata request sent by a client agent device; the processing module 720 is configured to execute that no cache node corresponding to the first directory exists in the tree cache structure stored in the apparatus, and the apparatus creates a cache node corresponding to the first directory in the tree cache structure under the condition that the apparatus is a home metadata server of a dynamic sub-tree where a sub-item of the first directory of the current request is located, reads content of the first metadata file stored in the storage device according to the first identifier, and stores the content of the first metadata file to the cache node corresponding to the first directory. It should be understood that the processing module 720 in the embodiments of the present invention may be implemented by a processor or processor-related circuit components, and the transceiver module 710 may be implemented by a transceiver or transceiver-related circuit components.

Please refer to fig. 8, which is a schematic structural diagram of an apparatus for constructing a metadata cache according to an embodiment of the present application. As shown in fig. 8, the apparatus 800 includes a processor 810, a memory 820, and a network interface 80. Optionally, the apparatus 800 further includes an input device 840, an output device 850, and a bus 860. The processor 80, the memory 820, the network interface 830, the input device 840 and the output device 860 are connected to each other via a bus 850. The memory 820 stores instructions or programs therein, and the processor 810 is configured to execute the instructions or programs stored in the memory 820. When the instructions or programs stored in the memory 820 are executed, the processor 810 is configured to perform the operations performed by the processing module 720 in the above-described method embodiment, and the network interface 830 is configured to perform the operations performed by the transceiver module 710 in the above-described embodiment.

It should be noted that the apparatus 700 or 800 provided in the embodiment of the present application may correspond to the metadata server in the metadata processing methods S501 to S505 provided in the embodiment of the present invention, and operations and/or functions of each module in the apparatus 700 or the apparatus 800 are respectively for implementing the corresponding flow of the method shown in fig. 5, and are not described herein again for brevity.

It should be understood that the processor mentioned in the embodiments of the present application may be a Central Processing Unit (CPU), and may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It will also be appreciated that the memory referred to in the embodiments of the application may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), double data rate SDRAM, enhanced SDRAM, SLDRAM, Synchronous Link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

It should be noted that when the processor is a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, the memory (memory module) is integrated in the processor.

It should be noted that the memory described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of the processes should be determined by their functions and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of constructing a metadata cache, the method comprising:

the method comprises the steps that a metadata server receives a metadata request sent by client proxy equipment, wherein the metadata request comprises a first identifier and a second identifier, the first identifier is an identifier of a first directory, the second identifier is an identifier of a subitem of the first directory, and the metadata request is used for requesting to process metadata information of the subitem;

if the cache node corresponding to the first directory does not exist in the tree cache structure stored by the metadata server, and the metadata server is a home metadata server of the dynamic subtree where the subitem is located, the metadata server creates the cache node corresponding to the first directory in the tree cache structure; wherein the creating, by the metadata server, a cache node corresponding to the first directory in the tree cache structure includes: the metadata server acquires a directory identification path from a subtree root directory of the dynamic subtree where the subitem is located to the first directory; the metadata server sequentially executes the following processing on each directory in the directory identification path according to the sequence from the root directory of the sub-tree to the first directory until the creation of the cache node corresponding to the first directory is completed: aiming at any directory in the directory identification path, if the tree-shaped cache structure does not have a cache node corresponding to the directory, creating a cache node corresponding to the directory in the tree-shaped cache structure;

and the metadata server reads a first metadata file stored in a storage device according to the first identifier, and stores the content of the first metadata file to a cache node corresponding to the first directory, wherein the first metadata file is the metadata file of the first directory, and the content of the first metadata file comprises metadata information of the sub-item.

2. The method of claim 1, wherein after the metadata server creates the cache node corresponding to the first directory in the tree cache structure, the method further comprises:

and the metadata server creates a cache node corresponding to the sub-item in the tree cache structure.

3. The method of claim 2, wherein after the metadata server saves the content of the first metadata file to the cache node corresponding to the first directory, the method further comprises:

the metadata server sets a self-untrusted mark and a sub-item trusted mark for a cache node corresponding to the first directory, sets a self-trusted mark and a sub-item untrusted mark for a cache node corresponding to the sub-item, and sets a self-untrusted mark and a sub-item untrusted mark for cache nodes corresponding to other directories newly created in a directory identification path from the subtree root directory to the first directory.

4. The method of claim 1, wherein after the metadata server receives the metadata request sent by the client agent device, the method further comprises:

if a cache node corresponding to the first directory exists in a tree-shaped cache structure stored by the metadata server and a subentry untrusted tag is set on the cache node corresponding to the first directory, the metadata server reads a first metadata file stored in the storage device according to the first identifier and stores the content of the first metadata file to the cache node corresponding to the first directory;

and the metadata server clears the sub-item non-credible mark of the cache node corresponding to the first directory and sets a sub-item credible mark for the cache node corresponding to the first directory.

5. The method of claim 4, wherein prior to the metadata server reading the first metadata file stored in the storage device, further comprising:

if the tree cache structure stored by the metadata server does not have the cache node corresponding to the subentry, the metadata server creates the cache node corresponding to the subentry in the tree cache structure;

after the metadata server stores the content of the first metadata file to the cache node corresponding to the first directory, the method further includes:

and the metadata server sets a self credible mark and a subitem incredible mark for the cache node corresponding to the subitem.

6. The method according to any one of claims 1 to 5, wherein before the metadata server creates a cache node corresponding to the first directory in the tree cache structure, further comprising:

the metadata server acquires a birth record of the first directory stored in a birth record server, wherein the birth record of the first directory comprises protection level information of the first directory;

the metadata server reads a first metadata file stored in a storage device, and the method comprises the following steps:

and the metadata server determines the storage position of the first metadata file in the storage device according to the first identifier and the protection level information of the first directory, and reads the first metadata file from the storage position.

7. The method of claim 6, wherein the metadata request further includes a file operation command;

and the metadata server processes the metadata information of the sub-items stored in the cache nodes of the first directory according to the file operation command.

8. The method of claim 1, wherein the metadata request further includes a file operation command;

after the metadata server receives the metadata request sent by the client agent device, the method further comprises:

and if the cache node corresponding to the first directory exists in the tree-shaped cache structure stored by the metadata server and the cache node corresponding to the first directory is provided with a subitem credible mark, the metadata server directly processes the metadata information of the subitem stored in the cache node of the first directory according to the file operation command.

9. The method of claim 8, wherein before the metadata server processes the metadata information of the child items stored in the cache nodes of the first directory, the method further comprises:

if the cache node corresponding to the sub-item does not exist in the tree-shaped cache structure stored by the metadata server, the metadata server creates the cache node corresponding to the sub-item in the tree-shaped cache structure, and sets a self-credible mark and a sub-item non-credible mark for the cache node corresponding to the sub-item.

10. The method according to claim 7 or 8, wherein after the metadata server processes the metadata information of the child item stored in the cache node of the first directory, the method further comprises:

and the metadata server sends the processed metadata information of the sub-items to the storage device.

11. An apparatus for constructing a metadata cache, the apparatus comprising:

a receiving and sending module, configured to receive a metadata request sent by a client agent device, where the metadata request includes a first identifier and a second identifier, the first identifier is an identifier of a first directory, the second identifier is an identifier of a child item of the first directory, and the metadata request is used to request processing of metadata information of the child item;

a processing module, configured to create a cache node corresponding to the first directory in a tree cache structure stored in the apparatus when the cache node corresponding to the first directory does not exist in the tree cache structure and the apparatus is a home metadata server of a dynamic sub-tree in which the sub-item is located;

the processing module is specifically configured to: acquiring a directory identification path from a subtree root directory of the dynamic subtree where the child item is located to the first directory; according to the sequence from the root directory of the sub-tree to the first directory, sequentially executing the following processing on each directory in the directory identification path until the creation of the cache node corresponding to the first directory is completed: aiming at any directory in the directory identification path, if the tree-shaped cache structure does not have a cache node corresponding to the directory, creating a cache node corresponding to the directory in the tree-shaped cache structure;

the processing module is further configured to read a first metadata file stored in a storage device according to the first identifier, and store the content of the first metadata file to a cache node corresponding to the first directory, where the first metadata file is a metadata file of the first directory, and the content of the first metadata file includes metadata information of the sub-item.

12. The apparatus of claim 11, wherein the processing module, after creating the cache node corresponding to the first directory in the tree cache structure, is further configured to:

and creating a cache node corresponding to the sub-item in the tree cache structure.

13. The apparatus of claim 12, wherein after the processing module saves the content of the first metadata file to the cache node corresponding to the first directory, the processing module further comprises:

setting an own untrusted mark and a subitem trusted mark for a cache node corresponding to the first directory, setting an own trusted mark and a subitem untrusted mark for a cache node corresponding to the subitem, and setting an own untrusted mark and a subitem untrusted mark for cache nodes corresponding to other directories newly created in a directory identification path from the subtree root directory to the first directory.

14. The apparatus of claim 11, wherein after the transceiver module receives the metadata request sent by the client agent device, the processing module is further configured to:

under the condition that a cache node corresponding to the first directory exists in a tree cache structure stored by the device and a subentry untrusted sign is set on the cache node corresponding to the first directory, reading a first metadata file stored in the storage equipment according to the first identity, and storing the content of the first metadata file to the cache node corresponding to the first directory;

and clearing the sub-item untrusted mark of the cache node corresponding to the first directory, and setting a sub-item trusted mark for the cache node corresponding to the first directory.

15. The apparatus of claim 14, wherein the processing module, prior to reading the first metadata file stored in the storage device, is further configured to:

when the tree cache structure stored by the device does not have the cache node corresponding to the sub item, the processing module creates the cache node corresponding to the sub item in the tree cache structure;

after the content of the first metadata file is saved to the cache node corresponding to the first directory, the processing module is further configured to:

and setting a self credible mark and a sub item non-credible mark for the cache node corresponding to the sub item.

16. The apparatus according to any of claims 11 to 15, wherein the processing module, before creating the cache node corresponding to the first directory in the tree cache structure, is further configured to:

acquiring a birth record of the first directory stored in a birth record server, wherein the birth record of the first directory comprises protection level information of the first directory;

the processing module reads a first metadata file stored in a storage device, and the processing module comprises:

and determining the storage position of the first metadata file in the storage device according to the first identifier and the protection level information of the first directory, and reading the first metadata file from the storage position.

17. The apparatus of claim 16, wherein the metadata request further comprises a file operation command;

and processing the metadata information of the sub-items stored in the cache nodes of the first directory according to the file operation command.

18. The apparatus of claim 11, wherein the metadata request further comprises a file operation command;

after the transceiver module receives the metadata request sent by the client agent device, the processing module is further configured to:

and under the condition that a cache node corresponding to the first directory exists in a tree cache structure stored by the device and a subitem credible mark is set on the cache node corresponding to the first directory, directly processing the metadata information of the subitem stored in the cache node of the first directory according to the file operation command.

19. The apparatus of claim 18, wherein the processing module, prior to processing the metadata information for the child entry maintained in the cache node of the first directory, is further configured to:

and under the condition that the cache node corresponding to the sub-item does not exist in the tree cache structure stored by the device, creating the cache node corresponding to the sub-item in the tree cache structure, and setting a self-credible mark and a sub-item non-credible mark for the cache node corresponding to the sub-item.

20. The apparatus according to claim 17 or 18, wherein after the processing module processes the metadata information of the sub-entry stored in the cache node of the first directory, the transceiver module is further configured to:

and sending the processed metadata information of the sub-items to the storage device.

21. An apparatus to construct a metadata cache, comprising at least one processor coupled with at least one memory:

the at least one processor configured to execute computer programs or instructions stored in the at least one memory to cause the apparatus to perform the method of any of claims 1-10.

22. A computer-readable storage medium, having stored thereon a computer program or instructions, which, when read and executed by a computer, cause the computer to perform the method of any one of claims 1 to 10.