CN117873967B

CN117873967B - Data management method, device, equipment and storage medium of distributed file system

Info

Publication number: CN117873967B
Application number: CN202410264177.6A
Authority: CN
Inventors: 赵明超; 张增冉; 廖旺冬; 张智; 李文龙; 罗维
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2024-03-08
Filing date: 2024-03-08
Publication date: 2024-05-17
Anticipated expiration: 2044-03-08
Also published as: CN117873967A

Abstract

The application discloses a data management method, device and equipment of a distributed file system and a storage medium, and belongs to the technical field of cloud storage. The scheme can be applied to scenes such as cloud storage, navigation and the like. Comprising the following steps: acquiring a file management request for a first file in a distributed file system, wherein the file management request is used for requesting to provide a service related to the first file; determining an access directory of first metadata based on a file management request, wherein the first metadata is used for managing a first file, and the access directory of the first metadata is used for indicating a plurality of metadata nodes to be accessed for acquiring the first metadata; determining first metadata based on a directory mapping table and an access directory of the first metadata, wherein the directory mapping table is used for representing a mapping relation between metadata and the access directory of the metadata, and the directory mapping table is stored in a hard disk; based on the first metadata, a service is performed with respect to the first file. The scheme helps to raise the metadata upper limit of the distributed file system.

Description

Data management method, device, equipment and storage medium of distributed file system

Technical Field

The present application relates to the field of cloud storage technologies, and in particular, to a data management method, apparatus, device, and storage medium for a distributed file system.

Background

In a distributed file system, metadata is implemented to manage files stored in data storage nodes. The metadata upper limit determines the upper limit of the size of the distributed file system.

In the related art, when metadata in the distributed file system reaches an upper limit, the number of main management nodes is increased by adding a new subset group, so that the upper limit of metadata of the whole distributed file system is improved.

However, metadata expansion by this method is costly.

Disclosure of Invention

The application provides a data management method, a device, equipment and a storage medium of a distributed file system. The technical scheme provided by the embodiment of the application is as follows.

According to an aspect of an embodiment of the present application, there is provided a data management method of a distributed file system, the method including the following steps.

Acquiring a file management request for a first file in the distributed file system, wherein the file management request is used for requesting to provide a service related to the first file;

determining an access directory of first metadata based on the file management request, wherein the first metadata is used for managing the first file in the distributed file system, and the access directory of the first metadata is used for indicating a plurality of metadata nodes which need to be accessed in sequence to acquire the first metadata;

determining the first metadata based on a directory mapping table and an access directory of the first metadata, wherein the directory mapping table is used for representing a mapping relation between metadata and the access directory of the metadata, and the directory mapping table is stored in a hard disk;

Based on the first metadata, a service is performed with respect to the first file.

According to an aspect of an embodiment of the present application, there is provided a data management apparatus of a distributed file system, the apparatus including the following modules.

A request acquisition module, configured to acquire a file management request for a first file in the distributed file system, where the file management request is used to request provision of a service related to the first file;

the directory determining module is used for determining an access directory of first metadata based on the file management request, wherein the first metadata is used for managing the first file in the distributed file system, and the access directory of the first metadata is used for indicating a plurality of metadata nodes which need to be accessed in sequence to acquire the first metadata;

the data determining module is used for determining the first metadata based on a directory mapping table and an access directory of the first metadata, wherein the directory mapping table is used for representing the mapping relation between metadata and the access directory of the metadata, and the directory mapping table is stored in a hard disk;

and a service execution module for executing a service related to the first file based on the first metadata.

According to an aspect of an embodiment of the present application, there is provided a computer apparatus including a processor and a memory in which a computer program is stored, the computer program being loaded and executed by the processor to implement the data management method of a distributed file system as described above.

According to an aspect of an embodiment of the present application, there is provided a computer apparatus including a processor and a memory, in which a computer program is stored, the computer program being loaded and executed by the processor to implement the data management method of the distributed file system described above.

According to an aspect of an embodiment of the present application, there is provided a computer-readable storage medium having stored therein a computer program loaded and executed by a processor to implement the data management method of a distributed file system as described above.

According to an aspect of an embodiment of the present application, there is provided a computer program product including a computer program stored in a computer-readable storage medium, from which a processor reads and executes the computer program to implement the data management method of a distributed file system as described above.

The technical scheme provided by the embodiment of the application has the beneficial effects that at least:

Compared with the prior art that the directory tree for managing the metadata is stored in the memory of the master management node, the addition of new metadata causes the data volume of the directory tree to be increased, so that the upper limit of the metadata is limited by the storage capacity of the memory. The embodiment of the application uses the directory mapping table to replace the directory tree stored in the memory in the related technology, and stores the directory mapping table through the hard disk. The method is favorable for reducing the restriction of the device hardware on the upper limit of the quantity of the metadata, is favorable for improving the upper limit of the quantity of the metadata, and has lower cost for expanding the metadata because the storage capacity of the hard disk is far larger than that of the memory.

Drawings

FIG. 1 is a schematic illustration of an implementation environment for an embodiment of the present application;

FIG. 2 is a schematic diagram of the inventive concept;

FIG. 3 is a flowchart of a method for data management of a distributed file system provided by an exemplary embodiment of the present application;

FIG. 4 is a schematic diagram of a directory tree provided by an exemplary embodiment of the present application;

FIG. 5 is a schematic diagram of a method of data management of a distributed file system according to another exemplary embodiment of the present application;

FIG. 6 is a schematic diagram of a data management method for a distributed file system according to yet another exemplary embodiment of the present application;

FIG. 7 is a schematic diagram of a metadata update process in the related art;

FIG. 8 is a schematic diagram of a metadata update process provided by an exemplary embodiment of the present application;

FIG. 9 is a graph comparing the effects of experiments conducted based on the protocol provided by the embodiments of the present application;

FIG. 10 is a block diagram of a data management apparatus of a distributed file system provided by an exemplary embodiment of the present application;

fig. 11 is a block diagram of a computer device according to an exemplary embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.

Cloud Technology (CT) refers to a hosting Technology that unifies serial resources such as hardware, software, network, etc. in a wide area network or a local area network to realize calculation, storage, processing and sharing of data. Cloud technology is a generic term for network technology, information technology, integration technology, management platform technology, application technology and the like based on cloud computing business model application. Background services of technical networking systems require a large amount of computing, storage resources, such as video websites, picture-like websites, and more portals.

Cloud Storage (CS) refers to a new concept that extends and evolves over Cloud computing concepts. Cloud storage may be implemented as a distributed file system.

Distributed file system (Distributed FILE SYSTEM, DFS) refers to a storage system that integrates a large number of different types of storage devices (also called storage nodes) in a network through application software or application interfaces to cooperatively work through functions of cluster application, grid technology, distributed storage file system and the like, and provides data storage and service access functions together. The physical storage resources managed in a distributed file system are not necessarily directly connected to a local node, but are instead distributed to different data storage nodes (which may be understood simply as a computer).

At present, the storage method of the distributed system is as follows: when creating logical volumes, each logical volume is allocated a physical storage space, which may be a disk composition of a certain storage device or of several storage devices.

A Database (Database) is understood to mean an electronic file cabinet, i.e. a place where electronic files are stored. The user can perform operations such as adding, inquiring, updating, deleting and the like on the data stored in the database.

The Database management system (Database MANAGEMENT SYSTEM, DBMS) is a computer software system designed for managing databases, and generally has basic functions of storage, interception, security, backup and the like.

The intelligent transportation system (INTELLIGENT TRAFFIC SYSTEM, ITS) is an integrated transportation system which is formed by effectively and comprehensively applying advanced scientific technologies (including but not limited to information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operation study, artificial intelligence and other technologies) to transportation, service control and vehicle manufacturing, and enhancing the connection among vehicles, roads and users.

FIG. 1 is a schematic diagram of an implementation environment for an embodiment of the present application. The implementation environment of the scheme can comprise: the terminal device 10, the master management node 20, the data storage node 30 and the slave management node 40, wherein the master management node 20, the data storage node 30 and the slave management node 40 are functional nodes in the distributed file system.

Terminal device 10 includes, but is not limited to, personal computers (Personal Computer, PCs), cell phones, tablet computers, wearable devices, intelligent voice interaction devices, intelligent appliances, vehicle terminals, aircraft, and like electronic devices. A client with a target application running on the terminal device 10. The terminal device 10 may also be directly referred to as a client (client) in the cloud storage domain.

The target application has a file management function. For example, the target application is used to provide file cloud storage and file management functions for the user. The target application may be a stand-alone application, a sub-application (i.e., applet) that runs in dependence on the parent application, or a functional web page that is presented in the browser. Types of target applications include, but are not limited to: cloud application class applications, shopping class applications, social class applications, interactive entertainment class applications, browser applications, content sharing class applications, virtual reality class applications, augmented reality class applications, etc., to which embodiments of the present application are not limited.

The terminal device 10 has at least a data transmitting/receiving function and a storage function. Alternatively, the terminal device 10 uploads a file to the distributed file system through the data transceiving function, downloads a file from the distributed file system, and instructs the distributed file system to perform editing operations on the file to change the file content.

The master management node 20, the data storage node 30, and the slave management node 40 are used to provide distributed file management services. Wherein master management node 20, data storage node 30, and slave management node 40 are all computer devices. Illustratively, in the case of running a master management process in a computer device, the computer device is the master management node 20; in the case of a computer device for storing files, the computer device is the data storage node 30. In the case of a distributed file system that is a Hadoop (an open source code framework written by program instructions), the master management node 20 is called Active naminode, the data storage node 30 is called Datanode, and the slave management node 40 is called Standby naminode. It should be noted that, the names of the master management node 20, the data storage node 30, and the slave management node 40 may be different in different distributed file systems, and the present application is not limited to the names of the three types of nodes in practical applications.

The master management node 20 may be understood as a server corresponding to the target application program, and is used for providing a background service for a client in the target application program, for example, the master management node 20 is a cloud server for providing a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, a middleware service, a domain name service, a security service, a content delivery network, (Content Delivery Network, CDN), and a basic cloud computing service such as a big data and artificial intelligence platform.

To achieve distributed storage, typically a distributed file system includes a plurality of data storage nodes 30, and the file uploaded by the user is divided into a plurality of data blocks (blocks), each of which is stored in at least one data storage node 30.

The slave management node 40 refers to a computer device for implementing the auxiliary management of the distributed file system. In the event of a downtime of master management node 20, slave management node 40 may be used in place of master management node 20 to promote stability of the distributed file system.

Master management node 20 may establish a communication connection with terminal device 10, data storage node 30, and slave management node 40. A data connection may be established between the terminal device 10 and the data storage node 30. In the event of a downtime of the master management node 20, the slave management node 40 may establish a communication connection with the terminal device 10, the data storage node 30.

In one application scenario, the terminal device 10 looks up the first file from the distributed file system. The example mainly includes the following processes: the terminal device 10 transmits a file management request regarding the first file to the master management node 20, and the master management node 20 determines and transmits first metadata for managing the first file to the terminal device 10 based on the file management request. The terminal device 10 determines at least one data storage node 30 storing data blocks of the first file based on the first metadata, and then obtains each data block into which the first file is divided from the at least one data storage node 30, and merges the data blocks to obtain the first file.

The master management node 20 needs to store metadata corresponding to all files in the distributed file system, and it can be seen that the upper limit of metadata stored in the distributed file system limits the upper limit of the number of files that can be managed by the distributed file system to a certain extent. After the target application program is opened for use in the actual application, the number of users is increased continuously, the number of files to be managed is increased, and the number of metadata is required to be expanded to meet the actual needs.

In the related art, a main means for raising the metadata upper limit of the distributed file system is to build a plurality of sub-clusters. That is, a plurality of sub-clusters are included in one distributed file system cluster, each sub-cluster including: the method comprises the steps of managing a master management node, managing a slave management node and storing files, wherein the different sub-clusters do not share any kind of nodes in the master management node, the slave management node and the storing files. Because the main management node in each sub-cluster has a respective metadata upper limit, the metadata upper limit of the whole distributed file system is improved by continuously adding new sub-clusters and introducing new main management nodes.

The plurality of sub-clusters are configured into one large cluster through a unified access layer. For example, multiple sets of sub-clusters are included under one access layer. Because the user interacts with the distributed file system through the access layer, i.e. the user can only perceive the access layer, the user sees an entire cluster from the view of the access layer by mounting the directories of the different clusters to the access layer.

However, increasing the overall metadata upper limit of a distributed file system by adding multiple sub-clusters has the following drawbacks. For example, the data management nodes of different sub-clusters are not in common, and it is therefore difficult to rename a directory in the data management node of one sub-cluster to the data management node of another sub-cluster. In addition, if metadata is too much in a certain subset group, metadata in the master management node of the subset group may exceed an upper limit. Too much metadata in a subset may also result in a longer time consuming restart process for the master management node in that subset. Therefore, further exploration of the upper limit lifting method of metadata is also required.

Fig. 2 is a schematic diagram of the inventive concept.

The scheme provided by the application is mainly used for expanding the upper limit of the quantity of the metadata which can be managed by the master management node, and reducing the restriction of the upper limit of the quantity of the metadata on the size of the distributed file system. Specifically, the embodiment of the application is based on the fact that the directory tree maintained in the memory of the master management node in the related technology is converted into the directory mapping table, and the directory mapping table is stored in the hard disk of the master management node. When the metadata of the corresponding file needs to be acquired based on the file management request of the client, the metadata is searched from the directory mapping table.

Compared with the prior art that the directory tree for managing the metadata is stored in the main management node memory, the data volume of the directory tree is increased in the process of adding the new metadata, and the new metadata cannot be added after the directory tree reaches a certain data size due to the restriction of the memory capacity. The embodiment of the application converts the directory tree into the directory mapping table and stores the directory mapping table in the hard disk, and the storage capacity of the hard disk is far greater than that of the memory, so that the method is beneficial to reducing the restriction of the device hardware on the upper limit of the quantity of the metadata and improving the upper limit of the quantity of the metadata.

In addition, in the restarting process of the master management node, the directory mapping table does not need to be loaded into the memory, so that the restarting time of the master management node is shortened, and the stability of the distributed file management service is improved.

Fig. 3 is a flowchart of a data management method of a distributed file system according to an exemplary embodiment of the present application. Illustratively, the subject of execution of the method may be the master management node 20 in fig. 1. The method is described below with a computer device as the main body for the convenience of description. As shown in FIG. 3, the method may include the following steps (310-340).

In step 310, a file management request for a first file in a distributed file system is obtained, the file management request being for requesting provision of a service with respect to the first file.

In some embodiments, a distributed file system is used to provide distributed storage and management services for files. Optionally, the distributed file system includes at least one data storage node, where the data storage nodes are configured to store a plurality of data blocks obtained by dividing a file, and the plurality of data blocks may be combined to obtain the complete file. The files stored in the distributed file system comprise files uploaded by users. Optionally, the distributed file system further includes a master management node, where metadata corresponding to each file stored in the distributed file system is stored in the master management node, where the metadata is used to manage the corresponding file.

Illustratively, the distributed file system is built based on Hadoop, referred to as Hadoop distributed file system (Hadoop Distributed FILE SYSTEM, HDFS). In this case, the master management node is called an Active naminode, and in the embodiment of the present application, the execution subject computer device is an Active naminode.

Optionally, the distributed file system provides a unified interface layer to the client such that the client interacts with the distributed file system through the interface layer. The distributed file system is used for providing corresponding file management service based on the file management request provided by the client.

In some embodiments, the file management request is for requesting management of the first file from the distributed file system. Alternatively, the first file refers to a file stored in a distributed file system, and as understood from the above embodiments, the first file is divided into a plurality of data blocks, and is stored in a plurality of data storage nodes in the form of data blocks.

The types of file management requests include, but are not limited to, at least one of: file edit request, file delete request, file find request. The file editing request is used for requesting to edit file contents of the first file; the file deletion request is used for requesting to delete the first file from the distributed file system; the file search request is used for requesting to acquire the first file from the distributed file system to the client, that is, downloading the first file from the distributed file system.

Optionally, the file management request includes a type indication identifier and a file identifier of the first file. Wherein the type indication identifier is used to characterize the type of service requested by the file management request, and the file identifier of the first file is used to uniquely identify the first file. The file identification of the first file includes, but is not limited to, at least one of: the name and number of the first file, the access directory of the metadata corresponding to the first file, and the like.

Step 320, determining an access directory of the first metadata based on the file management request, where the first metadata is used for managing the first file in the distributed file system, and the access directory of the first metadata is used for indicating a plurality of metadata nodes that need to be accessed in sequence to obtain the first metadata.

In some embodiments, the first metadata refers to metadata corresponding to the first file in the distributed file system. The distributed file system manages files stored therein through metadata.

Optionally, the first metadata is used for recording attribute information related to the first file. Illustratively, the first metadata includes storage addresses in the data storage node of respective data blocks into which the first file is divided. Illustratively, the first metadata further comprises at least one of: access rights of the first file, file owner identification, editing information of the first file. The access authority of the first file is used for indicating at least one user capable of accessing and acquiring the first file; the file owner identifies a user for characterizing the first file in possession of the first file, i.e., uploading the first file in the distributed file management system; editing information of the first file is used for recording modification of the first file in the storage process of the distributed file system.

Metadata in a distributed file system in the related art is stored in a memory of a computer device, and for convenience in managing metadata, metadata is stored in metadata nodes, that is, a plurality of metadata are stored in respective metadata nodes. The metadata nodes are interconnected to form a tree structure, which may be referred to as a directory tree.

The number of metadata nodes is larger than the total number of metadata in the distributed file system, and metadata nodes corresponding to the metadata belong to leaf nodes in the tree structure.

In this case, the first metadata may further include node information for recording other metadata nodes having a connection relationship with the first metadata node in the tree structure, and the first metadata node refers to a node of the tree structure for storing the first metadata.

In some embodiments, the access directory of the first metadata is used to find the first metadata, i.e. the access directory of the first metadata is an access index used to obtain the first metadata. The access directories to which different metadata correspond are different. Optionally, the access directory of the first metadata includes a plurality of subdirectories, and each subdirectory corresponds to one metadata node. The computer equipment sequentially accesses each sub-directory according to the arrangement sequence of each sub-directory in the access directory of the first metadata, and the first metadata node can be found so as to obtain the first metadata stored in the first metadata node.

Optionally, the access directory of the first metadata is used to indicate a node access path from a root node of the tree structure to the first metadata node, where the root node refers to a starting node in the tree structure, and the root node does not have a parent node in the tree structure. Illustratively, the access directory of the first metadata includes node names of a plurality of metadata nodes, that is, the access path of the first metadata is composed of the node names of the plurality of metadata nodes according to the access order of the metadata nodes in the tree structure.

For example, the access directory of a certain first metadata node is/dir 1/file1, where "/" indicates the name of the root node, "dir1" indicates the name of one metadata node, "file1" indicates the name of the first metadata node, and "/" between "dir1/file1" indicates the separator.

In one example, after obtaining the file management request, the computer device can directly obtain the access directory of the first metadata from the file management request, that is, the access directory of the first file carried in the file management request. In another example, after the file management request is obtained, the computer device generates an access directory of the first metadata based on the requested content of the file management request. For details on this part, reference is made to the following examples.

Step 330, determining the first metadata based on a directory mapping table and an access directory of the first metadata, wherein the directory mapping table is used for representing a mapping relationship between the metadata and the access directory of the metadata, and the directory mapping table is stored in the hard disk.

In some embodiments, a directory map refers to a map that provides a mapping relationship between metadata's access directory and metadata. That is, the correspondence relationship between the access directory of the metadata and the metadata can be determined through the directory mapping table so that the corresponding metadata can be found through the access directory of the metadata. Optionally, the directory mapping table includes metadata, that is, the computer searches metadata corresponding to the access directory of the first metadata according to the directory mapping table, so as to obtain the first metadata.

Illustratively, the directory map table belongs to a key-value table, wherein the key-value table refers to a lookup table composed of at least one key-value pair, and the key-value pair includes a key and a key value (value), through which a corresponding key value can be found in the key-value table. For example, the computer device determines a target key value according to the access directory of the first metadata, and takes a key value corresponding to the target key value as the first metadata. For details of this process, please refer to the following examples. By setting the directory mapping table in the form of a key value table, the speed of determining the first metadata from the directory mapping table based on the access directory of the first metadata is improved, and the response speed of the distributed file system to the file management request is improved.

In some embodiments, the directory map is stored in a hard disk. That is, during the operation of the master management node, the complete version of the directory mapping table does not need to be loaded into the memory. Optionally, the directory map is stored in a database, most of the data in the database being stored directly in the hard disk. Illustratively, the database is RocksDB (an efficient database software).

Optionally, in the case that the first metadata needs to be looked up based on the directory map, the computer device reads part of the content in the directory map from the hard disk into the memory for further operation. For example, the computer device reads the first metadata in the target mapping table from the hard disk into the memory to generate feedback information for the user to send to the client based on the first metadata. For details of this step, refer to the following examples.

In some embodiments, the directory map includes at least one map. Optionally, in the case that one mapping table is included in the directory mapping table, the access directory of the metadata corresponds to the first metadata between the mapping tables. Illustratively, the computer device directly looks up the access directory of the first metadata from the mapping table to obtain the first metadata. For example, the directory mapping table includes at least one key value pair using the access directory of the metadata as a key value, and the computer device searches the directory mapping table for the first metadata corresponding to the access directory of the first metadata using the access directory of the first metadata as a target key value.

Optionally, the directory mapping table includes at least two mapping tables, the computer device determines a node index of the first metadata node from the first mapping table according to the access directory of the first metadata, and searches the first metadata from the second mapping table according to the node index of the first metadata, and for details about this process, please refer to the following embodiments. The method is beneficial to improving the speed of searching and obtaining metadata from the catalog mapping table by setting at least two mapping tables, and is beneficial to reducing the overlapping information in the target mapping table by designing the mapping relation respectively represented by the at least two mapping tables in consideration of possible overlapping among different metadata access catalogs, thereby reducing unnecessary repeated information stored in the catalog mapping table and reducing the data volume of the catalog mapping table.

Step 340, based on the first metadata, performs a service with respect to the first file.

In some embodiments, the service with respect to the first file refers to a file management service performed with respect to the first file. The service for the first file includes at least one of: searching a first file, editing the first file, and deleting the first file.

In some embodiments, the data management method of the distributed file system further comprises: if the file management request is for requesting to change the first file, the computer device modifies the first metadata stored in the directory map based on the file management request; if the file management request is used for requesting to delete the first file, the computer equipment deletes the mapping relation related to the first metadata in the directory mapping table based on the file management; and if the file management request is used for requesting to access the first file, the computer equipment sends the first metadata to the client so that the client can acquire the file content of the first file from the file storage equipment in the distributed file system according to the first metadata.

Alternatively, the master management node may not directly act on the data storage node, and the master management node may feed back the first metadata to the client, so that the client instructs the data storage node to perform a corresponding operation on the first file based on the first metadata. Illustratively, the primary management node performing the service with respect to the first file includes feeding back the first metadata to the client.

For example, in the case that the service related to the first file is searching for the first file, the computer device sends the first metadata to the client, so that the client determines, according to the first metadata, storage addresses of data blocks into which the first file is divided in the data storage nodes, obtains the data blocks from the corresponding storage addresses, and finally obtains the first file.

As can be seen from the above embodiments, the type of service with respect to the first file is indicated by the file management request. The type of the service related to the first file is indicated according to the actual requirement of the client, and the present application is not described herein.

In summary, compared with the related art in which a directory tree for managing metadata is stored in a memory of a master management node, when new metadata is added, the data amount of the directory tree is increased, so that the upper limit of the metadata is limited by the storage capacity of the memory. The embodiment of the application uses the directory mapping table to replace the directory tree stored in the memory in the related technology, and stores the directory mapping table through the hard disk. The method is favorable for reducing the restriction of the device hardware on the upper limit of the quantity of the metadata, is favorable for improving the upper limit of the quantity of the metadata, and has lower cost for expanding the metadata because the storage capacity of the hard disk is far larger than that of the memory.

In some embodiments, step 320, as shown in FIG. 3, includes at least the following two implementations.

Implementation 1: the computer device reads the access directory of the first metadata from the file management request.

In this example, the file management request includes the first metadata, and the computer device directly reads the access directory of the first metadata from the file management request.

Implementation 2: the computer determines the node names of the metadata nodes based on the file management request; the computer equipment sorts the node names of the metadata nodes respectively to obtain an access directory of the first metadata.

In this example, the node names of the metadata nodes are used to uniquely identify the metadata nodes, with the node names of the different metadata nodes being different.

Optionally, the node names of the metadata nodes in different levels of the tree structure are different in form, the computer equipment determines the levels to which the metadata nodes respectively belong according to the node names of the metadata nodes, and sorts the node names of the metadata nodes from high to low based on the levels of the metadata nodes, so as to obtain the access directory of the first metadata.

Illustratively, a hierarchy mapping table is provided in the computer device, and for the node name of any metadata node, the computer device determines the hierarchy to which the metadata node belongs according to the hierarchy mapping table. For example, the tree structure includes 3 levels, the level mapping table indicates that the node name of the highest level is named "/", the node name of the second highest level has "dir", the node name of the lowest level has "file", and the computer device determines the arrangement order of the node names of the metadata in the access directory of the first metadata according to the level mapping table.

Since the complete metadata access directory is not included in the file management request, even if the file management request is hijacked, it is difficult for the hijacking device to determine the metadata access directory according to the file management request. Therefore, the access directory of the metadata is indirectly acquired through the file management request, and the security of the file management request is improved.

The following describes the first metadata lookup process in several embodiments.

As described in the above embodiment, if the directory mapping table includes at least two mapping tables, the efficiency of determining the first metadata according to the scheme provided by the present application is improved, and the description is given below in the case that the directory mapping table includes two mapping tables.

In some embodiments, metadata nodes corresponding to metadata included in the distributed file system are connected into a tree structure, and the directory mapping table includes an edge mapping table and a node mapping table, wherein the edge mapping table is used for representing a mapping relationship between two metadata nodes with parent-child relationships in the tree structure, and the node mapping table is used for representing a mapping relationship between metadata nodes and metadata stored by the metadata nodes.

In some embodiments, the tree structure includes at least one node, and connection relationships between the nodes, the nodes for storing information, the connection relationships between the nodes for characterizing the relevant relationships between the nodes. The connection between nodes is also called edge, edge. In the embodiment of the application, the nodes in the tree structure are metadata nodes.

Optionally, each node is set layer by layer from the root node in the tree structure, no connection relation is set between nodes of the same level, a structure of connection relation is allowed to be set between two nodes of adjacent levels, and two nodes with connection relation in the two adjacent levels are parent-child nodes.

Wherein, the root node of the tree structure does not have a parent node, and the leaf node of the tree structure does not have a child node. Illustratively, the metadata node corresponding to the metadata in the tree structure belongs to a leaf node. That is, metadata for managing files is stored in leaf nodes of the tree structure.

In a distributed file system, this tree structure is also referred to as a directory tree. Optionally, the access directory of the first metadata is used to characterize a directory path from the root node of the tree structure to the first metadata node for storing the first metadata.

Fig. 4 is a schematic diagram of a directory tree provided by an exemplary embodiment of the present application.

As shown in fig. 4, the "/" node is a root node in the directory tree, and a connection relationship exists between the root node "/" and the node "dir1", that is, the root node "/" and the node "dir1" are parent nodes, where the root node "/" is a parent node of the node "dir1", the node "dir1" is a child node of the root node "/", and the node "file1" is a leaf node in the directory tree. Illustratively, metadata for managing the file1 is stored in the node "file 1".

In some embodiments, a directory map is used to characterize the tree structure. Optionally, the directory mapping table includes an edge mapping table and a node mapping table, the edge mapping table is used for describing the connection relationship in the tree structure, that is, the edge mapping table is used for representing edge (edge) information between metadata nodes having the connection relationship in the tree structure, and the node mapping table is used for describing each metadata node included in the tree structure.

Illustratively, the edge mapping table stores node names of the metadata nodes, the node mapping table stores at least one metadata, the edge mapping table may correspond to the first mapping table in the above embodiment, and the node mapping table may correspond to the second mapping table in the above embodiment.

Optionally, the edge map and the node map are stored in a hard disk. In determining the first metadata, the computer device reads part of the contents in the edge mapping table and part of the contents in the node mapping table from the hard disk to the memory, and for details of this process, please refer to the following embodiments.

It should be clear that, the tree structure mentioned in the embodiment of the present application is for facilitating understanding of the scheme, and in the actual working process, the master management node in the embodiment of the present application does not need to maintain the tree structure, and only needs to maintain the directory mapping table stored in the hard disk. For example, the computer device only needs to maintain the edge mapping table and the node mapping table, and can determine the corresponding metadata according to the access paths of the edge mapping table and the node mapping table based on the metadata. Compared with the related art, in the practical application process, the directory tree structure does not exist in the master management node.

Optionally, the edge mapping table and the node mapping table belong to a key value table, that is, the computer device determines a target key value based on the access directory of the first metadata, performs a search operation from the edge mapping table or the node mapping table based on the target key value, and finally determines the first metadata. For details of this step, please refer to the following examples.

Fig. 5 is a schematic diagram of a data management method of a distributed file system according to another exemplary embodiment of the present application, where an execution body of the embodiment is a computer device, and the embodiment of the present application may include the following steps.

Step 310 obtains a file management request for a first file in the distributed file system, the file management request being for requesting provision of a service with respect to the first file.

For a detailed description of steps 310, 320, refer to the above embodiments.

Step 330 as shown in fig. 3 may be implemented as the following steps 333 and 336.

In step 333, a node index of the first metadata node is determined from the edge mapping table based on the access directory of the first metadata, where the first metadata node is a metadata node for storing the first metadata.

In some embodiments, the node name of the metadata node and the node index of the metadata node are both used to represent the metadata node. Wherein the node name of the metadata node is used to characterize the naming of the metadata node.

Illustratively, the name of a metadata node is related to the file managed by the metadata stored in that metadata node. For example, the name of the metadata node is the same as the file name managed by the metadata. Illustratively, the node name of the metadata node is generated by the computer device during the user's storage of the file to the distributed file system. For example, after receiving a new file to be stored, the computer device generates the name of the metadata node according to the naming convention.

In addition, the node name of the metadata node can be configured by a worker, and the source of the node name of the metadata node is not limited.

The node index of the metadata node is used to find the metadata node. Illustratively, the node index of the metadata node is represented by a number. For example, the node index of a certain metadata node is 21. The node index of the metadata may be referred to as an identity (Identity Document, ID) of the metadata node.

Optionally, the node names of the different metadata nodes are different, and the node indexes of the different metadata nodes are also different. For a certain metadata node, the node name of the metadata node corresponds to the node index of the metadata node one by one. Illustratively, the node name of the metadata node is known to the client and the node index of the metadata node is not known to the client, i.e., the client may carry the node name of the metadata node in the file management request. For example, an access directory of metadata is carried in the file management request, the access directory of metadata being composed of node names of at least one metadata node.

The present solution is used to manage at least one metadata in a tree-like structure by converting a directory tree used in the related art into an edge mapping table and a node mapping table. For example, combining the edge mapping table and the node mapping table can characterize the mapping relationship between the access directory of metadata and the corresponding metadata. Because the tree structure does not exist in the actual application process of the scheme, the node name of the metadata node in the embodiment of the application can be understood as the name of the metadata, and the node index of the metadata node can be understood as the index of the metadata.

In some embodiments, the first metadata node refers to a metadata node in a tree structure that stores first metadata. The node index of the first metadata is used to uniquely identify the first metadata node. Optionally, the node mapping table includes a mapping relationship between the first metadata and a node index of the first metadata node.

In one example, the edge mapping table includes a correspondence between the access directory of the first metadata and the node index of the first metadata node, that is, the computer device may directly find the node index of the first metadata node from the edge mapping table. Illustratively, the edge mapping table includes a key pair with an access directory of a certain metadata as a key.

In another example, optionally, the computer device determines a node index of the intermediate node based on the subdirectories included in the access directory of the first metadata, and updates the node index of the intermediate node based on the node index of the intermediate node and the next partial subdirectory until the node index of the first metadata node is determined. The node index of the intermediate node may be understood as the node index of the first auxiliary node in the following embodiments, and the next partial subdirectory may be understood as the second auxiliary node in the following embodiments, for details of this step, reference is made to the following embodiments.

Step 336, determining the first metadata from the node mapping table based on the node index of the first metadata node.

Optionally, the node mapping table includes a correspondence between node indexes of metadata nodes and metadata. The computer device directly obtains the first metadata from the node mapping table based on the node index of the first metadata node.

Optionally, the node mapping table includes a correspondence between a node index of the metadata node and a storage address of the metadata, and the computer device determines the storage address of the first metadata based on the node index of the first metadata node, and accesses the storage address to obtain the first metadata.

For details of step 340, please refer to the above embodiments, and detailed descriptions thereof are omitted.

Since there may be an overlap between access directories of different metadata, for example, metadata nodes corresponding to two metadata respectively have the same parent node in the tree structure. In this case, the access directories of the two metadata only differ in the node name of the last metadata node, and the rest are the same. The mapping relation between the access directory of the metadata and the metadata is reflected by the edge mapping table and the node mapping table together. On the one hand, the method is beneficial to reducing the information overlapping degree of the edge mapping table (for example, the edge mapping table does not need to comprise the complete access directory of metadata), reducing the data volume of the directory mapping table and reducing the storage space occupied by the storage directory mapping table.

On the other hand, compared with the access directory of metadata, the node index of the metadata node corresponding to the metadata to be searched is relatively simple, so that the speed of determining the metadata to be searched from the node index table according to the node index of the metadata is faster. According to the embodiment of the application, the edge mapping table is set, so that the reading speed of the computer equipment to the edge mapping table is improved, and the searching efficiency of searching the metadata is improved.

The process of determining node indexes of corresponding metadata nodes based on access directories of metadata is described in several exemplary embodiments.

In some embodiments, the access directory of the first metadata includes node names of respective metadata nodes, two adjacent metadata nodes in the metadata nodes are parent-child nodes in the tree structure, the edge mapping table includes a mapping relationship between a parent-child node combination and node indexes of child nodes, and the parent-child node combination includes node indexes of the parent nodes and node names of child nodes corresponding to the parent nodes.

Optionally, the node names of the metadata nodes have an arrangement order in the access directory of the first metadata, the arrangement order being used to simulate an order in which the computer device accesses the metadata nodes one by one from the root directory of the tree structure and finally reaches the first metadata node.

Illustratively, an access directory of a certain first metadata is/dir 2/file3 (as can be understood with reference to fig. 4), where "/", "dir2" and "file3" are node names of each of 3 metadata nodes, where the access directory of the first metadata indicates that if the metadata node "file3" needs to be accessed, the computer device needs to access the root node "/", then access the metadata node "dir2", and finally access the metadata node "file3", which indicates that the arrangement sequence between "/", "dir2" and "file3" is "/", "dir2" and "file3".

Optionally, two adjacent metadata nodes in the plurality of metadata nodes refer to two adjacent metadata nodes in the access directory of metadata at positions of node names.

For convenience of description, two adjacent metadata nodes among the plurality of metadata nodes are respectively referred to as a first node and a second node, wherein the first node is a parent node of the second node, and the second node is a child node of the first node. In the access path of the first metadata, the node name of the first node is arranged before the node name of the second node, and the node name of the first node is adjacent to the node name of the second node. That is, the premise of accessing the second node in the tree structure is to access the first node. Before accessing the first metadata node, the second node needs to be accessed, and then the second node is accessed based on the information provided by the first node.

In some embodiments, an edge mapping table is used to characterize the mapping between parent-child node combinations and child nodes. Optionally, the parent-child node combination is used to characterize a parent node and a child node corresponding to the parent node, such as the first node and the second node in the previous paragraph. For example, the parent-child node combination includes the node index of the parent node and the node name of the child node.

Optionally, the edge map representation includes a mapping relationship between a parent-child node combination and a node index of the child node. Illustratively, the edge mapping table belongs to a key value table, and the edge mapping table comprises a plurality of edge key value pairs. For any one of a plurality of edge key value pairs, the key value in the edge key value pair is the node index of the father node and the node index of the child node, that is, the father-child node combination, and the key value is the node index of the child node. The computer equipment combines the access catalog of the first metadata and the edge mapping table, gradually updates the key value, and finally determines the node index of the first metadata node from the edge mapping table. The process will be described below.

In some embodiments, the determining the node index of the first metadata node from the edge mapping table based on the access directory of the first metadata in sub-step 333 may include several sub-steps whose execution bodies are computer devices, such as the master management node in the above embodiments.

In step 333-a, a root node of the plurality of metadata nodes is used as a first auxiliary node, a node name of a second auxiliary node is determined based on an access directory of the first metadata, the root node is a top node in the tree structure, a node index of the root node is known, and the second auxiliary node is a child node of the first auxiliary node.

In some embodiments, the plurality of metadata nodes includes a plurality of node names included in an access directory of the first metadata corresponding to the metadata nodes, respectively. Since the node names of the metadata nodes can uniquely identify the metadata nodes, a plurality of metadata nodes can be understood as all metadata nodes indicated by the access directory of the first metadata. As can be appreciated based on the above embodiments, the plurality of metadata nodes includes metadata nodes that need to be traversed before accessing the first metadata node in the tree structure, and the first metadata node.

In some embodiments, the root node corresponds to the first metadata node in the tree structure, starting from the root node, and is able to access any metadata node in the tree structure. The access directory of the first metadata includes a node name of the root node. Optionally, the access directory of each metadata in the distributed file system includes a node name of the root node, and the node name of the root node is arranged in a top position in the access directory of the metadata.

Optionally, in the case that the root node is the first auxiliary node, the second auxiliary node refers to a metadata node adjacent to the node name of the root node in the access directory of the first metadata. In this case, the second auxiliary node refers to a child node having the root node as a parent node. It is understood by the tree structure that there is a direct connection between the first auxiliary node and the second auxiliary node.

For example, the access directory of the first metadata is/dir 2/file3, where "/", "dir2" and "file3" are node names of 3 metadata nodes, respectively, "/" is a node name of a root node, and in the case that the root node is the first auxiliary node, the second auxiliary node is a metadata node whose node name is "dir 2".

The node name of the root node is included in the access directory of any of the first metadata, and thus the node index of the root node is known to the computer device. Illustratively, the root node has a node index of "0" and the computer device has stored therein: the node index of the root node is 0.

Sub-step 333-c, searching the node index corresponding to the first parent-child node combination from the edge mapping table based on the first parent-child node combination formed by the node index of the first auxiliary node and the node name of the second auxiliary node, as the node index of the second auxiliary node.

Optionally, the first parent-child node combination includes a node index of the first auxiliary node and a node name of the second auxiliary node. The computer device illustratively combines the node index of the first auxiliary node and the node name of the second auxiliary node in a dictionary to obtain a first parent-child node combination. For example, the first parent-child node combination may be represented as (parentInodeId, CHILDNAME), where parentInodeId represents the node index of the first auxiliary node and CHILDNAME represents the node name of the second auxiliary node.

After determining the first parent-child combination, the computer device searches the node index corresponding to the first parent-child node combination from the edge mapping table as the node index of the second auxiliary node. For example, if the node index corresponding to the first parent-child combination is "2" in the edge mapping table, the computer device determines that the node index of the second auxiliary node is "2".

333-E determining the node index of the second auxiliary node as the node index of the first metadata node if the second auxiliary node is the last metadata node included in the access directory of the first metadata; otherwise, the second auxiliary node is taken as a new first auxiliary node, and the step of determining the node name of the second auxiliary node based on the access catalog of the first metadata is performed again.

In some embodiments, after determining the second auxiliary node, the computer device determines whether the second auxiliary node is the last metadata node of the plurality of metadata nodes based on the access directory of the first metadata. Optionally, the last metadata node refers to the access directory of the first metadata, and the node names are arranged in the last metadata node. I.e. the last metadata node is the first metadata.

The computer device determines a node name of the first metadata according to the access directory of the first metadata, and determines whether the second auxiliary node is a last metadata node of the plurality of metadata nodes according to the node name of the first metadata and the node name of the second auxiliary node. If the second auxiliary node is the last metadata node of the plurality of metadata nodes (i.e., the node name of the second auxiliary node is the same as the node name of the first metadata node), the computer device determines the node index of the second auxiliary node as the node index of the first metadata node.

If the second auxiliary node is not the last metadata node of the plurality of metadata nodes (i.e., the node name of the second auxiliary node is different from the node name of the first metadata node), the computer device takes the second auxiliary node as a new first auxiliary node and determines the node name of the new second auxiliary node based on the access directory of the first metadata. The computer device then obtains a new first parent-child node group based on the node index of the new first auxiliary node and the node name combination of the new second auxiliary node, so as to determine the node index of the new second auxiliary node from the edge map, and so on, until the second auxiliary node is the last metadata node of the plurality of metadata nodes.

In the embodiment of the application, the corresponding relation between the parent-child node combination and the node index of the child node is reflected through the edge mapping table. Because the manifestation of the parent-child node combinations is relatively simple, the computer device indexes the child nodes from the edge map based on the parent-child node combinations faster. The method is beneficial to shortening the time consumption of constructing a parent-child node combination by the computer equipment through the access directory of the metadata and the edge mapping table and carrying out iterative search on the node indexes of all child nodes, thereby accelerating the speed of the node indexes of the metadata nodes determined in the process. By the method, the data read-write performance of the main management node is improved, and the response speed of the distributed file system to the file management request is guaranteed.

The process of looking up the first metadata from the node map is described in several embodiments below.

In some embodiments, the node map table includes node indices of at least one metadata node and metadata corresponding to the node indices of the metadata node.

Optionally, the node mapping table belongs to a key value table. The node mapping table comprises a plurality of node key value pairs, and each node key value pair corresponds to different metadata nodes in the tree structure. Illustratively, for any one of a plurality of node key pairs, the key in the node key pair is the node index of the metadata node, and the key is the metadata stored in the metadata node.

Illustratively, for any one of a plurality of node key pairs, the key in the node key pair is the node index of the metadata node, the key is the storage address of the metadata in the hard disk, or the key is the storage address of the metadata in the memory.

Step 336, determining the first metadata from the node mapping table based on the node index of the first metadata node may include the following sub-steps, where the execution subject of the sub-steps is a computer device, that is, the master management device in the above embodiment.

In a substep 336-a, metadata corresponding to the node index of the first metadata node is looked up from the node mapping table.

Optionally, the computer device uses the node index of the first metadata node as a key value, and includes a key value pair of the node index of the first metadata node in the node mapping table, so as to read a key value from the key value pair, and obtain the first metadata.

In substep 336-b, the metadata corresponding to the node index of the first metadata node is determined to be the first metadata.

In the embodiment of the application, the node mapping table is used for representing the mapping relation between the node index of the metadata node and the metadata stored by the metadata node. The node index of the metadata node is represented in a simpler form, for example, the node index of the metadata node can be represented by a number, and the time for searching the node index of the metadata node from the node mapping table is shorter. By setting the node mapping table in this way, the speed of determining metadata from the node mapping table based on the node index of the metadata node is facilitated.

The process of determining the node index of a metadata node from an edge map is described below by way of one example. In this example, the directory mapping table includes an edge mapping table and a node mapping table, the edge mapping table includes a mapping relationship between a parent-child node combination and node indexes of child nodes, the node mapping table includes a mapping relationship between at least one node index of a metadata node and metadata corresponding to the node index of the metadata node, and an access directory of the first metadata is known to be/dir 1/file1.

TABLE 1 partial content in edge mapping tables

In table 1, key (parentInodeId, CHILDNAME) represents that the key value in the edge map is a parent-child node combination (node index of parent node, node name of child node), and value (childInodeId) represents that the key value in the edge map is node index of child node.

Table 2 partial contents of node map table

In table 2, key (inoeded) indicates a node index in which a key value in the node map is a metadata node, value (Metadata) indicates that a key value in the node map includes metadata stored in the metadata node.

Based on this background, determining the node index of the first metadata node mainly includes the following steps (A10-A80).

Step A10, taking a root node in a plurality of metadata nodes as a first auxiliary node, and determining the node name of a second auxiliary node by the computer equipment based on the access catalog of the first metadata.

In this step, the first auxiliary node is the root node, and the node index of the first auxiliary node is known as 0.

Based on the access directory/dir 1/file1 of the first metadata, the computer device determines that the node name of the second auxiliary node is dir1.

Step A20, the computer equipment searches the node index corresponding to the first parent-child node combination from the edge mapping table based on the first parent-child node combination formed by the node index of the first auxiliary node and the node name of the second auxiliary node, and uses the node index as the node index of the second auxiliary node.

In this step, the first parent-child node combination is (0, "dir 1"), and the computer device performs a lookup in the edge mapping table based on the first parent-child node combination (0, "dir 1") to obtain table 3.

Table 3 correspondence table of first parent-child node combinations and node indexes of child nodes

As shown in table 3, the computer device determines that the node index of the child node corresponding to the first parent-child node combination (0, "dir 1") is 1. That is, the node index of the second auxiliary node is 1. As can be appreciated from comparing table 1, table 3 is part of table 1.

In step a30, the computer device determines whether the second auxiliary node "dir1" is the last metadata node included in the access directory of the first metadata, and determines that the second auxiliary node "dir1" is not the last first metadata node.

In step a40, the computer device uses the second auxiliary node "dir1" as a new first auxiliary node, and determines that the node name of the new second auxiliary node is "file1" based on the access directory/dir 1/file1 of the first metadata.

In step a50, the computer device searches the node index corresponding to the new first parent-child node combination from the edge mapping table based on the new first parent-child node combination formed by the node index of the new first auxiliary node "dir1" and the node name of the new second auxiliary node "file1", and uses the node index as the node index of the new second auxiliary node "file 1".

In this step, the new first parent-child node combination representation (1, "file 1"), based on which the computer device looks up in the edge map table (table 1), results in table 4 below.

Table 4 new correspondence table of first parent-child node combinations and child node indexes

As shown in table 4, the computer device determines that the node index of the child node corresponding to the new first parent-child node combination (0, "dir 1") is 1, that is, the node index of the new second auxiliary node is 1. As can be appreciated from comparing table 1, table 4 is part of table 1.

In step a60, the computer device determines whether the new second auxiliary node "file1" is the last metadata node included in the access directory of the first metadata, and determines that the second auxiliary node "file1" is the last first metadata node among the plurality of metadata nodes.

In step a70, the computer device determines that the node index of the first metadata node is 3.

In step a80, the computer device searches the node mapping table for metadata corresponding to the node index 3 of the first metadata node.

In this step, the computer device searches in the node mapping table based on the node index 2 of the first metadata node, to obtain the following table 5; the computer device determines metadata corresponding to node index 2 of the first metadata node according to table 5.

Table 5 correspondence table of node index of first metadata node and metadata

As shown in table 5, the computer device determines that the metadata corresponding to the node index "3" of the first metadata node is "inode { id:3, parent_id:1, name:" file1", is_direction: false, & gt, i.e. the first metadata is found. As can be appreciated from comparing table 2, table 5 is part of table 2. The computer device determines metadata corresponding to the node index of the first metadata node as first metadata.

Compared with the prior art that a tree structure (directory tree) for managing metadata is stored in a memory, the upper limit of the quantity of the metadata is limited by the storage capacity of the memory, the embodiment of the application simulates the tree structure to manage the metadata through the edge mapping table and the node mapping table, and the edge mapping table and the node mapping table can be stored in a hard disk. In this way the difficulty of raising the upper limit of metadata is reduced, since the storage capacity in the hard disk is relatively loose. In addition, in the process of determining metadata according to the access catalog of the metadata, the edge mapping table and the node mapping table are matched with each other, so that the read-write performance of the master management node is not greatly influenced, and timely response of the distributed file system to the file management request is guaranteed.

The following describes a method for further improving the read-write performance of a master management node by several exemplary embodiments.

In some embodiments, substep 333-c further comprises the following steps before searching the node index of the second auxiliary node from the edge map based on the node index of the first auxiliary node and the node name of the second auxiliary node.

In step 333-b1, if the sub-directory cache table cached in the memory includes a first sub-directory set corresponding to the node index of the first auxiliary node, the computer device obtains the first sub-directory set, where the first sub-directory set includes at least one piece of sub-directory information, each piece of sub-directory information is used to represent a corresponding sub-node of the first auxiliary node in the tree structure, and the sub-directory information includes a node name of the sub-node and a node index of the sub-node.

In some embodiments, a sub-directory cache table is set in the memory, where the sub-directory cache table includes a sub-directory set corresponding to at least one history auxiliary node, where the history auxiliary node is a metadata node that is used as the first auxiliary node in a process of determining a node index of the first metadata node. Optionally, the history auxiliary node further includes a metadata node that was used as the first auxiliary node in determining the node index of the history first metadata node.

Optionally, the child directory includes child nodes corresponding to the first auxiliary node in the tree structure, and other child nodes taking the child nodes corresponding to the first auxiliary node in the tree structure as parent nodes. That is, for the sub-directory cache table, a corresponding one of the first auxiliary nodes in the tree structure refers to any one of the metadata nodes that can be accessed in the tree structure after the first auxiliary node is accessed.

Illustratively, the computer device looks up the node index of the first auxiliary node in the sub-directory cache table, if the first sub-directory set corresponding to the node index of the first auxiliary node is looked up in the sub-directory cache table, the computer device performs sub-step 333-b2 below, and if the first sub-directory set corresponding to the node index of the first auxiliary node is not looked up in the sub-directory cache table, the computer device performs sub-step 333-c above.

In the substep 333-b2, if the at least one piece of sub-directory information includes sub-directory information of the second auxiliary node, the computer device determines a node index of the second auxiliary node from the sub-directory information of the second auxiliary node, and skips the step of searching for the node index of the second auxiliary node from the edge mapping table based on the node index of the first auxiliary node and the node name of the second auxiliary node.

Optionally, after the first sub-directory set is acquired, the computer device searches for the sub-directory information of the second auxiliary node from the first sub-directory set based on the node name of the second auxiliary node, and if the first sub-directory base includes the sub-directory information of the second auxiliary node, the computer device directly acquires the node index of the second auxiliary node from the sub-directory information of the second auxiliary node. If the first sub-directory base does not include sub-directory information for the second auxiliary node, the computer device performs step 333-c, and searches the edge mapping table for the node index of the second auxiliary node.

Table 6 subdirectory cache table

Wherein, as shown in table 6, the node index is used to characterize the node index of at least one history auxiliary node included in the sub-directory cache table. For example, the set of subdirectories corresponding to the history auxiliary node index 1 is { < dir1,20>, < dir2,21>, < dir3,23>, < file,24> }, where < dir1,20> represents one subdirectory information, specifically, < dir1,20> represents a child node for which the node name is dir1, and the node index of the child node is 20.

In some embodiments, if the first sub-directory set does not include the sub-directory information of the second auxiliary node, the computer device performs sub-step 333-c, after searching the node index of the second auxiliary node from the edge map, generating the sub-directory information of the second auxiliary node based on the node name of the second auxiliary node and the node name of the second auxiliary node, and adding the sub-directory information of the second auxiliary node to the first sub-directory set of the first auxiliary node.

Optionally, the sub-directory set has an upper capacity limit, if the first sub-directory set of the first auxiliary node reaches the upper capacity limit before the sub-directory information of the second auxiliary node is added to the first sub-directory set of the first auxiliary node, determining at least one piece of sub-directory information to be deleted from the first sub-directory set of the first auxiliary node, deleting the at least one piece of sub-directory information to be deleted, and adding the sub-directory information of the second auxiliary node to the first sub-directory set of the first auxiliary node.

Illustratively, the subdirectory information to be deleted includes at least one of the following: the lowest frequency subdirectory information is accessed and the earliest time subdirectory information is added to the first subdirectory set.

The speed of accessing the sub-directory cache table stored in the memory is faster than that of accessing the edge mapping table in the hard disk, so that the read-write speed of the main management node is improved, and the response speed of the main management node to the file management request is improved.

The following describes a method for improving the read-write speed of the master management node in the embodiment of the present application through several embodiments.

In some embodiments, substep 336 further comprises the following steps, prior to determining the first metadata from the node mapping table based on the node index of the first metadata node.

In step 334, if the node cache table cached in the memory includes the node index of the first metadata node, metadata corresponding to the node index of the first metadata node is obtained from the node cache table, the metadata corresponding to the node index of the first metadata node is determined to be the first metadata, and the node cache table is used for storing at least one metadata historically searched from the node mapping table.

In some embodiments, the node cache table is in a form consistent with the node map table, the node cache table being stored in memory, the capacity of the node cache table being less than the capacity of the node map table. Optionally, the node cache table includes metadata corresponding to a node index of at least one first metadata node that is accessed most frequently in the first period of time and a node index of the at least one first metadata node. The node Cache table is also referred to as a node Cache (Inode Cache).

Table 7 node cache table

As shown in table 7, the node cache table includes metadata corresponding to the node index 1 of the metadata node.

In the substep 335, if the node index of the first metadata node is not included in the node cache table, the step of determining the first metadata from the node mapping table based on the node index of the first metadata node is performed.

Illustratively, after determining the node index of the first metadata node, the computer device looks up the node index of the first metadata node in the node cache table; if the node cache table comprises the node index of the first metadata node, reading metadata corresponding to the node index of the first metadata node from the node cache table; if the node cache table does not include the node index of the first metadata node, the computer device executes substep 336 to obtain the first metadata from the node mapping table of the hard disk.

In some embodiments, if the node index of the first metadata node is not included in the node cache table, the computer device performs a substep 336 of retrieving the first metadata from the node map table in the hard disk. The computer device then adds the node index of the first metadata node and the first metadata to a node cache table.

Optionally, if the node cache table reaches the capacity upper limit, the computer device determines a node index to be deleted from the node cache table, deletes the node index and metadata corresponding to the node index from the node cache table, and adds the node index and the first metadata of the first metadata node to the node cache table.

Illustratively, the deleted node index includes at least one of: the node index with the lowest frequency is searched in the node cache table, and the earliest node index in the node cache table.

Because the speed of reading metadata from the hard disk to the memory is slower, before the metadata is searched in the node mapping table in the hard disk, the metadata is searched in the memory cache table cached in the memory, so that the time consumption of reading and writing the metadata is saved, the reading and writing speed of a main management node is improved, and the response speed of the main management node to a file management request is improved.

Another method for determining the first metadata will be described by way of several exemplary embodiments.

In some embodiments, step 330 further includes the following steps, the execution subject of which is a computer device, before determining the first metadata based on the directory map and the access directory of the first metadata.

Step 328, the first metadata is determined based on the access directory of the first metadata and the directory cache table stored in the memory.

In some embodiments, the directory cache table includes an edge cache table for storing at least one set of parent-child nodes historically looked up from an edge key table.

In some embodiments, the directory cache table refers to a partial directory map table stored in memory. Optionally, the directory cache table includes a partial sub-table of the directory map table that is most frequently accessed. The form of the directory mapping table is consistent with that of the directory cache table, and details mapped by the directory mapping table are referred to the above embodiments, and will not be described herein.

Optionally, the directory cache table includes at least one of: edge cache table, node cache table, subdirectory cache table. The edge cache table uses a partial sub-table comprising an edge mapping table, and the node cache table comprises a partial sub-table of the edge mapping table. Illustratively, the directory cache table includes an edge cache table. Illustratively, the directory cache table includes a node cache table. Illustratively, the directory cache table includes a sub-directory cache table therein. Illustratively, the directory cache table includes an edge cache table and a node cache table. The details of the sub-directory cache table are referred to in the above embodiments, and will not be described herein.

Optionally, the storage in the directory cache table is maintained based on a least recently Used (LEAST RECENTLY Used, LRU) mechanism. That is, for any one of the cache tables included in the directory cache table (e.g., the edge cache table, the node cache table in the above embodiment), in the case where the cache table reaches the upper limit of capacity, the computer device deletes the content of the cache table having the lowest access frequency and the earliest writing time, so as to add new content to the cache table.

Considering that the amount of metadata included in the node map table is larger than the amount of data of the stored information in the directory cache table, and that the amount of metadata included in the distributed file system is larger, the hit efficiency of the node cache table may be lower. Under the condition of the shortage of the storage capacity of the memory, an edge cache table and a subdirectory cache table can be preferentially arranged in the memory so as to obtain larger read-write benefits.

In some embodiments, the directory cache table includes an edge cache table for storing at least one set of parent-child nodes historically looked up from an edge key table. In this case, step 328 determines the first metadata based on the access directory of the first metadata and the directory cache table stored in the memory, including the following sub-steps whose execution subject is a computer device.

In a substep 328-a, a node index of a first metadata node is determined from the edge cache table based on the access directory of the first metadata, the first metadata node being for storing the first metadata.

In some embodiments, in this step, the computer device uses a root node of the plurality of metadata nodes as the first auxiliary node, and determines a node name of the second auxiliary node based on the access directory of the first metadata. Then, the computer equipment searches the node index of the child node corresponding to the first parent-child node combination from the edge cache table based on the first parent-child node combination formed by the node index of the first auxiliary node and the node name of the second auxiliary node, and the node index is used as the node index of the second auxiliary node; if the second auxiliary node is the last metadata node included in the access catalog of the first metadata, the computer equipment determines the node index of the second auxiliary node as the node index of the first metadata node; otherwise, the computer device takes the second auxiliary node as a new first auxiliary node, and starts execution again from the step of determining the node name of the second auxiliary node based on the access directory of the first metadata.

The specific implementation of this step is similar to the sub-steps 333-a, 333-c and 333-e in the above embodiments, and specific reference is made to the above embodiments, and details thereof will not be repeated here.

Optionally, in the step, if the computer device cannot search the node index corresponding to the first parent-child node combination from the edge cache table, that is, the edge cache table does not include the first parent-child node combination, the computer device searches the node index corresponding to the first parent-child node combination from the edge mapping table stored in the hard disk. For example, if the computer device cannot find the node index corresponding to the first parent-child node combination from the edge cache table, after finding the node index corresponding to the first parent-child node combination from the edge mapping table stored in the hard disk, adding the node index corresponding to the first parent-child node combination to the edge memory table.

Optionally, in the case that the directory cache table includes a subdirectory cache table, if the computer device cannot find a node index corresponding to the first parent-child node combination from the edge cache table, the computer device finds a subdirectory set corresponding to the node index of the first auxiliary node from the subdirectory cache, and tries to determine the node index of the second auxiliary node from the subdirectory set.

Substep 328-b, determines the first metadata based on the node index of the first metadata node.

Optionally, in the case that the directory cache table includes the node cache table, the computer device determines whether the node cache table includes the node index of the first metadata, and if the node cache table includes the node index of the first metadata node, the computer device acquires metadata corresponding to the node index of the first metadata node from the node cache table, and determines the metadata corresponding to the node index of the first metadata node as the first metadata. If the node index of the first metadata node is not included in the node cache table, the computer device starts to execute the step of determining the first metadata from the node mapping table based on the node index of the first metadata node.

Optionally, in the case that the node cache table is not included in the directory cache table, the computer device determines the first metadata from the node mapping table based on the node index of the first metadata node. For the specific implementation of this step, please refer to the above examples, which are not limited herein.

If step 328 fails to determine the first metadata based on the access directory and the directory cache table of the first metadata, the computer device begins executing the step of determining the first metadata from the access directory based on the directory map table and the first metadata.

According to the method, before the first metadata is searched from the hard disk based on the access catalog of the first metadata, the first metadata is tried to be searched from the catalog cache table of the memory, and the method is beneficial to improving the reading and writing efficiency of the master management node file.

The process of searching the first metadata based on the access catalog of the first metadata is described below by way of one embodiment, and the present example mainly includes the following steps, whose execution subject is a computer device.

And step B10, after determining the access catalog of the first metadata, the computer equipment takes the root node in the metadata nodes as a first auxiliary node, determines the node name of a second auxiliary node based on the access catalog of the first metadata, and searches the node index of the second auxiliary node from the edge cache table based on a first parent-child node combination consisting of the node index of the first auxiliary node and the node name of the second auxiliary node.

In step B20, if the node index of the second auxiliary node is included in the edge cache table, the computer device directly obtains the node index of the second auxiliary node from the memory, and directly executes step B60.

And step B30, if the edge cache table does not comprise the node index of the second auxiliary node, the computer equipment searches a first sub-directory set corresponding to the node index of the first auxiliary node from the sub-directory cache table, and searches the node index of the second auxiliary node from the first sub-directory set based on the node name of the second auxiliary node.

Optionally, step B30 may also be implemented as: the computer equipment searches a first subdirectory set corresponding to the node index of the first auxiliary node from the subdirectory cache table, determines the node name of the first metadata node based on the access directory of the first metadata, and searches the node index of the second auxiliary node from the first subdirectory set based on the node name of the first metadata node. If the first subdirectory set includes subdirectory information of the first metadata node, then the node directly obtaining the first metadata node is then executed starting from step B70, based on the subdirectory information of the first metadata node.

And step B40, if the sub-directory cache table comprises a first sub-directory set and the first sub-directory set comprises sub-directory information of the second auxiliary node, acquiring a node index of the second auxiliary node from the sub-directory information of the second auxiliary node, and directly executing the step B60.

And step B50, if the sub-directory cache table does not comprise the first sub-directory set or the first sub-directory set does not comprise the sub-directory information of the second auxiliary node, searching the node index of the second auxiliary node from the edge mapping table based on the first parent-child node combination.

Step B60, if the second auxiliary node is the last metadata node included in the access directory of the first metadata, determining the node index of the second auxiliary node as the node index of the first metadata node; otherwise, the second auxiliary node is taken as a new first auxiliary node, and the step of determining the node name of the second auxiliary node based on the access catalog of the first metadata is performed again.

Step B70, searching the node index of the first metadata node from the node cache table.

And step B80, if the node cache table comprises the node index of the first metadata node, acquiring metadata corresponding to the node index of the first metadata node from the node cache table, and determining the metadata corresponding to the node index of the first metadata node as the first metadata.

Step B90, if the node cache table does not include the node index of the first metadata node, searching metadata corresponding to the node index of the first metadata node from the node mapping table; and determining the metadata corresponding to the node index of the first metadata node as the first metadata.

For details of this embodiment, please refer to the above embodiment, and detailed description thereof is omitted.

After the distributed file system is used for a period of time, metadata is increased, and after the metadata size exceeds 100 hundred million, if the computer device directly reads the metadata from the directory mapping table stored in RocksDB of the hard disk when the user reads the data, the efficiency is low. The present embodiment uses a cache to keep part of the table contents (recently read data+frequently read data) in the directory map table that has been read or edited by the computer device as a directory cache table in memory. When a user inquires a certain metadata, the metadata is firstly searched from the directory cache table. If the metadata exists in the directory mapping table of the memory, the metadata is directly returned to the client, and RocksDB is not required to be queried in the hard disk, so that the reading performance of the main management node is improved obviously.

Fig. 6 is a schematic diagram of a data management method of a distributed file system according to still another exemplary embodiment of the present application. As shown in fig. 6, the present embodiment includes several steps whose execution subjects are computer devices, that is, the master management node in the above embodiment.

For a detailed description of steps 310-340, refer to the above embodiments, and are not described herein.

Step 350, if the file management request is used to change or delete the first file, adding the file management request to a system log, where the system log is used to record at least one file management request; the version number of the system log is updated.

In some embodiments, the system log includes at least one file management request received to characterize the distributed file system running process. For synchronized updates of metadata based on system logs at the distributed file system. The system log is also called an edit log.

Alternatively, in the case where a file management request is used to request a query file, the master management node will not add the file management request to the system log, or the master management node will not update the version number of the system log. Alternatively, in the case where the file management request is for changing or deleting the first file, the master management node adds the file management request to the system log, and updates the version number of the system log. For example, after adding a file management request to the system log, the master management node updates the version number of the system log.

In some embodiments, the version number of the system log is used to identify a version of the system log. If the version numbers of the two system logs are different, the log contents of the two system logs are different; if the two system logs have the same version numbers, the log content of the two system logs is the same.

By the method, meaningless updating of the system log and the version number of the system log can be reduced, and maintenance pressure of a master management node on the system log in the distributed file system can be relieved.

In some embodiments, the version number is represented by a numerical value. Optionally, the value of the version number after the update is greater than the value of the version number before the update. Illustratively, the value of the updated version number is increased by k, which is a positive number, compared to the version number before the update. Optionally, the value of the updated version number is smaller than the value of the version number before updating. Illustratively, the value of the updated version number is reduced by k, which is a positive number, compared to the version number before the update.

In some embodiments, the version number is associated with the time at which the file management request was added to the system log. Optionally, the version number of the system log is equal to the time when the file management request is added to the system log. Illustratively, if a certain file management request is added to the system log at the time point of 12:03:10, the updated version number is 120310. This approach is suitable for use at idle in a distributed file system, since the distributed file system may receive multiple file management requests in a short period of time.

Step 360, comparing the version number with the backup metadata management device to determine a first system log, wherein the version number of the first system log is the latest version number; the first system log is synchronized with the backup metadata management device.

The backup metadata management device is also referred to as a slave management node. For a detailed description of the slave management node, reference is made to the above embodiments. During the working process of the master management node, the slave management node is in a dormant state, i.e. the slave management node does not need to respond to the file management request.

In order to improve stability of the distributed file system, the problem that the whole distributed file system cannot be normally used due to errors of the master management node is avoided, a data synchronization process exists between the slave management node and the master management node, and metadata stored in the slave management node and metadata stored in the master management node are consistent through the data synchronization process. Optionally, the master management node and the slave management node have respective system logs, and both management nodes update the metadata stored in each management node through the system logs.

Optionally, the first system log refers to a system log with the latest version number among the multiple versions of system logs. For example, if the value of the version number after updating is greater than the value of the version number before updating, the first system log is the system log with the largest version number; if the value of the updated version number is smaller than the value of the version number before updating, the first system log is the system log with the minimum version number.

The first system log is exemplified by the system log with the version number closest to the current time among the system logs with the versions.

In this step, the master management node compares the version numbers of the system logs with the slave management nodes to determine a first system log from the system logs stored in the master management node and the system logs stored in the slave management nodes. If the slave management node has the first system log and the master management node does not have the first system log, the master management node acquires the first system log from the slave management node and deletes the system body logs with other version numbers in the master management node. If the master management node has the first system log and the slave management node does not have the first system log, the slave management node requests the master management node to acquire the first system log, and deletes the system log with other version numbers in the slave management node.

In step 370, if a system log having another version number than the first system log is stored, the system log having another version number is deleted.

In some embodiments, after step 370, the method further comprises: and the master management node updates the content related to the first metadata in the directory mapping table according to the first system log. This step may be accomplished, for example, by a playback thread in the master management node. The playback thread is used to update metadata stored in the master management node based on file management requests in the system log.

Optionally, in the case that the file management request is used to delete the first file, the master management node obtains the file management request according to the first system log, and deletes the content related to the first metadata in the directory map. Illustratively, the first metadata-related content includes at least one of: the node index of the first metadata node, the node name of the first metadata node, and the parent-child node combination comprising the first metadata node.

For example, in the case where the directory map includes an edge map, the master management node deletes the node index of the child node corresponding to the parent-child node combination including the parent-child node of the first metadata node in the edge map. For another example, in the case where the directory map includes a node map, the master management node deletes the node index including the first metadata node in the node map, and the first metadata.

Optionally, in the case that the file management request is used to edit the first file, the master management node obtains the file management request according to the first system log, and edits the content related to the first metadata in the directory map. For example, when the data block divided into the edited first file changes or the data management node in which the data block is stored changes, the master management node generates updated first metadata, and when the directory map table includes the node map table, the master management node replaces the first metadata corresponding to the node index of the first metadata node with the updated first metadata.

The process of updating metadata by the slave management node based on the system log is the same as that of the master management node, and specific reference is made to the above embodiment, and details are not repeated here.

Compared with the prior art that Fsimage (memory snapshot of the memory of the management node at a certain checking time point) is used to realize metadata synchronization in the master management node and the slave management node, in the embodiment of the application, fsimage is not needed to be loaded in the system memory, which is beneficial to improving the restarting time of the master management node.

Moreover, the data volume of the system log is smaller than Fsimage, which is helpful to reduce the transmission bandwidth occupied by the master management node and the slave management node in the data synchronization process.

Fig. 7 is a schematic diagram of a metadata update process in the related art.

Both Active and Standby Namenodes accept log files, except that Standby Namenodes merge the log files and the local old Fsimage files into a new Fsimage file (Fsimage file contains all the Namenodes, and all the metadata in Fsimage need to be loaded into memory every time Namenodes restart). Finally, standby naminode issues a new Fsimage to Active Namenod, thus having a new Fsimage on both sides.

Wherein the distributed file system persists metadata into Fsimage, which is loaded from Fsimage to memory each time the cluster is started. I.e., fsimage, can be understood as a memory snapshot taken of the system memory from a certain checkpoint. Fsimage the latest metadata checkpoint is saved, and at startup of the distributed file system, the master management node loads Fsimage information, which contains information of all directories and files of the entire distributed file system, i.e. all metadata. For a certain file, metadata for managing the file includes data block description information, modification time, access time and the like; the directory includes modification time, access right control information (the group to which the directory belongs), and the like.

Fig. 8 is a schematic diagram of a metadata update process provided by an exemplary embodiment of the present application.

As shown in fig. 8, the rebuilt naminode no longer needs Fsimage because the metadata would be written to the directory map table stored in RocksDB, and no more write to Fsimage is needed.

When an Active naminode (which can provide service to the outside) receives a write operation, it is necessary to write into a memory and update editlog (system log) in Journal (file log). The Active Namenode has a playback editlog thread inside, which can periodically and batchly play back RocksDB data, and update txid (the serial number of editlog) of the Active Namenode when the corresponding editlog is played back.

Wherein Journal is responsible for storing editlog in the namespace of the distributed file system. The system log receives the edit log from the Active Namenode and persists it on the local disk. Thus, even if the Active Namenode fails, the Standby Namenode can still obtain the latest edit log from JournalNode to ensure the consistency and durability of the data.

The Standby Namenode also has an internal thread (corresponding to a backup of Active Namenode) and plays back editlog (System Log) periodically. And updates txid (version number) that was newly played back.

When editlog is deleted, txid of the Active Namenode and the Standby Namenode are compared, and one of them is selected as the old txid. And based on it, editlog older than it can be deleted.

Fig. 9 is a graph comparing effects of experiments performed based on the scheme provided by the embodiment of the present application.

As shown in fig. 9, in the related art, 5 hundred million metadata occupies 350GB of memory, and in the scheme provided by the present application, 100 hundred million data only requires 60GB of memory overhead. In the scheme provided by the application, the restart time of the management node (such as the master management node Namenode) of the metadata is 65 minutes, and in the scheme provided by the application, the restart time of the Namenode of 100 hundred million data is only 4 minutes. At 8: in the case of a read-write ratio of 1, the use of the directory map has little effect on the read-write performance of the naminode.

The following are examples of the apparatus of the present application that may be used to perform the method embodiments of the present application. For details not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the method of the present application.

Fig. 10 is a block diagram illustrating a data management apparatus of a distributed file system according to an exemplary embodiment of the present application. The apparatus may be implemented as all or part of a data management device of a distributed file system by software, hardware, or a combination of both. The apparatus 1000 may include: a request acquisition module 1010, a catalog determination module 1020, a data determination module 1030, and a service execution module 1040.

A request acquisition module 1010 is configured to acquire a file management request for a first file in the distributed file system, where the file management request is used to request provision of a service related to the first file.

The directory determining module 1020 is configured to determine, based on the file management request, an access directory of first metadata, where the first metadata is used to manage the first file in the distributed file system, and the access directory of the first metadata is used to indicate a plurality of metadata nodes that need to be accessed sequentially to obtain the first metadata.

The data determining module 1030 is configured to determine the first metadata based on a directory mapping table and an access directory of the first metadata, where the directory mapping table is used to characterize a mapping relationship between metadata and the access directory of metadata, and the directory mapping table is stored in a hard disk.

A service execution module 1040 for executing a service with respect to the first file based on the first metadata.

In some embodiments, the metadata nodes corresponding to the metadata included in the distributed file system are connected into a tree structure, the directory mapping table includes an edge mapping table and a node mapping table, wherein the edge mapping table is used for representing a mapping relationship between two metadata nodes with parent-child relationships in the tree structure, and the node mapping table is used for representing a mapping relationship between the metadata nodes and metadata stored in the metadata nodes; the data determination module 1030 includes: an index determining unit, configured to determine, from the edge mapping table, a node index of a first metadata node, where the first metadata node is a metadata node for storing the first metadata, based on an access directory of the first metadata; and the metadata determining unit is used for determining the first metadata from the node mapping table based on the node index of the first metadata node.

In some embodiments, the access directory of the first metadata includes node names of each of the plurality of metadata nodes, two adjacent metadata nodes in the plurality of metadata nodes are parent-child nodes in the tree structure, the edge mapping table includes a mapping relationship between a parent-child node combination and node indexes of child nodes, and the parent-child node combination includes node indexes of parent nodes and node names of child nodes corresponding to the parent nodes; the index determination unit includes: a name determining subunit, configured to determine, with a root node of the plurality of metadata nodes as a first auxiliary node, a node name of a second auxiliary node based on an access directory of the first metadata, where the root node is a top-level node in the tree structure, a node index of the root node is known, and the second auxiliary node is a child node of the first auxiliary node; a combination determining subunit, configured to search, from the edge mapping table, a node index corresponding to the first parent-child node combination based on a first parent-child node combination formed by the node index of the first auxiliary node and the node name of the second auxiliary node, as the node index of the second auxiliary node; an index determining subunit, configured to determine, when the second auxiliary node is a last metadata node included in the access directory of the first metadata, a node index of the second auxiliary node as a node index of the first metadata node; otherwise, the second auxiliary node is taken as a new first auxiliary node, and the step of determining the node name of the second auxiliary node based on the access catalog of the first metadata is started to be executed again.

In some embodiments, the apparatus 1000 further includes a first cache lookup module, configured to obtain, in a case that a first sub-directory set corresponding to a node index of the first auxiliary node is included in a sub-directory cache table cached in the memory, the first sub-directory set including at least one piece of sub-directory information, where each piece of sub-directory information is used to characterize a corresponding one of the sub-nodes of the first auxiliary node in the tree structure, and the piece of sub-directory information includes a node name of the sub-node and a node index of the sub-node; and under the condition that the at least one piece of sub-directory information comprises the sub-directory information of the second auxiliary node, determining the node index of the second auxiliary node from the sub-directory information of the second auxiliary node, and skipping over the step of searching the node index of the second auxiliary node from the edge mapping table based on the node index of the first auxiliary node and the node name of the second auxiliary node.

In some embodiments, the node mapping table includes at least one node index of the metadata node and metadata corresponding to the node index of the metadata node; the metadata determination unit includes: searching metadata corresponding to the node index of the first metadata node from the node mapping table; and determining metadata corresponding to the node index of the first metadata node as the first metadata.

In some embodiments, the apparatus 1000 further includes a second cache lookup module, configured to, when a node cache table cached in the memory includes a node index of the first metadata node, obtain metadata corresponding to the node index of the first metadata node from the node cache table, determine the metadata corresponding to the node index of the first metadata node as the first metadata, where the node cache table is configured to store at least one metadata historically looked up from the node mapping table; and under the condition that the node index of the first metadata node is not included in the node cache table, the step of determining the first metadata from the node mapping table is started to be executed from the node index based on the first metadata node.

In some embodiments, the apparatus 1000 further includes a third cache lookup module configured to determine the first metadata based on an access directory of the first metadata and a directory cache table stored in a memory; if the first metadata cannot be determined based on the access directory of the first metadata and the directory cache table, the step of determining the first metadata is started to be executed from the access directory of the first metadata based on the directory mapping table and the first metadata.

In some embodiments, the directory cache table includes an edge cache table for storing at least one set of parent-child nodes historically looked up from an edge key table; the third cache searching module is configured to determine a node index of a first metadata node from the edge cache table based on the access directory of the first metadata, where the first metadata node is used to store the first metadata; the first metadata is determined based on a node index of the first metadata node.

In some embodiments, the apparatus 1000 further comprises a metadata management module for modifying the first metadata stored in the directory map table based on the file management request in case the file management request is for requesting a modification of the first text; deleting a mapping relation related to the first metadata in the directory mapping table based on the file management in a case that the file management request is used for requesting deletion of the first file; and sending the first metadata to a client under the condition that the file management request is used for requesting to access the first file, so that the client can acquire the file content of the first file from the file storage device in the distributed file system according to the first metadata.

In some embodiments, the apparatus 1000 further includes a log synchronization module configured to add the file management request to a system log, where the system log is configured to record at least one of the file management requests if the file management request is used to change or delete the first file; updating the version number of the system log; comparing the version numbers with the backup metadata management equipment to determine a first system log, wherein the version number of the first system log is the latest version number; synchronizing the first system log with the backup metadata management device; if a system log having another version number than the first system log is stored, deleting the system log having the other version number.

In some embodiments, the directory determining module 1020 is configured to read an access directory of the first metadata from the file management request; or determining the node names of the metadata nodes based on the file management request, and sequencing the node names of the metadata nodes to obtain the access directory of the first metadata. It should be noted that, in the apparatus provided in the foregoing embodiment, when implementing the functions thereof, only the division of the foregoing functional modules is used as an example, in practical application, the foregoing functional allocation may be implemented by different functional modules, that is, the content structure of the device is divided into different functional modules, so as to implement all or part of the functions described above.

In addition, the apparatus and the method embodiments provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the apparatus and the method embodiments are detailed in the method embodiments and are not repeated herein. The beneficial effects of the device provided in the foregoing embodiments are described with reference to the method side embodiments, and are not repeated herein.

Fig. 11 is a block diagram illustrating a computer device according to an exemplary embodiment of the present application. The computer device 1100 may be the primary management node in the distributed file system described above.

In general, the computer device 1100 includes: a processor 1101 and a memory 1102.

The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1101 may be implemented in at least one hardware form of digital signal Processing (DIGITAL SIGNAL Processing, DSP), field programmable gate array (Field Programmable GATE ARRAY, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 1101 may also include a main processor, which is a processor for processing data in an awake state, also referred to as a central processor (Central Processing Unit, CPU), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1101 may be integrated with an image processor (Graphics Processing Unit, GPU) for rendering and rendering of content to be displayed by the display screen. In some embodiments, the processor 1101 may also include an artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) processor for processing computing operations related to machine learning.

Memory 1102 may include one or more computer-readable storage media, which may be tangible and non-transitory. Memory 1102 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 1102 stores at least one program that is loaded and executed by processor 1101 to implement the data management method of the distributed file system provided by the method embodiments described above.

The embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, the computer program being loaded and executed by a processor to implement the data management method of the distributed file system provided in the above method embodiments.

The computer readable medium may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes random access Memory (Random Access Memory, RAM), read-Only Memory (ROM), erasable programmable Read-Only Memory (EPROM), electrically erasable programmable Read-Only Memory (ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory, EEPROM), flash Memory or other solid state Memory technology, DVD (Digital Video Disc, high density digital video disk) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the ones described above.

Embodiments of the present application also provide a computer program product, which includes a computer program stored in a computer readable storage medium, from which a processor reads and executes the computer program to implement the data management method of the distributed file system provided in the above method embodiments.

It should be understood that references herein to "a plurality" are to two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

It should be noted that, before collecting the relevant data of the user and in the process of recording the relevant data of the user (such as the file stored by the user and the editing information recorded in the editing process of the file by the user), the application can display a prompt interface, a popup window or output voice prompt information, wherein the prompt interface, the popup window or the voice prompt information is used for prompting the user to collect the relevant data currently, so that the application only starts to execute the relevant step of acquiring the relevant data of the user after acquiring the confirmation operation of the user on the prompt interface or the popup window, otherwise (i.e. when the confirmation operation of the user on the prompt interface or the popup window is not acquired), the relevant step of acquiring the relevant data of the user is ended, i.e. the relevant data of the user is not acquired. In other words, in the method, the management of the files stored in the distributed file system by the user is performed according to the requirements of the relevant national laws and regulations, the informed consent or the independent consent of the personal information body is obtained under the condition that the user agrees and authorizes, the subsequent data use and processing actions are carried out within the scope of the laws and regulations and the authorization of the personal information body, and the collection, use and processing of the relevant user data need to comply with the relevant laws and regulations and standards of the relevant country and region.

The foregoing description of the preferred embodiments of the application is not intended to limit the application to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and principles of the application are intended to be included within the scope of the application.

Claims

1. A method of data management for a distributed file system, the method comprising:

Acquiring a file management request for a first file in the distributed file system, wherein the file management request is used for requesting to provide a service related to the first file, the distributed file system comprises a plurality of metadata, and metadata nodes corresponding to the metadata are connected into a tree structure;

Determining an access directory of first metadata based on the file management request, wherein the first metadata is used for managing the first file in the distributed file system, the access directory of the first metadata is used for indicating a plurality of metadata nodes which need to be accessed in sequence for acquiring the first metadata, the access directory of the first metadata comprises node names of the metadata nodes, and two adjacent metadata nodes in the metadata nodes are parent-child nodes in the tree structure;

Taking a root node in the metadata nodes as a first auxiliary node, determining a node name of a second auxiliary node based on an access catalog of the first metadata, wherein the root node is a top-level node in the tree structure, the node index of the root node is known, and the second auxiliary node is a child node of the first auxiliary node;

Searching a node index corresponding to the first parent-child node combination from an edge mapping table included in a directory mapping table as the node index of the second auxiliary node based on a first parent-child node combination formed by the node index of the first auxiliary node and the node name of the second auxiliary node, wherein the directory mapping table is used for representing the mapping relation between metadata and an access directory of the metadata, the directory mapping table is stored in a hard disk, the edge mapping table includes a mapping relation between the parent-child node combination and the node index of the child node, and the parent-child node combination includes the node index of the parent node and the node name of the child node corresponding to the parent node;

If the second auxiliary node is the last metadata node included in the access catalog of the first metadata, determining the node index of the second auxiliary node as the node index of the first metadata node; otherwise, taking the second auxiliary node as a new first auxiliary node, and executing the step of determining the node name of the second auxiliary node again from the access catalog based on the first metadata, wherein the first metadata node is a metadata node for storing the first metadata;

Determining the first metadata from a node mapping table included in the directory mapping table based on a node index of the first metadata node, wherein the node mapping table is used for representing a mapping relation between the metadata node and metadata stored by the metadata node;

2. The method of claim 1, wherein the searching the edge mapping table for the node index of the second auxiliary node based on the node index of the first auxiliary node and the node name of the second auxiliary node further comprises:

If a first sub-directory set corresponding to the node index of the first auxiliary node is included in a sub-directory cache table cached in the memory, acquiring the first sub-directory set, wherein the first sub-directory set comprises at least one piece of sub-directory information, each piece of sub-directory information is used for representing a corresponding sub-node of the first auxiliary node in the tree structure, and the sub-directory information comprises the node name of the sub-node and the node index of the sub-node;

And if the at least one piece of sub-directory information comprises the sub-directory information of the second auxiliary node, determining a node index of the second auxiliary node from the sub-directory information of the second auxiliary node, and skipping over the step of searching the node index of the second auxiliary node from the edge mapping table based on the node index of the first auxiliary node and the node name of the second auxiliary node.

3. The method of claim 1, wherein the node map comprises at least one of a node index of the metadata node and metadata corresponding to the node index of the metadata node;

the determining the first metadata from the node mapping table based on the node index of the first metadata node includes:

searching metadata corresponding to the node index of the first metadata node from the node mapping table;

And determining metadata corresponding to the node index of the first metadata node as the first metadata.

4. The method of claim 1, wherein the determining the node name of the second auxiliary node based on the access directory of the first metadata with the root node of the plurality of metadata nodes as the first auxiliary node further comprises:

if the node cache table cached in the memory comprises the node index of the first metadata node, acquiring metadata corresponding to the node index of the first metadata node from the node cache table, determining the metadata corresponding to the node index of the first metadata node as the first metadata, wherein the node cache table is used for storing at least one metadata historically searched from the node mapping table;

And if the node index of the first metadata node is not included in the node cache table, starting to execute the step of determining the node name of the second auxiliary node by taking the root node in the metadata nodes as the first auxiliary node and based on the access catalog of the first metadata.

5. The method of claim 1, wherein the determining the node name of the second auxiliary node based on the access directory of the first metadata with the root node of the plurality of metadata nodes as the first auxiliary node further comprises:

determining the first metadata based on an access directory of the first metadata and a directory cache table stored in a memory;

and if the first metadata cannot be determined based on the access directory of the first metadata and the directory cache table, starting to execute the step of determining the node name of the second auxiliary node based on the access directory of the first metadata by taking the root node of the metadata nodes as the first auxiliary node.

6. The method of claim 5, wherein the directory cache table comprises an edge cache table for storing at least one set of parent-child nodes historically looked up from an edge key table;

The determining the first metadata based on the access directory of the first metadata and the directory cache table stored in the memory includes:

Determining a node index of a first metadata node from the edge cache table based on the access directory of the first metadata, wherein the first metadata node is used for storing the first metadata;

The first metadata is determined based on a node index of the first metadata node.

7. The method according to claim 1, wherein the method further comprises:

if the file management request is used for requesting to change the first file, modifying the first metadata stored in the directory mapping table based on the file management request;

If the file management request is used for requesting to delete the first file, deleting the mapping relation related to the first metadata in the directory mapping table based on the file management request;

And if the file management request is used for requesting to access the first file, the first metadata is sent to the client so that the client can acquire the file content of the first file from the file storage device in the distributed file system according to the first metadata.

8. The method of claim 7, wherein the method further comprises:

If the file management request is used for changing or deleting the first file, adding the file management request into a system log, wherein the system log is used for recording at least one file management request;

Updating the version number of the system log;

Comparing the version numbers with the backup metadata management equipment to determine a first system log, wherein the version number of the first system log is the latest version number;

Synchronizing the first system log with the backup metadata management device;

and if the local end equipment stores the system logs with other version numbers except the first system log, deleting the system logs with other version numbers.

9. The method of claim 1, wherein determining the access directory for the first metadata based on the file management request comprises:

reading an access directory of the first metadata from the file management request;

Or alternatively

And determining the node names of the metadata nodes based on the file management request, and sequencing the node names of the metadata nodes to obtain the access directory of the first metadata.

10. A data management apparatus for a distributed file system, the apparatus comprising:

A request acquisition module, configured to acquire a file management request for a first file in the distributed file system, where the file management request is used to request to provide a service related to the first file, and the distributed file system includes a plurality of metadata, where metadata nodes corresponding to the metadata are connected to form a tree structure;

A directory determining module, configured to determine, based on the file management request, an access directory of first metadata, where the first metadata is used to manage the first file in the distributed file system, the access directory of the first metadata is used to indicate a plurality of metadata nodes that need to be accessed in sequence to obtain the first metadata, the access directory of the first metadata includes node names of the plurality of metadata nodes, and two adjacent metadata nodes in the plurality of metadata nodes are parent-child nodes in the tree structure;

the data determining module is used for determining a node name of a second auxiliary node based on an access catalog of the first metadata by taking a root node in the metadata nodes as a first auxiliary node, wherein the root node is a top-level node in the tree structure, the node index of the root node is known, and the second auxiliary node is a child node of the first auxiliary node;

The data determining module is further configured to search, based on a first parent-child node combination formed by node indexes of the first auxiliary node and node names of the second auxiliary node, for a node index corresponding to the first parent-child node combination from an edge mapping table included in a directory mapping table as the node index of the second auxiliary node, where the directory mapping table is used to characterize a mapping relationship between metadata and an access directory of the metadata, the directory mapping table is stored in a hard disk, the edge mapping table includes a mapping relationship between the node indexes of the parent-child node combination and the child node, and the parent-child node combination includes a node index of a parent node and a node name of the child node corresponding to the parent node;

The data determining module is further configured to determine, when the second auxiliary node is a last metadata node included in the access directory of the first metadata, a node index of the second auxiliary node as a node index of the first metadata node; otherwise, taking the second auxiliary node as a new first auxiliary node, and executing the step of determining the node name of the second auxiliary node again from the access catalog based on the first metadata, wherein the first metadata node is a metadata node for storing the first metadata;

The data determining module is further configured to determine, from a node mapping table included in the directory mapping table, the first metadata based on a node index of the first metadata node, where the node mapping table is used to characterize a mapping relationship between the metadata node and metadata stored by the metadata node;

11. A computer device comprising a processor and a memory, the memory having stored therein a computer program that is loaded and executed by the processor to implement the method of data management of a distributed file system as claimed in any one of claims 1 to 9.

12. A computer readable storage medium, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the data management method of a distributed file system according to any one of claims 1 to 9.

13. A computer program product, characterized in that it comprises a computer program stored in a computer readable storage medium, from which a processor reads and executes the computer program for implementing a data management method of a distributed file system according to any of claims 1 to 9.