CN112965935A

CN112965935A - Data processing method and distributed file system

Info

Publication number: CN112965935A
Application number: CN202110194237.8A
Authority: CN
Inventors: 任岚晖
Original assignee: Beijing Star Net Ruijie Networks Co Ltd
Current assignee: Beijing Star Net Ruijie Networks Co Ltd
Priority date: 2021-02-20
Filing date: 2021-02-20
Publication date: 2021-06-15

Abstract

The application discloses a data processing method and a distributed file system, wherein the method is suitable for front-end equipment in the distributed file system and comprises the following steps: acquiring a first operation request aiming at a first target file, wherein the first operation request comprises: path information corresponding to the first target file; matching a target directory corresponding to the path information by using a path index table according to a longest matching principle; in the path index table, inquiring whether the fragmentation state of the target directory is fragmentation or not, and if so, inquiring the fragmentation type corresponding to the target directory; and if the fragmentation type is a first preset type, determining a first target metadata server for executing the task corresponding to the first operation request according to a first fragmentation algorithm corresponding to the first preset type. According to the technical scheme provided by the embodiments of the application, the access efficiency of the user for accessing the directory or the file can be improved.

Description

Data processing method and distributed file system

Technical Field

The present application belongs to the field of storage technologies, and in particular, to a data processing method and a distributed file system.

Background

In the prior art, two types of data exist in a file system according to different storage contents: file system metadata, i.e., including directory/file attributes, extended attributes, permissions, including portions for managing file data; file system data: i.e. the file data that is really valid for the user.

The file system metadata recursively constructs a file system directory tree through the hierarchical relationship between the parent directory and the child directories. In a conventional single-machine file system, file system metadata and file system data are commonly stored persistently in the same or multiple disks of the same physical node. While the storage space of a single physical node is usually not too large (tens of TBs). Even if the metadata of the file system and the data of the file system can be stored in the same physical node by means of the extended interface, the metadata and the data need to be stored in different physical nodes and the metadata needs to be fragmented for the reasons of data redundancy, data reliability, throughput performance and the like.

In the directory fragmentation scheme in the prior art, generally, consistent hash calculation is performed on absolute path names of files or directories, the files or directories are hashed to different storage nodes in a distributed system, when a user accesses the files or directories, the distributed system accesses the storage nodes one by one according to an access request of the user to search for corresponding directories, the searching process is complicated, and the efficiency of accessing the files or directories by the user is low.

Disclosure of Invention

The embodiment of the application provides an implementation scheme different from that in the prior art, so that the access efficiency of a user to metadata is improved.

Specifically, in an embodiment of the present application, a data processing method is provided, which is applied to a front-end device in a distributed file system, and includes: acquiring a first operation request aiming at a first target file, wherein the first operation request comprises: path information corresponding to the first target file; matching a target directory corresponding to the path information by using a path index table according to a longest matching principle; in the path index table, inquiring whether the fragmentation state of the target directory is fragmentable, if so, inquiring the fragmentation type corresponding to the target directory; and if the fragment type is a first preset type, determining a first target metadata server for executing the task corresponding to the first operation request according to a first fragment algorithm corresponding to the first preset type.

Optionally, the method further comprises:

and if the fragment type is a second preset type, determining a second target metadata server according to the target directory and the path index table, and sending the first operation request to the second target metadata server, so that the second target metadata server determines a third target metadata server for executing the task corresponding to the first operation request by using the first operation request.

Optionally, determining, according to a first fragmentation algorithm corresponding to the first preset type, a first target metadata server for executing a task corresponding to the first operation request includes:

determining a lower directory corresponding to the target directory from the path information;

calculating to obtain a first calculation result through the first fragmentation algorithm by using the lower directory;

determining an index path corresponding to the first target file with the target directory according to the first calculation result;

and searching the first target metadata server corresponding to the index path from the path index table.

Optionally, the path information includes a file name of the first target file, and when the type of the first operation request is a creation request, the second target metadata server determines, by using the first operation request, a third target metadata server that executes a task corresponding to the first operation request, where the third target metadata server includes:

and calculating to obtain a second calculation result through a second fragmentation algorithm corresponding to the second preset type according to the file name, and determining the third target metadata server according to the second calculation result.

Optionally, the method further comprises:

according to the first calculation result, adjusting the path information in the first operation request to obtain a second operation request;

and sending the second operation request to the first target metadata server so that the first target metadata server executes the task corresponding to the first operation request based on the second operation request.

Optionally, before the first operation request for the first target file is obtained, the method further includes:

receiving management configuration information, wherein the management configuration information comprises the fragment type aiming at the target directory;

setting the fragment type in the path index table based on the management configuration information.

Optionally, the method further comprises:

receiving a virtual directory setting instruction for the target directory;

setting a plurality of pieces of virtual directory information corresponding to the target directory according to the virtual directory setting instruction;

wherein the plurality of virtual directory information includes the index path.

Optionally, when the directory corresponding to the first target file is a highest-level subdirectory of the target directory and the third target metadata server has executed the task corresponding to the first operation request, the method further includes:

acquiring updating information aiming at the path index table sent by configuration equipment;

updating the path index table by using the updating information;

and the update information is obtained by the configuration device according to feedback information sent by the third target metadata server after executing the task corresponding to the first operation request.

Optionally, the first target metadata server includes a plurality of virtual metadata servers;

searching the first target metadata server corresponding to the index path from the path index table comprises:

determining a corresponding target virtual metadata server with the index path according to the path index table;

determining the first target metadata server based on the target virtual metadata server.

Optionally, the first preset type includes hash fragmentation or fragmentation according to time information; the second preset type comprises random shards.

Optionally, the type of the first operation request includes any one of: create requests, delete requests, modify requests, and query requests.

In another embodiment of the present application, a data processing method is provided, which is applied to a first target metadata server in a distributed file system, and includes: receiving a first operation request sent by front-end equipment, wherein the first operation request comprises: path information corresponding to the first target file; executing a task corresponding to the first operation request; after the first operation request is obtained by the front-end device, a path index table is used, a target directory corresponding to the path information is matched according to a longest matching principle, a fragmentation state of the target directory is queried in the path index table to be fragmentable, and when the fragmentation type corresponding to the target directory is queried to be a first preset type, a first target metadata server executing a task corresponding to the first operation request is determined according to a first fragmentation algorithm corresponding to the first preset type and then forwarded to the first target metadata server.

In another embodiment of the present application, a data processing apparatus is provided, which is suitable for a front-end device in a distributed file system, and includes: an obtaining module, configured to obtain a first operation request for a first target file, where the first operation request includes: path information corresponding to the first target file; the matching module is used for matching a target directory corresponding to the path information according to the longest matching principle by using a path index table; the query module is used for querying whether the fragmentation state of the target directory is fragmentable or not in the path index table, and if so, querying the fragmentation type corresponding to the target directory; and the determining module is used for determining a first target metadata server for executing the task corresponding to the first operation request according to a first fragmentation algorithm corresponding to the first preset type if the fragmentation type is the first preset type.

Optionally, the determining module is further configured to: and if the fragment type is a second preset type, determining a second target metadata server according to the target directory and the path index table, and sending the first operation request to the second target metadata server, so that the second target metadata server determines a third target metadata server for executing the task corresponding to the first operation request by using the first operation request.

Optionally, when the determining module is configured to determine, according to the first fragmentation algorithm corresponding to the first preset type, a first target metadata server for executing a task corresponding to the first operation request, the determining module is specifically configured to:

Optionally, the path information includes a file name of the first target file, and when the type of the first operation request is a creation request, the second target metadata server is specifically configured to, when determining, by using the first operation request, a third target metadata server that executes a task corresponding to the first operation request:

Optionally, the apparatus is further configured to:

Optionally, before the obtaining module obtains the first operation request for the first target file, the apparatus is further configured to:

Optionally, the apparatus is further configured to:

receiving a virtual directory setting instruction for the target directory;

wherein the plurality of virtual directory information includes the index path.

Optionally, when the directory corresponding to the first target file is a highest-level subdirectory of the target directory and the third target metadata server has executed the task corresponding to the first operation request, the apparatus is further configured to:

updating the path index table by using the updating information;

when the determining module is configured to search the first target metadata server corresponding to the index path from the path index table, the determining module is specifically configured to:

In another embodiment of the present application, there is provided a data processing apparatus adapted to a first target metadata server in a distributed file system, including: a receiving module, configured to receive a first operation request sent by a front-end device, where the first operation request includes: path information corresponding to the first target file; the execution module is used for executing the task corresponding to the first operation request; after the first operation request is obtained by the front-end device, a path index table is used, a target directory corresponding to the path information is matched according to a longest matching principle, a fragmentation state of the target directory is queried in the path index table to be fragmentable, and when the fragmentation type corresponding to the target directory is queried to be a first preset type, a first target metadata server executing a task corresponding to the first operation request is determined according to a first fragmentation algorithm corresponding to the first preset type and then forwarded to the first target metadata server.

In another embodiment of the present application, there is provided a distributed file system including: the system comprises front-end equipment, configuration equipment and at least one server; wherein: the front-end equipment is used for acquiring a first operation request aiming at a first target file, and the first operation request comprises: path information corresponding to the first target file; matching a target directory corresponding to the path information by using a path index table according to a longest matching principle; in the path index table, inquiring whether the fragmentation state of the target directory is fragmentable, if so, inquiring the fragmentation type corresponding to the target directory; if the fragmentation type is a first preset type, determining a first target metadata server for executing a task corresponding to the first operation request according to a first fragmentation algorithm corresponding to the first preset type; a first target metadata server in the at least one server, configured to execute a task corresponding to the first operation request; the configuration equipment is used for acquiring configuration instructions of related personnel, setting the path index table according to the configuration instructions, and sending the path index table to the front-end equipment when acquiring an index table acquisition request of the front-end equipment.

The application provides a new scheme different from the prior art, and the front-end equipment in the distributed file system can determine a server actually processing a task corresponding to a received first operation request according to a path index table and path information corresponding to a first target file by combining the determined fragment type of a target directory corresponding to the path information, so that the determination efficiency of determining the server for processing the task corresponding to the first operation request is improved, and the access efficiency of a user for accessing the file or the directory is further improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts. In the drawings:

fig. 1 is a schematic structural diagram of a distributed file system according to an embodiment of the present application;

fig. 2a is a schematic flowchart of a data processing method according to an embodiment of the present application;

FIG. 2b is a diagram illustrating a structure of a directory tree in a distributed file system according to an embodiment of the present application;

FIG. 2c is a diagram illustrating a structure of a directory tree in another distributed file system according to an embodiment of the present application;

fig. 3 is a schematic flowchart of a data processing method according to another embodiment of the present application;

fig. 4 is a schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a data processing apparatus according to another embodiment of the present application.

Detailed Description

Before the technical solutions provided by the embodiments of the present application are described, a brief description of specific terms in this document will be provided.

MDS: metadata Server, Metadata Server;

v-MDS: virtual-Metadata Server, virtual Metadata Server.

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise, and "the" plural "generally includes at least two, but does not preclude the inclusion of at least one. It should be understood that the descriptions of "first," second, "etc. herein are used for distinguishing between different elements, devices, etc., and are not intended to indicate a sequential order, nor is it intended to be construed as limiting the types of" first "and" second. The words "if", as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to a monitoring", depending on the context. Similarly, the phrase "if it is determined" or "if it is monitored (a stated condition or event)" may be interpreted as "when determining" or "in response to determining" or "when monitoring (a stated condition or event)" or "in response to monitoring (a stated condition or event)", depending on the context.

It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a good or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such good or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a commodity or system that includes the element.

The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Fig. 1 is a schematic structural diagram of a distributed file system according to an exemplary embodiment of the present application, where the system includes: front-end equipment 11, configuration equipment 12 and at least one server 13; wherein:

the front-end device 11 is configured to obtain a first operation request for a first target file, where the first operation request includes: path information corresponding to the first target file; matching a target directory corresponding to the path information by using a path index table according to a longest matching principle; in the path index table, inquiring whether the fragmentation state of the target directory is fragmentable, if so, inquiring the fragmentation type corresponding to the target directory; if the fragmentation type is a first preset type, determining a first target metadata server for executing a task corresponding to the first operation request according to a first fragmentation algorithm corresponding to the first preset type;

a first target metadata server in the at least one server 13, configured to execute a task corresponding to the first operation request;

the configuration device 12 is configured to obtain a configuration instruction of a relevant person, set the path index table according to the configuration instruction, and send the path index table to the front-end device when an index table obtaining request of the front-end device is obtained. Moreover, the configuration device 12 may also provide services for addition, deletion, modification and search of one or more index entries in the path index table by related personnel.

Further, if the fragment type is a second preset type, the front-end device 11 is further configured to determine a second target metadata server according to the target directory and the path index table, and send the first operation request to the second target metadata server, so that the second target metadata server determines, by using the first operation request, a third target metadata server that executes a task corresponding to the first operation request;

a second target metadata server in the at least one server 13 is configured to determine, when the fragmentation type is a second preset type, the third target metadata server by using the first operation request;

in some optional embodiments of the present application, the first operation request, which is acquired by the front-end device and is directed to the first target file, may be determined based on a received third operation request triggered by an operation of a user on the first target file, and when the type of the third operation request is a deletion request, a modification request, and an inquiry request, the first operation request may be the same as the third operation request; when the type of the third operation request creates a request, the first operation request may be determined according to the file name of the first target file generated according to the preset rule and the third operation request.

In other optional embodiments of the present application, when the first operation request is any one of a creation request, a deletion request, a modification request, and an inquiry request, the front-end device may directly obtain the first operation request from the user device, where when the type of the first operation request is the creation request, the user device may generate a file name of the first target file according to a preset rule in advance, and generate the first operation request based on the file name of the first target file.

In some optional embodiments of the present application, the front-end device 11 may be one or more devices connected to a user device (e.g., a computer, a mobile device) used by a user, and when the user performs operations such as adding a file, deleting a file, modifying a file, querying a file, and the like based on the user device, the user device sends a corresponding operation request to the front-end device 11, so that the front-end device 11 determines a server for executing a corresponding task according to the operation request received from the user device.

Further, the first target metadata server may be a server that is directly determined by a front-end device to execute a task corresponding to the first operation request in the distributed file system, and the third target metadata server may be a server that is further determined by a second target metadata server to execute the task corresponding to the first operation request after the second target metadata server is determined by the front-end device in the distributed file system.

The first target metadata server, the second target metadata server, and the third target may specifically be entity devices corresponding to the physical nodes, and may also be virtual servers operating on the entity devices, where one entity device (hardware device) may operate one or more virtual servers. The front-end device may be the same device as a metadata server that executes a task corresponding to the operation request sent by the front-end device, or may be a device that is only used to receive the operation request sent by the user device in the distributed file system; the configuration device may be the same device as a metadata server that executes a task corresponding to an operation request sent by the front-end device, or may be a device that is dedicated to acquiring a configuration instruction of a related person in a distributed file system, sets the path index table according to the configuration instruction, and sends the path index table to the front-end device when acquiring an index table acquisition request of the front-end device; the present application is not limited thereto.

For the execution principle and the interaction process of each component unit in the embodiment of the system, such as the front-end device, the first target metadata server, the second target metadata server, the third target metadata server, and the configuration device, reference may be made to the following description of each method embodiment.

Fig. 2a is a schematic flowchart of a data processing method according to an exemplary embodiment of the present application, where the method is applied to a front-end device in a distributed file system, and includes:

201. acquiring a first operation request aiming at a first target file, wherein the first operation request comprises: path information corresponding to the first target file;

202. matching a target directory corresponding to the path information by using a path index table according to a longest matching principle;

203. in the path index table, inquiring whether the fragmentation state of the target directory is fragmentable, if so, inquiring the fragmentation type corresponding to the target directory;

2031. and if the fragment type is a first preset type, determining a first target metadata server for executing the task corresponding to the first operation request according to a first fragment algorithm corresponding to the first preset type.

Specifically, the first target file may be a file or a directory, and the type of the first operation request may include any one of the following: requests conforming to the POSIX (Portable Operating System Interface) standard, such as a creation request, a deletion request, a modification request, and a query request.

The path information corresponding to the first target file may include at least one of the following: the file name corresponding to the first target file and the superior directory information of the first target file.

Specifically, when the path information corresponding to the first target file includes a file name corresponding to the first target file and superior directory information of the first target file, the path information corresponding to the first target file is its own directory; in addition, when the first target file has no upper directory, the path information corresponding to the first target file may include only the file name of the first target file.

It should be noted that, in a case that the first operation request is a creation request, because a target server that creates the first target file is not determined yet, creation of the first target file in the user equipment is not completed, and a file name of the first target file is not stored in the path index table, at this time, the file name of the first target file may be generated by the front end device according to a preset rule, and the front end device further determines the first operation request according to the file name of the first target file and a third operation request received from the user equipment, where the third operation request may be triggered by a corresponding operation of the user on the user equipment side. In addition, the first operation request may also be directly obtained by the front-end device directly from the user equipment, which is not limited in this application.

The target directory is obtained by matching the path information corresponding to the first target file in the path index table according to the longest matching principle.

The fragment type corresponding to the target directory may be obtained based on a path index table, where the target directory and the fragment type corresponding to the target directory are stored in the path index table.

In some optional embodiments of the present application, the first preset type may be a type in which a calculation process corresponding to the first fragmentation algorithm involved is relatively simple, or a type in which the input is the same and the output is the same when the first fragmentation algorithm is calculated for multiple times; such as hash-slicing or slicing according to time information.

Further, the method further includes step 2032: if the fragment type is a second preset type, determining a second target metadata server according to the target directory and the path index table, and sending the first operation request to the second target metadata server, so that the second target metadata server determines a third target metadata server for executing a task corresponding to the first operation request by using the first operation request; wherein the second preset type may include random fragmentation.

Optionally, for the fragment type of the target directory, when the service applicable to the target directory requires a large number of files or sub-directories under the target directory, the fragment type corresponding to the target directory may be set as a hash fragment, when the service applicable to the target directory requires a small number of files or sub-directories under the target directory, the fragment type corresponding to the target directory may be set as a random fragment, and when the files or sub-directories under the target directory need to be fragmented according to creation time, the fragment type corresponding to the target directory may be set as a fragment according to time information.

Correspondingly, when the first preset type is Hash fragmentation, the first fragmentation algorithm is Hash algorithm, and when the first preset type is fragmentation according to time information, the first fragmentation algorithm is preset algorithm related to time information.

Further, in step 2031, determining, according to the first fragmentation algorithm corresponding to the first preset type, a first target metadata server for executing a task corresponding to the first operation request may include the following steps:

20311. determining a lower directory corresponding to the target directory from the path information;

20312. calculating by using the subordinate directory through the first fragmentation algorithm to obtain a first calculation result, and determining an index path corresponding to the first target file with the target directory according to the first calculation result;

20313. and searching the first target metadata server corresponding to the index path from the path index table.

Specifically, the subordinate directory described in 20311 above may be a highest-level subdirectory under the target directory, and in some optional embodiments of the application, as shown in table 1, if the path information included in the first operation request is: a/home/user 0/work, wherein "work" is a file name corresponding to the first target file, at this time, according to the longest matching principle, a target directory corresponding to the matched path information in the path index table is "/home", at this time, it can be known by querying the path index table that the type of the segment corresponding to the/home is a hash segment, and then a lower directory of the/home is determined based on the path information "/home/user 0/work": "user 0", using/home's lower directory: "user 0", which is calculated by the hash algorithm to obtain a first calculation result "1"; and further determining an index path corresponding to the first target file according to the '1' and the '/home': and/home/1. Therefore, the metadata server with the IP address of 192.168.1.11 corresponding to the index path/home/1 is found through the path index table, that is, the first target metadata server of the task corresponding to the first operation request.

Table 1 example of path index when the fragmentation type corresponding to the target directory is hash type

Index item sequence number	Indexing paths	Fragmentation status	Type of fragment	vMDS	MDS-IP
						1	/	N	N/A	0	192.168.1.10
2	/home	Y	HASH		0							192.168.1.10
						3	/home/1	N	N/A	1	192.168.1.11
4	/home/2	N	N/A	2	192.168.1.12
						5	/home/3	N	N/A	2	192.168.1.12
...	...	...	...	...	...

In table 1, the index path is used for matching, by the front-end device, a target directory corresponding to path information corresponding to the first target file based on the index path, where the fragmentation state refers to a fragmentation state corresponding to the index path, and when the fragmentation state is "N", it indicates that the corresponding directory (i.e., the index path) is not fragmentable, and when the fragmentation state is "Y", it indicates that the corresponding directory is fragmentable; when the fragmentation type is 'N/A', the fragmentation type is represented as no fragmentation type, when the fragmentation type is 'RANDOM', the fragmentation type is represented as RANDOM fragmentation, in addition, when the fragmentation type is 'HASH', the fragmentation type is represented as Hash fragmentation, and when the fragmentation type is 'TIME', the fragmentation type is represented as fragmentation according to TIME information. Table 1 is only an example of the path index table, and the content included in the path index table may not be limited to table 1, which is not limited in this application. The vMDS in table 1 indicate the identifiers of the virtual metadata servers corresponding to the directories, and the MDS-IP indicates the addresses of the hardware devices (i.e., metadata servers) where the virtual metadata servers corresponding to the directories are located.

Further, in step 20312, determining, according to the first calculation result, an index path corresponding to the first target file according to the target directory may include:

and combining the first calculation result with a target directory according to a directory format to obtain an index path corresponding to the first target file.

For example: according to the above "1" and "/home", determining the index path corresponding to the first target file includes:

combining "1" with "/home" yields "/home/1" in a directory format (i.e., in a format that includes a "/" designation).

Further, after determining the first target metadata server, the front-end device sends the first operation request to the first target metadata server, so that the first target metadata server performs corresponding processing based on the first operation request.

Specifically, the corresponding processing by the first target metadata server based on the first operation request is related to the type to which the first operation request belongs. For example: the first operation request is a creation request, and when the first operation request indicates that a first target file is created, the first target metadata server creates the first target file; the first operation request is a deletion request, and when the deletion of the first target file is indicated, the first target metadata server deletes the first target file; the first operation request is a modification request, and when the modification of the first target file is indicated, the first target metadata server modifies the first target file according to the relevant instruction of the user; the first operation request is a query request, and when the first operation request indicates that a first target file is queried, the first target metadata server displays the first target file for a user to query.

After the front-end device sends the first operation request to a second target metadata server, when the type of the first operation request is a creation request, the second target metadata server determines, by using the first operation request, a third target metadata server that executes a task corresponding to the first operation request, including:

and calculating a second calculation result through a second fragmentation algorithm corresponding to the second preset type according to the file name of the first target file included in the path information, and determining the third target metadata server according to the second calculation result.

Specifically, the second target metadata server also stores a fragment type corresponding to the target directory, and specifically, the fragment type corresponding to the target directory can be obtained from metadata of the target directory, the fragment type can be a random type, the second fragment algorithm is a random algorithm, and the file name of the first target file can be used as the input of a random function corresponding to the random algorithm, so as to obtain a second calculation result; the corresponding relationship between the second calculation result and the third target metadata server may be pre-stored in the second target metadata server, and the second target metadata server may determine the third target metadata server according to the corresponding relationship.

In the application, the front-end equipment in the distributed file system can realize the functions of inquiring the target directory, the fragment state and the fragment type corresponding to the target directory through the stored path index table, and process some simple calculations to determine a first target metadata server; after the second target metadata server obtains the first operation request, the fragment type corresponding to the target directory can be verified according to the metadata stored by the second target metadata server, so that the effect of improving the accuracy of the fragments can be achieved.

In some optional embodiments of the present application, a path index table may also be stored in the second target metadata server, and the fragment type corresponding to the target directory is determined according to the path index table, which is not limited in this application.

Further, the determining, by the second target metadata server, the third target metadata server according to the second calculation result may specifically include: determining the identifier of the virtual metadata server corresponding to the target directory according to the second calculation result, and further determining a third target metadata server corresponding to the first target file according to the identifier of the virtual metadata server and a preset corresponding relation stored in the second target metadata server; wherein the correspondence includes a correspondence of the identifier of the virtual metadata server and a third target metadata server.

In other optional embodiments of the present application, if the first operation request is a creation request, and a file name of the first target file generated by the front-end device according to the preset rule is "user 0", the file name is not stored in the path index table.

Specifically, as shown in table 2, if the first operation request is a create request, and the path information included in the first operation request is: and/home/user 0 (where the path index table does not include "user 0"), the path index table is used, and a target directory corresponding to the path information is matched as "/home" according to a longest matching rule, at this time, the fragment type corresponding to the home is queried as a random fragment according to the path index table, at this time, a second target metadata server corresponding to the home, that is, a metadata server with an address of 192.168.1.10 is determined according to the home and the path index table, and the first operation request is sent to the second target metadata server.

TABLE 2 example of path index representation when the fragmentation type corresponding to the target directory is a random type

Index item sequence number

Indexing paths

Fragmentation status

Type of fragment

vMDS

MDS-IP

1

/

N

N/A

0

192.168.1.10

2

/home

Y

RANDOM

0

192.168.1.10

Correspondingly, as further shown in fig. 2b, directory information corresponding to a plurality of files may be stored in the second target metadata server vMDS-0, vMDS-1 may refer to a third target metadata server (virtual metadata server) determined according to a random algorithm, and vMDS-2 refers to other virtual metadata servers. Specifically, the second target metadata server may include therein metadata of a directory "/", and metadata of subdirectories corresponding to the "/", such as metadata corresponding to "etc", "home", and "tmp". Here, "etc", "home", and "tmp" are directory names corresponding to the subdirectories of "/".

Further, a new file or a new directory is created according to the scheme of the present application, and a directory tree as shown in fig. 2c can be created, and specific creating manners can be referred to the foregoing, and are not described herein again.

In other optional embodiments in this application, after the front-end device performs step 202, and determines the target directory, if the fragmentation state corresponding to the target directory is "N" according to the path index table, that is, the target directory fragmentation type is "N/a", that is, when the target directory is not fragmented, the method further includes:

and determining a fourth target metadata server for executing the task corresponding to the first operation request according to the path index table, and sending the first operation request to the fourth target metadata server to enable the fourth target metadata server to execute the task corresponding to the first operation request. And when the fourth target metadata server determines that the target directory is not fragmented, the fourth target metadata server directly determines a server for executing the task corresponding to the first operation request according to the path index table.

Referring to table 3, if the path information included in the first operation request is: and/home/user 0/doc, where "doc" is a file name corresponding to the first target file, at this time, according to the longest matching rule, in the path index table, the target directory corresponding to the matched path information is "/home/user 0", at this time, the path index table is queried, and it may be determined that the fourth target metadata server executing the first operation request is a server with an address of 192.168.1.11.

TABLE 3 path index table example when target directory is not fragmented

Index item sequence number	Indexing paths	Fragmentation status	Type of fragment	vMDS	MDS-IP
						1	/	N	N/A	0	192.168.1.10
2	/home	Y	RANDOM		0							192.168.1.10
						3	/home/user0	N	N/A	1	192.168.1.11
4	/home/user1	N	N/A	2	192.168.1.12
						5	/home/user2	N	N/A	2	192.168.1.12
6	/home/user3	N	N/A	1	192.168.1.11
						7	/home/user4	N	N/A	0	192.168.1.10
...	...	...	...	...	...

Further, if the fragment type is a first preset type, in order to enable the first target metadata server to determine that its task is a task corresponding to the first operation request after receiving the first operation request, the method further includes the following steps:

s1, according to the first calculation result, adjusting the path information in the first operation request to obtain a second operation request;

and S2, sending the second operation request to the first target metadata server, so that the first target metadata server executes the task corresponding to the first operation request based on the second operation request.

Specifically, the step S1 may specifically include: and inserting the first calculation result between the target directory and the directory name of a lower directory (highest subdirectory) corresponding to the target directory in the path information according to a directory format to obtain intermediate virtual directory information corresponding to the first target file, and generating a second operation request according to the intermediate virtual directory information, wherein the second operation request comprises the intermediate virtual directory information. Accordingly, the first target metadata server may determine whether the second operation request has a preset virtual identifier, for example: the aforementioned "1" is used to determine whether to directly execute the task corresponding to the first operation request based on the second operation request.

For example: referring to table 1, if the path information included in the first operation request is: a/home/user 0/work, wherein "work" is a file name corresponding to the first target file, at this time, according to the longest matching principle, a target directory corresponding to the matched path information in the path index table is "/home", at this time, it can be known by querying the path index table that the type of the segment corresponding to the/home is a hash segment, and then a lower directory of the/home is determined based on the path information "/home/user 0/work": "user 0", using/home's lower directory: "user 0", which is calculated by the hash algorithm to obtain a first calculation result "1"; "1" is inserted between "/home" and "user 0" in the "/home/user 0/work" in a directory format (i.e., in a format including a "/" identifier), resulting in "/home/1/user 0/work". Wherein, "/home/1/user 0/work" is the intermediate virtual directory information corresponding to the first target file with the corresponding file name work. And further generating a second operation request according to the intermediate virtual directory information, so that the first target metadata server executes the task corresponding to the first operation request based on the second operation request.

It should be noted that the first operation request in the present application may further include other information, for example: the device identifier of the user equipment, the user information, the attribute information of the first object file, the size information, and the like, which are not limited in this application.

In other alternative embodiments of the present application, the first target metadata server may further determine whether itself may execute a task corresponding to the first operation request according to the locally stored metadata, which is not limited in this application.

Further, before obtaining the first operation request for the first target file, the method further includes the following steps:

s11, receiving management configuration information, wherein the management configuration information comprises the fragment type aiming at the target directory;

s12, setting the segment type in the path index table based on the management configuration information.

Specifically, the management configuration information may be determined directly according to a configuration instruction sent by a relevant person, and may also be received from a configuration device. In some scenarios, when a related person deploys and plans the distributed file system, and deploys the first target file, or deploys the upper directory of the first target file, the related person actually specifies the main purpose and file mode (size, distribution, and characteristics) of the file, and may configure different fragment types for directories corresponding to different files based on the purpose and file mode of different files. Specifically, when a file is created, the fragmentation status and the fragmentation type of its corresponding directory may be directly configured. For example: when creating the target directory, the fragment state and the fragment type of the target directory are directly set.

Further, the method further comprises the steps of:

s20, receiving a virtual directory setting instruction aiming at the target directory;

s21, setting a plurality of pieces of virtual directory information corresponding to the target directory according to the virtual directory setting instruction;

wherein the plurality of virtual directory information includes the index path.

Specifically, the virtual directory is configured to segment the target directory to achieve balanced storage of resources. The virtual directory setting instruction can be sent to the front-end device by the terminal device based on an operation instruction of a relevant person, and the directory identifier of the virtual directory included in the target directory and the number of the virtual directories can be set by a user. For example, the directory identifier of the virtual directory may be 1, 2, 3, 4, and so on.

Further, in some optional embodiments of the present application, when the first preset type is fragmentation according to time information, the first fragmentation algorithm is an algorithm related to time information, for example: the method comprises the following steps of performing modular calculation or surplus calculation and the like, wherein the algorithm can be used for realizing that new files created in the same week are stored in the same virtual directory related to time information, or new files created in the same day are stored in the same virtual directory related to time information; in particular the identification of the virtual directory may be month information, or year information.

Further, after the third target metadata server has executed the task corresponding to the first operation request, the method further includes the following steps:

s31, acquiring the updating information aiming at the path index table sent by the configuration equipment;

s32, updating the path index table by using the updating information;

the update information is obtained by the configuration device according to feedback information sent by the first target metadata server or the third target metadata server after executing the task corresponding to the first operation request. The third target metadata server may be the second target metadata server itself, or may be another metadata server.

Specifically, after executing the task corresponding to the first operation request, the third target metadata server may further send feedback information to the configuration device, where the feedback information may be used to instruct the configuration device to update the path index table. For example: if the type of the first operation request is a create request, after the third target metadata server creates the first target file, the feedback information may include: and indicating the configuration equipment to newly create an index item to update the identifier of the path index table and the index item to be added corresponding to the first target file.

Further, if the type of the first operation request is a deletion request, after the third target metadata server deletes the first target file, the feedback information may include: and indicating the configuration equipment to delete the index item to update the prompt of the path index table and the index item to be deleted corresponding to the first target file.

In other optional embodiments of the present application, if the type of the first operation request is a deletion request, after the third target metadata server deletes the first target file, the feedback information may further be: and indicating the configuration equipment to delete the index item to update the prompt of the path index table and the index item sequence number corresponding to the first target file.

Correspondingly, if the server executing the task corresponding to the first operation request is the fourth target metadata server, it may also send corresponding feedback information to the configuration device, so that the configuration device updates the path index table, and the specific processing is similar to that of the third target metadata server, and is not described herein again.

Further, in some optional embodiments of the present application, the update information may be a path index table updated by the configuration device on a path index table stored in the configuration device according to the feedback information received from the third target metadata server; the update information may also include only index items that need to be updated, such as: an index entry needs to be added to the path index table or an index entry needs to be deleted from the original path index table.

Specifically, in the path index table, as in table 1, each row may be referred to as an index entry, and one index entry may specifically include: index item sequence number, index path, fragmentation state corresponding to the index path, fragmentation type corresponding to the index path, vMDS corresponding to the index path, and MDS-IP. If the first operation request is a creation request, and the third target metadata server successfully creates the first target file, generating a corresponding index item according to a preset rule, and sending feedback information including the index item to the configuration equipment; the configuration device adds the index entry in an original path index table according to the received feedback information, wherein in the added index entry, the index entry serial number is increased by 1 on the original maximum serial number, and the path information corresponding to the first target file is added into an index path column of the index entry, the fragmentation state corresponding to the path information corresponding to the first target file is set as a default value 'N', and the fragmentation type corresponding to the path information corresponding to the first target file is 'N/A'; and respectively filling the virtual metadata server identification corresponding to the path information corresponding to the first target file determined according to the method and the address of the corresponding metadata server into a vMDS column and an MDS-IP column of the path index table to form a new path index table.

By the scheme, after a new file or directory is created, the fragmentation state corresponding to the path information corresponding to the new file is set as the default value 'N', so that the usability of a user is met, and the locality of a directory tree is also maintained; so that the parent directory and the child directories in the directory tree are on one hardware device or virtual server as much as possible, and the overhead of the hardware device or virtual server in synchronizing the metadata is further reduced. In addition, the number of subdirectories of the designated directory stored by the front-end device can be monitored through a monitoring program in the front-end device, the first target metadata server, the second target metadata server, the third target metadata server or the fourth target metadata server, when the number of the subdirectories exceeds a preset value, the fragmentation state of the designated target is switched to be Y, and the corresponding fragmentation type is set, so that the load balance of each metadata server is realized.

Further, if the first operation request is a deletion request, after the third target metadata server successfully deletes the first target file, the configuration device may delete the index item directly according to the indication configuration device to update the prompt of the path index table, and delete the corresponding index item according to the index item sequence number corresponding to the first target file, so as to obtain a new path index table.

The path index table may be unchanged when the type of the request for the first operation is a query request or a modification request.

Further, the first target metadata server comprises one or more virtual metadata servers;

determining the first target metadata server based on the target virtual metadata server. Each virtual metadata server is responsible for providing metadata services of a plurality of directory subtrees, and provides a uniform file system name space for the outside in a cooperative manner.

Further, in the solution of the present application, the metadata stored in the same virtual metadata server a in the distributed file system may be repeatedly stored in multiple hardware devices, that is, the hardware device where the virtual metadata server a is located may be multiple, for example: the hardware device B and the hardware device C may both have a virtual metadata server a running therein, and when the hardware device B needs further expansion, or capacity reduction, or abnormal downtime, the distributed file system may update the correspondence between the virtual metadata server a and the hardware device B, which is stored in advance, to the correspondence between the virtual metadata server a and the hardware device C.

Further, when the first target metadata server or the second target metadata server queries the local metadata after acquiring the first operation request, and fails to query the target directory, the method further includes:

and feeding back a query failure prompt to the front-end equipment so that the front-end equipment acquires the latest path index table from the configuration equipment, and the front-end equipment determines the first target metadata server or the second target metadata server again according to the latest path index table.

Further, the foregoing S31 and S32 may be executed only when the directory corresponding to the first target file is the highest-level subdirectory of the target directory, which is not limited in the present application.

Further, in the distributed file system of the present application, file system data corresponding to the metadata may also be stored, where the file system data and the metadata may provide services separately, or may provide services on the same physical node (hardware device), and similarly, the file system data may also adopt a fragmentation policy, and a specific fragmentation mode is similar to a metadata fragmentation mode, and the fragmentation mode of the file system data may be stored in the metadata of the corresponding directory, which is not described herein again. In addition, the file system data may be fragmented in other manners, specifically, the file system data may be fragmented by adjusting an offset between an address of a server where the file system data is stored and an address of a server where the upper directory is located, and the specific fragmentation manner is not limited in this application.

In addition, in the scheme of the application, the metadata in the distributed file system can be distributed in a balanced manner in real time, the fragmentation strategy is configured in advance, and the fragmentation strategy gradually increases along with the gradual access of users, so that the metadata 0 migration can be realized while the balanced distribution is ensured, and the method is suitable for a super-large-scale distributed file system; and the complicated steps caused by the fact that the metadata are migrated again due to unbalanced metadata distribution in the later period are avoided, and the access efficiency of the user for accessing the metadata is further improved.

In addition, in the scheme of the application, the front-end device in the distributed file system may determine, according to the index table and the path information of the first target file, a server that actually processes a task corresponding to the received first operation request in combination with the fragment type of the target directory corresponding to the determined path information, so that the determination efficiency of determining the server that processes the task corresponding to the first operation request is improved, and the access efficiency of the user for accessing the file or the directory is further improved. In addition, the fragmentation scheme provided by the application is various, not only can the usability of the user be met, but also the locality of the directory tree can be kept, and the expenditure of hardware equipment or a virtual metadata server in metadata synchronization can be further reduced.

Fig. 3 is a schematic flowchart of another data processing method, which is applied to a first target metadata server in a distributed file system, according to an exemplary embodiment of the present application, and includes the following steps:

301. receiving a first operation request sent by front-end equipment, wherein the first operation request comprises: path information corresponding to the first target file;

302. executing a task corresponding to the first operation request;

after the first operation request is obtained by the front-end device, a path index table is used, a target directory corresponding to the path information is matched according to a longest matching principle, a fragmentation state of the target directory is queried in the path index table to be fragmentable, and when the fragmentation type corresponding to the target directory is queried to be a first preset type, a first target metadata server executing a task corresponding to the first operation request is determined according to a first fragmentation algorithm corresponding to the first preset type and then forwarded to the first target metadata server.

For the specific implementation of the present embodiment, reference is made to the foregoing description, and further description is omitted here.

An exemplary embodiment of the present application further provides a data processing method, which is applicable to a second target metadata server in a distributed file system, and may include at least the following steps:

s01, receiving a first operation request sent by a front-end device, where the first operation request includes: path information corresponding to the first target file;

and S02, determining a third target server for executing the task corresponding to the first operation request by using the first operation request.

Specifically, after receiving the first operation request, the second target metadata server is specifically configured to: determining a fragment type corresponding to the parent directory information corresponding to the first target file by using local metadata information and the path information, if the fragment type is a second preset type, calculating to obtain a second calculation result through a second fragment algorithm corresponding to the second preset type according to the file name corresponding to the first target file, and determining a third target metadata server according to the second calculation result;

further, the second target metadata server is further configured to send the first operation request to the third target metadata server, so that the third target metadata server executes a task corresponding to the first operation request;

after the first operation request is obtained by the front-end device, a path index table is used, a target directory corresponding to the path information is matched according to a longest matching principle, whether the fragmentation state of the target directory is fragmentable or not is inquired in the path index table, and if yes, the fragmentation state corresponding to the target directory is forwarded to a second target metadata server when the fragmentation type corresponding to the target directory is a second preset type.

Fig. 4 is a schematic flow chart of another data processing method according to an exemplary embodiment of the present application, which may include at least the following steps:

401. a client (located in a front-end device) receives a metadata operation request;

402. the client uses a path index table of local cache to carry out longest matching search according to the path name in the metadata operation request, and determines a metadata server capable of processing the metadata operation request;

403. the client sends the metadata operation request to a metadata server capable of processing the current metadata operation request;

404. the metadata server determines whether the metadata server can determine a target server executing a task corresponding to the metadata operation request according to the received metadata operation request, if so, the following step 405 is executed, and if not, the following steps 4041 to 4042 are executed;

4041. sending prompt information to the client to prompt the client to acquire the latest path index table from the self-configuration equipment again;

4042. after receiving the prompt message, the client acquires the latest path index table from the configuration equipment so as to update the local path index table;

405. the metadata server judges whether the target server is the metadata server, if so, the following step 406 is executed, and if not, the following step 4051 is executed;

4051. the metadata server sends the metadata operation request to a target server so that the target server processes the metadata operation request and responds to the client;

406. the metadata server directly processes the metadata operation request and responds to the client.

Specifically, the metadata operation request may be the aforementioned first operation request; the metadata server may be the first target metadata server, the second target metadata server, the third target metadata server, or the fourth target metadata server.

Further, when the metadata operation request is a request for creating a file or deleting a file, in the method, the target server also updates the path index table stored locally after creating the file or deleting the file, and sends the updated path index table to the configuration device.

For the specific implementation of this embodiment, reference is made to the foregoing description, and further description is omitted here.

Fig. 5 is a schematic structural diagram of another data processing apparatus according to an exemplary embodiment of the present application, where the apparatus is specifically applicable to a front-end device in a distributed file system, and specifically includes: an acquisition module 51, a matching module 52, a query module 53 and a determination module 54; wherein:

an obtaining module 51, configured to obtain a first operation request for a first target file, where the first operation request includes: path information corresponding to the first target file;

the matching module 52 is configured to match a target directory corresponding to the path information according to a longest matching rule by using a path index table;

the query module 53 is configured to query, in the path index table, whether the fragmentation state of the target directory is fragmentable, and if so, query a fragmentation type corresponding to the target directory;

the determining module 54 is configured to determine, if the fragmentation type is a first preset type, a first target metadata server that executes a task corresponding to the first operation request according to a first fragmentation algorithm corresponding to the first preset type.

Optionally, the determining module 54 is further configured to: and if the fragment type is a second preset type, determining a second target metadata server according to the target directory and the path index table, and sending the first operation request to the second target metadata server, so that the second target metadata server determines a third target metadata server for executing the task corresponding to the first operation request by using the first operation request.

Optionally, when the determining module 54 is configured to determine, according to the first fragmentation algorithm corresponding to the first preset type, a first target metadata server for executing a task corresponding to the first operation request, specifically:

Optionally, the apparatus is further configured to:

Optionally, before the obtaining module 51 obtains the first operation request for the first target file, the apparatus is further configured to:

Optionally, the apparatus is further configured to:

receiving a virtual directory setting instruction for the target directory;

wherein the plurality of virtual directory information includes the index path.

updating the path index table by using the updating information;

when the determining module 54 is configured to search the first target metadata server corresponding to the index path from the path index table, specifically configured to:

Fig. 6 is a schematic structural diagram of another data processing apparatus, which is specifically applicable to a first target metadata server in a distributed file system, according to an exemplary embodiment of the present application, and includes: a receiving module 61 and an executing module 62; wherein:

a receiving module 61, configured to receive a first operation request sent by a front-end device, where the first operation request includes: path information corresponding to the first target file;

the execution module 62 is configured to execute a task corresponding to the first operation request;

Accordingly, embodiments of the present application also provide a computer-readable storage medium storing a computer program, where the computer program can implement the steps or functions of the data processing method provided in the foregoing embodiments when executed by a computer.

A specific implementation process of the scheme in the embodiment of the present application when the types of the first operation request are a creation request, a deletion request, a modification request, and a query request, respectively, is described below with reference to specific scenarios:

if the file name of the first target file is 'work', the upper-level directory of the first target file is 'home/user 0';

for the first operation request to create the first target file, the first operation request may include upper-level directory information (e.g., home/user0) of the first target file, and a file name (e.g., work) corresponding to the first target file, which is generated by the front end device/or the user device based on the creation request according to a preset rule, where path information corresponding to the first target file is: "/home/user 0/work".

After the front-end equipment receives the first operation request, if the self-stored path index table is utilized, the matched target directory is 'home' according to the longest matching principle, the fragmentation state of the target directory 'home' is inquired to be fragmentable in the path index table, and when the fragmentation type corresponding to the 'home' is inquired to be hash fragmentation, the lower directory of the home is utilized: "user 0", which is calculated by the hash algorithm to obtain a first calculation result "1"; and further determining an index path corresponding to the first target file according to the '1' and the '/home': the method comprises the steps that a/home/1 is carried out, so that a first target metadata server corresponding to an index path (/ home/1) is found through a path index table; sending the first operation request to a first target metadata server, and enabling the first target metadata server to execute an operation of creating a first target file with a file name of 'work';

if the front-end equipment utilizes a path index table stored by the front-end equipment, according to the longest matching principle, a matched target directory is 'home/user 0', and when the fragment type corresponding to 'home/user 0' is queried to be a random fragment, a second target metadata server is determined according to the target directory '/home/user 0' and the path index table, a first operation request is sent to the second target metadata server, work is calculated by the second target metadata through a random algorithm to obtain a corresponding server identifier, a third target metadata server is determined according to the server identifier, the first operation request is sent to the third target metadata server, and the third target metadata server executes the operation of creating a first target file with the file name of 'work';

if the front-end equipment uses a path index table stored by the front-end equipment, according to the longest matching principle, the matched target directory is 'home/user 0', and the fragment type corresponding to 'home/user 0' is inquired to be non-fragmentable, a fourth target metadata server is determined according to the target directory '/home/user 0' and the path index table, a first operation request is sent to the fourth target metadata server, and the fourth target metadata server executes an operation of creating a first target file with the file name of 'work'.

Application scenario two: when the first operation request is to delete a first target file with a file name of "work" and a superior directory of "/home/user 0", path information corresponding to the first target file is as follows: "/home/user 0/work";

if the front-end equipment utilizes a path index table stored by the front-end equipment, the matched target directory is 'home' according to the longest matching principle, the fragmentation state of the target directory 'home' is inquired to be fragmentable in the path index table, and the fragmentation type corresponding to the 'home' is inquired to be Hash fragmentation, the lower-level directory of the home is utilized: "user 0", which is calculated by the hash algorithm to obtain a first calculation result "1"; and further determining an index path corresponding to the first target file according to the '1' and the '/home': the method comprises the steps that a/home/1 is carried out, so that a first target metadata server corresponding to an index path (/ home/1) is found through a path index table; sending the first operation request to a first target metadata server (that is, a first target file with a file name of work is stored in the first target metadata server), so that the first target metadata server executes an operation of deleting the first target file with the file name of work;

if the front-end device uses a path index table stored in the front-end device, according to the longest matching principle, the matched target directory is 'home/user 0', and the fragmentation state of the target directory 'home/user 0' is queried to be non-fragmentation in the path index table, a fourth target metadata server (that is, a first target file with the file name of work is stored in the fourth target metadata server) is determined according to the path index table, so that the fourth target metadata server deletes the first target file.

Application scenario three: when the first operation request is to modify a first target file with a file name of "work" and an upper-level directory of "/home/user 0", path information corresponding to the first target file is as follows: "/home/user 0/work";

if the front-end equipment utilizes a path index table stored by the front-end equipment, the matched target directory is 'home' according to the longest matching principle, the fragmentation state of the target directory 'home' is inquired to be fragmentable in the path index table, and the fragmentation type corresponding to the 'home' is inquired to be Hash fragmentation, the lower-level directory of the home is utilized: "user 0", which is calculated by the hash algorithm to obtain a first calculation result "1"; and further determining an index path corresponding to the first target file according to the '1' and the '/home': the method comprises the steps that a/home/1 is carried out, so that a first target metadata server corresponding to an index path (/ home/1) is found through a path index table; sending the first operation request to a first target metadata server (that is, a first target file with a file name of work is stored in the first target metadata server), so that the first target metadata server executes an operation of modifying the first target file with the file name of work;

if the front-end device uses a path index table stored in the front-end device, according to the longest matching principle, the matched target directory is 'home/user 0', and the fragmentation state of the target directory 'home/user 0' is queried to be non-fragmentable in the path index table, a fourth target metadata server (that is, a first target file with a file name of work is stored in the fourth target metadata server) is determined according to the path index table, so that the fourth target metadata server executes an operation of modifying the first target file.

Application scenario three: when the first operation request is a first target file with a file name of "work" and a superior directory of "/home/user 0/", the path information corresponding to the first target file is: "/home/user 0/work";

if the front-end device uses a path index table stored in the front-end device, according to the longest matching principle, the matched target directory is 'home/user 0', and the fragmentation state of the target directory 'home/user 0' is queried to be non-fragmentation in the path index table, a fourth target metadata server (that is, a first target file with a file name of work is stored in the fourth target metadata server) is determined according to the path index table, so that the fourth target metadata server displays the first target file according to a user instruction.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. A data processing method is characterized in that the method is suitable for front-end equipment in a distributed file system and comprises the following steps:

acquiring a first operation request aiming at a first target file, wherein the first operation request comprises: path information corresponding to the first target file;

matching a target directory corresponding to the path information by using a path index table according to a longest matching principle;

in the path index table, inquiring whether the fragmentation state of the target directory is fragmentable, if so, inquiring the fragmentation type corresponding to the target directory;

and if the fragment type is a first preset type, determining a first target metadata server for executing the task corresponding to the first operation request according to a first fragment algorithm corresponding to the first preset type.

2. The method of claim 1, further comprising:

3. The method of claim 1, wherein determining a first target metadata server for executing a task corresponding to the first operation request according to a first fragmentation algorithm corresponding to the first preset type comprises:

4. The method according to claim 2, wherein the path information includes a file name of the first target file, and when the type of the first operation request is a creation request, the second target metadata server determines, by using the first operation request, a third target metadata server that executes a task corresponding to the first operation request, including:

5. The method of claim 3, further comprising:

6. The method of claim 1, wherein prior to obtaining the first operation request for the first target file, the method further comprises:

7. The method of claim 3, further comprising:

receiving a virtual directory setting instruction for the target directory;

wherein the plurality of virtual directory information includes the index path.

8. The method of claim 2, wherein when the directory corresponding to the first target file is a highest-level subdirectory of the target directory and the third target metadata server has executed the task corresponding to the first operation request, the method further comprises:

updating the path index table by using the updating information;

9. The method of claim 3, wherein the first target metadata server comprises a plurality of virtual metadata servers;

10. The method of claim 4, wherein the first preset type comprises hash fragmentation or fragmentation by time information; the second preset type comprises random shards.

11. The method of claim 1, wherein the type of the first operation request comprises any one of: create requests, delete requests, modify requests, and query requests.

12. A data processing method, adapted to a first target metadata server in a distributed file system, comprising:

receiving a first operation request sent by front-end equipment, wherein the first operation request comprises: path information corresponding to the first target file;

executing a task corresponding to the first operation request;

13. A data processing apparatus, adapted to a front-end device in a distributed file system, comprising:

an obtaining module, configured to obtain a first operation request for a first target file, where the first operation request includes: path information corresponding to the first target file;

the matching module is used for matching a target directory corresponding to the path information according to the longest matching principle by using a path index table;

the query module is used for querying whether the fragmentation state of the target directory is fragmentable or not in the path index table, and if so, querying the fragmentation type corresponding to the target directory;

and the determining module is used for determining a first target metadata server for executing the task corresponding to the first operation request according to a first fragmentation algorithm corresponding to the first preset type if the fragmentation type is the first preset type.

14. A data processing apparatus adapted for use with a first target metadata server in a distributed file system, comprising:

a receiving module, configured to receive a first operation request sent by a front-end device, where the first operation request includes: path information corresponding to the first target file;

the execution module is used for executing the task corresponding to the first operation request;

15. A distributed file system, comprising: the system comprises front-end equipment, configuration equipment and at least one server; wherein:

the front-end equipment is used for acquiring a first operation request aiming at a first target file, and the first operation request comprises: path information corresponding to the first target file; matching a target directory corresponding to the path information by using a path index table according to a longest matching principle; in the path index table, inquiring whether the fragmentation state of the target directory is fragmentable, if so, inquiring the fragmentation type corresponding to the target directory; if the fragmentation type is a first preset type, determining a first target metadata server for executing a task corresponding to the first operation request according to a first fragmentation algorithm corresponding to the first preset type;

a first target metadata server in the at least one server, configured to execute a task corresponding to the first operation request;

the configuration equipment is used for acquiring configuration instructions of related personnel, setting the path index table according to the configuration instructions, and sending the path index table to the front-end equipment when acquiring an index table acquisition request of the front-end equipment.