CN109739815B

CN109739815B - File processing method, system, device, equipment and storage medium

Info

Publication number: CN109739815B
Application number: CN201910024516.2A
Authority: CN
Inventors: 申航; 陈二奇; 田燕; 包能辉
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2019-01-10
Filing date: 2019-01-10
Publication date: 2021-08-17
Anticipated expiration: 2039-01-10
Also published as: CN109739815A

Abstract

The disclosure relates to a file processing method, a system, a device, equipment and a storage medium, belonging to the technical field of internet, wherein the method is applied to a distributed storage system, and comprises the following steps: receiving a deletion request sent by a client, wherein the deletion request carries an original file identifier of a target file, and the deletion request is used for indicating a deletion command to be called for the target file; based on the deletion request, calling a specified command to replace the deletion command; and in the process of executing the specified command, modifying the original file identifier into a target file identifier, wherein the target file identifier comprises a target storage path and a target file name, the target file name comprises the original file identifier, and the target storage path is stored in a specified directory set by the distributed storage system. The method and the device adopt the designated command to replace the deleting command, modify the file identification of the target file, do not really delete the target file, provide a mode of deleting the target file in the distributed file system, expand the application range and improve the reliability.

Description

File processing method, system, device, equipment and storage medium

Technical Field

The present disclosure relates to the field of internet technologies, and in particular, to a method, a system, an apparatus, a device, and a storage medium for processing a file.

Background

Storing files is a basic function of a client, and the client usually stores a large amount of files for a user to use. However, with the gradual expansion of the information scale, various types of files are layered endlessly, and a user needs to store more files in the process of using a client, and at this time, the files stored before are likely to need to be deleted.

Since the file cannot be restored due to complete deletion of the file, subsequent use by a user may be affected. Therefore, the client can preset a recycle bin, and replace the delete command with a command to move to the recycle bin. When a user triggers a deletion request for a target file, the client does not call a deletion command but calls a command for moving to the recycle bin when responding based on the deletion request, so that the target file is moved to the recycle bin by executing the command for moving to the recycle bin. And when the user triggers a recovery request for the target file in the recycle bin subsequently, the client recovers the target file based on the recovery request.

The foregoing deletion method is only applicable to the client, but a distributed storage system is proposed at present, and the distributed storage system also has a requirement for deleting a file, so it is urgently needed to provide a method for deleting a file, which is applied to the distributed storage system.

Disclosure of Invention

The present disclosure provides a file processing method, system, apparatus, device, and storage medium, which can overcome the problems in the related art.

According to a first aspect of the embodiments of the present disclosure, there is provided a file processing method applied to a distributed storage system, the method including:

receiving a deletion request sent by a client, wherein the deletion request carries an original file identifier of a target file, and the deletion request is used for indicating that a deletion command is called for the target file;

based on the deletion request, calling a specified command to replace the deletion command;

and in the process of executing the specified command, modifying the original file identifier into a target file identifier, wherein the original file identifier comprises an original storage path and an original file name, the target file identifier comprises a target storage path and a target file name, the target file name comprises the original file identifier, and the target storage path is stored in a specified directory set by the distributed storage system.

In one possible implementation, the method further includes:

receiving a recovery request sent by the client, wherein the recovery request carries the target file identifier;

modifying the target file identifier into the original file identifier based on the recovery request;

in a possible implementation manner, in the process of executing the specified command, modifying an original file identifier of the target file into a target file identifier includes:

in the process of executing the specified command, acquiring the name of the target file according to the original file identifier;

acquiring the specified directory as a primary directory, and acquiring a secondary directory corresponding to the time period to which the current time belongs under the primary directory;

generating the target storage path according to the primary directory and the secondary directory;

and modifying the original file identifier into the target file identifier according to the target storage path and the target file name.

In a possible implementation manner, the obtaining of the secondary directory corresponding to the time period to which the current time belongs under the primary directory includes:

when the secondary directory corresponding to the time period to which the current moment belongs is determined not to be included in the primary directory, creating the secondary directory in the primary directory;

and when the secondary directory corresponding to the time period to which the current moment belongs is determined to be included in the primary directory, acquiring the secondary directory.

In one possible implementation, the generating the target storage path according to the primary directory and the secondary directory includes:

acquiring a third-level directory with the number of the file identifiers under the second-level directory not reaching a preset number;

and generating the target storage path according to the primary directory, the secondary directory and the tertiary directory.

In one possible implementation, the method further includes:

and deleting the file corresponding to the file identifier of which the survival time under the specified directory reaches the preset time.

In a possible implementation manner, the secondary directory of each file under the specified directory corresponds to the modification time of the file identifier of the file;

deleting the files corresponding to the file identifiers of which the survival time duration reaches the preset time duration in the appointed directory, wherein the deleting comprises the following steps:

and deleting the secondary directory with the survival time reaching the preset time length in the appointed directory and the file corresponding to the file identifier under the secondary directory.

In one possible implementation manner, the delete command is a delete command in a unlink function, and the specified command is a rename command.

According to a second aspect of the embodiments of the present disclosure, a distributed storage system is provided, where the distributed storage system includes a metadata server and a data server, the data server is used for storing a file, and the metadata server is used for storing a file identifier of the file;

the metadata server is further configured to receive a deletion request sent by a client, where the deletion request carries an original file identifier of a target file, and the deletion request is used to instruct to call a deletion command for the target file;

the metadata server is further used for calling a specified command to replace the deletion command based on the deletion request;

the metadata server is further configured to modify the original file identifier into a target file identifier in a process of executing the specified command, where the original file identifier includes an original storage path and an original file name, the target file identifier includes a target storage path and a target file name, the target file name includes the original file identifier, and the target storage path is stored in a specified directory set by the distributed storage system.

According to a third aspect of the embodiments of the present disclosure, there is provided a file processing apparatus applied to a metadata server, the apparatus including:

a deletion request receiving unit configured to receive a deletion request sent by the client, where the deletion request carries an original file identifier of a target file, and the deletion request is used to instruct to call a deletion command for the target file;

a calling unit configured to call a specified command to replace the deletion command based on the deletion request;

a first modification unit, configured to modify an original file identifier into a target file identifier in a process of executing the specified command, where the original file identifier includes an original storage path and an original file name, the target file identifier includes a target storage path and a target file name, the target file name includes the original file identifier, and the target storage path is stored in a specified directory set by the distributed storage system.

In one possible implementation, the apparatus further includes:

a recovery request receiving unit configured to receive a recovery request sent by the client, where the recovery request carries the target file identifier;

a second modification unit configured to modify the target file identifier to the original file identifier based on the recovery request.

In one possible implementation, the first modifying unit includes:

the name acquisition subunit is configured to acquire the name of the target file according to the original file identifier in the process of executing the specified command;

the directory acquisition subunit is configured to acquire the specified directory as a primary directory and acquire a secondary directory corresponding to a time period to which the current time belongs under the primary directory;

a path generation subunit configured to generate the target storage path according to the primary directory and the secondary directory;

and the modifying subunit is configured to modify the original file identifier into the target file identifier according to the target storage path and the target file name.

In one possible implementation, the directory retrieval subunit is further configured to:

In one possible implementation, the path generation subunit is further configured to:

In one possible implementation, the apparatus further includes:

and the deleting unit is configured to delete the file corresponding to the file identifier of which the survival time duration reaches the preset time duration under the specified directory.

In a possible implementation manner, the secondary directory of each file under the specified directory corresponds to the modification time of the file identifier of the file; the deletion unit includes:

and the deleting subunit is configured to delete the secondary directory in the specified directory, the survival time of which reaches the preset time, and the file corresponding to the file identifier in the secondary directory.

According to a fourth aspect of embodiments of the present disclosure, there is provided a metadata server including:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

based on the deletion request, calling a specified command to replace the specified command of the deletion command;

According to a fifth aspect of embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium having instructions therein, which when executed by a processor of a metadata server, enable the metadata server to perform a file processing method, the method comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

According to a sixth aspect of embodiments of the present disclosure, there is provided a computer program product, wherein instructions which, when executed by a processor of a metadata server, enable the metadata server to perform a file processing method, the method comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

According to the file processing method, the system, the device, the equipment and the storage medium provided by the embodiment of the disclosure, the original file identification of the target file is carried by the deletion request which is sent by the client side, the deletion request is used for indicating that the deletion command is called to the target file, based on the deletion request, the specified command is called to replace the deletion command, and the original file identification is modified into the target file identification in the process of executing the specified command. The original file identification comprises an original storage path and an original file name, the target file identification comprises a target storage path and a target file name, the target file name comprises the original file identification, and the target storage path is stored in a specified directory set by the distributed storage system. Therefore, when the distributed storage system receives the deletion request of the target file, the distributed storage system does not call the deletion command any more, but calls the specified command, modifies the file identifier of the target file, and does not execute the subsequent deletion process any more, so as to ensure that the target file is still stored in the distributed storage system and cannot be deleted. And original file identification is reserved in the target file identification, so that the target file can be conveniently searched or restored subsequently. The method and the device for deleting the target file in the distributed storage system have the advantages that the application range is expanded, and the reliability is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a block diagram illustrating one implementation environment in accordance with an exemplary embodiment.

FIG. 2 is a flow diagram illustrating a file processing method according to an example embodiment.

FIG. 3 is a flow diagram illustrating a file processing method according to an example embodiment.

FIG. 4 is a block diagram illustrating a file processing apparatus according to an example embodiment.

Fig. 5 is a schematic diagram illustrating a structure of a metadata server according to an exemplary embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

Fig. 1 is a schematic diagram illustrating a structure of an implementation environment according to an exemplary embodiment, where the implementation environment includes a client 101 and a distributed storage system 102, the distributed storage system 102 includes a metadata server 1021 and a data server 1022, the client 101 is connected to the metadata server 1021, and the metadata server 1021 is connected to the data server 1022.

The client 101 may be a terminal or a server. The data server 1022 is configured to store a file, and the metadata server 1021 is configured to store a file identifier of the file, and is further configured to store metadata information of the file, where the metadata information is used to describe the file stored in the data server 1022, and may include multiple items of information such as a file type, a storage time, and index information.

The server may be a server, or a server cluster composed of a plurality of servers, or a cloud computing service center, etc.

The client 101 may access files in the data server 1022 through the metadata server 1021, delete files in the data server 1022, or perform other operations on files in the data server 1022.

Fig. 2 is a flowchart illustrating a file processing method according to an exemplary embodiment, which is applied to a distributed storage system, and as shown in fig. 2, includes the following steps:

in step 201, a deletion request sent by a client is received, where the deletion request carries an original file identifier of a target file, and the deletion request is used to instruct to invoke a deletion command on the target file.

In step 202, based on the delete request, a specified command is invoked to replace the delete command.

In step 203, in the process of executing the designated command, the original file identifier is modified into a target file identifier, the original file identifier includes an original storage path and an original file name, the target file identifier includes a target storage path and a target file name, the target file name includes the original file identifier, and the target storage path is stored in a designated directory set in the distributed storage system.

According to the method provided by the embodiment of the disclosure, a deletion request sent by a client is received, the deletion request carries an original file identifier of a target file, the deletion request is used for indicating that a deletion command is called for the target file, based on the deletion request, a specified command is called to replace the deletion command, and the original file identifier is modified into the target file identifier in the process of executing the specified command. The original file identification comprises an original storage path and an original file name, the target file identification comprises a target storage path and a target file name, the target file name comprises the original file identification, and the target storage path is stored in a specified directory set by the distributed storage system. Therefore, when the distributed storage system receives a deletion request of the target file, the distributed storage system does not call the deletion command any more, but calls the specified command to modify the file identifier of the target file, and does not execute the subsequent deletion process any more, so as to ensure that the target file is still stored in the distributed storage system and cannot be deleted. And original file identification is reserved in the target file identification, so that the target file can be conveniently searched or restored subsequently. The method and the device for deleting the target file in the distributed storage system have the advantages that the application range is expanded, and the reliability is improved.

In one possible implementation, the method further comprises:

receiving a recovery request sent by a client, wherein the recovery request carries a target file identifier;

and modifying the target file identification into the original file identification based on the recovery request.

In a possible implementation manner, modifying an original file identifier of a target file into a target file identifier during execution of a specified command includes:

acquiring a specified directory as a primary directory, and acquiring a secondary directory corresponding to a time period to which the current time belongs under the primary directory;

generating a target storage path according to the first-level directory and the second-level directory;

and modifying the original file identifier into a target file identifier according to the target storage path and the target file name.

In one possible implementation manner, obtaining the secondary directory corresponding to the time period to which the current time belongs under the primary directory includes:

when the secondary directory corresponding to the time period to which the current moment belongs is not included in the primary directory, creating the secondary directory under the primary directory;

In one possible implementation, generating the target storage path according to the primary directory and the secondary directory includes:

acquiring a third-level directory with the number of file identifications under the second-level directory not reaching a preset number;

and generating a target storage path according to the first-level directory, the second-level directory and the third-level directory.

In one possible implementation, the method further comprises:

In a possible implementation manner, the secondary directory of each file under the specified directory corresponds to the file identifier modification time of the file;

deleting the files corresponding to the file identifiers of which the survival time reaches the preset time in the appointed directory, wherein the deleting comprises the following steps:

and deleting the secondary directory with the survival time reaching the preset time and the file corresponding to the file identifier under the secondary directory in the appointed directory.

In one possible implementation, the delete command is a delete command in a unlink function, and the designated command is a rename command.

Fig. 3 is a flowchart illustrating a file processing method according to an exemplary embodiment, which is applied to a metadata server in the distributed storage system shown in fig. 1, and includes the following steps, as shown in fig. 3:

in step 301, a deletion request sent by a client is received.

In one possible implementation manner, each client may display a file management interface, where the file management interface includes at least one file identifier provided in the metadata server, and the at least one file identifier corresponds to at least one file in the data server, and a user may view the file identifier of each file through the file management interface, and may also access or delete the corresponding file through the file identifier in the metadata server.

The embodiment of the present disclosure describes a process of deleting a target file by taking the target file as an example, where the target file may be an audio file, a video file, a picture file, a text file, or other types of files, and the target file is stored in a data server in advance. When the metadata server receives a deletion request sent by the client, wherein the deletion request is used for indicating that a deletion command is called for a target file, the metadata server should call the deletion command, and deletion of the target file is realized by executing the deletion command.

The deletion request carries an original file identifier of the target file, wherein the original file identifier comprises an original storage path and an original file name, and the deletion request can also comprise index information, and the index information is used for indicating the identity of the target file and corresponds to the target file one by one. For example, the index information may be a storage address of the target file in the data server, or may also be index information corresponding to the storage address of the target file in the data server, and the like.

Regarding the determination manner of the original file identifier, the original storage path and the original file name in the original file identifier may be set by the data server by default according to the file type when the target file is created, or may be set by the user. For example: the original file name is "3. jpg", "file. doc", etc.

In a possible implementation manner, a user triggers a file creation request on a file management interface displayed by a client, the file creation request carries an original storage path and an original file name, the client sends the file creation request to a metadata server when receiving the file creation request, the metadata server creates metadata information of a file according to a directory where the original storage path is located when receiving the file creation request, the file creation request is sent to a data server, and the data server creates the target file and automatically generates the original file name based on the original file name.

As to the triggering of the delete request, in one possible implementation, the delete request may be triggered by the user through the client. And in the process that the user accesses the files through the client, the client acquires the file identification of each file in the data server through the metadata server and displays the file identification to the user. When a user wants to delete a target file, a deletion request for an original file identifier of the target file is triggered, and after receiving the deletion request, a client sends the deletion request to a metadata server and the metadata server receives the deletion request.

For example, a delete command is input by a user on a terminal, and a file identifier corresponding to the delete command is specified to be an original file identifier of the target file, the client determines to receive a deletion request for the target file, and sends the deletion request to the metadata server. Or when the user carries out right-click operation on the original file identifier of the target file on the client, the client detects the right-click operation, then a deletion option for the original file identifier of the target file is displayed, when the user carries out click operation on the deletion option, the client detects the click operation, determines that a deletion request for the target file is received, and sends the deletion request to the metadata server.

In another possible implementation, the delete request may be automatically triggered by the metadata server as needed. For example, the metadata server sets a preset time length, and also counts the storage time length of each file in the data server, and when it is detected that the storage time length of a target file reaches the preset time length, a deletion request for the target file is triggered. Alternatively, the server may trigger the delete request in other manners.

In step 302, based on the delete request, a specified command is invoked to replace the delete command.

In the related art, when a metadata server receives a deletion request, an original file name of a target file is determined according to an original file identifier carried in the deletion request, the deletion request carrying the original file name is sent to a data server, and when the data server receives the deletion request, the file carrying the original file name is deleted; or the metadata server determines the index information of the target file according to the original file identifier carried in the deletion request, sends the deletion request carrying the index information to the data server, and the data server deletes the target file corresponding to the index information when receiving the deletion request. The index information may be a storage address of the target file in the data server, or may also be index information corresponding to the storage address of the target file in the data server.

In the embodiment of the present disclosure, for each file to be deleted, when the metadata server receives a deletion request, if the file is directly deleted, the file cannot be restored, and the file cannot be accessed subsequently, which may possibly affect the normal use of the file. To avoid this, the metadata server, upon receiving the deletion request, calls a specified command to replace the deletion command based on the deletion request. That is, the metadata server will not call the delete command any more, but call the specified command, modify the file identifier of the target file, and will not execute the subsequent delete process any more, so as to ensure that the target file is still stored in the data server and will not be deleted.

In one possible implementation, the delete command is a delete command in an unlink function, and the specified command is a rename command. The metadata server may replace an unlink command that needs to be called in the process of deleting the file with a rename command, and in the unlink stage, when the metadata server calls the unlink command based on the deletion request, the rename command is called, and the following renaming operation in step 303 is performed.

In step 303, during the process of executing the specified command, the original file identifier of the target file is modified to the target file identifier.

When the metadata server calls the designated command, the designated command starts to be executed, in the execution process, an original file identifier of the target file is obtained, the original file identifier comprises an original storage path and an original file name, the original storage path is modified into the target storage path, so that after modification, the target file identifier comprises the target storage path and the target file name, the target file name comprises the original file identifier, and the target storage path is stored in a designated directory set by the metadata server.

The appointed directory can be preset by the metadata server, which is equivalent to that the metadata presets a special appointed directory, when a deletion request is triggered each time, the storage path of the files to be deleted but not actually deleted is stored under the appointed directory, that is, the file identifications of the files are stored under the appointed directory, so that when a user accesses the metadata server through a client, the file identifications triggering the deletion request can only be displayed under the appointed directory, but not under the original directory, the effect of deleting the files is realized, and the user can conveniently manage the files.

Wherein, the process that the metadata server modifies the original file identifier of the target file into the target file identifier comprises the following steps 3031-3035:

in step 3031, during the process of executing the designated command, the name of the target file is obtained according to the original file identifier.

The metadata server may obtain an original storage path and an original file name in the original file identifier, and combine the original storage path and the original file name to form a target file name, for example: and setting the original file names in the original storage path, and combining to obtain the target file names, or setting the original file names in the original storage path, and combining to obtain the target file names. In the combining process, in order to distinguish the original storage path from the original file name, a separator may be added between the original storage path and the original file name, such as: /, +, spaces, etc.

The target file name is related to the original file identifier, and the original storage path and the original file name of the target file can be recorded, so that the target file can be restored again according to the target file identifier in the subsequent process.

In step 3032, the specified directory is obtained as the primary directory.

In the embodiment of the disclosure, in order to facilitate the metadata server to manage a large number of files, the metadata server creates a multi-level directory, and the multi-level paths form a storage path, and each storage path corresponds to one or more files.

Wherein the designated directory is a primary directory. The primary directory may be configured through an mds _ recycle _ root _ dir function, or may be configured in other manners, and during the configuration process, a name of the primary directory, such as a large recycle bin, a small recycle bin, a backup folder, or other names, may be set.

In step 3033, the secondary catalog corresponding to the time period to which the current time belongs under the primary catalog is obtained.

If the storage paths of all files are stored in the first-level directory, when a certain file needs to be searched, the files need to be checked one by one, and a large amount of calculation is caused. Therefore, in order to search files conveniently, a plurality of secondary catalogues are created under the primary catalog, and each secondary catalog corresponds to different time periods respectively, so that file identifications of deletion time belonging to different time periods are stored under different secondary catalogues. The deletion time of any file identifier may represent the time of requesting to delete a file, and may be the time when the metadata server receives a deletion request carrying the file identifier, or the time when the file identifier is modified into another file identifier.

When the metadata server modifies the original file identifier into the target file identifier, the time period to which the current moment belongs can be obtained, and the secondary directory corresponding to the time period under the primary directory is obtained, so that the target storage path of the target file is stored under the secondary directory.

The time duration of the time period corresponding to each secondary directory may be equal or different. In one possible implementation manner, the metadata server respectively records the number of files corresponding to each historical time period in the historical period, that is, the number of files belonging to the historical time period at the deletion time. When the number of the files corresponding to a certain historical time period is determined to be larger than the preset number, the files required to be deleted in the historical time period are more, at the moment, the metadata server divides the time period corresponding to the historical time period in the current cycle into a plurality of time periods, so that the time lengths of the time periods are all smaller than the time length of the historical time period, and correspondingly, a corresponding secondary directory is created according to the time periods divided in the current cycle. Therefore, the division with finer granularity can be realized in the time period of deleting files frequently, and more file identifications can be stored together in the time period of deleting files less frequently, so that the quantity of the file identifications in different secondary directories is ensured to be balanced as much as possible. Wherein, the period can be one day, 1 hour or 10 minutes, etc.

When the metadata server acquires the secondary directory, it will first determine whether the time period under the primary directory includes the time period under the current time, and according to the difference of the determination results, the manner of acquiring the secondary directory includes but is not limited to the following two manners:

firstly, when the secondary directory corresponding to the time period to which the current time belongs is determined not to be included in the primary directory, the secondary directory is created in the primary directory.

And the metadata server searches a secondary directory corresponding to the time period under the primary directory based on the time period to which the current time belongs, and if the metadata server determines that the secondary directory corresponding to the time period to which the current time belongs is not included under the primary directory, a secondary directory is created under the primary directory, wherein the secondary directory is the secondary directory corresponding to the time period to which the current time belongs.

For example: and when the current time is three afternoons of 12-month and 4-month in 2018, if the time period of the current time is 12-month and 4-month in 2018, searching whether a secondary catalog corresponding to 12-month and 4-day in 2018 exists under the primary catalog, and if the secondary catalog is determined not to be included under the primary catalog, creating the secondary catalog.

And secondly, when the secondary directory corresponding to the time period to which the current time belongs is determined to be included in the primary directory, the secondary directory is obtained.

And the metadata server searches a secondary directory corresponding to the time period under the primary directory based on the time period to which the current time belongs, and acquires the secondary directory if the metadata server determines that the secondary directory corresponding to the time period to which the current time belongs is included under the primary directory.

For example: and when the current time is three afternoons of 12-month and 4-month in 2018, if the time period of the current time is 12-month and 4-month in 2018, searching whether a secondary catalog corresponding to 12-month and 4-month in 2018 exists in the primary catalog, and if the secondary catalog is determined to be included in the primary catalog, acquiring the secondary catalog.

In step 3034, a target storage path is generated based on the primary directory and the secondary directory.

The metadata server may combine the primary directory and the secondary directory to form a target storage path. For example: when the primary directory is $ { mds _ recycle _ root _ dir }, and the secondary directory is $ { date _ rmday }, the target storage path may be $ { mds _ recycle _ root _ dir }/$ { date _ rmday }.

In one possible implementation, if the storage paths of all files are stored in the secondary directory, the process of searching the files is time-consuming and labor-consuming. Therefore, the secondary directory can be further refined, namely, a tertiary directory can be created under the secondary directory, and the number of file identifications under each tertiary directory can reach the preset number at most. For example, the preset number may be 1000, or 10000, or 100000, or may be another number.

Then, the metadata server obtains a tertiary directory in which the number of the file identifiers in the secondary directory does not reach the preset number, so that the file identifiers are stored in the tertiary directory subsequently. The method for obtaining the third-level directory includes, but is not limited to, the following three ways:

first, if the tertiary directory does not exist under the secondary directory, the tertiary directory is created.

And secondly, establishing the latest number of the file identifiers in the tertiary catalog in all the tertiary catalogs under the secondary catalog, and if the number does not reach the preset number, acquiring the tertiary catalog. If the number has reached a preset number, another tertiary directory is created under the secondary directory.

And thirdly, pre-creating a plurality of tertiary catalogs, acquiring the quantity of the file identifiers in each tertiary catalog after receiving the deletion request, and acquiring the tertiary catalog if the quantity of the file identifiers in any tertiary catalog does not reach the preset quantity.

After the metadata server obtains the third-level directory, the first-level directory, the second-level directory, and the third-level directory may be combined to generate a target storage path, for example: when the primary directory is $ { mds _ recycle _ root _ dir }, the secondary directory is $ { date _ rmday }, and the tertiary directory is $ { frag _ num }, the target storage path is: $ mds _ recycle _ root _ dir }/$ { date _ rmday }/$ { frag _ num }.

In step 3035, the original file identifier is modified into the target file identifier according to the target storage path and the target file name.

And after the metadata server acquires the target storage path and the target file name, combining the target storage path and the target file name to generate a target file identifier.

Such as: the target storage path is $ { mds _ recycle _ root _ dir }/$ { date _ rmday }, and the target file name is $ { wait _ dense }, so that a target file identifier comprising a tertiary directory is obtained: $ mds _ recycle _ root _ dir }/$ { date _ rmday }/$ { waitingrm _ dentry }, which means that the target file identifier is stored in the three-level directory.

Or the target storage path is $ { mds _ recycle _ root _ dir }/$ { date _ rmday }/{ frag _ num }, and the target file name is $ { wait _ density }, then the target file identifier including the four-level directory is obtained: $ mds _ recycle _ root _ dir }/$ { date _ rmday }/{ frag _ num }/$ { waitingrm _ dentry }, which means that the target file identifier is stored in the four-level directory.

In step 304, a recovery request sent by the client is received, and the target file identifier is modified to the original file identifier based on the recovery request.

The recovery request carries index information of the target file, and the metadata server can modify the target file identifier into the original file identifier based on the recovery request of the target file when receiving the recovery request.

In a possible implementation manner, the terminal may display the modified target file identifier in the file management interface, when the user needs to restore the target file, a restoration request may be triggered for the target file, and the client, when receiving the restoration request, sends the restoration request to the metadata server and is received by the metadata server.

For example, when a user right-clicks the target file on the client, the client detects the right-click operation, and then displays the recovery option of the target file, and when the user clicks the recovery option, the client detects the click operation, determines that a recovery request for the target file is received, and sends the recovery request to the metadata server.

The target file identifier comprises an original file identifier, so that after the metadata server acquires the target file, the original file identifier is extracted from the target file identifier, and the target file identifier of the target file is modified into the original file identifier. At this time, the storage path of the target file is modified into the original storage path, and the recovery of the target file is realized.

It should be noted that, the embodiment of the present disclosure only describes the process of deleting and restoring the target file, and when other files need to be deleted or restored, similar manners may be used, and details are not described here.

According to the method provided by the embodiment of the disclosure, a deletion request sent by a client is received, the deletion request carries an original file identifier of a target file, the deletion request is used for indicating that a deletion command is called for the target file, based on the deletion request, a specified command is called to replace the deletion command, and the original file identifier is modified into the target file identifier in the process of executing the specified command. The original file identification comprises an original storage path and an original file name, the target file identification comprises a target storage path and a target file name, the target file name comprises the original file identification, and the target storage path is stored in a specified directory set by the distributed storage system. Therefore, when the distributed storage system receives a deletion request of the target file, the distributed storage system does not call the deletion command any more, but calls the specified command to modify the file identifier of the target file, and does not execute the subsequent deletion process any more, so as to ensure that the target file is still stored in the distributed storage system and cannot be deleted, and the original file identifier is reserved in the target file identifier, thereby facilitating the subsequent search or recovery of the target file. The method and the device have the advantages that the target file is deleted in the distributed storage system, so that the application range is expanded, and the reliability is improved;

moreover, after the target file is moved to the designated storage space, the target file name comprises the original file identifier, and when a recovery request is received, the target file identifier of the target file can be recovered to the original file identifier again, so that the storage path of the target file is recovered to the original storage path again, the recovery of the target file is realized, and the reliability is improved.

The method provided by the embodiment of the disclosure can also delete the file in the data server to release the occupied storage space.

In one possible implementation manner, the file corresponding to the file identifier whose lifetime has reached the preset duration under the specified directory is deleted.

And in the process of modifying the file identifier, the metadata server also records the modification time of the file identifier as the time of requesting to delete the file, judges the survival time of each modified file identifier in the specified directory according to the modification time and the current time of the file identifier, and deletes the file corresponding to the file identifier when the survival time is determined to reach the preset time. Namely, the metadata server sends a deletion request to the data server, the deletion request carries a file identifier, and when the data server receives the deletion request, the corresponding file is deleted according to the file identifier. Or, the deletion request carries index information corresponding to the file identifier, and the data server deletes the corresponding file according to the index information when receiving the deletion request.

In another possible implementation manner, in the process of modifying the file identifier, the metadata server deletes the secondary directory in the specified directory, for which the lifetime has reached the preset duration, and the file corresponding to the file identifier in the secondary directory based on the time period and the current time corresponding to the secondary directory. The metadata server deletes a secondary directory with the survival time reaching the preset time, acquires a file identifier under the secondary directory, sends a deletion request to the data server, wherein the deletion request carries the file identifier, and deletes the corresponding file according to the file identifier when the data server receives the deletion request. Or, the deletion request carries index information corresponding to the file identifier, and the data server deletes the corresponding file according to the index information when receiving the deletion request.

The modification time of the file identifier under each secondary directory belongs to the time period corresponding to the secondary directory, the life time of the secondary directory refers to the difference between the current time and the creation time of the secondary directory, and can represent the difference between the current time and the deletion request time of the file to a certain extent, so that the secondary directory with the life time reaching the preset time and the file corresponding to the file identifier under the secondary directory can be deleted, and the files which are requested to be deleted by a user for a long time and are not recovered can be deleted in batches without being deleted one by one.

The preset time period can be 3 days, one week or one month, or can be any time period meeting the requirement. The preset duration may be configured by the mds _ recycle _ clear _ interval function, or may also be configured in other configurations.

In addition, the metadata server may delete the file by deleting the tertiary directory, for example, the metadata server may delete the tertiary directory with the earliest creation time or the tertiary directory with the largest file identifier under any one or more secondary directories according to the sequence of the creation time, or may also adopt other manners.

Regarding the time of deleting the file in the data server, the metadata server may set a deletion period in advance, and each time the start time of the deletion period is reached, the metadata server starts to delete the file corresponding to the file identifier to be deleted in the designated directory. The period may be 1 day, one week, or other periods as well. Accordingly, the starting time of each cycle may be 1 am of each day, monday of each week, or other starting time that is configurable by the mds _ recycle _ clear _ begin _ time function.

When deleting a file corresponding to a file identifier under a specified directory, the metadata server may set a gc thread, and delete the file through the gc thread or delete the file through a purgeque function.

According to the method provided by the embodiment of the disclosure, after the metadata server modifies the file identifier, the file identifier with longer lifetime can be deleted according to the lifetime of the modified file identifier or the lifetime of the secondary directory where the modified file identifier is located, so that the file corresponding to the file identifier in the data server is deleted, more storage spaces in the metadata server and the data server are released, resources of the storage space are saved, the user does not need to manually delete the file identifier, and the operation of the user is simplified.

It should be noted that, the embodiment of the present disclosure has been described only by taking the execution subject as the metadata server, in another possible implementation manner, the distributed storage system may further include other management servers for managing files, and the method provided by the embodiment of the present disclosure may also be executed by other management servers in the distributed storage system.

Fig. 4 is a block diagram illustrating a file processing apparatus according to an exemplary embodiment, applied to a metadata server, as illustrated in fig. 4, the apparatus including:

a deletion request receiving unit 401 configured to receive a deletion request sent by a client, where the deletion request carries an original file identifier of a target file, and the deletion request is used to instruct to invoke a deletion command on the target file;

a calling unit 402 configured to call a specified command to replace the deletion command based on the deletion request;

a first modifying unit 403, configured to modify, during the execution of the specified command, an original file identifier into a target file identifier, where the original file identifier includes an original storage path and an original file name, the target file identifier includes a target storage path and a target file name, the target file name includes an original file identifier, and the target storage path is stored in a specified directory set in the distributed storage system.

According to the device provided by the embodiment of the disclosure, a deletion request sent by a client is received, the deletion request carries an original file identifier of a target file, the deletion request is used for indicating that a deletion command is called for the target file, based on the deletion request, a specified command is called to replace the deletion command, and the original file identifier is modified into the target file identifier in the process of executing the specified command. The original file identification comprises an original storage path and an original file name, the target file identification comprises a target storage path and a target file name, the target file name comprises the original file identification, and the target storage path is stored in a specified directory set by the distributed storage system. Therefore, when the distributed storage system receives the deletion request of the target file, the distributed storage system does not call the deletion command any more, but calls the specified command to modify the file identifier of the target file, and does not execute the subsequent deletion process any more, so as to ensure that the target file is still stored in the distributed storage system and cannot be deleted. And original file identification is reserved in the target file identification, so that the target file can be conveniently searched or restored subsequently. The method and the device for deleting the target file in the distributed storage system have the advantages that the application range is expanded, and the reliability is improved.

In one possible implementation, the apparatus further includes:

the recovery request receiving unit is configured to receive a recovery request sent by a client, and the recovery request carries a target file identifier;

and the second modification unit is configured to modify the target file identifier into the original file identifier based on the recovery request.

In one possible implementation, the first modifying unit 403 includes:

the directory acquisition subunit is configured to acquire the specified directory as a primary directory and acquire a secondary directory corresponding to the time period to which the current time belongs under the primary directory;

a path generation subunit configured to generate a target storage path according to the primary directory and the secondary directory;

and the modifying subunit is configured to modify the original file identifier of the target file into the target file identifier according to the target storage path and the target file name.

In one possible implementation, the apparatus further includes:

and the deleting unit is configured to delete the file corresponding to the file identifier of which the survival time duration reaches the preset duration under the specified directory.

In a possible implementation manner, the modification time of the secondary directory of each file under the directory and the file identifier of the file is specified; the deletion unit includes:

With regard to the apparatus in the above-described embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.

Fig. 5 is a schematic structural diagram illustrating a metadata server 500 according to an exemplary embodiment, where the metadata server 500 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 501 and one or more memories 502, where the memory 502 stores at least one instruction, and the at least one instruction is loaded and executed by the processors 501 to implement the methods provided by the above method embodiments.

Certainly, the metadata server may further have components such as a wired or wireless network interface, a keyboard, and an input/output interface, so as to perform input and output, and the metadata server may further include other components for implementing the functions of the device, which is not described herein again.

The metadata server 500 may be used to perform the steps performed by the metadata server in the above-described file processing method.

In an exemplary embodiment, there is also provided a non-transitory computer readable storage medium having instructions therein, which when executed by a processor of a metadata server, enable the metadata server to perform a file processing method, the method comprising:

receiving a deletion request sent by a client, wherein the deletion request carries an original file identifier of a target file, and the deletion request is used for indicating a deletion command to be called for the target file;

in the process of executing the specified command, the original file identification is modified into a target file identification, the original file identification comprises an original storage path and an original file name, the target file identification comprises a target storage path and a target file name, the target file name comprises the original file identification, and the target storage path is stored in a specified directory set by the distributed storage system.

In an exemplary embodiment, there is also provided a computer program product, instructions in which, when executed by a processor of a metadata server, enable the metadata server to perform a file processing method, the method comprising:

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A file processing method is applied to a distributed storage system, the distributed storage system comprises a metadata server and a data server, the metadata server is used for storing file identification of a file, the data server is used for storing the file, and the method comprises the following steps:

the metadata server receives a deletion request sent by a client, wherein the deletion request carries an original file identifier of a target file, and the deletion request is used for indicating that a deletion command is called for the target file;

the metadata server invoking a rename command to replace the delete command based on the delete request;

in the process that the metadata server executes the renaming rename command, the original file identifier is modified into a target file identifier, the original file identifier comprises an original storage path and an original file name, the target file identifier comprises a target storage path and a target file name, the target file name comprises the original file identifier, and the target storage path is stored in a specified directory set by the metadata server, so that when a user accesses the metadata server through the client, the file identifier triggering a deletion request is displayed in the specified directory, and the effect of deleting a file is realized, wherein when the life time of the file identifier in the specified directory reaches the preset time, the corresponding file in the data server is deleted;

the method further comprises the following steps: the metadata server receives a recovery request sent by the client, modifies the target file identifier into an original file identifier based on the recovery request, modifies the storage path of the target file into an original storage path, and realizes recovery of the target file, wherein the recovery request carries index information of the target file;

modifying the original file identifier into a target file identifier in the process of executing the renaming rename command by the metadata server, wherein the modifying comprises the following steps:

in the process that the metadata server executes the renaming command, the target file name is obtained according to the original file identifier; acquiring the specified directory as a primary directory, and acquiring a secondary directory corresponding to the time period to which the current time belongs under the primary directory; generating the target storage path according to the primary directory and the secondary directory; and modifying the original file identifier into the target file identifier according to the target storage path and the target file name.

2. The method of claim 1, wherein the obtaining, by the metadata server, the secondary directory corresponding to the time period to which the current time belongs under the primary directory comprises:

when the secondary directory corresponding to the time period to which the current time belongs is determined not to be included in the primary directory, the metadata server creates the secondary directory in the primary directory;

and when the secondary directory corresponding to the time period to which the current moment belongs is determined to be included in the primary directory, the metadata server acquires the secondary directory.

3. The method of claim 1, wherein the metadata server generates the target storage path based on the primary directory and the secondary directory, comprising:

the metadata server acquires a tertiary catalog of which the number of file identifications under the secondary catalog does not reach a preset number;

and the metadata server generates the target storage path according to the primary directory, the secondary directory and the tertiary directory.

4. The method according to claim 1, wherein the secondary directory of each file under the specified directory corresponds to the modification time of the file identifier of the file; the method comprises the following steps:

5. The method according to any of claims 1-4, wherein the delete command is a delete command in a unlink function.

6. The distributed storage system is characterized by comprising a metadata server and a data server, wherein the data server is used for storing files, and the metadata server is used for storing file identifiers of the files;

the metadata server is further used for calling a renaming rename command to replace the deletion command based on the deletion request;

the metadata server is further configured to modify the original file identifier into a target file identifier in a process of executing the renaming rename command, where the original file identifier includes an original storage path and an original file name, the target file identifier includes a target storage path and a target file name, the target file name includes the original file identifier, and the target storage path is stored in an assigned directory set by the metadata server, so that when a user accesses the metadata server through the client, a file identifier that has triggered a deletion request is displayed in the assigned directory, and a file deletion effect is achieved, where a file identifier in the assigned directory has a lifetime reaching a preset duration, and a corresponding file in the data server is deleted;

the metadata server is further configured to receive a recovery request sent by the client, where the recovery request carries the target file identifier;

the metadata server is further used for modifying the target file identifier into the original file identifier based on the recovery request;

the metadata server is further configured to, during the renaming rename command, obtain the name of the target file according to the original file identifier, obtain the specified directory as a primary directory, obtain a secondary directory corresponding to a time period to which the primary directory belongs at the current time, generate the target storage path according to the primary directory and the secondary directory, and modify the original file identifier into the target file identifier according to the target storage path and the name of the target file.

7. A file processing apparatus applied to a metadata server, wherein a distributed storage system includes the metadata server and a data server, the metadata server is configured to store a file identifier of a file, and the data server is configured to store the file, the apparatus comprising:

a deletion request receiving unit configured to receive a deletion request sent by a client, where the deletion request carries an original file identifier of a target file, and the deletion request is used to instruct to call a deletion command for the target file;

a calling unit configured to call a rename command to replace the delete command based on the delete request;

a first modification unit, configured to modify an original file identifier into a target file identifier in a process of executing the renaming rename command, where the original file identifier includes an original storage path and an original file name, the target file identifier includes a target storage path and a target file name, the target file name includes the original file identifier, and the target storage path is stored in an assigned directory set by the metadata server, so that when a user accesses the metadata server through the client, a file identifier that has triggered a deletion request is displayed in the assigned directory, and a file deletion effect is achieved, where when a lifetime of the file identifier in the assigned directory reaches a preset duration, a corresponding file in the data server is deleted;

the device further comprises: a recovery request receiving unit configured to receive a recovery request sent by the client, where the recovery request carries the target file identifier;

a second modification unit configured to modify the target file identifier to the original file identifier based on the recovery request;

the first modification unit includes:

the name acquisition subunit is configured to acquire the name of the target file according to the original file identifier in the process of executing the renaming rename command;

8. The apparatus of claim 7, wherein the directory retrieval subunit is further configured to:

9. The apparatus of claim 7, wherein the path generation subunit is further configured to:

10. The apparatus according to claim 7, wherein the secondary directory of each file under the specified directory corresponds to a modification time of the file identifier of the file; the first modification unit includes:

11. The apparatus according to any of claims 7-10, wherein the delete command is a delete command in a unlink function.

12. A metadata server, characterized in that the metadata server comprises:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to perform the file processing method of any one of claims 1 to 5.

13. A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of a metadata server, enable the metadata server to perform the file processing method of any one of claims 1-5.