CN114661240A

CN114661240A - Data processing method and storage system

Info

Publication number: CN114661240A
Application number: CN202210334469.3A
Authority: CN
Inventors: 谢东凯; 廖晨歌; 周沁; 张锋
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2022-03-30
Filing date: 2022-03-30
Publication date: 2022-06-24

Abstract

The application provides a data processing method and a storage system, and relates to the field of storage. The method comprises the following steps: the metadata server receives a first instruction from a recycle bin management server; identifying the first data as an abnormal state based on the first instruction; and in the case that the system overhead meets a preset condition, cleaning the first data which is identified as the abnormal state from the recycle bin. Based on the data processing method provided by the application, the expired data which is not cleaned is identified as the abnormal state, the charging space of the tenant is not counted, and the data which is identified as the abnormal state is cleaned under the condition that the system overhead meets the preset condition, so that the strategy of a recycle bin for cleaning the expired data is optimized, the influence on the system performance is reduced, the occupied time of the tenant on the storage space is not additionally increased, and unnecessary expense brought to the tenant is avoided.

Description

Data processing method and storage system

Technical Field

The present application relates to the field of storage, and in particular, to a data processing method and a storage system.

Background

When the tenant stores data by using the cloud space, the data can be deleted by mistake. In order to prevent the tenant data from being lost due to the fact that the tenant data is deleted by mistake, a recycle bin function is arranged in the cloud space. The data deleted by the tenant can enter the recycle bin, and the tenant can also reply the mistakenly deleted data from the recycle bin. Meanwhile, the recycle bin also has an automatic cleaning function, and when the retention time of the data in the recycle bin exceeds the time set by the tenant, the recycle bin can automatically clean out the overdue data.

After the recycle bin is started, the storage space occupied by the storage system for the data in the recycle bin is still charged according to the normal storage price, and after the data is cleaned from the storage space of the recycle bin, the storage space is released, and the charging is stopped. Therefore, if the recycle bin cannot clean out expired data timely, the charging space of the tenant can be occupied, and extra cost is added to the tenant. Therefore, the recycle bin needs to clean up stale data in real time. However, the clearing of the recycle bin data requires modification of the metadata of the file system, and also triggers deletion of the metadata and the data, which occupies a certain system overhead. If the background is also clearing the recycle bin data while the user is accessing the system, the overhead will be large, and the system performance will be affected, for example, the processing of creating, deleting, rewriting requests, etc. of the data link will be affected.

Therefore, it is necessary to provide a method for optimizing the policy of the recycle bin for cleaning the stale data, and reducing the influence of automatically cleaning the stale data on the tenant as much as possible.

Disclosure of Invention

The application provides a data processing method and a storage system, aiming at optimizing a strategy for clearing the overdue data of a recycle bin and reducing the influence of automatic clearing of the overdue data on tenants as much as possible.

In a first aspect, the present application provides a data processing method applied in a storage system, where the storage system is used to provide data storage services for one or more tenants, and the storage system includes a recycle bin management server and a metadata server, and the method includes: the metadata server receives a first instruction from the recycle bin management server, wherein the first instruction is used for indicating that first data in the recycle bin of the first tenant is marked as an abnormal state; the first data is data which is stored in the recycle bin for a preset time and is not cleaned, and the storage space occupied by the data which is marked as an abnormal state in the recycle bin is not counted into the charging space of the first tenant; the metadata server identifies the first data as an abnormal state based on the first instruction; and the metadata server cleans the first data which is identified as abnormal state from the recycle bin under the condition that the system overhead meets a preset condition.

In a second aspect, the present application provides a data processing method, including: the recycle bin management server generates a first instruction according to the time index of the first data; the first instruction is used for indicating that the first data are marked as abnormal states, and the storage space occupied by the data marked as the abnormal states in the recycle bin is not counted into the charging space of the first tenant; the recycle bin management server sends a first instruction to the metadata server.

In a third aspect, the present application provides a data processing method, applied to a metadata server, including: the metadata server receives a first instruction from the recycle bin management server, wherein the first instruction is used for indicating that first data in the recycle bin of the first tenant is identified as an abnormal state; the first data is data which is stored in the recycle bin for a preset time and is not cleaned, and the storage space occupied by the data which is marked as an abnormal state in the recycle bin is not counted into the charging space of the first tenant; the metadata server identifies the first data as an abnormal state based on the first instruction; and the metadata server cleans the first data which is identified as abnormal state from the recycle bin under the condition that the system overhead meets a preset condition.

In a fourth aspect, the present application provides a storage system for providing data storage services for one or more tenants, the storage system comprising a recycle bin management server and a metadata server; the recycle bin management server is used for sending a first instruction to the metadata server, wherein the first instruction is used for indicating that first data in the recycle bin of the first tenant is marked to be in an abnormal state; the first data is data which is stored in the recycle bin for a preset time and is not cleaned, and the storage space occupied by the data which is marked as an abnormal state in the recycle bin is not counted into the charging space of the first tenant; the metadata server is used for identifying the first data as an abnormal state based on the first instruction; and the data clearing device is used for clearing the first data from the recycle bin based on the second instruction under the condition that the system overhead meets a preset condition.

In a fifth aspect, the present application provides a metadata server for implementing the method of the second aspect.

In a sixth aspect, the present application provides a recycle bin management server for implementing the method in the third aspect.

In a seventh aspect, the present application provides an electronic device, including a processor, configured to invoke a computer program (also referred to as code, or instructions) to cause the electronic device to implement the method according to any one of the first to third aspects.

In an eighth aspect, the present application provides a computer readable storage medium comprising a computer program which, when run on a computer, causes the computer to perform the method of any one of the first to third aspects.

In a ninth aspect, the present application provides a computer program product comprising: a computer program (which may also be referred to as code, or instructions), which when executed, causes a computer to perform the method of any of the first to third aspects.

In the embodiment of the application, the first data which is expired but not cleaned is identified as the abnormal state through the metadata server, the paid space of the tenant is not counted, and the data which is identified as the abnormal state is cleaned under the condition that the system overhead meets the preset condition, so that the strategy of a recycle bin for cleaning the expired data is optimized, the influence on the system performance is reduced, the occupied time of the tenant on the storage space is not additionally increased, and unnecessary expense brought to the tenant is avoided.

Drawings

Fig. 1 is a schematic structural diagram of a storage system according to an embodiment of the present application;

fig. 2 is a schematic flowchart of deleting metadata according to an embodiment of the present application;

fig. 3 is a schematic flow chart of space release provided in an embodiment of the present application;

fig. 4 is a schematic structural diagram of a log-structured merge Tree LSM-Tree according to an embodiment of the present application;

FIG. 5 is a schematic flow chart diagram of a data processing method provided by an embodiment of the present application;

FIG. 6 is a flowchart illustrating a recycle bin management service and a metadata service provided in an embodiment of the present application;

FIG. 7 is a flowchart illustrating a recycle bin directory structure query according to an embodiment of the present disclosure;

fig. 8 is a schematic block diagram of a recycle bin management server provided in an embodiment of the present application;

FIG. 9 is a schematic block diagram of a metadata server provided by an embodiment of the present application;

fig. 10 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.

Detailed Description

To make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments.

Thus, the following detailed description of the embodiments of the present application, as presented in the figures, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

In the embodiments of the present application, the terms "first", "second", and the like are used for distinguishing similar objects, and are not necessarily used for describing a particular order or sequence. Furthermore, the terms "comprises" and any variations thereof, are intended to cover non-exclusive inclusions.

Fig. 1 is a schematic structural diagram of a storage system according to an embodiment of the present application. As shown in fig. 1, the storage system 110 includes a recycle bin management server 111 and a metadata server 112. The recycle bin management server 111 may be used to generate instructions instructing the metadata server 112 to perform a cleanup action for the metadata in the recycle bin; the metadata server 112 includes a key-value (KV) database, which is a database storing data by key values, each key corresponds to a unique value and is suitable for querying the data through a primary key.

In the embodiment of the present application, the KV database stores therein data of each tenant, for example, including, but not limited to, metadata of a file system and metadata of a recycle bin. Different tenants rent different storage spaces, using different file systems and recycle bins. Of course, the same tenant may use multiple file systems, which is not limited in this application. For the sake of distinction and explanation, the file system and the recycle bin are distinguished at the granularity of tenants.

It should be understood that the recycle bin management server 111 and the metadata server 112 are shown separately in fig. 1 for ease of distinction only. In an actual scenario, the recycle bin management server 111 and the metadata server 112 may be two independent physical devices, or may be two different logical partitions on the same physical device. The embodiments of the present application do not limit this.

In order to prevent the data stored in the storage system by the tenant from being deleted by mistake and lost, a recycle bin function is designed in the storage system so that the tenant can recover the deleted data by mistake. Meanwhile, the recycle bin also has an automatic cleaning function, and when the retention time of the data in the recycle bin exceeds the preset time, the recycle bin management server sends an instruction to the metadata server to instruct the metadata server to clean out the expired data.

At present, a storage system charges the storage space occupied by data in a recycle bin according to a normal storage price, and after the data is cleaned from the storage space of the recycle bin, the storage space is released, and the charging is stopped. Therefore, if the recycle bin cannot clean out expired data timely, the charging space of the tenant can be occupied, and extra cost is added to the tenant. Therefore, the recycle bin needs to clean up stale data in real time. However, clearing the recycle bin data requires modifying the metadata of the file system, and also triggers deletion of the metadata and the data, which takes up a certain overhead.

The cleaning of the recycle bin data includes two flows of deleting metadata and freeing space. The process of deleting metadata is shown in fig. 2, and the process of freeing space is shown in fig. 3.

Specifically, the process of deleting the metadata may specifically include:

step 201, deleting file object metadata;

step 202, deleting the file object index record;

step 203, modifying the modification time and the statistical information of the father catalog;

step 204, modifying the statistical information of the file system;

step 205, recording a file deletion event.

The process of releasing the space may specifically include:

step 301, acquiring a file deletion event;

Step 302, determining the storage position of data according to the file layout (layout) record;

step 303, deleting data;

step 304, modifying the storage management record of the system;

step 305, delete the layout record of the file.

Currently, a log structured merge Tree (LSM-Tree) structure is widely used by a disk to store data. The structure of the LSM-Tree is shown in FIG. 4. A Random Access Memory (RAM) and a disk (disk) are shown in the structure, wherein the disk may be further divided into multiple layers (levels), such as level 0, level 1, and level 2 shown in the figure. Wherein the total file size of each layer is predefined, and the size of the next layer is larger than that of the previous layer. When the memory is full, the files in the memory are written to the disk. The files in the magnetic disk are merged layer by layer, once the total size of the files in a certain layer exceeds a predefined threshold value, one file and the files in the next layer can be selected to be merged (compact).

In the above two flows, the modify and delete operations, i.e. steps 201, 202, 203, 204, 303, 304, 305, are all written to a memory table (memory table) of the RAM, and data is actually modified or deleted after multiple mergers. The query request needs to traverse the memtable of the RAM, the sorted string table (SStable) of level 0 in the disk, and the SStable containing query data at the lower layer. Therefore, if the backend is also clearing the recycle bin data while the user is accessing the system, the overhead will be large, and the system performance will be affected, for example, the processing of creating, deleting, rewriting requests and the like of the data link will be affected.

In view of this, the present application provides a data processing method, when data in a recycle bin is expired, on one hand, the expired data is identified and is not counted into a charging space of a tenant, and on the other hand, the expired data is cleaned up until a storage system enters a valley period. Therefore, the strategy of clearing the overdue data by the recycle bin is optimized, the influence on the system performance is reduced, the occupied time of the tenant on the storage space is not additionally increased, and unnecessary expenses of the tenant are avoided.

The data processing method and the storage system provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

It should be noted that the data processing method provided by the embodiment of the present application can be applied to a storage system. The storage system can be used for providing storage services for one or more tenants, and each tenant can use the storage space provided by the storage system to store respective data. The storage system may include a recycle bin management server and a metadata server, and the following embodiments will describe the method provided by the embodiments of the present application by taking an interaction between the two as an example.

For convenience of differentiation and illustration, the method provided by the embodiment of the present application is described below with the first data of the first tenant as an example. The first data is data which is stored in the recycle bin for a preset time and is not cleaned. Simply stated, the first data is the data that has expired without being cleaned. It is to be understood that the first data may be data from any tenant, and is defined herein as data of the first tenant for ease of distinction only.

Fig. 5 is a schematic flowchart of a data processing method according to an embodiment of the present application. As shown in fig. 5, the method may include steps 501 to 504. The individual steps of fig. 2 are explained in detail below.

In step 501, the recycle bin management server generates a first command according to the time index of the first data.

As described above, the first data is data that has been stored in the recycle bin for a predetermined period of time but has not been cleaned. The preset duration may be set by a tenant or may be default by the system, which is not limited in this application. The recycle bin management server can judge whether the data is expired and not cleaned according to the time when the data is put into the recycle bin and the preset time length.

Here, the data is put into the recycle bin, which may mean that the user deletes the data from the file system, and in order to avoid the deletion error, the system automatically moves the data into the recycle bin to be temporarily stored for a period of time. In the embodiment of the present application, the operation of deleting data from the file system may be recorded, for example, when the time that data 1 (i.e., an example of the first data) is put into the recycle bin is 2021 year 6 month 15 day 1, it may be recorded as 2021061501; for another example, when data 2 (i.e., another example of the first data) is placed in the recycle bin at 6/15/2/2021, it can be recorded as 2021061502.

Since the tenant may delete the unwanted data at any time, i.e., new data may be stored in the recycle bin at any time. Thus, the recycle bin management server may periodically count the data in the recycle bin to determine newly generated stale uncleaned data.

In an embodiment of the application, the time index of the first data may be related to a time at which the first data was placed in the recycle bin. Each time index may correspond to a time node. In this way, the recycle bin management server may generate a first instruction based on the time index of the first data to instruct the metadata server to identify the first data as an abnormal state through the first instruction.

Here, the time index may be created at different time lengths for granularity, for example, in hours, one time index is created for each hour corresponding to the data placed in the recycle bin within that hour.

For example, when data 1 is placed in the recycle bin at 2021, 6, month, 15, day 1, the corresponding time index is 1024; data 2 was placed into the recycle bin at 2021, 6, 15, 2, which corresponds to a time index of 1025.

In the embodiment of the present application, the granularity of the time index may be the same as the charging mode. For example, if the tenant charges for using the storage space are charged in hours, the granularity of the time index may be hours. Of course, the tenant may be charged for using the storage space by using other time lengths as a unit, and the time index may be distinguished by using other time lengths as a granularity, which is not limited in this application.

Optionally, step 501 comprises: the recycle bin management server determines a time index of first data according to a preset duration and a first mapping relation, wherein the first mapping relation comprises a corresponding relation between at least one time index and at least one time, and the time in the at least one time represents the time when the data under the corresponding time index is put into the recycle bin; the recycle bin management server generates a first instruction according to the time index of the first data.

Table 1 shows an example of the first mapping relationship.

TABLE 1

Time index	Time
		1024	2021061501
1025	2021061502

It should be understood that the first mapping shown in table 1 is given based on data 1 and data 2 in the above example, but this should not limit the present application in any way. The present application does not limit the specific content in the first mapping relationship.

Taking the mapping relationship between the time index 1024 and the time 2021061501 as an example, assuming that the preset time duration is 1 day, when the data 1 with the time 2021061501 is stored in the recycle bin at 6 month, 16 day 1 in 2021, the time duration reaches the preset time duration, the recycle bin management server determines the time index 1024 of the data 1 from the time 2021061501 according to the first mapping relationship, and further generates a first instruction according to the time index 1024 of the data 1.

The first mapping relationship may be recorded in a metadata server, for example, metadata of a recycle bin. The recycle bin management server can read related data in the metadata server through an access interface provided by the metadata server, further determine a time index of the first data, and further generate a first instruction according to the time index of the first data. The storage space occupied by the data identified as abnormal in the recycle bin does not account for the first tenant's billing space. That is, the storage space occupied by the expired uncleaned data is not taken into account in the charging space of the tenant. In other words, the tenant does not need to pay extra for stale uncleaned data. In contrast, the storage space occupied by the data identified as normal in the recycle bin is taken into account in the charging space of the first tenant, to which the tenant needs to pay. Here, the normal state data is data that has not expired in the recycle bin, that is, data that has not been stored in the recycle bin for a predetermined time.

In step 502, the recycle bin management server sends a first instruction to the metadata server. Accordingly, the metadata server receives a first instruction from the recycle bin management server.

In step 503, the metadata server identifies the first data as abnormal based on the first instruction.

The recycle bin management server may trigger the metadata server to identify the first data as the abnormal state by sending a first instruction to the metadata server.

Optionally, step 503 comprises: the metadata server identifies the state of the first data in the second mapping relation as an abnormal state based on the first instruction, the second mapping relation comprises a corresponding relation between at least one time index and at least one state, the at least one state comprises a normal state and an abnormal state, and the storage space occupied by the data identified as the normal state in the recycle bin is counted into the charging space of the first tenant.

An example, the second mapping is a mapping of time index 1024 to state. Before the metadata server receives the first instruction, the second mapping relationship of data 1 is: time index 1024, state "normal"; after the metadata server receives the first instruction, the second mapping relation of the data 1 is changed into: time index 1024, state "abnormal state".

Table 2 shows an example of the second mapping relationship.

TABLE 2

Time index	Status of state
		1024	Abnormal state
1025	Normal state

It should be understood that the first mapping relationship and the second mapping relationship may be stored in two independent mapping relationships or may be combined into one mapping relationship for storage. For example, table 3 shows another example of the first mapping relationship and the second mapping relationship.

TABLE 3

Time index	Status of state	Time
			1024	Abnormal state	2021061501
1025	Normal state	2021061502

Optionally, the first instructions further indicate an occupancy of the storage space by the first data; after step 501, the method further comprises: and the metadata server excludes the occupation amount of the data to be cleaned to the storage space from the occupation amount of the file system and the recycle bin of the first tenant to the storage space to obtain the charging space of the first tenant.

Here, the data to be cleaned may include the first data whose expiration is determined in the above-described step 501, and the data whose expiration has been determined previously but has not been cleaned. Since the data are stored in the recycle bin for a time that reaches or exceeds a preset time, the data should be cleared. However, due to the introduction of the mechanism introduced by the present application, the metadata server cleans the system overhead only when the system overhead meets a preset condition, and therefore the storage time in the recycle bin may exceed or exceed a preset time period. In order to avoid unnecessary expense for the tenants, the occupation amount of the storage space by the data to be cleaned is excluded from the charging space of the first tenant. That is, the charging space of the tenant is storage space occupied by the file system of the tenant + storage space occupied by the recycle bin-storage space occupied by the data to be cleaned.

An example, the unit of storage space is Gigabytes (GB). If the first instruction indicates that the data 1 occupies 1GB of the storage space; the file system of the first tenant occupies 10GB, the recycle bin occupies 5G, and the metadata processor counts the data to be cleaned and occupies 2GB before receiving the first instruction. After receiving the first instruction, the metadata processor may add up the storage space 1GB occupied by the data 1 to the storage space 2GB occupied by the previously counted data to be cleaned, to obtain 3 GB. The charging space of the first tenant is: 10GB +5GB-3 GB-12 GB.

Optionally, the abnormal state includes a frozen state and a clear state.

It should be understood that the frozen state and the clear state both belong to abnormal states, and the storage space occupied by the data identified as the frozen state or the clear state is not taken into account in the charging space. In the present embodiment, the frozen state and the clear state are used to distinguish the states before and after metering.

Illustratively, step 503 includes: the metadata server marks the state of the first data as a frozen state before excluding the occupation amount of the first data on the storage space from the occupation amounts of the file system and the recycle bin of the first tenant; and after the occupation amount of the first data on the storage space is eliminated from the occupation amounts of the file system and the recycle bin of the first tenant, the metadata server identifies the state of the first data as a cleaning state.

For example, after receiving the first instruction, the metadata server identifies the time index 1024 of data 1 as the frozen state in the second mapping relationship before performing the action of cleaning data 1. And the metadata server accumulates the storage space occupied by the data 1 to the storage space occupied by the data to be cleaned, and after updating the charging space, the time index 1024 of the data 1 in the second mapping relation is marked as a cleaning state.

Therefore, the atomicity of the processing process of the metadata processor on the first data can be ensured, and the phenomenon that the metering result of the charging space is wrong due to repeated accumulation of the storage space occupied by the first data can be avoided under the condition of requesting replay.

It should be understood that the frozen state and the cleaning state are only two states introduced for the convenience of distinguishing before and after billing, and should not constitute any limitation to the present application. The present application does not exclude the possibility of introducing other states to distinguish between the two states before and after charging.

In step 504, in case the overhead satisfies a preset condition, the first data identified as abnormal is cleaned up from the recycle bin.

Since a certain system overhead is required to clean data from the recycle bin, which affects the functions of creating, deleting, reading and writing data links in the storage system, the cleaning of data is preferably performed in the valley period of the storage system.

Whether the storage system is in the valley period or not can be judged according to the system overhead. If the overhead is low, it can be considered to be in the valley period. Therefore, the metadata server can monitor the current system overhead in real time, and in the case of low system overhead, the data which is identified as abnormal state is cleaned up from the recycle bin. Here, the data identified as the abnormal state includes first data.

As described above, in order to distinguish the states before and after metering, the frozen state and the purge state are introduced separately. It will be appreciated that, since the data needs to be metered into the billing space before being cleaned, the cleaned data is typically data in the cleaning state, with the introduction of the frozen state and the cleaning state.

Here, the overhead includes overhead of the foreground and overhead of the background. The system overhead of the foreground specifically includes system overhead caused by user access, for example, system overhead brought by operations such as writing, deleting, modifying, querying and the like; the overhead of the background may specifically include the aforementioned overhead caused by clearing the expired data in the recycle bin.

The metadata server can clean the data to be cleaned from the recycle bin under the condition that the system overhead of the foreground and/or the system overhead of the background are small.

Illustratively, the overhead of the foreground may be measured by the Input Output Per Second (IOPS) of the foreground, the Query Per Second (QPS) of the foreground, and so on. For example, a threshold is set for foreground IOPS or QPS, for example, foreground IOPS corresponds to a first preset threshold, and foreground QPS corresponds to a second preset threshold.

If the foreground IOPS is taken as a judgment basis, the metadata server can determine that the system overhead of the foreground is low under the condition that the foreground IOPS is lower than a first preset threshold; if the foreground QPS is used as a determination criterion, the metadata server may determine that the system overhead of the foreground is relatively small under the condition that the foreground QPS is lower than the second preset threshold.

It should be understood that foreground IOPS and foreground QPS are just two examples of overhead for characterizing the foreground, and the present application includes but is not limited to such.

The overhead of the background can be measured by the amount of data that is not merged. For example, a third preset threshold is set for the amount of data that is not merged. And if the uncombined data amount is lower than a third preset threshold, the metadata database determines that the system overhead of the background is low.

The metadata server can clear the data to be cleared from the recycle bin under the condition that the system overhead of any one of the foreground and the background is small, and can also clear the data to be cleared from the recycle bin under the condition that the system overhead of both the foreground and the background is small.

That is, the preset conditions include one or more of the following: the input and output quantity per second of the foreground is lower than a first preset threshold, the query rate per second of the foreground is lower than a second preset threshold, and the data quantity which is not combined in the memory is higher than a third preset threshold. In practical applications, which item or items are selected, and whether to add more preset conditions may be set by the operator, which is not limited in this application. The metadata server may execute the step 204 according to the state of each data in the second mapping relationship, or may execute the step 204 in response to a call from the recycle bin management server.

Optionally, before step 504, the method further comprises: the recycle bin management server generates a second instruction based on the N data to be cleaned, wherein the second instruction is used for indicating the cleaning of the N data to be cleaned, the N data to be cleaned comprise the first data, and N is a preset value; the recycle bin management server sends a second instruction to the metadata server.

As each time the recycle bin management server counts, one or more overdue and uncleaned data can be determined, and after the overdue and uncleaned data are determined, the data can be marked as abnormal data to be cleaned. If the instruction is sent separately for each data, a large signaling overhead may be caused, and therefore N cleaning requests of the data to be cleaned may be aggregated in one instruction and sent to the metadata server. That is, the recycle bin management server may generate a second instruction to instruct to clean the N data to be cleaned based on the N data to be cleaned. N may be a default of the system or may be set by a tenant, which is not limited in this application.

It should be noted that the data may include data in a file and data in a directory. A data may specifically refer to data in a file (or data under a file directory) or data in a directory.

It should be further noted that, the recycle bin management server identifies the first data as an abnormal state, and the recycle bin management server calls the metadata server to clean the first data, which may be regarded as two independent processes, and the two processes are not coupled to each other.

As an example, FIG. 6 is a flow diagram of a recycle bin management service and a metadata service. Wherein the recycle bin management service may be provided by a recycle bin management server, the metadata service may be provided by a metadata server, and the metadata server may provide the metadata service in response to a call by the recycle bin management server.

In the recycle bin management service process, firstly, a file system is selected for cleaning, then an hour directory is obtained, the directory in cleaning is extracted, then files and subdirectories under the directory to be cleaned are obtained, and finally, a metadata service is called to execute cleaning. In the metadata service process, a file/subdirectory cleaning request is received firstly, then whether cleaning can be carried out is judged according to the directory state of the file/subdirectory, whether the cleaning request can be executed is judged according to indexes such as QPS (quick Path manager) of the current foreground request, level 0 file number and the like, finally the file/subdirectory is cleaned, and a cleaning result is returned to a recycle bin management server. If the cleaning is successful, the metadata service can be repeatedly called to execute the cleaning in order to ensure the successful cleaning; if the system is busy and the cleaning fails, another file system cleaning is selected.

In the above step, the hour list may be the contents shown in table 1, table 2 or table 3.

Optionally, the second instruction carries data identifiers of N pieces of data to be processed. In an embodiment of the present application, the data identification may include: a time index node (inode), a parent directory index node, a parent directory entry (dentryid), and a file name of the data to be cleaned.

The data identification is of the form: < hour directory inode > # < original parent directory inode > < dentryid > < original filename >, wherein the hour directory inode is an example of a time inode having an hour as a granularity.

For example, data of data having a file name of "file 2" is identified as "1024 #23_27_ file 2". For another example, data of data having a file name of "dir 4" is identified as "1025 #20_23_ dir 4".

It should be appreciated that each data identifier may be used to uniquely identify a data and determine the location of that data in the storage space. Thus, based on the data identifications of the N data to be processed, the metadata server can determine which data need to be deleted and their locations.

The metadata server counts the input and output requests and query requests of the foreground and the uncombined data amount of the background. The input/output request and the query request of the foreground can be used for judging whether the current storage system is busy or not, and the number of files which are not merged in the background can be used for predicting the performance influence of the input/output request of the subsequent foreground on the storage system. Therefore, whether the storage system is in the valley period can be judged by the three preset conditions.

As mentioned above, data identified as an abnormal state cannot be accessed, and data identified as a normal state can be accessed.

Here "inaccessible" and "accessible" are for the tenant. If the time length of the data 1 existing in the recycle bin does not reach the preset time length, the data 1 is still stored in the recycle bin, and the data 1 can be accessed to the tenant; if the time length of the data 1 existing in the recycle bin reaches the preset time length, the data 1 is supposed to be in a cleared state for the tenant, and although the data 1 may not be cleared in the metadata server temporarily, the tenant cannot access the data 1.

Optionally, the method further comprises: the metadata server receives a query request, wherein the query request carries a directory structure index of second data and is used for requesting to query the second data; the metadata server determines a time index of the first data according to the directory structure index of the second data and a third mapping relation, wherein the third mapping relation comprises a one-to-one corresponding relation of at least one directory structure index and at least one time index; the metadata server determines the state of the second data according to the second mapping relation; the metadata server refuses the query request under the condition that the state of the second data is an abnormal state; or the metadata server searches the second data based on the directory structure index and the time index under the condition that the state of the second data is a normal state.

Table 4 below shows an example of the third mapping relationship. The third mapping shown in table 4 may be referred to as a directory structure table.

TABLE 4

Directory structure index	Time index
			2	1025
20	1025
		22	1025
23	1024，1025
		26	1024

The flow of the recycle bin directory structure query is shown in fig. 7, and specifically, the flow of the recycle bin directory structure query may include:

step 701, acquiring an hour directory for storing deleted files/subdirectories of the directory;

step 702, filtering the hour catalogue in the abnormal state;

step 703, inquiring files and directories in the normal state hour directory.

For example, assume that the directory structure indices of data 1 and data 2 are both "23". If the tenant inquires '23' in 2021, 6 months, 16 days and 1 days, the metadata server can determine that the time indexes are 1024 and 1025 through the third mapping relation, then inquire the second mapping relation, the state corresponding to the time index 1024 is an abnormal state, and the state corresponding to the time index 1025 is a normal state. Then data 1 with time index 1024 is filtered out and data 2 with time index 1025 is returned to the tenant.

Based on the method, the first data which is expired but not cleaned is identified as the abnormal state through the metadata server, the paid space of the tenant is not counted, and the data which is identified as the abnormal state is cleaned under the condition that the system overhead meets the preset condition, so that the strategy of a recycle bin for cleaning the expired data is optimized, the influence on the system performance is reduced, the occupied time of the tenant on the storage space is not additionally increased, and unnecessary expense brought to the tenant is avoided.

Fig. 8 is a schematic block diagram of a recycle bin management server according to an embodiment of the present application. The recycle bin management server 800 may be the recycle bin management server 111 of fig. 1, or a server having the same function as it. As shown in fig. 8, the recycle bin management server 800 may include a processing module 810 and a transmitting module 820. The processing module 810 may be configured to generate a first instruction according to a time index of first data, where the first data is data that is stored in a recycle bin of a first tenant for a preset duration and is not cleared, the first instruction is used to indicate that the first data is identified as an abnormal state, and a storage space occupied by the data identified as the abnormal state in the recycle bin is not included in a charging space of the first tenant; the sending module 820 may be configured to send a first instruction to a metadata server.

Optionally, the processing module 810 may be further configured to determine a time index of the first data according to the preset duration and the first mapping relationship; the storage space occupied by the data identified as the abnormal state in the recycle bin is not counted into the charging space of the first tenant, the first mapping relation comprises a corresponding relation between at least one time index and at least one time, and the time in the at least one time represents the time when the data under the corresponding time index is stored in the recycle bin.

Optionally, the processing module 810 may be further configured to generate a second instruction based on the N data to be cleaned, where the second instruction is used to instruct to clean the N data to be cleaned, the N data to be cleaned includes the first data, and N is a preset value; the sending module 820 may be further configured to send a second instruction to the metadata server.

Fig. 9 is a schematic block diagram of a metadata server provided in an embodiment of the present application. The metadata server 900 may be the metadata server 112 of fig. 1, or a server having the same functions as it. As shown in fig. 9, the metadata server 900 may include a receiving module 910 and a processing module 920. The receiving module 910 may be configured to receive a first instruction from the recycle bin management server, where the first instruction is used to indicate that first data in the recycle bin of a first tenant is identified as an abnormal state, where the first data is data that is stored in the recycle bin for a preset time period but is not cleaned, and a storage space occupied by the data identified as the abnormal state in the recycle bin is not counted into a charging space of the first tenant; the processing module 920 may be configured to identify the first data as abnormal based on the first instruction; and in the case that the system overhead meets a preset condition, clearing the first data which are identified as abnormal states from the recycle bin.

Optionally, the processing module 920 may be further configured to eliminate the occupied amount of the data to be cleaned on the storage space from the occupied amounts of the file system and the recycle bin of the first tenant on the storage space, so as to obtain a charging space of the first tenant; the data to be cleaned includes first data.

Optionally, the processing module 920 is further configured to identify the state of the first data as a frozen state before excluding the occupancy of the storage space by the first data from the occupancy of the storage space by the file system and the recycle bin of the first tenant; after the occupation amount of the first data on the storage space is excluded from the occupation amounts of the file system and the recycle bin of the first tenant on the storage space, the state of the first data is identified as a clean state.

Optionally, the processing module 920 may be further configured to, based on the second instruction, clean the N data to be cleaned from the recycle bin when the overhead meets a preset condition.

Optionally, the receiving module 910 is further configured to receive a query request, where the query request carries a directory structure index of the second data, and is used to request to query the second data; the processing module 920 may be further configured to determine a time index of the first data according to the directory structure index of the second data and a third mapping relationship, where the third mapping relationship includes a one-to-one correspondence relationship between at least one directory structure index and at least one time index; determining the state of the second data according to the second mapping relation; rejecting the query request when the state of the second data is an abnormal state; or under the condition that the state of the second data is a normal state, searching the second data based on the directory structure index and the time index.

Fig. 10 is a schematic structural diagram of a data processing apparatus provided in the present application, which may be a recycle bin management server in one embodiment, and a metadata server in another embodiment. As shown in fig. 10, the data processing apparatus 1000 may include at least one processor 1010 for implementing the functions of data processing in the methods provided herein. For details, reference is made to the detailed description in the method example, which is not repeated herein.

The data processing apparatus 1000 may also include a memory 1020 for storing program instructions and/or data. The memory 1020 is coupled to the processor 1010. The coupling in this application is an indirect coupling or communication connection between devices, units or modules, and may be in an electrical, mechanical or other form, and is used for information interaction between the devices, units or modules. The processor 1010 may operate in conjunction with the memory 1020. Processor 1010 may execute program instructions stored in memory 1020. At least one of the at least one memory may be included in the processor.

The data processing apparatus 1000 may also include a communication interface 1030 for communicating with other devices via a transmission medium, such that the apparatus used in the data processing apparatus 1000 may communicate with other devices. The communication interface 1030 may be, for example, a transceiver, an interface, a bus, a circuit, or a device capable of performing transceiving functions. Processor 1010 may transmit and receive data and/or information using communications interface 1030 and may be used to implement the method of data processing in the corresponding embodiment of fig. 5.

The specific connection medium among the processor 1010, the memory 1020, and the communication interface 1030 is not limited in this application. In fig. 10, the processor 1010, the memory 1020, and the communication interface 1030 are connected by a bus 1040. The bus 1040 is shown in fig. 10 by a thick line, and the connection between other components is merely illustrative and not intended to be limiting. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 10, but this is not intended to represent only one bus or type of bus.

In the embodiments of the present application, the processor may be a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, and may implement or perform the methods, steps, and logic blocks disclosed in the present application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the present application may be embodied directly in a hardware processor, or in a combination of the hardware and software modules in the processor.

According to the method provided by the present application, the present application further provides an electronic device, which includes a processor for calling a computer program to make the electronic device execute the method executed by the recycle bin management server and/or the method executed by the metadata server in the embodiments shown in fig. 5 to 7. Optionally, the electronic device further comprises a memory for storing said computer program.

According to the method provided by the present application, there is also provided a computer-readable storage medium storing a computer program which, when run on a computer, causes the computer to perform the method performed by the recycle bin management server or the method performed by the metadata server in the embodiments shown in fig. 5 to 7.

In accordance with the methods provided herein, the present application also provides a computer program product comprising: computer program code. The computer program code, when run on a computer, causes the computer to perform the method performed by the recycle bin management server or the method performed by the metadata server in the embodiments shown in fig. 5 to 7.

The solutions provided in the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions described in accordance with the present application are generated, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, a network appliance, a terminal device or other programmable apparatus. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website, computer, server, or data center to another website, computer, server, or data center via wire, such as coaxial cable, fiber optic, Digital Subscriber Line (DSL), or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium such as a Digital Video Disc (DVD), or a semiconductor medium, among others.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data processing method applied to a storage system, wherein the storage system is used for providing storage services for one or more tenants, and the storage system comprises a recycle bin management server and a metadata server, and the method comprises the following steps:

the metadata server receives a first instruction from the recycle bin management server, wherein the first instruction is used for indicating that first data in a recycle bin of a first tenant is identified as an abnormal state; the first data is data which is stored in the recycle bin for a preset time and is not cleaned, and the storage space occupied by the data which is identified as an abnormal state in the recycle bin is not counted into the charging space of the first tenant;

the metadata server identifies the first data as abnormal based on the first instruction;

The metadata server clears the first data identified as abnormal from the recycle bin when the system overhead meets a preset condition.

2. The method of claim 1, wherein the first instruction carries a time index for the first data, the time index being related to a time at which the first data was placed in the recycle bin; and

the method further comprises the following steps:

the recycle bin management server determines a time index of the first data according to the preset duration and a first mapping relation, wherein the first mapping relation comprises a corresponding relation between at least one time index and at least one time, and the time in the at least one time represents the time when the data under the corresponding time index is put into the recycle bin;

and the recycle bin management server generates the first instruction according to the time index of the first data.

3. The method of claim 1, wherein the first instruction further indicates an occupancy of the storage space by the first data;

the method further comprises the following steps:

the metadata server excludes the occupation amount of the data to be cleaned on the storage space from the occupation amounts of the file system and the recycle bin of the first tenant on the storage space to obtain a charging space of the first tenant; the data to be cleaned comprises the first data.

4. The method of claim 3, wherein the abnormal state comprises a frozen state and a cleared state, and

the metadata server identifies the state of the first data as an abnormal state based on the first instruction, including:

the metadata server identifies the state of the first data as a frozen state before excluding the occupancy of the storage space by the first data from the occupancy of the storage space by the file system and the recycle bin of the first tenant;

the metadata server identifies the state of the first data as a clean state after excluding the occupancy of the storage space by the first data from the occupancy of the storage space by the file system and the recycle bin of the first tenant.

5. The method of claim 1, wherein the method further comprises:

the recycle bin management server generates a second instruction based on N data to be cleaned, wherein the second instruction is used for indicating to clean the N data to be cleaned, the N data to be cleaned comprise the first data, and N is a preset value;

the recycle bin management server sends the second instruction to the metadata server; and

The metadata server cleans the first data identified as abnormal state from the recycle bin under the condition that the system overhead meets a preset condition, and the method comprises the following steps:

and the metadata server cleans the N data to be cleaned from the recycle bin based on the second instruction under the condition that the system overhead meets a preset condition.

6. The method of claim 5, wherein the second instruction carries data identifiers of the N data to be cleaned, and each data identifier comprises a time index node, a parent directory entry and a file name of the data to be cleaned.

7. The method of claim 1, wherein the preset conditions include one or more of:

the input and output quantity of the foreground per second is lower than a first preset threshold,

the query rate per second of the foreground is lower than a second predetermined threshold, an

The amount of data not merged in the memory is lower than a third preset threshold.

8. The method of any of claims 1 to 7, wherein data identified as abnormal cannot be accessed and data identified as normal can be accessed.

9. The method of claim 8, wherein the method further comprises:

The metadata server receives a query request, wherein the query request carries a directory structure index of second data and is used for requesting to query the second data;

the metadata server determines a time index of the first data according to a directory structure index of the second data and a third mapping relation, wherein the third mapping relation comprises a one-to-one correspondence relation between at least one directory structure index and at least one time index;

the metadata server determines the state of the second data according to the second mapping relation;

the metadata server refuses the query request under the condition that the state of the second data is an abnormal state; or

And the metadata server searches the second data based on the directory structure index and the time index under the condition that the state of the second data is a normal state.

10. A storage system for providing data storage services for one or more tenants, the storage system comprising a recycle bin management server and a metadata server; wherein the content of the first and second substances,

the recycle bin management server is used for sending a first instruction to the metadata server, wherein the first instruction is used for indicating that first data in a recycle bin of a first tenant is identified to be in an abnormal state; the first data is data which is stored in the recycle bin for a preset time and is not cleaned, and the storage space occupied by the data which is identified as an abnormal state in the recycle bin is not counted into the charging space of the first tenant;

The metadata server is used for identifying the first data as an abnormal state based on the first instruction; and the data clearing device is used for clearing the first data from the recycle bin based on the second instruction under the condition that the system overhead meets a preset condition.

11. An electronic device, comprising a processor configured to invoke a computer program to cause the electronic device to implement the method of any one of claims 1 to 9.

12. A computer-readable storage medium, in which a computer program is stored which, when executed, implements the method of any one of claims 1 to 9.