CN111061689A

CN111061689A - File expiration processing method and device for object storage system and electronic equipment

Info

Publication number: CN111061689A
Application number: CN201911284504.XA
Authority: CN
Inventors: 刘鹏程
Original assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Current assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date: 2019-12-13
Filing date: 2019-12-13
Publication date: 2020-04-24

Abstract

The invention discloses a file expiration processing method of an object storage system, which is applied to a server side, wherein a metadata base of the server side comprises a metadata field for defining file expiration time, and the method comprises the following steps: receiving a file storage request sent by a client, wherein the storage request comprises first file expiration time of a file; adding the first file expiration time to the meta information field and storing the file; according to the expiration time of the first file corresponding to the meta-information field, deleting the corresponding expired file regularly; when an access request of a client to a file is received, if the request time of the access request is determined to exceed the first file expiration time, determining whether the file corresponding to the access request is deleted; and if the expired file corresponding to the access request is determined not to be deleted, deleting the expired file. The invention provides great flexibility to delete data, and simultaneously reduces the management and maintenance burden of a user, thereby improving the deletion efficiency.

Description

File expiration processing method and device for object storage system and electronic equipment

Technical Field

The present invention relates to the field of storage, and in particular, to a file expiration processing method and apparatus for an object storage system, and an electronic device.

Background

The object storage is distributed storage facing to the internet, supports reading and writing at any time and any place through a hypertext Transfer protocol/hypertext Transfer security protocol (Http/Http), supports a standard Rest API (Representational State Transfer Interface State Transfer Application program Interface), provides an infinite storage space for a client through a flat storage architecture, is a high-reliability, non-upper-limit and low-cost storage mode, and is suitable for storage of mass data.

In the object storage system, in order to save storage resources, some outdated files and the like need to be deleted, and in the related art, when a large quantity of outdated files are deleted, the deleting process is complex, the efficiency is low, and the maintenance cost is high.

Disclosure of Invention

An object of the present invention is to provide a new technical solution for a file expiration processing method of an object storage system, so as to solve at least some of the technical problems described above.

According to a first aspect of the present invention, there is provided a file expiration processing method for an object storage system, applied to a server, the method including:

receiving a file storage request sent by a client, wherein the storage request comprises first file expiration time of a file;

adding the first file expiration time to the meta information field and storing the file;

according to the expiration time of the first file corresponding to the meta-information field, deleting the corresponding expired file regularly;

when an access request of a client to a file is received, determining whether the request time of the access request exceeds the first file expiration time of the file corresponding to the access request;

if the request time of the access request exceeds the first file expiration time of the file corresponding to the access request, determining whether the file corresponding to the access request is deleted; and

if the expired file corresponding to the access request is determined not to be deleted, the expired file is deleted.

Optionally, the method further comprises:

receiving a modification request of the file expiration time sent by a client, wherein the modification request comprises a second file expiration time of a file corresponding to the modification request;

and updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time.

Optionally, the method further comprises:

before updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time, determining whether the request time of the modification request exceeds the first file expiration time of the file corresponding to the modification request; and the number of the first and second groups,

updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time, including:

and if the request time of the modification request is determined not to exceed the first file expiration time of the file corresponding to the modification request, updating the first file expiration time of the file corresponding to the modification request in the meta-information field according to the second file expiration time.

Optionally, the method further comprises:

when the file storage request sent by the client does not contain the first file expiration time of the file, determining the expiration time of the file according to a preset expiration time setting rule.

Optionally, periodically deleting the corresponding expired file according to the expiration time of the first file corresponding to the meta information field, including:

aiming at any file, judging whether the file reaches the first file expiration time of the file or not;

if the first file expiration time of the file is reached, storing the address information of the file in a deletion queue; and

and reading the message in the deletion queue through the deletion process so as to delete the file.

Optionally, receiving a file storage request sent by a client includes:

aiming at any file, receiving a plurality of file storage requests sent by a client, wherein each file storage request comprises first file expiration time of the file; and the number of the first and second groups,

adding a first file expiration time to the meta information field and storing the file, including:

and adding the first file expiration time in the file storage request of the last received file into the meta information field according to the time sequence of receiving the plurality of file storage requests, and storing the file.

According to a second aspect of the present invention, there is provided a file expiration processing method for an object storage system, applied to a client, the method including:

generating a file storage request containing a first file expiration time of a file;

and sending the storage request to a server side so that the server side adds the first file expiration time of the file into a meta-information field which is stored in a meta-database and used for defining the file expiration time, and periodically deleting the corresponding expired file according to the first file expiration time.

Optionally, generating a file storage request including a first file expiration time of the file includes:

and writing the expiration time of the file in the request header of the file storage request, so that the first file expiration time of the file is contained in the file storage request.

According to a third aspect of the present invention, there is provided a file expiration processing apparatus for an object storage system, which is applied to a server side, a meta database of the server side including a meta information field for defining a file expiration time, the apparatus including:

the first receiving module is used for receiving a file storage request sent by a client, wherein the storage request comprises first file expiration time of a file;

the storage module is used for adding the expiration time of the first file into the meta-information field and storing the file;

the first deleting module is used for deleting the corresponding expired files regularly according to the expiration time of the first files corresponding to the meta-information fields;

the first determining module is used for determining whether the request time of the access request exceeds the first file expiration time of the file corresponding to the access request when the access request of the client to the file is received;

the second determining module is used for determining whether the file corresponding to the access request is deleted or not under the condition that the first determining module determines that the request time of the access request exceeds the first file expiration time of the file corresponding to the access request; and

and the second deleting module is used for deleting the expired file corresponding to the access request under the condition that the second determining module determines that the expired file is not deleted.

Optionally, the apparatus further comprises:

the second receiving module is used for receiving a modification request of the file expiration time sent by the client, wherein the modification request comprises a second file expiration time of a file corresponding to the modification request; and

and the updating module is used for updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time.

Optionally, the apparatus further comprises:

a third determining module, configured to determine whether the request time of the modification request exceeds the first file expiration time of the file corresponding to the modification request before updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time;

the updating module is further used for updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time when the third determining module determines that the request time of the modification request does not exceed the first file expiration time of the file corresponding to the modification request.

Optionally, the apparatus further comprises:

the setting module is used for determining the expiration time of the file according to a preset expiration time setting rule when the first file expiration time of the file is not included in the file storage request sent by the client.

Optionally, the file expiration processing apparatus applied to the server side for the object storage system further includes: a fourth determining module and an address information storing module.

When the first deleting module deletes the corresponding expired file periodically according to the first file expiration time corresponding to the meta-information field, the method comprises the following steps:

for any file, the fourth determining module is configured to determine whether the file has reached a first file expiration time of the file;

if the first file expiration time of the file is reached, the address information storage module stores the address information of the file in a deletion queue; and

and reading the message in the deletion queue through a deletion process, and deleting the file by the first deletion module.

Optionally, the receiving, by the first receiving module, a file storage request sent by a client includes:

aiming at any file, a first receiving module receives a plurality of file storage requests sent by a client, and each file storage request comprises first file expiration time of the file; and the number of the first and second groups,

the storage module adds a first file expiration time to the meta-information field and stores the file, including:

and according to the time sequence of receiving the file storage requests, the storage module adds the first file expiration time in the file storage request of the file which is received last to the meta information field and stores the file.

According to a fourth aspect of the present invention, there is provided a file expiration processing apparatus for an object storage system, applied to a client, the apparatus comprising:

the generating module is used for generating a file storage request containing the first file expiration time of the file; and

the sending module is used for sending the storage request to the server side so that the server side adds the first file expiration time of the file into a meta-information field which is stored in a meta-database and used for defining the file expiration time, and deletes the corresponding expired file regularly according to the first file expiration time.

the generation module writes the expiration time of the file in a request header of a file storage request so that the file storage request comprises the first file expiration time of the file.

According to a fifth aspect of the present invention, there is provided an electronic apparatus comprising:

file expiration processing means for the object storage system; alternatively, the first and second electrodes may be,

a processor and a memory for storing executable instructions for controlling the processor to perform a method of file expiration handling for an object storage system according to the first aspect of the invention.

According to a sixth aspect of the present invention, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the file expiration processing method of the object storage system according to the first and second aspects of the present invention.

According to the embodiment of the invention, the metadata base of the server side of the object storage system comprises the metadata field for defining the file expiration time, so that a user only needs to set the file expiration time when uploading the file, or the server side of the object storage system can delete the expired file according to the file expiration time by modifying the file expiration time after uploading the file, thereby the deletion of the expired file is more flexible, and the burden of managing and maintaining the expired file by the user can be greatly relieved by timely deleting the expired file. Since the expiration time of the file is determined when the user uploads the file, the server side of the object storage system can uniformly schedule and intensively place the deleted data in the same time period, batch deletion can be performed on the bottom server after the deletion time is reached, and random deletion is converted into sequential deletion, so that the deletion efficiency is improved. In addition, due to asynchronous deletion operation of the server, some expired files may not be deleted until the expiration time is reached, and through the embodiment of the invention, the expired files are deleted in time when the access request of the user is received, so that delayed deletion of the expired files can be realized under the condition that the client is not aware, the expired files of the client are ensured not to be accessed by the user of the client any more, and the expired access safety of the expired files is improved.

Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.

Fig. 1 is a block diagram showing a hardware configuration of an electronic device that can implement an embodiment of the present invention.

Fig. 2 is a flowchart illustrating steps of a file expiration processing method applied to a server-side object storage system according to an embodiment of the present invention.

Fig. 3 is a flowchart illustrating steps of a file expiration processing method for an object storage system, applied to a client according to an embodiment of the present invention.

Fig. 4 is a flowchart of deleting an expired file in a periodic deletion manner according to an embodiment of the present invention.

Fig. 5 is a flowchart of deleting an expired file in a lazy delete mode when a request is made for accessing a file according to an embodiment of the present invention.

Fig. 6 is a block diagram illustrating a file expiration processing apparatus applied to a server-side object storage system according to an embodiment of the present invention.

Fig. 7 is a block diagram illustrating a file expiration processing apparatus for an object storage system, applied to a client according to an embodiment of the present invention.

Fig. 8 is a block diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

Fig. 1 is a block diagram showing a configuration of hardware of an electronic apparatus 1000 that can implement an embodiment of the present invention.

Fig. 1 is a block diagram showing a hardware configuration of an electronic apparatus 1000 that can implement an embodiment of the present invention. The electronic device 1000 may be a unitary server or a distributed server across multiple computers or computer data centers. The server may be of various types, such as, but not limited to, a storage server, a cloud database server, a cloud computing server, a cloud management server, a web server, a news server, a mail server, a message server, an advertisement server, a file server, an application server, an interaction server, a storage server, a database server, a proxy server, or the like of a distributed system. In some embodiments, each server may include hardware, software, or embedded logic components or a combination of two or more such components for performing the appropriate functions supported or implemented by the server. For example, a server, such as a blade server, a cloud server, etc., or may be a server group consisting of a plurality of servers, which may include one or more of the above types of servers, etc.

The electronic device 1000 may also be a laptop, desktop computer, cell phone, tablet computer, etc. implementing embodiments of the present invention.

As shown in fig. 1, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit CPU, a microprocessor MCU, or the like. The memory 1200 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 is capable of wired or wireless communication, for example, and may specifically include WiFi communication, bluetooth communication, 2G/3G/4G/5G communication, and the like. The display device 1500 is, for example, a liquid crystal display panel, a touch panel, or the like. The input device 1600 may include, for example, a touch screen, a keyboard, a somatosensory input, and the like. A user can input/output voice information through the speaker 1700 and the microphone 1800.

The electronic device shown in fig. 1 is merely illustrative and is in no way meant to limit the invention, its application, or uses. In an embodiment of the present invention, the memory 1200 of the electronic device 1000 is configured to store instructions, where the instructions are configured to control the processor 1100 to perform an expired file deletion method for any object storage provided by the embodiment of the present invention. It will be appreciated by those skilled in the art that although a plurality of means are shown for the electronic device 1000 in fig. 1, the present invention may relate to only some of the means therein, e.g. the electronic device 1000 may relate to only the processor 1100 and the storage means 1200. The skilled person can design the instructions according to the disclosed solution. How the instructions control the operation of the processor is well known in the art and will not be described in detail herein.

In an embodiment of the present invention, a file expiration processing method for an object storage system is provided, which is applied to a server side.

Referring to fig. 2, a flowchart illustrating a file expiration processing method for an object storage system according to an embodiment of the present invention is shown, where a metadata database of a server includes a metadata field for defining an expiration time of a file.

As shown in fig. 2, a file expiration processing method for an object storage system, which is applied to a server side in an embodiment of the present invention, includes the following steps:

step 202, receiving a file storage request sent by a client, wherein the storage request comprises a first file expiration time of the file;

step 204, adding the expiration time of the first file into the meta information field and storing the file;

step 206, according to the expiration time of the first file corresponding to the meta-information field, deleting the corresponding expired file periodically;

step 208, when receiving an access request of a client to a file, determining whether the request time of the access request exceeds the first file expiration time of the file corresponding to the access request;

step 210, if the request time of the access request is determined to exceed the first file expiration time of the file corresponding to the access request, determining whether the file corresponding to the access request is deleted; and

if the expired file corresponding to the access request is determined not to be deleted, the expired file is deleted, step 212.

In this embodiment, a server (storage server) in the object storage system may use a non-relational Structured Query Language (NoSQL) database to record a meta information field of a file, where the meta information field may include information such as a last modification time of the file, a file size, and the like. In the embodiment of the present application, a meta information field for defining the file expiration time may also be included in the meta database on the server side.

The client uploads a file to the server and sends a file storage request, and in step 202, the server receives the file storage request sent by the client, where the file storage request includes a first file expiration time of the file. The client side sets the expiration time of the file, the expiration time can be set from the granularity of the file instead of setting an expiration policy on the storage server, and the client side can flexibly specify the deletion policy conveniently.

In one example, the client may set an expiration time of a file in a request header of a file upload storage request through a PUT interface (an interface representing that the client sends data to the server or the server receives data) when the file is uploaded, where the interface is defined as follows:

PUT/{ObjectKey}？expire HTTP/1.1

Host:{BucketName}.{endpoint}

Date:{date}

x-kss-meta-expire:2592000

Authorization:{SignatureValue}

the PUT interface carries a custom request header, x-kss-meta-expire, which is the storage time of the file. Only files with TTL greater than 0 are allowed to be set, and if expire is set, lazy deletion will be triggered and returned to the client 404.

In some embodiments, multiple file storage requests are generated for the same file and sent to the server, as multiple modifications may occur by the client to the file within a short period of time. Based on this, the receiving of the file storage request sent by the client in step 202 may include:

aiming at any file, receiving a plurality of file storage requests sent by a client, wherein each file storage request comprises first file expiration time of the file; and

adding a first file expiration time to a meta-information field and storing the file, including:

and adding the expiration time of the first file in the file storage request of the file received last into the meta information field according to the time sequence of receiving the plurality of file storage requests, and storing the file.

That is, when a plurality of file storage requests sent by the client are received for any file, the server finally stores the first file expiration time in the last received file storage request in the meta information field according to the time sequence of receiving the plurality of file storage requests, and stores the file. For example, for file a, four file storage requests, namely a1, a2, a3, a4 and the like are received in chronological order, namely a4 is the last received file storage request, so that the first file expiration time in a4 is taken as the expiration time of the a file and is stored in the meta information field, and the a file is stored at the same time.

Then, in step 204, the server adds the expiration time of the received first file to the meta information field and stores the file.

Further, in step 206, the server may delete the expired file according to the first file expiration time corresponding to the meta information field. Illustratively, whether the current time reaches the first file expiration time of a certain file in the meta-information field is judged, if yes, the file can be determined to be an expired file, and a deletion operation is performed on the expired file. In some embodiments, the first file expiration time of any file may be determined according to the traffic condition of the server, for example, the first file expiration time may set an idle time period of the server (e.g., every morning), such as scanning a metadata database every morning, and if a certain file is found to be expired, the stored data of the file that has expired is deleted.

In some embodiments, for step 206, it may be implemented as: aiming at any file, judging whether the file reaches the first file expiration time of the file or not; if the first file expiration time of the file is reached, storing the address information of the file in a deletion queue; and the message in the deletion queue can be read by the deletion process to delete the file. The determination of whether the file has reached the first file expiration time of the file may be performed as follows:

when the file expiration time is a time interval, that is, a time period, the remaining lifetime TTL of the file is (file upload time + file expiration time) -the current access request time of the file, and when the remaining lifetime TTL of the file is less than or equal to 0, it is determined that the file has reached the first file expiration time of the file.

The remaining time-to-live of the file decreases with time, and the TTL of the file can be queried through a GET interface (which represents an interface for sending a request for data to a database), and an example of querying the TTL of the file using the GET interface is as follows:

GET/{ObjectKey}？ttl HTTP/1.1

Host:{BucketName}.{endpoint}

Date:{date}

Authorization:{SignatureValue}

an example response to the query request is:

HTTP/1.1 200 OK

Date:Fri,26Dec 2014 07:14:18 GMT

Last-Modified:Sun,1Jan 2009 12:00:00 GMT

Content-Length:124

Content-Type:text/plain

Connection:close

Server:Tengine

<ttl>

250000000

</ttl>

meanwhile, the file expiration time can also be checked by using a GET interface, and one example is as follows:

GET/{ObjectKey}？expire HTTP/1.1

Host:{BucketName}.{endpoint}

Date:{date}

x-kss-meta-expire:2592000

Authorization:{SignatureValue}

in this embodiment of the present application, it is considered that since the periodic deletion is asynchronous deletion, the deletion time cannot be guaranteed, and in order to achieve the effect of deleting the file at the client regularly, the server may perform a deletion operation when the expired file is requested, that is, perform step 208 to step 212: when an access request of a client to a file is received, determining whether the request time of the access request exceeds the first file expiration time of the file corresponding to the access request, if the request time of the access request exceeds the first file expiration time of the file corresponding to the access request, further determining whether the file corresponding to the access request is deleted, and if the request time of the access request exceeds the first file expiration time of the file corresponding to the access request, deleting the expired file. Based on the embodiment, the client side can not acquire the expired files, backlog caused by untimely deletion of the deletion queue is avoided, and data security is further improved.

According to the method and the device, when the request time of the access request is determined to exceed the first file expiration time of the file corresponding to the access request, and meanwhile, the expired file corresponding to the access request is determined not to be deleted, the expired file is deleted. And if the request time of the access request does not exceed the first file expiration time of the file corresponding to the access request, the file is not an expired file, deletion is not executed, and the normal access request of the file is executed. In addition, if it is determined that the request time of the access request exceeds the first file expiration time of the file corresponding to the access request, but the file is deleted, prompt information indicating that the file is deleted or cannot be accessed may be returned to the client, for example, return 404.

After the file is uploaded, the server side can also modify the file expiration time of the file:

specifically, a server receives a modification request of file expiration time sent by a client, wherein the modification request comprises second file expiration time of a file corresponding to the modification request; the server side can update the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time, so that the modification of the file expiration time can be realized.

In an example of modifying the file expiration time after uploading the file, an interface for modifying the file expiration time by the client through the PUT interface is defined as follows:

PUT/{ObjectKey}HTTP/1.1

Host:{BucketName}.{endpoint}

Date:{date}

x-kss-meta-expire:2592000

Authorization:{SignatureValue}

it should be noted that before updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time, it is determined whether the request time of the modification request has exceeded the first file expiration time of the file corresponding to the modification request, and updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time includes:

If the request time of the modification request is determined to exceed the first file expiration time of the file corresponding to the modification request, the modification request for the file expiration time of the file triggers deletion of the expired file because the file is an expired file. The corresponding operations as in steps 208 through 212 may continue.

In addition, when the file storage request sent by the client does not contain the first file expiration time of the file, the server side can determine the expiration time of the file according to a preset expiration time setting rule.

Specifically, determining the file expiration time according to the preset expiration time setting rule may include, but is not limited to, at least one of the following four cases:

firstly, when a file storage request sent by a client does not contain the first file expiration time of a file, a server sets the expiration time of the file to be never expired.

Secondly, when the file storage request sent by the client does not contain the first file expiration time of the file, the server determines the expiration time of the file according to the expiration times of the same or similar files. For example, the expiration time of the file may be determined based on user habits of the file, based on user habits of the type of file, or based on a conventional expiration time of the type of file.

Thirdly, when the file storage request sent by the client does not contain the first file expiration time of the file, the server determines the expiration time of the file according to the conventional storage time of the storage server.

Fourthly, aiming at the files without the set expiration time, the server side presets a given time value, and the file expiration time is determined according to the time value. For example, if the time value of the file is set to 2 hours, the server of the object storage system starts to calculate the time when the file is received, and the expiration time of the file is obtained after 2 hours.

In the following, an embodiment of the present invention for periodically deleting the expired files according to the expiration time of the first file corresponding to the meta-information field is described with reference to fig. 4,

As shown in fig. 4, in some embodiments, the file expiration time is set in the idle period of the server, and deleting the expired file by means of periodic deletion mainly includes:

first, the regular deletion program is started, and the regular deletion program starts to run (step 402).

Next, it is determined whether the server side of the object storage system is idle (step 404), for example, whether the current time is in the morning of each day.

If the server side of the object storage system does not reach the idle time, it indicates that the server side does not meet the starting condition of the timed deletion program, and needs to continue to wait for the starting condition of the timed deletion program (step 406).

If the server side of the object storage system has reached the idle time, it continues to determine whether the load of the object storage system is lower than the set threshold (step 410), and if the load of the system is not lower than the set threshold, it indicates that the system itself is still in a busy state, and is still not suitable for executing the regular deletion program, so it continues to wait for the condition of starting the regular deletion program (i.e., enter step 406).

After step 406, it is determined whether a condition for starting the regular deletion program is satisfied, where the condition for starting the regular deletion program includes determining whether the system is idle and has a low load (step 408), and if the system is idle and has a low load, the condition for starting the regular deletion program is satisfied and the regular deletion program starts to run (step 402).

After determining whether the load of the target storage system is lower than the set threshold (step 410), if the load of the system is lower than the set threshold, the periodic deletion procedure is continuously executed, the metadata database is scanned, and the file system records meeting the deletion condition are added to the deletion queue (step 412).

Next, the delete queue is read and the data in the back-end storage system is deleted (step 414).

Meanwhile, after step 412, it is detected whether the file of the target storage system is completely scanned (step 416), and if so, the scanning is stopped (step 418).

Further, after step 412, it may be determined simultaneously whether the load of the target storage system exceeds a threshold (step 420), and if so, the scan is stopped, the location of the current scan is recorded (step 422), and the scan is stopped (step 418).

Since the way of deleting the expired file periodically in step 206 is asynchronous deletion, the deletion time cannot be guaranteed, and in order to achieve the effect of deleting the file immediately by the user, the object storage system of the present application deletes the expired file when the expired file is requested, which is defined as an inactive deletion way (i.e., step 208-step 212). In the lazy deletion mode, when a client accesses a request file, the server executes active deletion in time under the condition that the file which has reached the expiration time is not deleted in time. The access request for the file includes all operations such as setting a file or querying a file, and examples of the access request for the file include, for example, the above PUT interface or GET interface to set or query the file.

In the following, a specific embodiment of a way for deleting an expired file when a client access request to the file is received in the present invention is described with reference to fig. 5.

As shown in fig. 5, the lazy deletion mode deleting the expired files of the object storage system includes:

at step 502, an access request for a file is first received, where the access request for the file may include a storage request or query, an expiration modification, etc. as described above.

Then, step 504 is entered to determine whether the request time of the access request exceeds the first file expiration time of the file corresponding to the access request (corresponding to step 208).

In a specific embodiment of determining that the request time of the access request exceeds the first file expiration time of the file corresponding to the access request, the TTL of the file may be calculated, where the remaining time to live TTL is (file upload time + file expiration time) -the current request time of the file (i.e., the request time of the access request), and if TTL is less than or equal to 0, it is determined that the current request time of the file is not within the set expiration time period, i.e., the file is an expired file, and then the file is deleted (corresponding to the foregoing steps 210 to 212), and the access operation on the file is returned to the client 404 (step 508).

If TTL is greater than 0, then it is determined that the current request time of the file is within the set expiration time period, and the file is a non-expired file, then according to the content of the access request, the corresponding file is returned or the corresponding operation is performed on the file (step 510).

The embodiment is executed on the server side, namely the server automatically deletes the file according to the specified expiration time when the client uploads the file, so that the management and maintenance of the client are reduced, and the maintenance cost of the client is reduced. Due to the fact that the expiration time of the files is determined when the files are uploaded by the client, the server can conduct unified scheduling and centralized placement on the deleted data in the same time period, batch deletion can be conducted on the server after the deletion time is reached, random deletion is converted into sequential deletion, and deletion efficiency is improved. And the server combines two deletion strategies of a regular deletion mode and an inert deletion mode, so that on one hand, the client cannot acquire the invalid file, and on the other hand, the extrusion of a deletion queue caused by asynchronous processing of regular deletion is avoided, and the expired file can still be accessed to influence data security after the expiration time. In the invention, the modification strategy of the expiration time is provided, and the client can modify the expiration time of the file before the expiration time is reached (namely the file is not deleted yet), so that the flexibility is higher. In other embodiments, if the expiration time is set to the idle period of the server, the load pressure of the server can be reduced, and the deletion efficiency can be improved.

In another embodiment of the present invention, a file expiration processing method for an object storage system is provided, which is applied to a client.

Please refer to fig. 3, which is a flowchart illustrating a file expiration processing method for an object storage system applied to a client according to an embodiment of the present invention.

As shown in fig. 3, a file expiration processing method for an object storage system, which is applied to a client according to an embodiment of the present invention, includes the following steps:

step 302, generating a file storage request containing the first file expiration time of a file; and

step 304, sending the storage request to a server, so that the server adds the first file expiration time of the file to a meta-information field stored in a meta-database and used for defining the file expiration time, and periodically deletes the corresponding expired file according to the first file expiration time.

In step 302, generating a file storage request including a first file expiration time of a file, includes:

and writing the expiration time of the file in a request header of the file storage request so that the file storage request comprises the first file expiration time of the file. The expiration time is set on the file granularity instead of the expiration policy on the storage server, so that the client can flexibly specify the deletion policy.

The expiration time of writing the file in the request header of the file storage request and other corresponding operations related to the client are described in detail in the above-mentioned file expiration processing method applied to the server for the object storage system, and are not described herein again.

In addition, the file expiration time can also be set as a date, and the same effect can be obtained by directly setting the file expiration date. For example, the upload time of the file is 2019, 9, 27, 12: 00, setting the file's expiration time to 60, then the file will be updated at 13, 9, 27 months, 2019: 00 expire, the file's expiration date may also be defined and set to 2019, 9, 27, 13: 00, and can also achieve corresponding effects. Under the condition that the expiration date of the file is defined, when the expiration time of the file is less than or equal to the current request time of the file, the file is indicated to be expired, and then the file is deleted.

According to the method and the device, a meta information field for defining the expiration time (expire) of the file is newly added in a cloud database of a server side of an object storage system, a file storage request sent by a client side is received, and the first file expiration time of the file contained in the storage request is added into the meta information field, so that the corresponding expired file can be deleted according to the file expiration time in the meta information field, and therefore the deletion strategy can be defined more flexibly and more directly.

In addition, compared with an expiration strategy of a storage space in the prior art, the method and the device cannot ensure the real-time property of deletion, and data can exist for a long time when the queue is overstocked, so that the data security is influenced.

Moreover, the file expiration time can be specified when the file is uploaded, so that the scheduling of the back end is facilitated, for example, the server can store files which are invalid at the same time together according to the expiration time in the file storage request, and the deleting efficiency can be improved. For the service of monitoring video and other scenes, when a user uploads a video file, the user already defines the deletion time of the file, and the object storage back-end server stores the file together according to the preset expiration time, so that the deletion efficiency can be improved. In the related art, the deletion strategy based on the life cycle of the storage space is set by the server after the file is uploaded, so that the hash of the file is discharged, and the deletion efficiency is low.

In another embodiment of the present invention, a file expiration processing apparatus for an object storage system, applied to a server side, is further provided, where a meta database of the server side includes a meta information field for defining a file expiration time.

As shown in fig. 6, the file expiration processing apparatus 2000 for an object storage system applied to a server side includes: a first receiving module 2100, a storage module 2200, a first deleting module 2300, a first determining module 2400, a second determining module 2500, and a second deleting module 2600.

A first receiving module 2100, configured to receive a file storage request sent by a client, where the storage request includes a first file expiration time of the file;

a storage module 2200 for adding the first file expiration time to the meta information field and storing the file;

a first deleting module 2300, configured to delete a corresponding expired file periodically according to a first file expiration time corresponding to the meta information field;

a first determining module 2400, configured to determine, when an access request of a client for a file is received, whether a request time of the access request exceeds a first file expiration time of the file corresponding to the access request;

a second determining module 2500, configured to determine whether the file corresponding to the access request is deleted if the first determining module determines that the request time of the access request exceeds the first file expiration time of the file corresponding to the access request; and

a second deleting module 2600, configured to delete the expired file corresponding to the access request if the second determining module determines that the expired file is not deleted.

In one example, the file expiration processing apparatus 2000 for an object storage system applied to a server side further includes: a second receiving module (not shown) and an updating module (not shown).

The second receiving module is used for receiving a modification request of the file expiration time sent by the client, wherein the modification request comprises the second file expiration time of the file corresponding to the modification request; and

In one example, the file expiration processing apparatus 2000 for an object storage system applied to a server side further includes: a third determining module (not shown in the figure).

and the updating module is further configured to update the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time when the third determining module determines that the request time of the modification request does not exceed the first file expiration time of the file corresponding to the modification request.

In one example, the file expiration processing apparatus 2000 for an object storage system applied to a server side further includes: a setup module (not shown in the figures).

In one example, the file expiration processing apparatus 2000 for an object storage system applied to a server side further includes: a fourth determination module (not shown in the figure) and an address information storage module (not shown in the figure).

In one example, the first receiving module 2100 receives a file storage request sent by a client, including:

the first receiving module 2100 receives a plurality of file storage requests sent by a client for any file, where each file storage request includes a first file expiration time of the file; and the number of the first and second groups,

the storage module 2200 adds a first file expiration time to the meta information field and stores the file, including:

In another embodiment of the present invention, a file expiration processing apparatus for an object storage system, which is applied to a client, is also provided.

As shown in fig. 7, the file expiration processing apparatus 3000 for an object storage system applied to a client includes: a generation module 3200 and a transmission module 3400.

A generating module 3200, configured to generate a file storage request including a first file expiration time of a file; and

a sending module 3400, configured to send the storage request to a server, so that the server adds the first file expiration time of the file to a meta information field stored in a meta database and used for defining file expiration time, and periodically deletes the corresponding expired file according to the first file expiration time.

In one example, generating a file storage request containing a first file expiration time for a file includes:

the generating module 3200 writes the expiration time of the file in a request header of the file storage request, so that the file storage request includes the first file expiration time of the file.

According to the method and the device, a meta information field for defining the expiration time (expire) of the file is newly added in a cloud database of a server side of an object storage system, a file storage request sent by a client side is received, and the first file expiration time of the file contained in the storage request is added into the meta information field, so that the corresponding expired file can be deleted according to the file expiration time corresponding to the meta information field, and therefore the deletion strategy can be defined more flexibly and more directly.

Moreover, the file expiration time can be specified when the file is uploaded, so that the scheduling of the back end is facilitated: several file data contents which are invalid at the same time are stored, so that the deleting efficiency can be improved. For the service of monitoring video and other scenes, when a user uploads a video file, the user already defines the deletion time of the file, and the object storage back-end server has preset expiration time and stores the file together, so that the deletion efficiency can be improved. In the prior art, the life cycle management strategy based on the storage space is usually set after the file is uploaded, so that the hash of the file is discharged, and the deletion efficiency is low.

According to still another embodiment of the present invention, there is provided an electronic apparatus, and the electronic apparatus 4000 may be the electronic apparatus 1000 shown in fig. 1. Fig. 8 is a block diagram of an electronic device according to an embodiment of the present invention. In one aspect, the electronic device 4000 may include the foregoing file expiration processing apparatus 2000 for an object storage system applied to a server or the foregoing file expiration processing apparatus 3000 for an object storage system applied to a client, for implementing the file expiration processing method for an object storage system according to any embodiment of the present invention.

On the other hand, as shown in fig. 8, the electronic device 4000 may include a memory 4200 and a processor 4400, the memory 4200 being configured to store executable instructions; the instruction is used to control the processor 4400 to execute the foregoing file expiration processing method for the object storage system applied to the server side and the file expiration processing method for the object storage system applied to the client side.

In this embodiment, the electronic device 4000 may be any electronic product having the memory 4200 and the processor 4400, such as a mobile phone, a tablet computer, a palmtop computer, a desktop computer, a notebook computer, a workstation, a game machine, and the like.

Finally, according to yet another embodiment of the present invention, there is also provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a file expiration processing method for an object storage system according to any embodiment of the present invention.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

The computer program instructions for carrying out operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A file expiration processing method for an object storage system is applied to a server side, and is characterized in that a meta-information field for defining file expiration time is included in a meta-database of the server side, and the method comprises the following steps:

if the request time of the access request is determined to exceed the first file expiration time of the file corresponding to the access request, determining whether the file corresponding to the access request is deleted; and

2. The method of claim 1, further comprising:

receiving a modification request of file expiration time sent by a client, wherein the modification request comprises second file expiration time of a file corresponding to the modification request; and

3. The method of claim 2, further comprising:

determining whether the request time of the modification request exceeds the first file expiration time of the file corresponding to the modification request before updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time; and the number of the first and second groups,

the updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time includes:

and if the request time of the modification request is determined not to exceed the first file expiration time of the file corresponding to the modification request, updating the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time.

4. The method of claim 1, further comprising:

when a file storage request sent by a client does not contain the first file expiration time of the file, determining the expiration time of the file according to a preset expiration time setting rule.

5. The method according to claim 1, wherein said periodically deleting the corresponding expired file according to the expiration time of the first file corresponding to the meta-information field comprises:

6. The method according to claim 1, wherein the receiving a file storage request sent by a client comprises:

the adding the first file expiration time to the meta information field and storing the file comprises:

and adding the first file expiration time in the file storage request of the file received last into the meta information field according to the time sequence of receiving the plurality of file storage requests, and storing the file.

7. A file expiration processing method for an object storage system is applied to a client, and is characterized by comprising the following steps:

generating a file storage request containing a first file expiration time of a file; and

and sending the storage request to a server side so that the server side adds the first file expiration time of the file to a meta-information field stored in a meta-database and used for defining the file expiration time, and periodically deleting the corresponding expired file according to the first file expiration time.

8. The method of claim 7, wherein generating a file storage request containing a first file expiration time for a file comprises:

and writing the expiration time of the file in a request header of the file storage request so that the file storage request comprises the first file expiration time of the file.

9. A file expiration processing device for an object storage system, applied to a server side, is characterized in that a meta-information field for defining file expiration time is included in a meta-database of the server side, and the device comprises:

the first deleting module is used for deleting the corresponding expired files periodically according to the expiration time of the first files corresponding to the meta-information fields;

the device comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for determining whether the request time of an access request exceeds the first file expiration time of a file corresponding to the access request when the access request of a client to the file is received;

a second determining module, configured to determine whether the file corresponding to the access request is deleted if the first determining module determines that the request time of the access request exceeds a first file expiration time of the file corresponding to the access request; and

10. The apparatus of claim 9, further comprising:

11. The apparatus of claim 10, further comprising:

the updating module is further configured to update the first file expiration time of the file corresponding to the modification request in the meta information field according to the second file expiration time when the third determining module determines that the request time of the modification request does not exceed the first file expiration time of the file corresponding to the modification request.

12. The apparatus of claim 9, further comprising:

13. A file expiration processing device for an object storage system, applied to a client, is characterized by comprising:

and the sending module is used for sending the storage request to a server side so that the server side adds the first file expiration time of the file to a meta information field which is stored in a meta database and used for defining the file expiration time, and periodically deletes the corresponding expired file according to the first file expiration time.

14. An electronic device, comprising:

the file expiration processing apparatus for an object storage system according to any one of claims 9 to 13; alternatively, the first and second electrodes may be,

a processor and a memory for storing executable instructions for controlling the processor to perform the file expiration processing method of the object storage system according to any one of claims 1 to 8.

15. A computer-readable storage medium, characterized in that a computer program is stored thereon, which when executed by a processor implements the file expiration processing method of the object storage system according to any one of claims 1 to 8.