CN107330061B - File deletion method and device based on distributed storage - Google Patents
File deletion method and device based on distributed storage Download PDFInfo
- Publication number
- CN107330061B CN107330061B CN201710516726.4A CN201710516726A CN107330061B CN 107330061 B CN107330061 B CN 107330061B CN 201710516726 A CN201710516726 A CN 201710516726A CN 107330061 B CN107330061 B CN 107330061B
- Authority
- CN
- China
- Prior art keywords
- deletion request
- deletion
- storage device
- object storage
- target file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012217 deletion Methods 0.000 title claims abstract description 276
- 230000037430 deletion Effects 0.000 title claims abstract description 276
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 58
- 238000005520 cutting process Methods 0.000 claims description 35
- 238000013507 mapping Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004883 computer application Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5044—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering hardware capabilities
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a file deleting method based on distributed storage, which comprises the following steps: generating a deletion request set of a target file aiming at the target file to be deleted; acquiring the residual processing capacity of the object storage device corresponding to each deletion request in the deletion request set; for each object storage device, determining whether to issue a deletion request to the object storage device according to the remaining processing capacity of the object storage device; if so, determining a deletion request to be sent from the deletion request set, and sending the determined deletion request to be sent to the object storage device, so that the object storage device executes corresponding file deletion operation based on the received deletion request. By applying the technical scheme provided by the embodiment of the invention, the working efficiency of bottom storage can be effectively improved, and meanwhile, the influence on the normal operation of other services in the object storage equipment can be avoided. The invention also discloses a file deleting device based on distributed storage, and the file deleting device has corresponding technical effects.
Description
Technical Field
The invention relates to the technical field of computer application, in particular to a file deleting method and device based on distributed storage.
Background
With the advent of the internet big data age, the data share occupied by application scenes such as video monitoring, broadcast and television media resources and the like is large, a large number of files are stored through a distributed storage system, and the required hardware resources are very large. Files on the underlying storage may have a certain timeliness, that is, some files may be deleted beyond a certain time, so that the corresponding underlying storage resources can be reused.
In a distributed storage system, if a file is to be deleted, the file belongs to synchronous deletion at the upper file system level, and the mapping object of the file on the bottom storage needs asynchronous deletion. If the file is large, e.g., exceeding 1TB, the file needs to be cut, e.g., into 4MB stripes, so that about 262144 files of the 1TB file are distributed on all underlying storage, and the underlying storage needs to asynchronously delete the divided files. However, if the current underlying storage is in a busy state with high pressure, when the mapping objects are deleted, other IO operations of the underlying storage may be affected too much by the mapping objects, and may even exceed the upper limit of the processing capacity of the underlying storage, so that a situation that any IO operation of the upper-layer file system cannot be responded to may be caused.
Therefore, how to delete files in the distributed storage system to avoid affecting the normal operation of the underlying storage services is a technical problem that needs to be solved urgently by those skilled in the art at present.
Disclosure of Invention
The invention aims to provide a file deleting method and device based on distributed storage, so as to ensure that all services stored at the bottom layer can normally run when files are deleted in a distributed storage system.
In order to solve the technical problems, the invention provides the following technical scheme:
a file deleting method based on distributed storage comprises the following steps:
generating a deletion request set of a target file to be deleted;
acquiring the residual processing capacity of the object storage device corresponding to each deletion request in the deletion request set;
for each object storage device, determining whether to issue a deletion request to the object storage device according to the remaining processing capacity of the object storage device;
if so, further determining a deletion request to be sent from the deletion request set, and sending the determined deletion request to be sent to the object storage device, so that the object storage device executes corresponding file deletion operation based on the received deletion request.
Preferably, the determining a deletion request to be sent from the deletion request set includes:
determining whether a first sum of deletion requests which are not processed and completed currently of each object storage device reaches a first upper limit of a preset deletion number;
and if not, determining a deletion request to be sent from the deletion request set according to the difference value between the first upper limit of the deletion number and the first sum.
Preferably, the generating a deletion request set of the target file for the target file to be deleted includes:
determining the size of a target file to be deleted;
if the size of the target file is larger than a preset threshold value, cutting the target file according to the size of a set strip to obtain a plurality of cutting objects;
and generating a deletion request corresponding to each cutting object.
Preferably, the method further comprises the following steps:
and in the process of cutting the target file, if the second sum of the generated deletion request of the cut object and the deletion request which is not issued currently reaches the second upper limit of the preset deletion quantity, the cutting operation is suspended.
Preferably, when it is determined not to send the deletion request to the object storage device, the method further includes:
and delaying the deletion requests corresponding to the object storage device in the deletion request set.
A file deletion apparatus based on distributed storage, the apparatus comprising:
the deletion request set generation module is used for generating a deletion request set of the target file aiming at the target file to be deleted;
a residual processing capacity obtaining module, configured to obtain a residual processing capacity of the object storage device corresponding to each deletion request in the deletion request set;
a deletion request determining module, configured to determine, for each object storage device, whether to issue a deletion request to the object storage device according to the remaining processing capacity of the object storage device, and if so, trigger the deletion request issuing module;
and the deletion request issuing module is used for determining a deletion request to be sent from the deletion request set and issuing the determined deletion request to be sent to the object storage device so that the object storage device executes corresponding file deletion operation based on the received deletion request.
Preferably, the deletion request issuing module is specifically configured to:
determining whether a first sum of deletion requests which are not processed and completed currently of each object storage device reaches a first upper limit of a preset deletion number;
and if not, determining a deletion request to be sent from the deletion request set according to the difference value between the first upper limit of the deletion number and the first sum.
Preferably, the deletion request set generating module is specifically configured to:
determining the size of a target file to be deleted;
if the size of the target file is larger than a preset threshold value, cutting the target file according to the size of a set strip to obtain a plurality of cutting objects;
and generating a deletion request corresponding to each cutting object.
Preferably, the cutting operation suspending module is further included for:
and in the process of cutting the target file, if the second sum of the generated deletion request of the cut object and the deletion request which is not issued currently reaches the second upper limit of the preset deletion quantity, the cutting operation is suspended.
Preferably, the system further comprises a delete request deferral module, configured to:
and when determining that the deletion request is not sent to the object storage device, delaying the deletion requests corresponding to the object storage device in the deletion request set.
By applying the technical scheme provided by the embodiment of the invention, a deletion request set of the target file is generated aiming at the target file to be deleted, the residual processing capacity of the object storage equipment corresponding to each deletion request in the deletion request set is obtained, whether the deletion request is issued to the object storage equipment is determined aiming at each object storage equipment according to the residual processing capacity of the object storage equipment, if so, the deletion request to be sent is further determined from the deletion request set, and the determined deletion request to be sent is issued to the object storage equipment, so that the object storage equipment executes the corresponding file deletion operation based on the received deletion request.
By acquiring the residual processing capacity of the object storage device, whether to send a deletion request to the object storage device and the deletion request to be sent are determined according to the residual processing capacity of the object storage device, so that the quantity of the deletion requests sent to the object storage device can be correspondingly reduced when the pressure of the object storage device is higher, and the quantity of the deletion requests sent to the object storage device can be correspondingly increased when the pressure of the object storage device is lower, so that the working efficiency of bottom storage can be effectively improved when files are deleted, particularly the batch deletion of ultra-large files, and the problem that other services cannot normally operate when the files are deleted due to the fact that the files are too large or the working pressure of the bottom storage is too high and the like can be avoided.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flowchart of an implementation of a file deletion method based on distributed storage according to the present invention;
fig. 2 is a schematic structural diagram of a file deleting device based on distributed storage according to the present invention.
Detailed Description
The core of the invention is to provide a file deleting method based on distributed storage, so that when deleting files, especially deleting oversized files in batch, the working efficiency of bottom storage can be effectively improved, and the problem that other services cannot normally operate when deleting files due to factors such as overlarge files or overlarge working pressure of bottom storage is avoided.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a flowchart of an implementation of a file deletion method based on distributed storage according to the present invention is shown. The method may comprise the steps of:
s101: and generating a deletion request set of the target file aiming at the target file to be deleted.
In practical application, a user can issue a deletion instruction according to needs. According to the deleting instruction of the user, the target file to be deleted can be determined. The target file may be a plurality of files or may be one file. When a deletion instruction sent by a user is received, the file can be synchronously deleted on a metadata layer so that the user can obtain a correct response, and then an asynchronous thread is started to truly delete the data on the underlying object storage device.
And after determining the target file to be deleted, generating a deletion request set of the target file aiming at the target file. The target file to be deleted may be a file stored on one object storage device, or may be a file stored on a plurality of different object storage devices. If the target file is a super large file exceeding 1TB, when the deletion request of the target file is generated, the target file can be cut, and a corresponding deletion request is generated for each cut file. Of course, if the target file is multiple files, a corresponding delete request may be generated for each target file. The generated deletion requests of the target files constitute a deletion request set. A deletion request queue may be formed for the generated set of deletion requests for the target file.
After the deletion request set of the target file is generated, the operation of step S102 may be performed.
S102: and obtaining the residual processing capacity of the object storage device corresponding to each deletion request in the deletion request set.
In step S101, a deletion request set of the target file is generated, and the deletion request set may include one or more deletion requests. And respectively determining the object storage device corresponding to each deletion request in the obtained deletion request set, for example, obtaining the object storage device corresponding to the deletion request through the index number of the deletion request.
After determining the object storage devices corresponding to each deletion request, the remaining processing capabilities of the object storage devices may be further obtained. One or more threads may be launched for acquiring the remaining processing power of each object storage device.
In the embodiment of the present invention, the remaining processing capacity may be determined according to the processing capacity upper limit of the object storage device and the number of IO requests to be processed by the object storage device in the current state, where the IO requests may include a delete request, a read request, a write request, and the like. For example, the upper limit of processing of a certain object storage device is 100ops (number of operations per second), the number of IO requests to be processed by the current object storage device is 40, and the remaining processing capacity of the object storage device is considered to be 60.
After obtaining the remaining processing capacity of the object storage device corresponding to each deletion request, the operation of S103 may be performed.
S103: and for each object storage device, determining whether to issue a deletion request to the object storage device according to the residual processing capacity of the object storage device.
After obtaining the remaining processing capacity of the object storage device, for each object storage device, it may be determined whether to issue a deletion request to the object storage device according to the remaining processing capacity of the object storage device. For example, a remaining processing capability threshold may be set, and if the remaining processing capability of the object storage device is higher than the remaining processing capability threshold, it is determined that a delete request may be issued to the object storage device. If the remaining processing capacity of the object storage device is lower than the threshold, it indicates that the remaining processing capacity of the object storage device is low, and if a delete request is still issued to the object storage device at this time, the normal operation of other services of the object storage device may be affected. So in this case no delete request may be issued to the object storage device.
The threshold of the remaining processing capability may be set and adjusted according to actual situations, for example, if it is required to ensure that the target storage device can normally operate when performing other IO request processing, or reduce the load of the target storage device to ensure that the front-stage code stream does not lose frames, a higher threshold of the remaining processing capability may be set. If it is determined that the deletion request is not issued to the object storage device, the deletion request may be deferred or not processed, i.e., the deletion request is not issued, as shown in fig. 1.
When it is determined that the deletion request is issued to the object storage device, the operation of S104 may be continued.
S104: and determining a deletion request to be sent from the deletion request set, and sending the determined deletion request to be sent to the object storage device, so that the object storage device executes corresponding file deletion operation based on the received deletion request.
For each object storage device, after determining that a deletion request is to be issued to the object storage device, a deletion request to be sent may be determined from a deletion request set. Specifically, the deletion request to be sent may be determined according to the remaining processing capacity of the object storage device and according to a preset ratio of the remaining processing capacity. For example, if the current remaining processing capacity of a certain object storage device is 60, which is higher than the set remaining processing capacity threshold 10, it may be determined that a deletion request is to be issued to the object storage device, and if the preset proportion is 20%, in this case, the number of deletion requests to be sent corresponding to the object storage device, which is determined from the deletion request set, is 12, and the deletion request to be sent may be determined according to this number.
For each object storage device, after determining the deletion request to be sent, the deletion request may be sent to the object storage device. After receiving the deletion request, the object storage device may delete the file corresponding to the received deletion request.
By applying the method provided by the embodiment of the invention, a deletion request set of the target file is generated aiming at the target file to be deleted, the residual processing capacity of the object storage device corresponding to each deletion request in the deletion request set is obtained, whether the deletion request is issued to the object storage device is determined aiming at each object storage device according to the residual processing capacity of the object storage device, if yes, the deletion request to be sent is further determined from the deletion request set, and the determined deletion request to be sent is issued to the object storage device, so that the object storage device executes the corresponding file deletion operation based on the received deletion request.
By acquiring the residual processing capacity of the object storage device, whether to send a deletion request to the object storage device and the deletion request to be sent are determined according to the residual processing capacity of the object storage device, so that the quantity of the deletion requests sent to the object storage device can be correspondingly reduced when the pressure of the object storage device is higher, and the quantity of the deletion requests sent to the object storage device can be correspondingly increased when the pressure of the object storage device is lower, so that the working efficiency of bottom storage can be effectively improved when files are deleted, particularly the batch deletion of ultra-large files, and the problem that other services cannot normally operate when the files are deleted due to the fact that the files are too large or the working pressure of the bottom storage is too high and the like can be avoided.
In a specific embodiment of the present invention, the determining, in step S104, a deletion request to be sent from a deletion request set may include the following steps:
the method comprises the following steps: determining whether a first sum of deletion requests which are not processed and completed currently of each object storage device reaches a first upper limit of a preset deletion number;
step two: and if not, determining the deletion request to be sent from the deletion request set according to the difference value between the first upper limit of the deletion number and the first sum.
For convenience of description, the two steps are combined.
To ensure that the subject storage device is able to handle a burst of IO requests, such an embodiment of the present invention may be employed. Determining the number of the currently unprocessed and completed deletion requests of each object storage device, and calculating the sum of the number of the currently unprocessed and completed deletion requests of all the object storage devices, so as to obtain a first sum, and obtaining the total residual processing capacity of the object storage devices through the first sum. The larger the first sum, the lower the overall remaining processing power of the object storage device.
And presetting a first upper limit of the deletion number, determining whether the first sum reaches the preset first upper limit of the deletion number, and if so, indicating that the total residual processing capacity of the object storage equipment is too low, and not suitable for issuing a deletion request. If the first sum is lower than the preset first upper limit of the deletion number, it is indicated that the total remaining processing capacity of the object storage device can meet the condition for processing the burst IO request, and the difference value can be obtained by subtracting the first sum from the first upper limit of the deletion number. According to the difference, the deletion request to be sent can be determined from the deletion request set, for example, the number of the deletion requests to be sent which is equal to the difference is determined from the deletion request set.
In an embodiment of the present invention, step S101 specifically includes the following steps:
the method comprises the following steps: determining the size of a target file to be deleted;
step two: if the size of the target file is larger than a preset threshold value, cutting the target file according to the size of the set strip to obtain a plurality of cutting objects;
step three: and generating a deletion request corresponding to each cutting object.
For convenience of description, the above steps are combined for illustration.
After determining the target file to be deleted, the size of the target file may be determined first. And comparing the size of the target file with a preset threshold value. If the size of the target file is larger than the preset threshold value, the target file may be a large file, and the target file may be cut according to the set stripe size to obtain a plurality of cut objects. The slice size can be set to 4MB size in general. For each cut object, a delete request corresponding to the respective cut object may be generated. The generated deletion requests may be arranged in the order of generation and placed in a queue to obtain a deletion request queue.
In a specific embodiment of the present invention, in the process of cutting the target file, if a second sum of the generated deletion request of the cut object and the deletion request that is not issued currently reaches a second upper limit of the preset deletion number, the cutting operation is suspended.
The number of the generated deletion requests can be controlled within a numerical value, which is called a second upper limit of the deletion number, in the process of cutting the target file, a second sum of the generated deletion requests of the cut object and the deletion requests which are not issued at present can be calculated, and when the second sum reaches the second upper limit of the preset deletion number, the cutting operation can be suspended.
The second upper limit of the deletion number can be set and adjusted according to actual occasions, for example, the number of the second upper limit is preset to ten thousand. When the second sum is calculated, the second sum may also be calculated according to an actual situation and according to a preset period, or when a receipt made by the object storage device for the deletion request is received, the second sum may also be calculated. When the object storage device successfully completes the issued deletion request and correctly sends the receipt, the object storage device can continue to issue the deletion request, so that the object file can be continuously cut until the object file is cut completely or the second sum reaches the second upper limit of the preset deletion number again.
It should be noted that the above embodiments do not affect the implementation of the present invention.
In a specific embodiment of the present invention, when determining not to send a deletion request to the object storage device, the method may further include:
and delaying the deletion requests corresponding to the object storage devices in the deletion request set.
For each object storage device, when the remaining processing capacity of the object storage device indicates that the stress of the object storage device is too high, the deletion requests corresponding to the object storage devices may be all deferred to be processed. Specifically, the deletion request queue may be detected, all deletion requests corresponding to the object storage device may be extracted, and all the deletion requests may be subjected to a deferred processing. These delete requests may be moved to the tail of the delete request queue, for example. Of course, when it is determined that the pressure of the object storage device is too high, and in the process of issuing the deletion requests in the sequence of the deletion request queue, when a deletion request corresponding to the object storage device occurs, the deletion request may be moved to the end of the queue. Therefore, the problem that the normal operation of other businesses is influenced by the excessive pressure of the object storage equipment can be avoided.
Corresponding to the above method embodiment, an embodiment of the present invention further provides a file deleting device based on distributed storage, and a file deleting device based on distributed storage described below and a file deleting method based on distributed storage described above may be referred to correspondingly.
Fig. 2 is a schematic structural diagram of a file deleting device based on distributed storage according to the present invention. The device includes:
a deletion request set generating module 201, configured to generate a deletion request set of a target file for the target file to be deleted;
a remaining processing capacity obtaining module 202, configured to obtain a remaining processing capacity of the object storage device corresponding to each deletion request in the deletion request set;
a deletion request determining module 203, configured to determine, for each object storage device, whether to issue a deletion request to the object storage device according to the remaining processing capacity of the object storage device, and if so, trigger the deletion request issuing module 204;
the deletion request issuing module 204 is configured to determine a deletion request to be sent from the deletion request set, and issue the determined deletion request to be sent to the object storage device, so that the object storage device executes a corresponding file deletion operation based on the received deletion request.
By applying the device provided by the embodiment of the invention, a deletion request set of the target file is generated aiming at the target file to be deleted, the residual processing capacity of the object storage device corresponding to each deletion request in the deletion request set is obtained, whether the deletion request is issued to the object storage device is determined aiming at each object storage device according to the residual processing capacity of the object storage device, if yes, the deletion request to be sent is further determined from the deletion request set, and the determined deletion request to be sent is issued to the object storage device, so that the object storage device executes the corresponding file deletion operation based on the received deletion request.
By acquiring the residual processing capacity of the object storage device, whether to send a deletion request to the object storage device and the deletion request to be sent are determined according to the residual processing capacity of the object storage device, so that the quantity of the deletion requests sent to the object storage device can be correspondingly reduced when the pressure of the object storage device is higher, and the quantity of the deletion requests sent to the object storage device can be correspondingly increased when the pressure of the object storage device is lower, so that the working efficiency of bottom storage can be effectively improved when files are deleted, particularly the batch deletion of ultra-large files, and the problem that other services cannot normally operate when the files are deleted due to the fact that the files are too large or the working pressure of the bottom storage is too high and the like can be avoided.
In a specific embodiment of the present invention, the deletion request issuing module 204 is specifically configured to:
determining whether a first sum of deletion requests which are not processed and completed currently of each object storage device reaches a first upper limit of a preset deletion number;
and if not, determining the deletion request to be sent from the deletion request set according to the difference value between the first upper limit of the deletion number and the first sum.
In a specific embodiment of the present invention, the deletion request set generating module 201 is specifically configured to:
determining the size of a target file to be deleted;
if the size of the target file is larger than a preset threshold value, cutting the target file according to the size of the set strip to obtain a plurality of cutting objects;
and generating a deletion request corresponding to each cutting object.
In one embodiment of the present invention, the cutting operation suspending module is further included for:
and in the process of cutting the target file, if the second sum of the generated deletion request of the cut object and the deletion request which is not issued currently reaches the second upper limit of the preset deletion quantity, the cutting operation is suspended.
In a specific embodiment of the present invention, the system further includes a delete request deferral module, configured to:
and when determining that the deletion request is not sent to the object storage device, delaying the deletion requests corresponding to the object storage device in the deletion request set.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The principle and the implementation of the present invention are explained in the present application by using specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present invention. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (8)
1. A file deleting method based on distributed storage is characterized by comprising the following steps:
generating a deletion request set of a target file to be deleted;
acquiring the residual processing capacity of the object storage device corresponding to each deletion request in the deletion request set;
for each object storage device, determining whether to issue a deletion request to the object storage device according to the remaining processing capacity of the object storage device;
if so, further determining a deletion request to be sent from the deletion request set, and sending the determined deletion request to be sent to the object storage device, so that the object storage device executes corresponding file deletion operation based on the received deletion request;
the determining a deletion request to be sent from the deletion request set includes:
determining whether a first sum of deletion requests which are not processed and completed currently of each object storage device reaches a first upper limit of a preset deletion number;
and if not, determining a deletion request to be sent from the deletion request set according to the difference value between the first upper limit of the deletion number and the first sum.
2. The method according to claim 1, wherein the generating a deletion request set of the target file for the target file to be deleted comprises:
determining the size of a target file to be deleted;
if the size of the target file is larger than a preset threshold value, cutting the target file according to the size of a set strip to obtain a plurality of cutting objects;
and generating a deletion request corresponding to each cutting object.
3. The method of claim 2, further comprising:
and in the process of cutting the target file, if the second sum of the generated deletion request of the cut object and the deletion request which is not issued currently reaches the second upper limit of the preset deletion quantity, the cutting operation is suspended.
4. The method of any of claims 1 to 3, when determining not to send a delete request to the object storage device, further comprising:
and delaying the deletion requests corresponding to the object storage device in the deletion request set.
5. A file deleting device based on distributed storage is characterized by comprising:
the deletion request set generation module is used for generating a deletion request set of the target file aiming at the target file to be deleted;
a residual processing capacity obtaining module, configured to obtain a residual processing capacity of the object storage device corresponding to each deletion request in the deletion request set;
a deletion request determining module, configured to determine, for each object storage device, whether to issue a deletion request to the object storage device according to the remaining processing capacity of the object storage device, and if so, trigger the deletion request issuing module;
a deletion request issuing module, configured to determine a deletion request to be sent from the deletion request set, and issue the determined deletion request to be sent to the object storage device, so that the object storage device executes a corresponding file deletion operation based on the received deletion request;
the deletion request issuing module is specifically configured to:
determining whether a first sum of deletion requests which are not processed and completed currently of each object storage device reaches a first upper limit of a preset deletion number;
and if not, determining a deletion request to be sent from the deletion request set according to the difference value between the first upper limit of the deletion number and the first sum.
6. The apparatus of claim 5, wherein the deletion request set generation module is specifically configured to:
determining the size of a target file to be deleted;
if the size of the target file is larger than a preset threshold value, cutting the target file according to the size of a set strip to obtain a plurality of cutting objects;
and generating a deletion request corresponding to each cutting object.
7. The apparatus of claim 6, further comprising a cutting operation pause module to:
and in the process of cutting the target file, if the second sum of the generated deletion request of the cut object and the deletion request which is not issued currently reaches the second upper limit of the preset deletion quantity, the cutting operation is suspended.
8. The apparatus of any of claims 5 to 7, further comprising a delete request deferral module to:
and when determining that the deletion request is not sent to the object storage device, delaying the deletion requests corresponding to the object storage device in the deletion request set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710516726.4A CN107330061B (en) | 2017-06-29 | 2017-06-29 | File deletion method and device based on distributed storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710516726.4A CN107330061B (en) | 2017-06-29 | 2017-06-29 | File deletion method and device based on distributed storage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107330061A CN107330061A (en) | 2017-11-07 |
| CN107330061B true CN107330061B (en) | 2021-02-02 |
Family
ID=60197946
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710516726.4A Active CN107330061B (en) | 2017-06-29 | 2017-06-29 | File deletion method and device based on distributed storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107330061B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108846032A (en) * | 2018-05-28 | 2018-11-20 | 郑州云海信息技术有限公司 | File delet method, device and equipment in a kind of storage system |
| CN108959399B (en) * | 2018-06-04 | 2022-07-15 | 平安科技（深圳）有限公司 | Distributed data deletion flow control method and device, electronic equipment and storage medium |
| CN110888844B (en) * | 2019-11-22 | 2023-03-21 | 浪潮电子信息产业股份有限公司 | Data deleting method, system, equipment and computer readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103392167A (en) * | 2012-12-28 | 2013-11-13 | 华为技术有限公司 | Caching method of distributed storage system, nodes and computer readable medium |
| CN103997540A (en) * | 2014-06-10 | 2014-08-20 | 深圳市友华通信技术有限公司 | Method for achieving distributed storage of network, storage system and customer premise equipment |
| CN106227469A (en) * | 2016-07-28 | 2016-12-14 | 乐视控股（北京）有限公司 | Data-erasure method and system for distributed storage cluster |
| CN106407214A (en) * | 2015-08-02 | 2017-02-15 | 郑建锋 | Distributed storage method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4696089B2 (en) * | 2007-03-30 | 2011-06-08 | 三菱電機インフォメーションシステムズ株式会社 | Distributed storage system |
-
2017
- 2017-06-29 CN CN201710516726.4A patent/CN107330061B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103392167A (en) * | 2012-12-28 | 2013-11-13 | 华为技术有限公司 | Caching method of distributed storage system, nodes and computer readable medium |
| CN103997540A (en) * | 2014-06-10 | 2014-08-20 | 深圳市友华通信技术有限公司 | Method for achieving distributed storage of network, storage system and customer premise equipment |
| CN106407214A (en) * | 2015-08-02 | 2017-02-15 | 郑建锋 | Distributed storage method and system |
| CN106227469A (en) * | 2016-07-28 | 2016-12-14 | 乐视控股（北京）有限公司 | Data-erasure method and system for distributed storage cluster |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107330061A (en) | 2017-11-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107330061B (en) | File deletion method and device based on distributed storage | |
| US10037348B2 (en) | Database management system with database hibernation and bursting | |
| US10055134B2 (en) | Data compression method and storage system | |
| CN110795284B (en) | Data recovery method, device and equipment and readable storage medium | |
| CN109582649B (en) | Metadata storage method, device and equipment and readable storage medium | |
| CN109597800B (en) | Log distribution method and device | |
| US20230305880A1 (en) | Cluster distributed resource scheduling method, apparatus and device, and storage medium | |
| CN116450287A (en) | Method, device, equipment and readable medium for managing storage capacity of service container | |
| CN107329832B (en) | Data receiving method and device | |
| CN108599774B (en) | Compression method, system, device and computer readable storage medium | |
| EP3396554A1 (en) | Backup control method and backup control device | |
| US20190028420A1 (en) | Message sending method and terminal device | |
| CN110647498A (en) | File storage method, device, equipment and computer readable storage medium | |
| CN112506698A (en) | Small file data reconstruction recovery method, system, terminal and storage medium | |
| US11194619B2 (en) | Information processing system and non-transitory computer readable medium storing program for multitenant service | |
| US11232228B2 (en) | Method and device for improving data storage security | |
| CN115718732A (en) | Disk file management method, device, equipment and storage medium | |
| CN107229424B (en) | Data writing method for distributed storage system and distributed storage system | |
| CN114564149A (en) | Data storage method, device, equipment and storage medium | |
| CN109582244B (en) | Metadata disk-dropping method, device, terminal and computer-readable storage medium | |
| CN111984406A (en) | Method, device and equipment for distributing CPU (Central processing Unit) resources in management process of virtualization platform | |
| CN112269763A (en) | File aggregation method and related device | |
| CN111625506A (en) | Distributed data deleting method, device and equipment based on deleting queue | |
| CN111405313A (en) | Method and system for storing streaming media data | |
| CN111104381A (en) | Log management method, device and equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20201231 Address after: Building 9, No.1, guanpu Road, Guoxiang street, Wuzhong Economic Development Zone, Wuzhong District, Suzhou City, Jiangsu Province Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd. Address before: Room 1601, floor 16, 278 Xinyi Road, Zhengdong New District, Zhengzhou City, Henan Province Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |