CN114116913A - Distributed storage system storage bucket processing method and device and terminal - Google Patents
Distributed storage system storage bucket processing method and device and terminal Download PDFInfo
- Publication number
- CN114116913A CN114116913A CN202210088473.6A CN202210088473A CN114116913A CN 114116913 A CN114116913 A CN 114116913A CN 202210088473 A CN202210088473 A CN 202210088473A CN 114116913 A CN114116913 A CN 114116913A
- Authority
- CN
- China
- Prior art keywords
- bucket
- name
- actual
- logical
- storage system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 238000000034 method Methods 0.000 claims abstract description 39
- 238000013500 data storage Methods 0.000 claims abstract description 31
- 238000013507 mapping Methods 0.000 claims abstract description 31
- 230000008569 process Effects 0.000 claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 26
- 238000011084 recovery Methods 0.000 claims abstract description 24
- 238000004064 recycling Methods 0.000 claims abstract description 12
- 238000012217 deletion Methods 0.000 claims description 21
- 230000037430 deletion Effects 0.000 claims description 21
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000003203 everyday effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000004083 survival effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of distributed storage system bucket processing, and particularly discloses a distributed storage system bucket processing method, a device and a terminal, wherein a logical bucket name is created at a client, an actual bucket corresponding to the created logical bucket name is created at a data storage layer, and mapping is performed between the logical bucket name and the actual bucket; when the client deletes the logical bucket name, the logical bucket name is immediately deleted, and meanwhile, the actual bucket corresponding to the data storage layer is added into a garbage recovery queue; and the garbage collection process deletes the actual barrel in the garbage collection queue. According to the invention, the operation of deleting the bucket is carried out by combining the bucket mapping with the garbage recycling, so that the timeliness of deleting the bucket can be realized, and the problem of failure in deleting the storage bucket can be solved.
Description
Technical Field
The invention relates to the field of distributed storage system bucket processing, in particular to a distributed storage system bucket processing method, a distributed storage system bucket processing device and a terminal.
Background
Currently, in a distributed object storage system, deleting a bucket is always a difficult problem. In general, a bucket is deleted, and the object in the bucket needs to be deleted before the bucket is deleted. If there are any objects in the bucket, the operation of deleting the bucket cannot be performed. Therefore, when the number of objects in the bucket is small, the objects in the bucket are deleted first, and then the bucket is deleted, but when the number of the objects in the bucket reaches a certain level (tens of millions or hundreds of millions), the deleting operation takes extra time.
In addition, when the number of objects in the bucket is too large, the current general practice is to set a life cycle process for the bucket, so that the bucket is deleted through the life cycle process, and when there is no object in the bucket, the bucket is deleted. But the lifecycle process executes the delete bucket operation with the following disadvantages: the execution of the life cycle process cannot be executed immediately, and the life cycle process is an independent thread which is executed once a day, so that the timeliness of deleting the bucket cannot be realized; the execution time of the life cycle process is 0: 00-06: 00 every day, which is mainly because the life cycle process consumes system resources, the life cycle process is determined to be executed once every day for 6 hours, so that the objects of the buckets in the time period may not be completely deleted, and the bucket deletion is failed.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method, an apparatus, and a terminal for processing buckets of a distributed storage system, where the operations of deleting buckets are performed by combining bucket mapping with garbage collection, so that timeliness of deleting buckets can be achieved, and a problem of failure in deleting buckets can be solved.
In a first aspect, a technical solution of the present invention provides a bucket processing method for a distributed storage system, including the following steps:
creating a logical bucket name at a client, creating an actual bucket corresponding to the created logical bucket name at a data storage layer, and mapping between the logical bucket name and the actual bucket; the operation of the client on the object under the name of the logical bucket is mapped to the actual bucket, and the actual operation on the object is completed in the actual bucket;
when the client deletes the logical bucket name, the logical bucket name is immediately deleted, and meanwhile, the actual bucket corresponding to the data storage layer is added into a garbage recovery queue;
and the garbage collection process deletes the actual barrel in the garbage collection queue.
Further, the operation of the client on the object under the name of the logical bucket comprises the operation of uploading and deleting the object.
Further, the method comprises the following steps:
when a new logical bucket name identical to the old logical bucket name is created again after a certain old logical bucket name is deleted, the name of a new actual bucket created in the data storage layer is different from the name of the old actual bucket.
Further, the name of the actual bucket is the name of the corresponding logical bucket name plus a random string.
Further, the garbage collection process deletes the actual barrel in the garbage collection queue, which specifically includes:
monitoring the idleness of the distributed storage system by a garbage recovery process;
and deleting the actual buckets in the garbage recycling queue when the idleness of the distributed storage system reaches a threshold value.
In a second aspect, the present invention provides a bucket handling apparatus for a distributed storage system, including,
a bucket creation module: creating a logical bucket name at a client, creating an actual bucket corresponding to the created logical bucket name at a data storage layer, and mapping between the logical bucket name and the actual bucket; the operation of the client on the object under the name of the logical bucket is mapped to the actual bucket, and the actual operation on the object is completed in the actual bucket;
a bucket deletion module: when the client deletes the logical bucket name, the logical bucket name is immediately deleted, and meanwhile, the actual bucket corresponding to the data storage layer is added into a garbage recovery queue;
storage bucket clearance module: and the garbage collection process deletes the actual barrel in the garbage collection queue.
In a third aspect, a technical solution of the present invention provides a terminal, including:
a memory for storing a distributed storage system bucket handler;
a processor configured to implement the steps of the distributed storage system bucket processing method as described in any one of the above when executing the distributed storage system bucket processing program.
Compared with the prior art, the method, the device and the terminal for processing the storage barrel of the distributed storage system have the following beneficial effects that: only one logical bucket name is created at the client, meanwhile, a real storage bucket is created in the data storage layer, mapping is carried out between the logical bucket name and the actual bucket, no object is contained in the logical bucket name, and the actual operation of the object is completed in the actual bucket. And then when deleting the storage bucket, deleting the name of the logical bucket, immediately deleting the storage bucket at the user angle, simultaneously putting the background actual bucket into a garbage recovery queue, and subsequently finishing the actual deletion of the storage bucket by a garbage recovery process to ensure that the storage bucket can be normally deleted, which is not perceived by the user and cannot influence the performance of the system when finishing the deletion of the storage bucket by the garbage recovery process. According to the invention, the operation of deleting the bucket is carried out by combining the bucket mapping with the garbage recycling, so that the timeliness of deleting the bucket can be realized, and the problem of failure in deleting the storage bucket can be solved.
Drawings
For a clearer explanation of the embodiments or technical solutions of the prior art of the present application, the drawings needed for the description of the embodiments or prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flowchart illustrating a method for processing buckets of a distributed storage system according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a mapping relationship between logical bucket names and actual buckets according to the present invention.
Fig. 3 is a schematic block diagram of a structure of a bucket processing apparatus of a distributed storage system according to a second embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a terminal according to a third embodiment of the present invention.
Detailed Description
Some terms related to the present invention are explained below.
Distributed storage refers to a group of computer nodes which communicate through a network and coordinate with each other to provide storage services.
The Object storage is Object-based storage, and the main operation Object is an Object (Object). The storage protocol is S3, Swift, etc.
The bucket is the container for storing the object in S3. Each object is stored in a bucket, and multiple objects may be stored in a bucket.
Gc (garpage collection), garbage collection, which means that garbage collection is performed periodically for useless objects in the object storage system.
Lc (life cycle), in the object storage system, the life cycle refers to setting a survival time period or a time point for a certain object, and after the time point is reached, the system will automatically delete the object.
The gateway service is a gateway based on the current popular RESTful protocol provided by an object storage system, is compatible with Amazon S3 and openstack Swift interfaces, analyzes, converts and processes http requests of clients conforming to S3 or Swift protocol, sends the http requests to a data storage layer, and sends data returned by the data storage layer to the clients for development and use of the clients of corresponding object storage.
The core of the invention is to provide a distributed storage system storage bucket processing scheme aiming at the problems of untimely deletion and deletion failure of the current storage bucket, a logical bucket name is established at a client, the logical bucket name is only a logical bucket name and does not contain an object, the logical bucket name can be deleted immediately, an actual bucket corresponding to the established logical bucket name is established at a data storage layer, mapping is carried out between the logical bucket name and the actual bucket, and the operation on the object is really completed in the actual bucket; when the client deletes the logical bucket name, the logical bucket name is immediately deleted, so that a user can sense that the storage bucket is deleted, and meanwhile, an actual bucket corresponding to the data storage layer is added into a garbage recovery queue; the garbage recycling process deletes the actual barrel in the garbage recycling queue to complete the real deletion of the storage barrel, and the deletion process can be carried out when the system is idle, so that the system performance cannot be influenced and the user cannot perceive the deletion. According to the scheme, the operation of deleting the barrel is performed by combining barrel mapping and garbage recycling, so that the timeliness of deleting the barrel can be realized, and the problem of failure in deleting the storage barrel can be solved.
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Example one
Fig. 1 is a flowchart illustrating a method for processing buckets of a distributed storage system according to a first embodiment of the present invention, which includes the following steps.
S101, a logic bucket name is created at a client, an actual bucket corresponding to the created logic bucket name is created in a data storage layer, and mapping is carried out between the logic bucket name and the actual bucket.
The logical bucket name created at the client is only a logical bucket name, and when the user creates a bucket, only a logical bucket name, such as bucket1, is created, but in the object store, a real bucket, i.e., an actual bucket, named "bucket 1+ random string" is created. And adding a middle layer of mapping of the logical bucket name and the actual bucket in the object storage system to realize the mapping relation of the logical bucket and the actual bucket.
Thus, the bucket name displayed to the user is the logical bucket of bucket1, but the operation on bucket1 and the object in the bucket is actually the operation on the actual bucket with the bucket name "bucket 1+ random string". That is, the operation of the client on the object under the name of the logical bucket is mapped to the actual bucket, and the actual operation on the object is completed in the actual bucket.
When a new logical bucket name is created, it amounts to creating a logical bucket name and an actual bucket, and mapping the logical bucket name to the actual bucket. All operations on the names of the logical buckets are mapped to the actual buckets, for example, objects are uploaded or deleted into the logical buckets, and the objects are actually uploaded or deleted into the actual buckets. For example, uploading an object into bucket1, actually saving the object into a bucket named "bucket 1+ random string", and no object is saved in bucket1, which is only a logical display to the outside.
S102, when the client deletes the logical bucket name, the logical bucket name is immediately deleted, and meanwhile, the actual bucket corresponding to the data storage layer is added into a garbage collection queue.
When a bucket needs to be deleted, the logical bucket name is deleted at the client first, and since the logical bucket name does not actually contain an object but only a logical display, the logical bucket name is immediately deleted from the perspective of the user, which is equivalent to that the bucket is immediately deleted.
When a logical bucket name is deleted, the mapping relation between the logical bucket name and the actual bucket is cut off, in this embodiment, the operation triggers that the actual bucket of the data storage layer is added into the garbage collection queue, the garbage collection process completes the actual deletion of the storage bucket subsequently, the process is completed in the background of the system, the system can automatically delete the storage bucket, and the user is unaware of the deletion, and other operations of the user are not affected.
S103, the garbage collection process deletes the actual barrel in the garbage collection queue.
The execution of the garbage collection process has no specific time and duration until all data in the garbage collection queue is deleted, so that the fact that all actual buckets can be normally deleted is guaranteed, and the deletion failure condition is avoided.
In order to reduce the influence on the system performance, the garbage collection process monitors the vacancy of the distributed storage system, and deletes the actual barrel in the garbage collection queue when the vacancy of the distributed storage system reaches a threshold value. Namely, the garbage collection process carries out deletion operation when the storage system is idle, thereby having no influence on the system performance and having no perception to users.
As shown in fig. 2, which is a schematic diagram of a mapping relationship between logical bucket names and actual buckets, in the present embodiment, for convenience of mapping, the name of an actual bucket is the name of a corresponding logical bucket name plus a random character string, for example, the name of a logical bucket is bucket1, and then the name of an actual bucket is bucket1+ random character string 1.
When a user deletes a logical bucket name at a client, the logical bucket name is immediately deleted, so that the user can immediately recreate a new logical bucket name with the same name, and at this time, the new logical bucket name is mapped to another new actual bucket, and the name of the new actual bucket is different from that of the old actual bucket before (the actual bucket corresponding to the deleted logical bucket name), so as to avoid being added to a garbage collection queue. For example, logical bucket name bucket1, mapped actual bucket name bucket1+ random string 1; if the logical bucket name bucket1 is created again after the bucket1 is deleted, the name of the mapped new actual bucket is bucket1+ random string 11.
In the bucket processing method of the distributed storage system provided by this embodiment, only one logical bucket name is created at the client, and meanwhile, a real bucket is created in the data storage layer, mapping is performed between the logical bucket name and the actual bucket, and the logical bucket name does not contain any object, and the actual operation of the object is completed in the actual bucket. And then when deleting the storage bucket, deleting the name of the logical bucket, immediately deleting the storage bucket at the user angle, simultaneously putting the background actual bucket into a garbage recovery queue, and subsequently finishing the actual deletion of the storage bucket by a garbage recovery process to ensure that the storage bucket can be normally deleted, which is not perceived by the user and cannot influence the performance of the system when finishing the deletion of the storage bucket by the garbage recovery process. According to the invention, the operation of deleting the bucket is carried out by combining the bucket mapping with the garbage recycling, so that the timeliness of deleting the bucket can be realized, and the problem of failure in deleting the storage bucket can be solved.
Example two
The second embodiment provides a bucket processing apparatus for a distributed storage system, and implements the bucket processing method for the distributed storage system of the first embodiment.
Fig. 3 is a schematic block diagram of a structure of a bucket processing apparatus of a distributed storage system according to a second embodiment of the present invention, which includes the following functional modules.
A bucket creation module: and creating a logical bucket name at the client, creating an actual bucket corresponding to the created logical bucket name at the data storage layer, and mapping between the logical bucket name and the actual bucket. And mapping the operation of the client on the object under the name of the logical bucket to the actual bucket, and finishing the actual operation on the object in the actual bucket.
A bucket deletion module: when the client deletes the logical bucket name, the logical bucket name is immediately deleted, and meanwhile, the actual bucket corresponding to the data storage layer is added into the garbage collection queue.
Storage bucket clearance module: and the garbage collection process deletes the actual barrel in the garbage collection queue.
The distributed storage system bucket processing apparatus of this embodiment is used to implement the foregoing distributed storage system bucket processing method, and therefore, a detailed implementation of this apparatus can be seen in the foregoing example section of the distributed storage system bucket processing method, and therefore, the detailed implementation of this apparatus may refer to the description of the corresponding section example, and will not be further described herein.
In addition, since the bucket processing apparatus of the distributed storage system of this embodiment is used to implement the foregoing bucket processing method of the distributed storage system, its role corresponds to that of the foregoing method, and is not described herein again.
EXAMPLE III
Fig. 4 is a schematic structural diagram of a terminal device 400 according to an embodiment of the present invention, including: a processor 410, a memory 420, and a communication unit 430. The processor 410 is configured to implement the distributed storage system bucket handler stored in the memory 420 by performing the following steps:
creating a logical bucket name at a client, creating an actual bucket corresponding to the created logical bucket name at a data storage layer, and mapping between the logical bucket name and the actual bucket; the operation of the client on the object under the name of the logical bucket is mapped to the actual bucket, and the actual operation on the object is completed in the actual bucket;
when the client deletes the logical bucket name, the logical bucket name is immediately deleted, and meanwhile, the actual bucket corresponding to the data storage layer is added into a garbage recovery queue;
and the garbage collection process deletes the actual barrel in the garbage collection queue.
According to the invention, only one logical bucket name is created at the client, meanwhile, a real storage bucket is created in the data storage layer, mapping is carried out between the logical bucket name and the actual bucket, no object is contained in the logical bucket name, and the actual operation of the object is completed in the actual bucket. And then when deleting the storage bucket, deleting the name of the logical bucket, immediately deleting the storage bucket at the user angle, simultaneously putting the background actual bucket into a garbage recovery queue, and subsequently finishing the actual deletion of the storage bucket by a garbage recovery process to ensure that the storage bucket can be normally deleted, which is not perceived by the user and cannot influence the performance of the system when finishing the deletion of the storage bucket by the garbage recovery process. According to the invention, the operation of deleting the bucket is carried out by combining the bucket mapping with the garbage recycling, so that the timeliness of deleting the bucket can be realized, and the problem of failure in deleting the storage bucket can be solved.
In some embodiments, when the processor 410 executes the bucket processing subroutine stored in the memory 420, the following steps may be specifically implemented: and mapping the operation of the client on the object under the name of the logical bucket to the actual bucket, and finishing the actual operation on the object in the actual bucket.
In some embodiments, when the processor 410 executes the bucket processing subroutine stored in the memory 420, the following steps may be specifically implemented: operations on objects under the logical bucket name at the client include upload and delete operations on the objects.
In some embodiments, when the processor 410 executes the bucket processing subroutine stored in the memory 420, the following steps may be specifically implemented: when a new logical bucket name identical to the old logical bucket name is created again after a certain old logical bucket name is deleted, the name of a new actual bucket created in the data storage layer is different from the name of the old actual bucket.
In some embodiments, when the processor 410 executes the bucket processing subroutine stored in the memory 420, the following steps may be specifically implemented: the name of the actual bucket is the name of the corresponding logical bucket name plus a random string.
In some embodiments, when the processor 410 executes the bucket processing subroutine stored in the memory 420, the following steps may be specifically implemented: monitoring the idleness of the distributed storage system by a garbage recovery process; and deleting the actual buckets in the garbage recycling queue when the idleness of the distributed storage system reaches a threshold value.
The terminal apparatus 400 includes a processor 410, a memory 420, and a communication unit 430. The components communicate via one or more buses, and those skilled in the art will appreciate that the architecture of the servers shown in the figures is not intended to be limiting, and may be a bus architecture, a star architecture, a combination of more or less components than those shown, or a different arrangement of components.
The memory 420 may be used for storing instructions executed by the processor 410, and the memory 420 may be implemented by any type of volatile or non-volatile storage terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The executable instructions in memory 420, when executed by processor 410, enable terminal 400 to perform some or all of the steps in the method embodiments described below.
The processor 410 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by operating or executing software programs and/or modules stored in the memory 420 and calling data stored in the memory. The processor may be composed of an Integrated Circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs connected with the same or different functions. For example, the processor 410 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.
A communication unit 430, configured to establish a communication channel so that the storage terminal can communicate with other terminals. And receiving user data sent by other terminals or sending the user data to other terminals.
Example four
The present invention also provides a computer storage medium, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).
A computer storage medium stores a distributed storage system bucket handler that when executed by a processor implements the steps of:
creating a logical bucket name at a client, creating an actual bucket corresponding to the created logical bucket name at a data storage layer, and mapping between the logical bucket name and the actual bucket; the operation of the client on the object under the name of the logical bucket is mapped to the actual bucket, and the actual operation on the object is completed in the actual bucket;
when the client deletes the logical bucket name, the logical bucket name is immediately deleted, and meanwhile, the actual bucket corresponding to the data storage layer is added into a garbage recovery queue;
and the garbage collection process deletes the actual barrel in the garbage collection queue.
According to the invention, only one logical bucket name is created at the client, meanwhile, a real storage bucket is created in the data storage layer, mapping is carried out between the logical bucket name and the actual bucket, no object is contained in the logical bucket name, and the actual operation of the object is completed in the actual bucket. And then when deleting the storage bucket, deleting the name of the logical bucket, immediately deleting the storage bucket at the user angle, simultaneously putting the background actual bucket into a garbage recovery queue, and subsequently finishing the actual deletion of the storage bucket by a garbage recovery process to ensure that the storage bucket can be normally deleted, which is not perceived by the user and cannot influence the performance of the system when finishing the deletion of the storage bucket by the garbage recovery process. According to the invention, the operation of deleting the bucket is carried out by combining the bucket mapping with the garbage recycling, so that the timeliness of deleting the bucket can be realized, and the problem of failure in deleting the storage bucket can be solved.
In some embodiments, when executed by the processor, the distributed storage system bucket processing sub-program stored in the readable storage medium may specifically implement: and mapping the operation of the client on the object under the name of the logical bucket to the actual bucket, and finishing the actual operation on the object in the actual bucket.
In some embodiments, when executed by the processor, the distributed storage system bucket processing sub-program stored in the readable storage medium may specifically implement: operations on objects under the logical bucket name at the client include upload and delete operations on the objects.
In some embodiments, when executed by the processor, the distributed storage system bucket processing sub-program stored in the readable storage medium may specifically implement: when a new logical bucket name identical to the old logical bucket name is created again after a certain old logical bucket name is deleted, the name of a new actual bucket created in the data storage layer is different from the name of the old actual bucket.
In some embodiments, when executed by the processor, the distributed storage system bucket processing sub-program stored in the readable storage medium may specifically implement: the name of the actual bucket is the name of the corresponding logical bucket name plus a random string.
In some embodiments, when executed by the processor, the distributed storage system bucket processing sub-program stored in the readable storage medium may specifically implement: monitoring the idleness of the distributed storage system by a garbage recovery process; and deleting the actual buckets in the garbage recycling queue when the idleness of the distributed storage system reaches a threshold value.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, and the storage medium can store program codes, and includes instructions for enabling a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, and the like) to perform all or part of the steps of the method in the embodiments of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The above disclosure is only for the preferred embodiments of the present invention, but the present invention is not limited thereto, and any non-inventive changes that can be made by those skilled in the art and several modifications and amendments made without departing from the principle of the present invention shall fall within the protection scope of the present invention.
Claims (7)
1. A distributed storage system bucket processing method is characterized by comprising the following steps:
creating a logical bucket name at a client, creating an actual bucket corresponding to the created logical bucket name at a data storage layer, and mapping between the logical bucket name and the actual bucket; the operation of the client on the object under the name of the logical bucket is mapped to the actual bucket, and the actual operation on the object is completed in the actual bucket;
when the client deletes the logical bucket name, the logical bucket name is immediately deleted, and meanwhile, the actual bucket corresponding to the data storage layer is added into a garbage recovery queue;
and the garbage collection process deletes the actual barrel in the garbage collection queue.
2. The distributed storage system bucket processing method of claim 1 wherein the operations on the objects under the logical bucket name at the client comprise upload and delete operations on the objects.
3. The distributed storage system bucket processing method of claim 2, further comprising the steps of:
when a new logical bucket name identical to the old logical bucket name is created again after a certain old logical bucket name is deleted, the name of a new actual bucket created in the data storage layer is different from the name of the old actual bucket.
4. The distributed storage system bucket processing method of claim 3, wherein the name of the actual bucket is the name of the corresponding logical bucket name plus a random string.
5. The method for processing the buckets of the distributed storage system according to claim 4, wherein the deleting operation performed by the garbage collection process on the actual buckets in the garbage collection queue specifically includes:
monitoring the idleness of the distributed storage system by a garbage recovery process;
and deleting the actual buckets in the garbage recycling queue when the idleness of the distributed storage system reaches a threshold value.
6. A distributed storage system bucket handling apparatus, comprising,
a bucket creation module: creating a logical bucket name at a client, creating an actual bucket corresponding to the created logical bucket name at a data storage layer, and mapping between the logical bucket name and the actual bucket; the operation of the client on the object under the name of the logical bucket is mapped to the actual bucket, and the actual operation on the object is completed in the actual bucket;
a bucket deletion module: when the client deletes the logical bucket name, the logical bucket name is immediately deleted, and meanwhile, the actual bucket corresponding to the data storage layer is added into a garbage recovery queue;
storage bucket clearance module: and the garbage collection process deletes the actual barrel in the garbage collection queue.
7. A terminal, comprising:
a memory for storing a distributed storage system bucket handler;
a processor for implementing the steps of the distributed storage system bucket processing method of any one of claims 1 to 5 when executing the distributed storage system bucket processing program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210088473.6A CN114116913A (en) | 2022-01-26 | 2022-01-26 | Distributed storage system storage bucket processing method and device and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210088473.6A CN114116913A (en) | 2022-01-26 | 2022-01-26 | Distributed storage system storage bucket processing method and device and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114116913A true CN114116913A (en) | 2022-03-01 |
Family
ID=80361337
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210088473.6A Pending CN114116913A (en) | 2022-01-26 | 2022-01-26 | Distributed storage system storage bucket processing method and device and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114116913A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105989195A (en) * | 2015-03-23 | 2016-10-05 | 国际商业机器公司 | Approach and system for processing data in database |
| CN109032509A (en) * | 2018-06-29 | 2018-12-18 | 郑州云海信息技术有限公司 | A kind of data-erasure method and relevant apparatus of distributed memory system |
| CN109784064A (en) * | 2017-11-15 | 2019-05-21 | 李明均 | Computer Data Security management system |
| CN111309437A (en) * | 2020-02-13 | 2020-06-19 | 苏州浪潮智能科技有限公司 | Asynchronous deletion method, system and device for logical volume |
| CN112799841A (en) * | 2021-01-29 | 2021-05-14 | 烽火通信科技股份有限公司 | Data object storage management method and device |
| CN113366424A (en) * | 2019-02-01 | 2021-09-07 | Emc Ip控股有限公司 | Scalable garbage collection for deduplication storage |
-
2022
- 2022-01-26 CN CN202210088473.6A patent/CN114116913A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105989195A (en) * | 2015-03-23 | 2016-10-05 | 国际商业机器公司 | Approach and system for processing data in database |
| CN109784064A (en) * | 2017-11-15 | 2019-05-21 | 李明均 | Computer Data Security management system |
| CN109032509A (en) * | 2018-06-29 | 2018-12-18 | 郑州云海信息技术有限公司 | A kind of data-erasure method and relevant apparatus of distributed memory system |
| CN113366424A (en) * | 2019-02-01 | 2021-09-07 | Emc Ip控股有限公司 | Scalable garbage collection for deduplication storage |
| CN111309437A (en) * | 2020-02-13 | 2020-06-19 | 苏州浪潮智能科技有限公司 | Asynchronous deletion method, system and device for logical volume |
| CN112799841A (en) * | 2021-01-29 | 2021-05-14 | 烽火通信科技股份有限公司 | Data object storage management method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11809726B2 (en) | Distributed storage method and device | |
| US10817386B2 (en) | Virtual machine recovery method and virtual machine management device | |
| CN111124277B (en) | Deep learning data set caching method, system, terminal and storage medium | |
| CN110825495A (en) | Container cloud platform recovery method, device, equipment and readable storage medium | |
| CN115292026B (en) | Management method, device and equipment of container cluster and computer readable storage medium | |
| CN111078404B (en) | Computing resource determining method and device, electronic equipment and medium | |
| CN113672350B (en) | Application processing method and device and related equipment | |
| CN111400041A (en) | Server configuration file management method and device and computer readable storage medium | |
| CN111399977A (en) | Virtual machine migration method, device, equipment and medium | |
| CN115469813A (en) | Data processing method and device, storage medium and electronic device | |
| CN112860787A (en) | Method for switching master nodes in distributed master-slave system, master node device and storage medium | |
| CN111049913B (en) | Data file transmission method and device, storage medium and electronic equipment | |
| CN115048190B (en) | Virtual machine snapshot management method, system and computer readable medium | |
| CN110716690B (en) | Data recovery method and system | |
| CN116483274A (en) | Online migration method, device, equipment and medium for distributed block storage volume | |
| CN114116913A (en) | Distributed storage system storage bucket processing method and device and terminal | |
| CN109766220A (en) | Back-up restoring method, device and the computer readable storage medium of application system | |
| CN109542841A (en) | The method and terminal device of data snapshot are created in cluster | |
| CN114356970A (en) | Storage system resource caching method and device | |
| CN114721876A (en) | Data backup method, device and medium | |
| CN114553859A (en) | BMC configuration management method and device, electronic equipment and storage medium | |
| CN110703988B (en) | Storage pool creating method, system, terminal and storage medium for distributed storage | |
| CN114328007A (en) | Container backup and restoration method, device and medium thereof | |
| CN110768855B (en) | Method and device for testing linkmzation performance | |
| CN112559126A (en) | Implementation method for permanent storage virtual machine restoration based on openstack cloud management platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220301 |