CN113419828B - Object storage life cycle management method and system - Google Patents
Object storage life cycle management method and system Download PDFInfo
- Publication number
- CN113419828B CN113419828B CN202110600500.9A CN202110600500A CN113419828B CN 113419828 B CN113419828 B CN 113419828B CN 202110600500 A CN202110600500 A CN 202110600500A CN 113419828 B CN113419828 B CN 113419828B
- Authority
- CN
- China
- Prior art keywords
- bucket
- fragment
- life cycle
- lifecycle
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 65
- 238000007726 management method Methods 0.000 title abstract description 26
- 239000012634 fragment Substances 0.000 claims abstract description 245
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000012545 processing Methods 0.000 claims abstract description 24
- 230000008569 process Effects 0.000 claims abstract description 12
- 238000012986 modification Methods 0.000 claims description 11
- 230000004048 modification Effects 0.000 claims description 11
- 238000012217 deletion Methods 0.000 claims description 7
- 230000037430 deletion Effects 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000012790 confirmation Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 14
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 238000013508 migration Methods 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Techniques For Improving Reliability Of Storages (AREA)
Abstract
The invention discloses a life cycle management method and a life cycle management system for object storage, wherein the life cycle management method for the object storage comprises the following steps: associating a plurality of bucket shard lifecycle substates within a bucket lifecycle state, wherein each bucket shard corresponds to at least one RGW object; when a bucket lifecycle is executed, all bucket fragment lifecycles within the bucket are executed using multiple threads with bucket fragments as a granularity according to the bucket fragment lifecycle substates. The technical scheme of the invention can solve the problems that the life cycle processing speed is slow and the user requirements are difficult to meet due to the fact that the single thread processes the bucket life cycle in the prior art.
Description
Technical Field
The present invention relates to the field of object storage technologies, and in particular, to a method and a system for managing a life cycle of object storage.
Background
With the development of the internet, the data volume of the internet is larger and larger, and distributed object storage is generated at the same time. As a new network storage architecture, the distributed object storage mode is suitable for storage of picture, video and other unstructured data, has the characteristics of high-speed direct access to a disk for block storage and distributed sharing of file storage, and is widely developed and applied to the Internet.
In an object storage architecture, a bucket is a storage container for objects, and all objects in the bucket are at the same logical level. The bucket has a bucket life cycle. By setting a specified object expiration rule for an object in which a prefix or a tag is specified in the bucket, an automatic expiration deletion operation for the object in the bucket can be realized.
In object storage, object deletion or migration within a bucket lifecycle is at bucket granularity, and even if lifecycle threads are multi-threaded, only one thread can operate the same bucket at the same time. The life cycle is still single threaded for a single bucket. Specifically, in the object store, eligible objects within the bucket can be automatically deleted through the bucket lifecycle. The information of the life cycle execution state of the bucket is stored in the object map omap of the metadata pool lc object and is used for recording the execution state of the life cycle of the bucket and the execution time of the life cycle. When the thread checks the life cycle state of the bucket, if the life cycle is not started to be executed, firstly modifying the life cycle state to be in execution, and then starting to delete the object in the bucket; when a bucket lifecycle state is active, other threads will not operate the bucket.
The above described lifecycle execution is bucket-granular, which results in only one thread executing the bucket's lifecycle execution at a time. When a bucket containing a large number of objects needs to be automatically deleted, the existing life cycle processing speed is slow, the processing efficiency is low, and the requirements of users are difficult to meet.
Disclosure of Invention
The invention provides a method and a system for managing a life cycle of object storage, and aims to solve the problems that in the prior art, a bucket is taken as a granularity, the life cycle of the bucket is executed by a single thread, the processing speed is slow, and the user requirements are difficult to meet.
According to a first aspect of the present invention, there is provided a life cycle management method of an object storage, comprising:
associating a plurality of bucket shard lifecycle substates within a bucket lifecycle state, wherein each bucket shard corresponds to at least one RGW object;
when a bucket lifecycle is executed, all bucket fragment lifecycles within the bucket are executed using multiple threads with bucket fragments as a granularity, according to the bucket fragment lifecycle substates.
Preferably, the method for managing a lifetime of an object storage, wherein the step of adding a bucket fragment lifetime sub-state in the bucket lifetime state, includes:
Storing life cycle execution state information corresponding to the bucket into an object index of a metadata pool;
storing life cycle execution state information corresponding to each barrel fragment into a barrel fragment index of an index pool;
and establishing the association relation between the bucket and the plurality of bucket fragments according to the bucket name and the bucket fragment index.
Preferably, the method for managing a life cycle of an object store further includes:
traversing all Rados objects in the metadata pool, and judging whether the life cycle state of the Rados objects is unexecuted or not;
and when the life cycle state of the Rados object is not executed, sequentially executing the bucket life cycle corresponding to the Rados object according to the bucket sequence of the object index record.
Preferably, the method for managing a lifetime of an object storage, which uses a plurality of threads to execute the lifetime of all the bucket fragments in the bucket with the bucket fragment as a granularity, includes:
when the bucket lifecycle is executed, respectively executing all corresponding bucket fragment lifecycles within the bucket using a plurality of threads to process the RGW object corresponding to each bucket fragment;
and when the bucket fragment life cycle of all the bucket fragments is executed, confirming that the bucket life cycle is executed.
Preferably, the method for managing a lifecycle of object storage, which uses a plurality of threads to respectively execute all corresponding lifecycle of bucket fragments in a bucket, includes:
Traversing the bucket fragment lifecycle substates of all bucket fragments in the bucket according to a preset bucket fragment traversal order when the bucket lifecycle is executed;
when the sub-state of the bucket fragment life cycle with the bucket fragments is not executed, executing the bucket fragment life cycle by using the corresponding thread;
modifying the bucket fragment lifecycle substate into execution to avoid other threads from processing the bucket fragment;
when all RGW objects in the bucket fragment are executed, the bucket fragment life cycle sub-state is modified to be completed.
According to a second aspect of the present invention, there is also provided a lifecycle management system for object storage, comprising:
a state association module for associating a plurality of bucket fragment lifecycle substates within a bucket lifecycle state, wherein each bucket fragment corresponds to at least one RGW object;
and the bucket fragment cycle execution module is used for executing all the bucket fragment life cycles in the bucket by using a plurality of threads with the bucket fragments as granularity according to the sub-states of the bucket fragment life cycles when the bucket life cycles are executed.
Preferably, in the object storage lifecycle management system, the state association module comprises:
the bucket information storage module is used for storing the life cycle execution state information corresponding to the bucket into an object index of the metadata pool;
The system comprises a bucket fragment information storage module, a bucket fragment index storage module and a data processing module, wherein the bucket fragment information storage module is used for storing life cycle execution state information corresponding to each bucket fragment into a bucket fragment index of an index pool;
and the relation establishing submodule is used for establishing the association relation between the bucket and the plurality of bucket fragments according to the bucket name and the bucket fragment index.
Preferably, the life cycle management system of the object storage further includes:
the object traversing module is used for traversing all Rados objects in the metadata pool and judging whether the life cycle state of the Rados objects is unexecuted or not;
and the bucket cycle execution module is used for sequentially executing the bucket life cycles corresponding to the Rados objects according to the bucket sequence of the object index records when the life cycle state of the Rados objects is not executed.
Preferably, in the life cycle management system of object storage, the bucket fragment cycle execution module includes:
a cycle execution submodule, configured to, when a bucket lifecycle is executed, respectively execute all corresponding bucket fragment lifecycles within the bucket using a plurality of threads to process an RGW object corresponding to each bucket fragment;
and the execution confirmation submodule is used for confirming that the bucket life cycle is executed when the bucket life cycle of all the bucket fragments is executed.
Preferably, in the life cycle management system of object storage, the cycle execution submodule includes:
the bucket fragment traversing submodule is used for traversing the bucket fragment life cycle substates of all the bucket fragments in the bucket according to a preset bucket fragment traversing order when the bucket life cycle is executed;
the bucket fragment execution submodule is used for executing the bucket fragment life cycle by using a corresponding thread when the bucket fragment life cycle substate with the bucket fragment is not executed;
the state modification submodule is used for modifying the life cycle sub-state of the barrel fragment into execution so as to avoid other threads from processing the barrel fragment;
and the state modification submodule is also used for modifying the life cycle sub-state of the barrel fragment into the finished state when all the RGW objects in the barrel fragment are executed.
In summary, according to the object storage lifecycle management scheme provided by the present application, a plurality of bucket fragment lifecycle sub-states are associated in the original bucket lifecycle state, so that when the bucket lifecycle is executed, the bucket fragments can be used as a granularity to invoke all bucket fragment lifecycles in a plurality of threads to execute the bucket. Because the granularity of the barrel fragments is finer, each barrel comprises a plurality of barrel fragments, and each thread only operates an object on one barrel fragment, so that the barrel fragments are processed by using a plurality of threads at the same time, namely, the efficient parallel processing of the object storage gateway RGW object is realized by the barrel fragments, the deleting or transferring speed of the object in the barrel is obviously improved, and the problems that the single thread executes the life cycle of the barrel and the processing speed is slow in the prior art are solved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a first method for managing a life cycle of an object storage according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a method for adding a sub-state of a bucket fragment lifecycle according to the embodiment shown in FIG. 1;
FIG. 3 is a flowchart illustrating a multi-threaded execution method of a bucket shard lifecycle according to the embodiment shown in FIG. 1;
FIG. 4 is a flowchart illustrating a method for performing a bucket fragment lifecycle of the embodiment shown in FIG. 3;
FIG. 5 is a flowchart illustrating a life cycle management method for object storage according to a second embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a lifecycle management system for a first object storage according to an embodiment of the present invention;
FIG. 7 is a block diagram illustrating a state association module provided in the embodiment shown in FIG. 6;
FIG. 8 is a schematic structural diagram of a lifecycle management system for a second type of object storage according to an embodiment of the present invention;
FIG. 9 is a block diagram illustrating a barrel fragment cycle execution module according to the embodiment shown in FIG. 6;
fig. 10 is a schematic structural diagram of a cycle execution submodule provided in the embodiment shown in fig. 9.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The main technical problems of the embodiment of the invention are as follows:
in the existing cycle management scheme for object storage, deletion or migration of objects in a bucket lifecycle is granular with buckets, and even if lifecycle threads are multi-threaded, only one thread can operate the same bucket at the same time. The life cycle is still single threaded for a single bucket. Because the lifecycle execution mode is bucket-granular, this results in only one thread executing the bucket's lifecycle execution operation at a time. When a bucket containing a large number of objects needs to be automatically deleted or migrated, the existing life cycle processing speed is slow, the processing efficiency is low, and the requirements of users are difficult to meet.
In order to solve the above problem, referring to fig. 1 in particular, fig. 1 is a schematic flowchart of a lifecycle management method for object storage according to an embodiment of the present invention. As shown in fig. 1, the method for managing the life cycle of the object storage includes:
s110: a plurality of bucket slice lifecycle sub-states are associated within a bucket lifecycle state, wherein each bucket slice corresponds to at least one RGW object. RGW objects, i.e. object storage gateway objects, are objects in buckets in the object storage architecture in the field of the present application. In the bucket life cycle, the RGW object in the bucket needs to be deleted or the storage category of the RGW object needs to be converted periodically according to the rule specified by the configuration. In this step, a plurality of bucket fragment lifecycle substates are associated within the bucket lifecycle state, so that with the bucket fragment as a granularity, lifecycle operations, including deleting or converting objects at regular times, can be performed on the RGW objects within the bucket fragment. The bucket lifecycle state includes the execution condition of the bucket lifecycle, and the bucket fragment lifecycle substates are similar.
As a preferred embodiment, as shown in fig. 2, the step of adding a bucket fragment lifecycle sub-state within a bucket lifecycle state specifically includes:
S111: storing life cycle execution state information corresponding to the bucket into an object index of a metadata pool; in the object storage architecture, the bucket life cycle refers to that an expiration rule of a top is set for an RGW object with a prefix or a tag specified in a bucket, so that an expiration deletion or conversion operation of the RGW object in the bucket can be realized. In the embodiment of the present application, the life cycle execution state information (including the execution state and execution time of the life cycle of the bucket) corresponding to the bucket is stored in the object index omap of the metadata pool lc object. Thus, when the bucket life cycle needs to be executed, the deletion or conversion operation of the RGW object in the bucket can be executed according to the life cycle execution state information in the object index in the metadata pool.
S112: storing life cycle execution state information corresponding to each barrel fragment into a barrel fragment index of an index pool; the bucket fragment is a fragment of an index object of a bucket in the object storage gateway RGW, and is used for recording objects in the bucket and related information of the objects. In the embodiment of the application, the life cycle execution state information corresponding to the bucket fragment is stored in the omap of the index pool bucket fragment, so that the execution state of the RGW object in the bucket fragment can be obtained when the life cycle of the bucket fragment is executed by inquiring the life cycle execution state corresponding to each bucket fragment in the index omap of the prime reason bucket fragment.
S113: and establishing the association relationship between the bucket and the plurality of bucket fragments according to the bucket name and the bucket fragment index.
In the technical solution provided by the embodiment of the present application, the lifecycle execution state information corresponding to the bucket is stored in the object index of the metadata pool, and storing the lifecycle execution state information corresponding to each bucket shard into the bucket shard index of the index pool, and finally, establishing the association relationship between the bucket and a plurality of bucket fragments according to the same name and the index of the bucket fragments, so that when the life cycle of the bucket is executed and the RGW object in the bucket is deleted or converted, the corresponding life cycle execution state of the bucket can be searched in the object index in the metadata pool, and at the moment, according to the association relationship between the bucket and the bucket fragment established by the bucket name and the bucket fragment index, the life cycle execution state information of each barrel fragment in the barrel can be inquired in the barrel fragment index of the index pool, and at the moment, the RGW object in the barrel fragment can be deleted or converted according to the life cycle execution state information of the barrel fragment. Because a bucket corresponds to multiple bucket fragments, it is possible to perform lifecycle operations using a number of threads corresponding to the bucket fragments.
The lifecycle management method provided in the embodiment shown in fig. 1 further includes the following steps after step S110 is processed:
S120: when a bucket lifecycle is executed, all bucket fragment lifecycles within the bucket are executed using multiple threads with bucket fragments as a granularity, according to the bucket fragment lifecycle substates. In the embodiment of the present application, since one bucket includes a plurality of bucket fragments, by executing all the life cycles of the bucket fragments in the bucket with the bucket fragments as granularity, the RGW objects in the bucket fragments can be deleted or converted in parallel by using a plurality of threads corresponding to the number of the bucket fragments. When a bucket lifecycle is executed, the bucket fragmentation is taken as a granularity such that a single thread processes a single bucket fragment. In the object storage service, the bucket fragment defaults to 128, so that the operation of deleting objects in one bucket simultaneously by 128 life cycle threads can be realized at most.
Specifically, as a preferred embodiment, as shown in fig. 3, the step S120: using the bucket fragment as a granularity and using a plurality of threads to execute the life cycle of all the bucket fragments in the bucket, the method comprises the following steps:
s121: when a bucket lifecycle is executed, all corresponding bucket slice lifecycles within the bucket are executed separately using multiple threads to process the RGW object corresponding to each bucket slice. In the embodiment of the present application, the bucket lifecycle is executed with the bucket fragments as a granularity, so that one thread corresponds to one bucket fragment, and the bucket includes a plurality of bucket fragments, so that the plurality of threads can be used to respectively execute all corresponding bucket fragment lifecycles in the bucket, and delete or convert the RGW objects in the bucket fragments. The execution efficiency of the whole bucket life cycle is improved.
S122: and when the bucket fragment life cycle of all the bucket fragments is executed, confirming that the bucket life cycle is executed. Because a bucket includes multiple bucket fragments, it can be confirmed that the bucket life cycle is completed when the bucket fragment life cycles of all the bucket fragments are completed.
In the technical scheme provided by the embodiment of the application, because one bucket comprises a plurality of bucket fragments, the embodiment of the application takes the bucket fragments as granularity, and one thread correspondingly processes one bucket fragment when executing the life cycle of the bucket, so that when executing the life cycle of the bucket, the life cycle of the bucket is equivalently split into a plurality of bucket fragments, namely, the life cycles of the plurality of bucket fragments can be processed in parallel by using a plurality of threads, and the deleting or converting efficiency of RGW objects in the bucket fragments is improved.
As a preferred embodiment, as shown in fig. 4, the step S121: using a plurality of threads to respectively execute all corresponding bucket fragment lifecycles in the bucket, including:
s1211: when the bucket lifecycle is executed, the bucket fragment lifecycle substates of all bucket fragments within the bucket are traversed in a preset bucket fragment traversal order. The bucket fragment lifecycle substates include unexecuted, in execution, and completed. By traversing the bucket fragment lifecycle substates of all bucket fragments in the bucket, the operations of deleting or converting the RWG object can be performed for all unexecuted bucket fragments in the bucket.
S1212: when the bucket fragment lifecycle sub-state for which there is a bucket fragment is unexecuted, the bucket fragment lifecycle is executed using the corresponding thread. In the embodiment of the application, when the bucket fragment lifecycle sub-state of the bucket fragment is unexecuted, it indicates that the plurality of RGW objects corresponding to the bucket fragment are not deleted or converted, and at this time, the corresponding thread is used to execute the bucket fragment lifecycle, so that the deletion or conversion action can be executed on the unexecuted RGW objects in the bucket fragment, and further, the execution efficiency of the bucket is improved in a parallel execution manner.
S1213: the bucket fragment lifecycle substate is modified to be in execution to avoid other threads from processing the bucket fragment. When the sub-state of the bucket fragment life cycle is modified to be in execution, the fact that the thread processes the bucket fragment life cycle is indicated, and other threads cannot process the bucket fragment at the moment, so that other threads can be used for processing other bucket fragments in the bucket, and the overall execution efficiency of the bucket life cycle is improved.
S1214: when all RGW objects in the bucket fragment are executed, the bucket fragment life cycle sub-state is modified to be completed.
In summary, according to the method for managing the life cycle of the object storage provided by the embodiment of the present application, a plurality of bucket fragment life cycle sub-states are associated in the original bucket life cycle state, so that through the bucket fragment life cycle sub-states, when the bucket life cycle is executed, the bucket fragments can be used as the granularity, and a plurality of threads are invoked to execute all the bucket fragment life cycles in the bucket. Because the granularity of the barrel fragments is finer, each barrel comprises a plurality of barrel fragments, and each thread only operates an object on one barrel fragment, so that the barrel fragments are processed by using a plurality of threads at the same time, namely, the efficient parallel processing of the object storage gateway RGW object is realized by the barrel fragments, the deleting or transferring speed of the object in the barrel is obviously improved, and the problems that the single thread executes the life cycle of the barrel and the processing speed is slow in the prior art are solved.
As a preferred embodiment, as shown in fig. 5, the lifecycle management method for object storage according to the embodiment of the present application further includes the following steps in addition to the above steps:
s210: traversing all Rados objects in the metadata pool, and judging whether the life cycle state of the Rados objects is unexecuted or not; one metadata pool includes a plurality of Rados objects, Rados (scalable, automated Distributed Object Store), that is, a novel network storage architecture mentioned in the background of the present application, in which one Rados Object corresponds to a plurality of buckets. Thus, in traversing all Rados objects in the metadata pool, it can be determined whether there is a lifecycle state of Rados objects as unexecuted. In the embodiment of the present application, for example, the objects lc.1 to lc.x in the lc pool can be traversed in a randomly generated order.
S220: and when the life cycle state of the Rados object is not executed, sequentially executing the bucket life cycle corresponding to the Rados object according to the bucket sequence of the object index record. For example, when looking up the state of Rados object-lc.x, if the first operation lc.x is performed, the lifecycle state of all buckets on lc.x is modified to be unexecuted, and the lifecycle state corresponding to the bucket fragments in the buckets is also modified to be unexecuted at this time. The bucket sequence is recorded in the object index record of Rados object-lc.x, and the life cycle state in the bucket can be executed in sequence according to the bucket sequence. In addition, in the embodiment of the present application, the bucket is further divided into a plurality of bucket fragments, and a plurality of threads are used to perform the deleting or converting operation of the RGW object with the bucket fragments as the granularity, that is, step S120 shown in fig. 1 is executed.
In addition, based on the same concept of the above method embodiment, the embodiment of the present invention further provides a patent of a lifecycle management system for object storage, which is used to implement the above method of the present invention, and because the principle of solving the problem of the embodiment of the lifecycle management system is similar to that of the above method, the embodiment of the lifecycle management system at least has all the beneficial effects brought by the technical solutions of the above embodiments, and details are not repeated here.
Referring to fig. 6, fig. 6 is a schematic structural diagram of a lifecycle management system for object storage according to an embodiment of the present invention. As shown in fig. 6, the life cycle management system of the object storage includes:
a state association module 110 configured to associate a plurality of bucket fragment lifecycle sub-states within a bucket lifecycle state, wherein each bucket fragment corresponds to at least one RGW object.
The bucket fragment cycle execution module 120 is configured to execute all bucket fragment lifecycles in the bucket using multiple threads with bucket fragments as a granularity according to the bucket fragment lifecycle substates when the bucket lifecycle is executed.
In summary, according to the method for managing the life cycle of the object storage provided by the embodiment of the present application, a plurality of bucket fragment life cycle sub-states are associated in the original bucket life cycle state, so that through the bucket fragment life cycle sub-states, when the bucket life cycle is executed, the bucket fragments can be used as the granularity, and a plurality of threads are invoked to execute all the bucket fragment life cycles in the bucket. Because the granularity of the barrel fragments is finer, each barrel comprises a plurality of barrel fragments, and each thread only operates an object on one barrel fragment, so that the barrel fragments are processed by using a plurality of threads at the same time, namely, the efficient parallel processing of the object storage gateway RGW object is realized by the barrel fragments, the deleting or transferring speed of the object in the barrel is obviously improved, and the problems that the single thread executes the life cycle of the barrel and the processing speed is slow in the prior art are solved.
As a preferred embodiment, as shown in fig. 7, in the lifecycle management system for object storage provided in the embodiment of the present application, the state association module 110 includes:
a bucket information storage module 111, configured to store life cycle execution state information corresponding to a bucket into an object index of a metadata pool;
a bucket fragment information storage module 112, configured to store life cycle execution state information corresponding to each bucket fragment into a bucket fragment index of the index pool;
the relationship establishing submodule 113 is configured to establish an association relationship between a bucket and a plurality of bucket fragments according to the bucket name and the bucket fragment index.
As a preferred embodiment, as shown in fig. 8, the lifecycle management system for object storage according to the embodiment of the present application further includes, in addition to the above modules:
an object traversing module 130, configured to traverse all Rados objects in the metadata pool, and determine whether a life cycle state of a Rados object is unexecuted;
and a bucket cycle executing module 140, configured to, when there is a lifetime state of the Rados object that is not executed, sequentially execute bucket lifetimes corresponding to the Rados object according to a bucket order of the object index record.
As a preferred embodiment, as shown in fig. 9, the barrel slice period execution module 120 of the embodiment of the present application includes:
A cycle execution sub-module 121, configured to, when the bucket lifecycle is executed, respectively execute all corresponding bucket fragment lifecycles within the bucket using a plurality of threads to process the RGW object corresponding to each bucket fragment;
and an execution confirmation sub-module 122, configured to, when the bucket fragment lifecycle of all the bucket fragments is executed completely, confirm that the bucket lifecycle is executed completely.
As a preferred embodiment, as shown in fig. 10, in the embodiment of the present application, the cycle execution submodule shown in fig. 9 includes:
a bucket fragment traversal submodule 1211 configured to traverse bucket fragment lifecycle substates of all bucket fragments in the bucket in a preset bucket fragment traversal order when the bucket lifecycle is executed;
the bucket fragment execution sub-module 1212 is configured to execute the bucket fragment lifecycle using the corresponding thread when the bucket fragment lifecycle sub-state in which the bucket fragment exists is unexecuted;
a state modification submodule 1213, configured to modify the bucket fragment lifecycle sub-state into execution, so as to avoid processing the bucket fragment by other threads;
the state modification sub-module 1213 is further configured to modify the bucket fragment lifecycle sub-state to be completed when all RGW objects within the bucket fragment are completely executed.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (8)
1. A method for lifecycle management for object storage, comprising:
associating a plurality of bucket shard lifecycle substates within a bucket lifecycle state, wherein each bucket shard corresponds to at least one RGW object;
when the bucket lifecycle is executed, executing all bucket fragment lifecycles in the bucket using a plurality of threads with the bucket fragments as granularity according to the bucket fragment lifecycle substates;
the step of adding a bucket fragment lifecycle sub-state within a bucket lifecycle state comprises: storing life cycle execution state information corresponding to the bucket into an object index of a metadata pool;
Storing life cycle execution state information corresponding to each barrel fragment into a barrel fragment index of an index pool;
establishing an association relationship between a bucket and a plurality of bucket fragments according to the bucket name and the bucket fragment index, searching a life cycle execution state corresponding to the bucket in an object index in a metadata pool, and executing deletion or conversion operation on an RGW object in the bucket fragment according to the life cycle execution state information of the bucket fragment;
traversing all Rados objects in the metadata pool, and judging whether the life cycle state of the Rados objects is unexecuted or not;
and when the life cycle state of the Rados object is not executed, sequentially executing the bucket life cycle corresponding to the Rados object according to the bucket sequence of the object index record.
2. The method of claim 1, wherein the step of using multiple threads to execute the lifecycle of all bucket shards within a bucket at bucket shard granularity comprises:
when the bucket lifecycle is executed, respectively executing all corresponding bucket fragment lifecycles in the bucket by using a plurality of threads to process the RGW object corresponding to each bucket fragment;
and when the bucket fragment life cycle of all the bucket fragments is executed, confirming that the bucket life cycle is executed.
3. The method of claim 2, wherein the step of using the plurality of threads to respectively execute all corresponding bucket shard lifecycles within a bucket comprises:
traversing the bucket fragment lifecycle substates of all bucket fragments in the bucket according to a preset bucket fragment traversal order when the bucket lifecycle is executed;
when the sub-state of the bucket fragment life cycle with the bucket fragments is not executed, executing the bucket fragment life cycle by using a corresponding thread;
modifying the bucket fragment lifecycle substate to be in execution to avoid other threads from processing the bucket fragment;
and when the execution of all RGW objects in the barrel fragment is finished, modifying the life cycle sub-state of the barrel fragment to be finished.
4. A lifecycle management system for object storage, comprising:
a state association module for associating a plurality of bucket fragment lifecycle substates within a bucket lifecycle state, wherein each bucket fragment corresponds to at least one RGW object;
and the bucket fragment cycle execution module is used for executing all the bucket fragment life cycles in the bucket by using a plurality of threads with the bucket fragments as granularity according to the bucket fragment life cycle sub-state when the bucket life cycle is executed.
5. The object store lifecycle management system of claim 4, wherein the state association module comprises:
the bucket information storage module is used for storing the life cycle execution state information corresponding to the bucket into an object index of the metadata pool;
the bucket fragment information storage module is used for storing the life cycle execution state information corresponding to each bucket fragment into the bucket fragment indexes of the index pool;
and the relation establishing submodule is used for establishing the incidence relation between the barrel and the plurality of barrel fragments according to the barrel name and the barrel fragment index.
6. The object store lifecycle management system according to claim 5, further comprising:
the object traversing module is used for traversing all Rados objects in the metadata pool and judging whether the life cycle state of the Rados objects is unexecuted or not;
and the bucket cycle execution module is used for sequentially executing the bucket life cycles corresponding to the Rados objects according to the bucket sequence of the object index records when the life cycle state of the Rados objects is not executed.
7. The object storage lifecycle management system of claim 4, wherein the bucket shard cycle execution module comprises:
A cycle execution submodule, configured to, when a bucket lifecycle is executed, respectively execute all corresponding bucket fragment lifecycles within the bucket using a plurality of threads to process an RGW object corresponding to each bucket fragment;
and the execution confirmation submodule is used for confirming that the execution of the bucket life cycle is finished when the execution of the bucket life cycle of all the bucket fragments is finished.
8. The object store lifecycle management system according to claim 7, wherein the cycle execution submodule, comprising:
the barrel fragment traversal submodule is used for traversing the barrel fragment life cycle sub-states of all barrel fragments in the barrel according to a preset barrel fragment traversal order when the barrel life cycle is executed;
the bucket fragment execution submodule is used for executing the bucket fragment life cycle by using a corresponding thread when the bucket fragment life cycle substate with the bucket fragment is not executed;
a state modification submodule, configured to modify a sub-state of the bucket fragment lifecycle into execution, so as to avoid processing the bucket fragment by other threads;
and the state modification submodule is also used for modifying the life cycle substate of the barrel fragment into the finished state when the execution of all the RGW objects in the barrel fragment is finished.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110600500.9A CN113419828B (en) | 2021-05-31 | 2021-05-31 | Object storage life cycle management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110600500.9A CN113419828B (en) | 2021-05-31 | 2021-05-31 | Object storage life cycle management method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113419828A CN113419828A (en) | 2021-09-21 |
| CN113419828B true CN113419828B (en) | 2022-07-29 |
Family
ID=77713395
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110600500.9A Active CN113419828B (en) | 2021-05-31 | 2021-05-31 | Object storage life cycle management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113419828B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115934363A (en) * | 2023-03-07 | 2023-04-07 | 苏州浪潮智能科技有限公司 | Method, system and device for emptying bucket of distributed object storage system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111339033A (en) * | 2020-02-29 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Method, system and terminal for realizing object storage life cycle |
| CN111913925A (en) * | 2019-05-08 | 2020-11-10 | 厦门网宿有限公司 | Data processing method and system in distributed storage system |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017035395A1 (en) * | 2015-08-25 | 2017-03-02 | Huawei Technologies Co., Ltd. | System and method for network function virtualization resource management |
| CN108776682B (en) * | 2018-06-01 | 2021-06-22 | 紫光西部数据(南京)有限公司 | Method and system for randomly reading and writing object based on object storage |
| CN112417036A (en) * | 2018-09-10 | 2021-02-26 | 华为技术有限公司 | Method and device for processing metadata of object in distributed storage system |
| CN111857890B (en) * | 2020-07-24 | 2021-08-31 | 苏州浪潮智能科技有限公司 | Service processing method, system, device and medium |
| CN112269530A (en) * | 2020-10-10 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Data storage management method and related device |
| CN112416892A (en) * | 2020-12-10 | 2021-02-26 | 四川警察学院 | Emergency video data cloud storage system |
-
2021
- 2021-05-31 CN CN202110600500.9A patent/CN113419828B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913925A (en) * | 2019-05-08 | 2020-11-10 | 厦门网宿有限公司 | Data processing method and system in distributed storage system |
| CN111339033A (en) * | 2020-02-29 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Method, system and terminal for realizing object storage life cycle |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113419828A (en) | 2021-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102831120B (en) | A kind of data processing method and system | |
| US10235474B2 (en) | In-memory graph analytics system that allows memory and performance trade-off between graph mutation and graph traversal | |
| CN106776783B (en) | Unstructured data memory management method and system | |
| CN104461744A (en) | Resource allocation method and device | |
| CN109145055B (en) | Data synchronization method and system based on Flink | |
| CN114077602B (en) | Data migration method and device, electronic equipment and storage medium | |
| US20240104059A1 (en) | Method for Service Processing and System, Device, and Medium | |
| CN109885642B (en) | Hierarchical storage method and device for full-text retrieval | |
| CN113419828B (en) | Object storage life cycle management method and system | |
| CN104601562A (en) | Interactive method and system of game server and database | |
| CN110134646B (en) | Knowledge platform service data storage and integration method and system | |
| CN102724301B (en) | Cloud database system and method and equipment for reading and writing cloud data | |
| CN114969165B (en) | Data query request processing method, device, equipment and storage medium | |
| CN110119389B (en) | Writing operation method of virtual machine block equipment, snapshot creation method and device | |
| WO2022253165A1 (en) | Scheduling method, system, server and computer readable storage medium | |
| CN114579303A (en) | Industrial Internet service data processing method, equipment and medium | |
| CN114579059A (en) | File data migration method, device, equipment and storage medium | |
| CN110222105B (en) | Data summarization processing method and device | |
| CN114528078A (en) | Data migration method, device, equipment and medium for object storage | |
| CN107463374B (en) | Subset model extraction method and system based on inheritance and association automatic deduction | |
| TW202036326A (en) | Database migration method, device and apparatus and computer readable medium | |
| CN117176743B (en) | Data processing method, apparatus, device, readable storage medium, and program product | |
| CN117371413B (en) | Organization information viewing method and system | |
| CN116820712A (en) | Thread pool management method, device, equipment and medium | |
| CN116644922A (en) | FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |