CN110874182A - Processing method, device and equipment for stripe index - Google Patents
Processing method, device and equipment for stripe index Download PDFInfo
- Publication number
- CN110874182A CN110874182A CN201811015213.6A CN201811015213A CN110874182A CN 110874182 A CN110874182 A CN 110874182A CN 201811015213 A CN201811015213 A CN 201811015213A CN 110874182 A CN110874182 A CN 110874182A
- Authority
- CN
- China
- Prior art keywords
- index
- stripe
- strip
- target object
- recorded
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 11
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000012545 processing Methods 0.000 claims description 17
- 238000012217 deletion Methods 0.000 abstract description 10
- 230000037430 deletion Effects 0.000 abstract description 10
- 238000012544 monitoring process Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000000717 retained effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a processing method, a device and equipment of a stripe index, comprising the following steps: determining a target object needing to be deleted; acquiring a searched stripe index of a stripe storing the target object; when determining that other object identifiers except the target object identifier are recorded in the searched strip index, setting the state of the target object as a designated mark; and deleting the searched strip index when the state of the object corresponding to each object identifier recorded in the searched strip index is determined to be the assigned mark. The method provided by the application can prevent the problem that the metadata of the undeleted object is lost due to deletion of the strip index.
Description
Technical Field
The present application relates to the field of storage, and in particular, to a method, an apparatus, and a device for processing a stripe index.
Background
An object storage system generally includes a storage node and a management node. A storage node, also called an OSD (object storage Device) Device, stores objects. The management node, also called MDS (metadata server) device, is configured to store metadata of an object stored in the OSD device, where the metadata of the object may indicate object information such as a location where the object is stored and an input time of the object.
In general, the metadata of an object may include an object index and a stripe index. The object index includes at least: object identification, stripe identification of the stripe storing the object, and object write time, etc. The strip index at least comprises information such as block identification on the strip and object identification of an object stored in the block.
The object storage system can delete some specified objects periodically, when deleting, the storage node and the management node can simultaneously execute a deletion strategy, the storage node deletes the object to be deleted, and the management node deletes the metadata of the object to be deleted. Such as the management node deleting the object index of the object and the stripe index of the stripe storing the object. However, when the deleted tape index also stores other objects, the metadata of the other objects is lost.
Disclosure of Invention
In view of this, the present application provides a method, an apparatus, and a device for processing a stripe index, so as to prevent a problem that metadata of an undeleted object is lost due to deletion of the stripe index.
Specifically, the method is realized through the following technical scheme:
according to a first aspect of the present application, there is provided a method for processing a stripe index, where the method is applied to a management node in an object storage system, and the method includes:
determining a target object to be deleted;
searching a stripe index of a stripe storing the target object;
detecting whether other object identifications except the target object identification are recorded in the searched strip index;
if yes, setting a designated mark for the target object identification;
and deleting the strip index when determining that each object identifier recorded in the strip index is provided with the assigned mark.
Optionally, the method further includes:
and if the searched strip index only records the target object identification, deleting the strip index.
Optionally, the determining a target object to be deleted includes:
determining an object index meeting specified conditions in locally recorded object indexes;
and taking the object indicated by the determined object index as an object to be deleted.
Optionally, the object index records at least an object writing time;
the determining, from the locally recorded object indexes, an object index that satisfies a specified condition includes:
subtracting a preset time length from the current time to determine a deleting time;
and searching the object index of which the object writing time is before the deleting time, and determining the searched object index as the object index meeting the specified condition.
Optionally, the strip index records at least an object identifier;
the searching the stripe index of the stripe storing the target object comprises:
and searching the strip index containing the key word in the recorded strip indexes by taking the object identifier of the target object as the key word.
According to a second aspect of the present application, there is provided a stripe index processing apparatus, which is applied to a management node in an object storage system, and includes:
a determination unit configured to determine a target object to be deleted;
the searching unit is used for searching the strip index of the strip storing the target object;
a detecting unit, configured to detect whether other object identifiers other than the target object identifier are recorded in the searched strip index;
the setting unit is used for setting a specified mark for the target object identification if the target object identification is the same as the target object identification;
and the processing unit is used for deleting the strip index when determining that each object identifier recorded in the strip index is provided with the assigned mark.
Optionally, the processing unit is further configured to delete the stripe index if the found stripe index only records the target object identifier.
Optionally, the determining unit is specifically configured to determine, from the locally recorded object indexes, an object index that meets a specified condition; and taking the object indicated by the determined object index as an object to be deleted.
Optionally, the object index records at least an object writing time;
the determining unit is specifically configured to determine the deleting time by subtracting a preset time length from the current time when determining the object index meeting the specified condition in the locally recorded object indexes; and searching the object index of which the object writing time is before the deleting time, and determining the searched object index as the object index meeting the specified condition.
Optionally, the strip index records at least an object identifier;
the searching unit is specifically configured to search the recorded stripe indexes for the stripe indexes that include the keyword, with the object identifier of the target object as the keyword.
According to a third aspect of the present application, there is provided an object management apparatus comprising a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to perform the processing method of the stripe index provided by the first aspect.
According to a fourth aspect of the present application, there is provided a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method of processing a stripe index provided by the first aspect.
The application provides a processing method of a strip index, a management node can set a designated mark for an object needing to be deleted, and when the strip index is deleted, the management node deletes the strip index when determining that each object recorded by the strip index is set with the designated mark. Otherwise, the stripe index is retained.
Since the management node deletes the strip index only when it is determined that each object recorded on the strip index is provided with the designation flag, it is possible to prevent a problem that metadata of an undeleted object is lost due to deletion of the strip index.
Drawings
FIG. 1 is a schematic view of a strap shown in an exemplary embodiment of the present application;
FIG. 2 is a flow chart illustrating a method for processing a stripe index according to an exemplary embodiment of the present application;
FIG. 3 is a diagram illustrating a hardware configuration of an object management device according to an exemplary embodiment of the present application;
fig. 4 is a block diagram of a slice index processing apparatus according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
Before introducing the stripe index processing method provided by the present application, concepts related to the present application are introduced.
1) Stripes, objects, blocks
In an object storage system, stored data, files, and the like are all referred to as objects.
In an object storage system, a storage node OSD and a management node MDS are typically included. A storage node refers to a storage device for storing objects. The storage node may comprise a number of disks. The N disks of the plurality of disks may form a disk array. The total storage space of the N disks is divided into N stripes, each stripe including N blocks, the N blocks for storing objects.
As shown in fig. 1, the disk array shown in fig. 1 is a disk array composed of 6 disks. Each column in fig. 1 represents a disk and each row represents a stripe, each stripe having 6 blocks.
When the object is stored, if the size of the object is larger than one block on the strip, the object is split into N sub-objects, and the N sub-objects are stored by using the N blocks on the strip. If the size of the object is smaller than a block on the stripe, then a block on the stripe is used to store the object.
2) Object index
An object index table is configured on a management node in the object storage system. The object index table includes a plurality of object indexes, each object index corresponding to an object stored in a storage node. The object index may indicate object information such as a storage location with the object.
At least the object id of the object, the writing time of the object, the stripe id for storing the object, etc. are recorded in the object index, but other contents, such as the size of the object, etc., may also be included in the object index, which is only exemplary and not specifically limited herein.
When the object storage system needs to store the received object, the storage node may store the object in a specified stripe, and the management node may generate an object index of the object.
For example, assume that object 1 needs to be stored, assume that the storage node stores object 1 on block 1 and block 2 of stripe 1 at 2018/05/1619: 00.
The management node may generate an object index 1 for object 1, the generated object index 1 being shown in table 1.
| Object identification | Storing the stripe identification of the object | Time of writing |
| Object 1 | Strip 1 | 2018/05/16 19:00 |
TABLE 1
Of course, other contents, such as the size of the object, etc., may also be included in table 1, and the object index is only exemplarily illustrated here and is not specifically limited.
3) Stripe index
In order to facilitate the management of the stripe, a stripe index table is also configured on the management node in the object storage system. The band index table records the band indexes of all bands. At least: a block identification of a block on the stripe, an object identification of an object the block stores.
For example, assume that object 1 needs to be stored, assume that the storage node stores object 1 on block 1 and block 2 of stripe 1. Assuming that object 2 also needs to be stored, assume that the storage node stores object 2 on block 3 on stripe 1.
At this time, the stripe index of stripe 1 on the management node is as shown in table 2.
TABLE 2
Of course, the strip indexes shown in table 2 may also include other contents, and the strip indexes are only exemplarily illustrated and not specifically limited herein.
Typically, the object storage system will periodically delete some objects to free up space to store newly received objects. When deleting an object, the storage node OSD may delete the object from the block of the stripe. The management node MDS may delete the metadata of the object that needs to be deleted, for example, the MDS may delete the object index and the stripe index of the object that needs to be deleted.
However, when the deleted tape index also contains other object identifiers, the metadata of other objects is lost when the tape index is directly deleted.
For example, suppose the storage system were to delete object 1, object 1 being stored on block 1 and block 2 of stripe 1, and object 2 also being stored on block 3 of stripe 1. And object 2 is an object that does not require deletion.
When deleting object 1, the OSD may delete object 1 from blocks 1 and 2 of stripe 1. Meanwhile, the MDS may delete the object index for object 1 and delete the stripe index for stripe 1. However, since the object 2 is also stored in the stripe index of the stripe 1, and the object 2 is an object that does not need to be deleted, the stripe index of the stripe 1 is deleted, so that the stripe index of the object 2 is lost, which causes the metadata of the object 2 to be lost.
In view of this, the present application provides a method for processing a stripe index, where a management node may set a specified flag for an object that needs to be deleted, and when deleting the stripe index, the management node deletes the stripe index only when determining that each object of the stripe index record is set with the specified flag. Otherwise, the stripe index is retained.
Since the management node deletes the strip index only when it is determined that each object recorded on the strip index is provided with the designation flag, it is possible to prevent a problem that metadata of an undeleted object is lost due to deletion of the strip index.
Referring to fig. 2, fig. 2 is a flowchart illustrating a stripe index processing method according to an exemplary embodiment of the present application. The processing method of the stripe index can be applied to a management node of an object storage system and can comprise the following steps. The management node may periodically perform steps 201 to 203.
Step 201: the management node may determine the target object to be deleted.
In the object storage system, the object storage system periodically deletes some data that is written earlier to free some space to store the newly written object.
In implementation, the management node may search an object index satisfying a specified condition in an object index list recorded locally, and use an object indicated by an object identifier recorded in the searched object index as an object to be deleted.
When searching for an object index that meets the specified condition, an optional implementation manner is that the management node subtracts a preset time length from the current time to obtain the deletion time. Then, the management node may search for an object index whose write time is before the deletion time, and then use an object indicated by the object identifier recorded in the searched object index as a target object to be deleted.
It should be noted that the preset time period is a time period set by a developer according to actual experience, and is not specifically limited herein.
It should be further noted that the management node may also search some specified object indexes as the object indexes meeting the specified condition, and here, only one example of searching the object indexes meeting the specified condition is described, and the object indexes searching the specified condition is not specifically limited.
Step 202: the management node may look up the stripe index of the stripe storing the target object.
In implementation, the management node may use the object identifier of the target object as a key, and search the strip index list for the strip index recorded with the key.
Step 203: the management node can detect whether other object identifiers except the target object identifier are recorded in the searched strip index.
Step 204: if other object identifiers except the target object identifier are recorded in the searched strip index, the management node can set a designated mark for the target object identifier.
If the found stripe index also records other object identifiers except the target object identifier, the management node can set a designated mark for the target object identifier.
For example, the management node may add a specified flag to the target object identifier in the found stripe index, where the specified flag indicates that the target object is an object to be deleted.
If the searched stripe index only records the target object identifier, the management node may delete the searched stripe index from the stripe index list. In addition, the management node can also delete the object index corresponding to the target object.
Of course, the management node may also delete the object index corresponding to the target object when determining the target object to be deleted, and the present application does not specifically limit the time for deleting the object index of the target object, as long as the management node deletes the object index corresponding to the target object after determining the target object to be deleted.
Step 205: and when the management node determines that each object identifier recorded in the searched strip index is provided with the assigned mark, deleting the searched strip index.
In an alternative implementation manner, after performing step 202 and step 203 on each target object to be deleted, the management node may traverse the list of stripe indexes and sequentially read the stripe indexes including the object identifiers of the objects to be deleted.
The management node may perform the following steps for each stripe index read.
Here, the read stripe is taken as the above-mentioned searched stripe index as an example to explain
The management node may detect whether the object identifier of each object of the searched strip index record is set with a specific flag.
If each object in the searched stripe index is configured with the designated mark, the management node can delete the stripe index. In addition, the management node may also delete the object index of the target object.
If some object identifications are configured with the specified marks or are not configured with the specified marks in the object identifications recorded in the searched strip index, the searched strip index is reserved. In addition, the management node may also delete the object index of the target object.
It should be noted that the management node may also delete the object index corresponding to the target object when determining the target object to be deleted, and the present application does not specifically limit the time for deleting the object index of the target object, as long as the management node deletes the object index corresponding to the target object after determining the target object to be deleted.
It should be noted that, it is generally easy to think of a way to start a monitoring task in the background, and delete the stripe index when the monitoring task in the background monitors that all object identifiers in the stripe index are configured with the specified tag. And the program of the current CPU calling the memory only sets a specified mark for the object identifier of the object to be deleted periodically. In short, the setting of the designated mark and the monitoring task of the background do not have relevance, and the setting of the designated mark and the monitoring task of the background respectively perform respective tasks. This is disadvantageous in that, when there are a large number of strip indexes in which a plurality of object identifiers are recorded, a background monitoring task becomes a burden on the MDS, and the performance of the MDS is improved.
Therefore, what is used in this application is: after the operations of step 202 to step 204 are completed for the target object that needs to be deleted. And sequentially reading the strip indexes containing the object identifications of the objects to be deleted from the strip index list, and deleting the strip indexes when determining that the object identifications of the objects recorded by the strip indexes are all configured with the assigned marks. The deleting operation of the strip index is not completed by a background, but the operation of deleting the strip index is executed after the program in the memory is called by the CPU and the appointed marks are set on the object identifications of all the target objects. The background monitoring task does not need to be independently set, so that the problem that the background task consumes resources of the MDS equipment is avoided, and the performance of the equipment can be improved by adopting the method provided by the application.
The following describes in detail a processing method of a slice index provided in the present application by way of specific examples.
Step 1: and the management node determines the object to be deleted.
The management node may determine the object to be deleted by using the method described in step 201, which is not described herein again.
Assume that there are 4 objects to be deleted, object 1, object 2, object 3, and object 4, as determined by the management node.
Step 2: the management node can search the strip index corresponding to each object to be deleted.
Specifically, the management node may use the object identifier of the object to be deleted as a key, search for the stripe index including the key, and determine the found stripe index as the stripe index corresponding to the object to be deleted.
For example, assume that blocks 1-3 on stripe 1 store object 1 and blocks 5-6 on stripe 1 store object 2. Only object 1 and object 2 are stored on stripe 1.
The management node may use the stripe index 1 in which the object identifiers of the object 1 and the object 2 are recorded as the stripe indexes corresponding to the object 1 and the object 2, where the stripe index 1 is shown in table 3.
TABLE 3
Assume that blocks 1-5 on stripe 2 store object 3 and that only object 3 is stored on stripe 2.
The management node may use the stripe index 2 in which the object identifier of the object 3 is recorded as the stripe index corresponding to the object 3, where the stripe index 2 is shown in table 4.
TABLE 4
It is assumed that the blocks 1 to 2 on the band 3 store the object 4, the blocks 4 to 6 on the band 3 store the object 5, and the object 5 is an object that does not need to be deleted. The stripe 3 stores only the object 4 and the object 5.
The management node may identify the object identification stripe index 3, in which the object 4 is recorded, as the stripe index of the object 4. The band index 3 is shown in table 5.
TABLE 5
And step 3: for each object to be deleted, the management node may check whether an object identifier of another object except the object to be deleted is recorded in the stripe index corresponding to the object to be deleted, and if so, set the object identifier of the object to be deleted to set a specified flag. If not, the stripe index is deleted.
For example, for an object 1 to be deleted, the management node may check whether other object identifiers besides the object 1 identifier are recorded in the stripe index 1 corresponding to the object 1, in this example, since the identifier of the object 2 is also recorded in the stripe index 1, the management node adds a specified flag to the object 1 in the stripe index 1.
For the object 2 to be deleted, the management node may check whether other object identifiers except the object 2 identifier are recorded in the stripe index 1 corresponding to the object 2, in this example, because the identifier of the object 1 is also recorded in the stripe index 1, the management node adds a specified mark to the object 2 in the stripe index 1.
Table 6 shows the stripe index 1 of the management node after adding the specific mark to the object 1 and the object 2 in the stripe index 1.
TABLE 6
For the object 3 to be deleted, the management node may check whether other object identifiers besides the object 3 identifier are recorded in the strip index 2 corresponding to the object 3, in this example, since only the object 3 identifier is recorded in the strip index 2, the management node may delete the strip index of the object 3.
For the object 4 to be deleted, the management node may check whether other object identifiers besides the identifier of the object 4 are recorded in the strip index 3 corresponding to the object 4, in this example, because the identifier of the object 5 is also recorded in the strip index 3, the management node adds a specified mark to the object 4 in the strip index 3. The band index 3 to which the specified mark is added to the object 4 is shown in table 7.
TABLE 7
And 4, step 4: after the management node completes the operation of step 3 on the object to be deleted, the management node may search for the strip index including the object identifier of the object to be deleted. For each found stripe index, the management node may check whether each object identifier of the stripe index record is configured with a specified flag, if so, delete the stripe index, and if not, retain the stripe index.
For example, in this example, the management node may acquire that the stripe indexes recorded with the multiple object identifiers are stripe index 1 and stripe index 3.
For the stripe index 1, the management node may check whether the object id of each object in the stripe index 1 is configured with a specific flag, in this example, the object 1 and the object 2 recorded in the stripe index 1 are configured with specific flags, so the management node deletes the stripe index 1.
For the stripe index 3, the management node may check whether each object in the stripe index 3 is configured with the specified flag, in this example, the object 4 recorded in the stripe index 3 is configured with the specified flag, but the object 5 recorded in the stripe index 3 is not configured with the specified flag, so the management node may retain the stripe index 3.
As can be seen from the above description, the management node may set a specified flag for an object to be deleted, and when deleting the strip index, the management node deletes the strip index when determining that the object identifier of each object in the strip index record is configured with the specified flag. Otherwise, the stripe index is retained.
Since the management node deletes the strip index only when determining that the object identifier of each object recorded on the strip index is configured with the specified mark, the problem that the metadata of the undeleted object is lost due to deletion of the strip index can be prevented.
Referring to fig. 3, fig. 3 is a hardware architecture diagram of an object management device in which a stripe index processing apparatus according to an exemplary embodiment of the present application is located. The object management apparatus includes: a communication interface 301, a processor 302, a machine-readable storage medium 303, and a bus 304; wherein the communication interface 301, the processor 302, and the machine-readable storage medium 303 communicate with each other via a bus 304. The processor 302 may perform the above-described stripe index processing method by reading and executing machine-executable instructions in the machine-readable storage medium 303 corresponding to the stripe index processing control logic.
The machine-readable storage medium 303 referred to herein may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and the like. For example, the machine-readable storage medium 303 may include at least one of the following storage media: volatile memory, non-volatile memory, other types of storage media. The volatile Memory may be a Random Access Memory (RAM), and the nonvolatile Memory may be a flash Memory, a storage drive (e.g., a hard disk drive), a solid state disk, and a storage disk (e.g., a compact disk, a DVD).
Referring to fig. 4, fig. 4 is a block diagram illustrating a stripe index deleting apparatus according to an exemplary embodiment of the present application, which may be applied to a management node in an object storage system, and may include the following elements.
A determining unit 401, configured to determine a target object to be deleted;
a searching unit 402, configured to search a stripe index of a stripe storing the target object;
a detecting unit 403, configured to detect whether other object identifiers than the target object identifier are recorded in the searched stripe index;
a setting unit 404, configured to set a specified flag for the target object identifier if yes;
a processing unit 405, configured to delete the strip index when it is determined that each object identifier recorded in the strip index is provided with the assigned flag.
Optionally, the processing unit 405 is further configured to delete the stripe index if the found stripe index only records the target object identifier.
Optionally, the determining unit 401 is specifically configured to determine, from among locally recorded object indexes, an object index that meets a specified condition; and taking the object indicated by the determined object index as an object to be deleted.
Optionally, the object index records at least an object writing time;
the determining unit 401, when determining an object index meeting a specified condition from among locally recorded object indexes, is specifically configured to subtract a preset duration from a current time to determine a deletion time; and searching the object index of which the object writing time is before the deleting time, and determining the searched object index as the object index meeting the specified condition.
Optionally, the strip index records at least an object identifier;
the searching unit 402 is specifically configured to search the recorded stripe indexes for the stripe indexes that include the keyword, with the object identifier of the target object as the keyword.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (12)
1. A processing method of a stripe index is applied to a management node in an object storage system and comprises the following steps:
determining a target object to be deleted;
searching a stripe index of a stripe storing the target object;
detecting whether other object identifications except the target object identification are recorded in the searched strip index;
if yes, setting a designated mark for the target object identification;
and deleting the strip index when determining that each object identifier recorded in the strip index is provided with the assigned mark.
2. The method of claim 1, further comprising:
and if the searched strip index only records the target object identification, deleting the strip index.
3. The method of claim 1, wherein determining the target object to be deleted comprises:
determining an object index meeting specified conditions in locally recorded object indexes;
and taking the object indicated by the determined object index as an object to be deleted.
4. The method according to claim 3, wherein the object index records at least an object writing time;
the determining, from the locally recorded object indexes, an object index that satisfies a specified condition includes:
subtracting a preset time length from the current time to determine a deleting time;
and searching the object index of which the object writing time is before the deleting time, and determining the searched object index as the object index meeting the specified condition.
5. The method of claim 1, wherein the strip index records at least an object identifier;
the searching the stripe index of the stripe storing the target object comprises:
and searching the strip index containing the key word in the recorded strip indexes by taking the object identifier of the target object as the key word.
6. An apparatus for processing a stripe index, the apparatus being applied to a management node in an object storage system, comprising:
a determination unit configured to determine a target object to be deleted;
the searching unit is used for searching the strip index of the strip storing the target object;
a detecting unit, configured to detect whether other object identifiers other than the target object identifier are recorded in the searched strip index;
the setting unit is used for setting a specified mark for the target object identification if the target object identification is the same as the target object identification;
and the processing unit is used for deleting the strip index when determining that each object identifier recorded in the strip index is provided with the assigned mark.
7. The apparatus according to claim 6, wherein the processing unit is further configured to delete the stripe index if the found stripe index only records the target object id.
8. The apparatus according to claim 6, wherein the determining unit is specifically configured to determine, among locally recorded object indexes, an object index that satisfies a specified condition; and taking the object indicated by the determined object index as an object to be deleted.
9. The apparatus of claim 8, wherein the object index records at least an object writing time;
the determining unit is specifically configured to determine the deleting time by subtracting a preset time length from the current time when determining the object index meeting the specified condition in the locally recorded object indexes; and searching the object index of which the object writing time is before the deleting time, and determining the searched object index as the object index meeting the specified condition.
10. The apparatus of claim 6, wherein the stripe index records at least an object identification;
the searching unit is specifically configured to search the recorded stripe indexes for the stripe indexes that include the keyword, with the object identifier of the target object as the keyword.
11. An object management device comprising a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to perform the method of any one of claims 1 to 5.
12. A machine-readable storage medium having stored thereon machine-executable instructions which, when invoked and executed by a processor, cause the processor to perform the method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811015213.6A CN110874182B (en) | 2018-08-31 | 2018-08-31 | Processing method, device and equipment for strip index |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811015213.6A CN110874182B (en) | 2018-08-31 | 2018-08-31 | Processing method, device and equipment for strip index |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110874182A true CN110874182A (en) | 2020-03-10 |
| CN110874182B CN110874182B (en) | 2023-12-26 |
Family
ID=69715476
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811015213.6A Active CN110874182B (en) | 2018-08-31 | 2018-08-31 | Processing method, device and equipment for strip index |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110874182B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625502A (en) * | 2020-05-28 | 2020-09-04 | 浙江大华技术股份有限公司 | Data reading method and device, storage medium and electronic device |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050257083A1 (en) * | 2004-05-13 | 2005-11-17 | Cousins Robert E | Transaction-based storage system and method that uses variable sized objects to store data |
| KR20060067119A (en) * | 2004-12-14 | 2006-06-19 | 한국전자통신연구원 | Crash recovery system and method for a distributed file server using object based storage |
| CN101393532A (en) * | 2007-03-29 | 2009-03-25 | 赛门铁克公司 | Removal of data from a single instancing data archive |
| US8549223B1 (en) * | 2009-10-29 | 2013-10-01 | Symantec Corporation | Systems and methods for reclaiming storage space on striped volumes |
| CN103605726A (en) * | 2013-11-15 | 2014-02-26 | 中安消技术有限公司 | Method and system for accessing small files, control node and storage node |
| US20140214782A1 (en) * | 2013-01-31 | 2014-07-31 | Futurewei Technologies, Inc. | Distributed Storage Object Delete |
| US9047312B1 (en) * | 2011-03-30 | 2015-06-02 | Amazon Technologies, Inc. | System and method for deletion of extraneous delete marker objects |
| JP2015210815A (en) * | 2014-04-25 | 2015-11-24 | ネイバー コーポレーションＮＡＶＥＲ Ｃｏｒｐｏｒａｔｉｏｎ | Database management method and database management system |
| CN105912664A (en) * | 2016-04-11 | 2016-08-31 | 华为技术有限公司 | Method and equipment for file processing |
| WO2017206436A1 (en) * | 2016-06-03 | 2017-12-07 | 杭州海康威视数字技术股份有限公司 | Method, device and system for recovering space on online hard drive |
| CN107562368A (en) * | 2016-06-30 | 2018-01-09 | 杭州海康威视数字技术股份有限公司 | A kind of data processing method and device |
| CN107704203A (en) * | 2017-09-27 | 2018-02-16 | 郑州云海信息技术有限公司 | It polymerize delet method, device, equipment and the computer-readable storage medium of big file |
| CN107870940A (en) * | 2016-09-28 | 2018-04-03 | 杭州海康威视数字技术股份有限公司 | A kind of file memory method and device |
| CN107958079A (en) * | 2017-12-14 | 2018-04-24 | 郑州云海信息技术有限公司 | Aggregate file delet method, system, device and readable storage medium storing program for executing |
-
2018
- 2018-08-31 CN CN201811015213.6A patent/CN110874182B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10599344B1 (en) * | 2004-05-13 | 2020-03-24 | Primos Storage Technology Llc | Transaction-based storage system and method that uses variable sized objects to store data |
| US20050257083A1 (en) * | 2004-05-13 | 2005-11-17 | Cousins Robert E | Transaction-based storage system and method that uses variable sized objects to store data |
| KR20060067119A (en) * | 2004-12-14 | 2006-06-19 | 한국전자통신연구원 | Crash recovery system and method for a distributed file server using object based storage |
| CN101393532A (en) * | 2007-03-29 | 2009-03-25 | 赛门铁克公司 | Removal of data from a single instancing data archive |
| US8549223B1 (en) * | 2009-10-29 | 2013-10-01 | Symantec Corporation | Systems and methods for reclaiming storage space on striped volumes |
| US9047312B1 (en) * | 2011-03-30 | 2015-06-02 | Amazon Technologies, Inc. | System and method for deletion of extraneous delete marker objects |
| US20150261749A1 (en) * | 2011-03-30 | 2015-09-17 | Amazon Technologies, Inc. | System and method for deletion of extraneous delete marker objects |
| US20140214782A1 (en) * | 2013-01-31 | 2014-07-31 | Futurewei Technologies, Inc. | Distributed Storage Object Delete |
| CN103605726A (en) * | 2013-11-15 | 2014-02-26 | 中安消技术有限公司 | Method and system for accessing small files, control node and storage node |
| JP2015210815A (en) * | 2014-04-25 | 2015-11-24 | ネイバー コーポレーションＮＡＶＥＲ Ｃｏｒｐｏｒａｔｉｏｎ | Database management method and database management system |
| CN105912664A (en) * | 2016-04-11 | 2016-08-31 | 华为技术有限公司 | Method and equipment for file processing |
| WO2017206436A1 (en) * | 2016-06-03 | 2017-12-07 | 杭州海康威视数字技术股份有限公司 | Method, device and system for recovering space on online hard drive |
| CN107562368A (en) * | 2016-06-30 | 2018-01-09 | 杭州海康威视数字技术股份有限公司 | A kind of data processing method and device |
| CN107870940A (en) * | 2016-09-28 | 2018-04-03 | 杭州海康威视数字技术股份有限公司 | A kind of file memory method and device |
| CN107704203A (en) * | 2017-09-27 | 2018-02-16 | 郑州云海信息技术有限公司 | It polymerize delet method, device, equipment and the computer-readable storage medium of big file |
| CN107958079A (en) * | 2017-12-14 | 2018-04-24 | 郑州云海信息技术有限公司 | Aggregate file delet method, system, device and readable storage medium storing program for executing |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625502A (en) * | 2020-05-28 | 2020-09-04 | 浙江大华技术股份有限公司 | Data reading method and device, storage medium and electronic device |
| CN111625502B (en) * | 2020-05-28 | 2024-02-23 | 浙江大华技术股份有限公司 | Data reading method and device, storage medium and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110874182B (en) | 2023-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110149803B (en) | Data storage method, system and terminal equipment | |
| US10331349B2 (en) | Shrinking virtual hard disk image | |
| US9183268B2 (en) | Partition level backup and restore of a massively parallel processing database | |
| CN108021513B (en) | Data storage method and device | |
| US20090063534A1 (en) | Data lineage in data warehousing environments | |
| CN107015985B (en) | Data storage and acquisition method and device | |
| US20150046645A1 (en) | Method, Storage System, and Program for Spanning Single File Across Plurality of Tape Media | |
| TW201941189A (en) | Recording device, reading device, recording method, recording program storage medium, reading method, reading program storage medium, and magnetic tape | |
| CN109783274B (en) | Disk snapshot management method and device and readable storage medium | |
| CN113157600A (en) | Space allocation method of shingled hard disk, file storage system and server | |
| CN110874182B (en) | Processing method, device and equipment for strip index | |
| CN108984343B (en) | Virtual machine backup and storage management method based on content analysis | |
| KR20120082176A (en) | Data processing method of database management system and system thereof | |
| CN109165305B (en) | Characteristic value storage and retrieval method and device | |
| US20120011099A1 (en) | File backup apparatus and method | |
| US20080162591A1 (en) | Method of Logging Transactions and a Method of Reversing a Transaction | |
| US9116915B1 (en) | Incremental scan | |
| CN112765170B (en) | Embedded time sequence data management method and device | |
| CN110471623B (en) | Hard disk file writing method, device, computer equipment and storage medium | |
| CN115858471A (en) | Service data change recording method, device, computer equipment and medium | |
| CN110196785B (en) | Data backup management method and device and electronic equipment | |
| JP2010044571A (en) | Database device, data management method and program therefore | |
| CN113254273A (en) | Method, system, device and medium for real-time recovery of principal metadata | |
| CN113625964B (en) | NandFlash-based sequential storage method, electronic device and storage medium | |
| CN116055589B (en) | Data management method and device and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |