CN117762336A - Regular deletion method, device and equipment for Ceph object - Google Patents
Regular deletion method, device and equipment for Ceph object Download PDFInfo
- Publication number
- CN117762336A CN117762336A CN202311786623.1A CN202311786623A CN117762336A CN 117762336 A CN117762336 A CN 117762336A CN 202311786623 A CN202311786623 A CN 202311786623A CN 117762336 A CN117762336 A CN 117762336A
- Authority
- CN
- China
- Prior art keywords
- ceph
- objects
- target
- storage
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000012217 deletion Methods 0.000 title claims description 33
- 230000037430 deletion Effects 0.000 title claims description 32
- 230000000737 periodic effect Effects 0.000 claims description 27
- 238000010276 construction Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 description 4
- 239000007787 solid Substances 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method, a device and equipment for periodically deleting Ceph objects, wherein the method comprises the steps of reading HASH values of the Ceph objects stored by OSD in a distributed object storage system; when at least two Ceph objects with the same HASH value are detected, selecting a target Ceph object from the at least two Ceph objects, and reading the storage position of the target Ceph object; and constructing a preset object recorded with a storage position for each reference Ceph object except the target Ceph object, and deleting each reference Ceph object. According to the embodiment of the application, the Ceph objects with the same object content in the distributed object storage system are searched at regular time, and then the reference Ceph objects are replaced by the preset objects recorded with the storage addresses of the target Ceph objects, so that repeated data in the distributed object storage system are reduced and the space utilization rate of the distributed object storage system is improved under the condition that the reference Ceph objects can be used.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, and a device for periodically deleting a Ceph object.
Background
The distributed Object storage system includes a plurality of storage nodes that can map data to be stored onto Object storage devices (objects-based Storage Device, OSD) of a set of different storage nodes. However, with the development of cloud computing, big data and internet of things, the amount of data to be stored in a distributed object storage system is rapidly increased, and repeated content exists among many data, which results in redundancy or repeated data in the objects stored in the distributed system, and further, the storage space of the distributed system is wasted.
There is thus a need for improvements and improvements in the art.
Disclosure of Invention
The technical problem to be solved by the application is to provide a method, a device and equipment for periodically deleting Ceph objects aiming at the defects of the prior art.
In order to solve the above technical problem, a first aspect of the embodiments of the present application provides a periodic deletion method for a Ceph object, where the periodic deletion method for a Ceph object specifically includes:
reading HASH values of the Ceph objects stored by the OSD in the distributed object storage system;
when at least two Ceph objects with the same HASH value are detected, selecting a target Ceph object from the at least two Ceph objects, and reading the storage position of the target Ceph object;
and constructing a preset object recorded with the storage position for each reference Ceph object except the target Ceph object in at least two Ceph objects, and deleting each reference Ceph object.
The method for periodically deleting the Ceph objects, wherein selecting a target Ceph object from at least two Ceph objects specifically comprises:
reading the storage time of each Ceph object in at least two Ceph objects;
and selecting the earliest stored Ceph object from at least two Ceph objects according to the storage time as a target Ceph object.
The method for periodically deleting a Ceph object, wherein the constructing a preset object recorded with the storage position for each reference Ceph object except for a target Ceph object in at least two Ceph objects specifically includes:
for each reference Ceph object of the at least two Ceph objects except for the target Ceph object, creating an empty object for the reference Ceph object;
and recording the storage position in the empty object to obtain a preset object.
The method for periodically deleting a Ceph object, wherein when at least two Ceph objects with the same HASH value are detected, selecting a target Ceph object from the at least two Ceph objects, and reading a storage location of the target Ceph object, the method further comprises:
constructing a deduplication counter for the target Ceph object;
reading the number of objects of the reference Ceph object, and configuring the number of objects in the deduplication counter.
The method for periodically deleting the Ceph object, wherein the method further comprises the following steps:
when the preset object in the OSD is deleted, searching the target Ceph object based on the storage position of the preset object record, and reducing the deduplication counter of the searched target Ceph object by 1.
The method for periodically deleting the Ceph object, wherein the method further comprises the following steps:
when a target Ceph object in a distributed object storage system is deleted, reading all preset objects recorded with storage addresses of the target Ceph object;
selecting a candidate object from all the read preset objects, and storing the object content of the target Ceph object into the candidate object to obtain a new target Ceph object;
and modifying the storage addresses of all preset object records except the candidate object in all preset objects into the storage address of the candidate object.
The method for periodically deleting the Ceph objects, wherein the HASH value of each Ceph object stored in the OSD in the read distributed object storage system is specifically:
and reading the HASH value of the object recorded by the RocksDB in BuleFS at a preset time through the MGR process to obtain the HASH value of each Ceph object.
The second aspect of the embodiments of the present application provides a periodic deletion apparatus for a Ceph object, where the periodic deletion apparatus for a Ceph object specifically includes:
the reading module is used for reading HASH values of the Ceph objects stored by the OSD in the distributed object storage system;
the selecting module is used for selecting a target Ceph object from at least two Ceph objects when at least two Ceph objects with the same HASH value are detected, and reading the storage position of the target Ceph object;
the construction module is used for constructing preset objects recorded with the storage positions for each reference Ceph object except the target Ceph object in at least two Ceph objects, and deleting each reference Ceph object. A third aspect of the present embodiments provides a computer-readable storage medium storing one or more programs executable by one or more processors to implement steps in a periodic deletion method for a Ceph object as described in any one of the above.
A fourth aspect of the present embodiment provides a terminal device, including: a processor and a memory;
the memory has stored thereon a computer readable program executable by the processor;
the processor, when executing the computer-readable program, implements the steps in the periodic deletion method for a Ceph object as described in any one of the above.
The beneficial effects are that: compared with the prior art, the embodiment of the application provides a method, a device and equipment for periodically deleting Ceph objects, wherein the method comprises the steps of reading HASH values of the Ceph objects stored by OSD in a distributed object storage system; when at least two Ceph objects with the same HASH value are detected, selecting a target Ceph object from the at least two Ceph objects, and reading the storage position of the target Ceph object; and constructing a preset object recorded with the storage position for each reference Ceph object except the target Ceph object in at least two Ceph objects, and deleting each reference Ceph object. According to the embodiment of the application, the Ceph objects with the same object content in the distributed object storage system are searched at regular time, and then the reference Ceph objects are replaced by the preset objects recorded with the storage addresses of the target Ceph objects, so that repeated data in the distributed object storage system are reduced and the space utilization rate of the distributed object storage system is improved under the condition that the reference Ceph objects can be used.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a periodic deletion method for a Ceph object according to an embodiment of the present application.
Fig. 2 is a flowchart of step S20 in a periodic deletion method for a Ceph object according to an embodiment of the present application.
Fig. 3 is a flowchart of step S20 in a periodic deletion method for a Ceph object according to an embodiment of the present application.
Fig. 4 is a schematic structural diagram of a periodic deleting device for a Ceph object according to an embodiment of the present application.
Fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present application.
Detailed Description
The embodiment of the application provides a method, a device and equipment for periodically deleting Ceph objects, which are used for making the purposes, technical schemes and effects of the application clearer and more definite, and the application is further described in detail below by referring to the drawings and the embodiments. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein includes all or any element and all combination of one or more of the associated listed items.
It will be understood by those skilled in the art that all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs unless defined otherwise. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
It should be understood that the sequence number and the size of each step in this embodiment do not mean the sequence of execution, and the execution sequence of each process is determined by the function and the internal logic of each process, and should not constitute any limitation on the implementation process of the embodiment of the present application.
It has been found that a distributed Object storage system includes a plurality of storage nodes, and that data to be stored can be mapped onto Object storage devices (objects-based Storage Device, OSD) of a set of different storage nodes. However, with the development of cloud computing, big data and internet of things, the amount of data to be stored in a distributed object storage system is rapidly increased, and repeated content exists among many data, which results in redundancy or repeated data in the objects stored in the distributed system, and further, the storage space of the distributed system is wasted.
In order to solve the above-mentioned problem, in the embodiment of the present application, HASH values of the Ceph objects stored in the OSD in the distributed object storage system are read; when at least two Ceph objects with the same HASH value are detected, selecting a target Ceph object from the at least two Ceph objects, and reading the storage position of the target Ceph object; and constructing a preset object recorded with the storage position for each reference Ceph object except the target Ceph object in at least two Ceph objects, and deleting each reference Ceph object. According to the embodiment of the application, the Ceph objects with the same object content in the distributed object storage system are searched at regular time, and then the reference Ceph objects are replaced by the preset objects recorded with the storage addresses of the target Ceph objects, so that repeated data in the distributed object storage system are reduced and the space utilization rate of the distributed object storage system is improved under the condition that the reference Ceph objects can be used.
The application will be further described by the description of embodiments with reference to the accompanying drawings.
The embodiment provides a method for periodically deleting a Ceph object, as shown in fig. 1, including:
s10, reading HASH values of the Ceph objects stored by the OSD in the distributed object storage system.
In particular, the distributed object storage system may include a plurality of storage nodes, each of which may include an OSD storage area, and the OSD storage area may include a plurality of OSDs. The OSD is used to store a file to be written, where the OSD corresponds to a hard disk, for example, the OSD may use a solid state drive SSD (Solid State Drive, SSD), which is also called a solid state hard disk, and may also use a conventional magnetic disk, such as a Serial Attached SCSI (SAS) disk, a Serial SATA (SATA) disk, or the like.
In the embodiment of the application, the distributed object storage system is a Ceph object storage system, where the Ceph object storage system provides a standard RESTful API (Web application program interface based on HTTP protocol). In addition, the triggering condition of the periodic deletion of the Ceph object is that the client initiates a periodic deletion request of the Ceph object at a preset time, so that the distributed object storage system can delete the Ceph object periodically based on the periodic deletion request, or the distributed object storage system automatically initiates the periodic deletion operation of the Ceph object every preset time, and the like.
Further, in the distributed object storage system, the HASH value of the Ceph object is recorded in the RocksDB in the BuleFS, and therefore, when the HASH value of the Ceph object needs to be obtained, the HASH value of each Ceph object stored in the distributed object storage system can be obtained by accessing the RocksDB in the BuleFS. Based on this, the HASH value of each Ceph object stored in the OSD in the distributed object storage system is specifically: and reading the HASH value of the object recorded by the RocksDB in BuleFS at a preset time through the MGR process to obtain the HASH value of each Ceph object. Wherein the object HASH value recorded by the RocksDB in the BuleFS is generated when the Ceph object is written into the distributed object storage system and recorded in the RocksDB in the BuleFS. That is, when a Ceph object is written into an OSD in a distributed object storage system, HASH values of the Ceph object are simultaneously calculated and the calculated HASH values are recorded in a RocksDB record in a BuleFS, so that HASH values of all Ceph objects stored in the distributed system can be obtained by accessing the RocksDB record in the BuleFS.
S20, when at least two Ceph objects with the same HASH value exist, selecting a target Ceph object from the at least two Ceph objects, and reading the storage position of the target Ceph object.
Specifically, there may be two or more than two Ceph objects having the same HASH value, and the HASH values of the respective Ceph objects are the same. That is, the same Ceph object exists in the distributed object storage system. The target Ceph object is one of at least two Ceph objects, and one of the at least two Ceph objects other than the target Ceph object is noted as a reference Ceph object. That is, at least two Ceph objects are divided into one target Ceph object and several reference Ceph objects. The storage position is the storage address of the target Ceph object in the distributed object storage system at the current moment, and the target Ceph object can be accessed according to the storage position. That is, the object content of the target Ceph object can be read according to the storage address.
It should be noted that there may be at least two Ceph objects with the same HASH value, which may be one group or multiple groups. That is, after the HASH value of each Ceph object is read, all HASH values may be compared, and a group of Ceph objects having HASH values may be obtained, or a plurality of groups of Ceph objects having HASH values may be obtained, or a group of Ceph objects having HASH values may not be obtained, wherein when a group of Ceph objects having HASH values is not obtained, the deletion operation for the Ceph objects is directly ended; when there are one or more groups, deletion operations are performed for each group, respectively, and the deletion operations performed for each group are the same, a group will be described as an example.
In some implementations, the process of acquiring at least two Ceph objects having the same HASH value may be: after the HASH value of each Ceph object is read, grouping the Ceph objects stored by all the OSDs in the distributed object storage system according to the HASH value to obtain a plurality of Ceph object groups, and then selecting the Ceph object groups with the number of objects being greater than or equal to 2 to obtain at least two Ceph objects with the same HASH value. In addition, when the Ceph objects are periodically deleted, all the Ceph objects stored in the distributed object storage system may be periodically deleted, or the Ceph objects in each OSD in the distributed object storage system may be periodically deleted. That is, when the HASH value of each Ceph object is read, HASH values of all the OSD stored Ceph objects may be read to re-delete the distributed object storage system; the HASH value of the Ceph object stored in the OSD can also be read to re-delete the Ceph object in the OSD; the HASH values of the Ceph objects stored in each OSD can be read respectively, so as to re-delete the Ceph objects in each OSD in parallel.
Further, when at least two Ceph objects with the same HASH value exist, one Ceph object may be randomly selected from the at least two Ceph objects as a target Ceph object, one Ceph object may be selected as a target Ceph object according to the storage time, one Ceph object may be selected as a target Ceph object according to the reading times, and the like. When one Ceph object is selected as the target Ceph object according to the reading times, the Ceph object with the highest reading times can be used as the target Ceph object, so that the times of searching the target Ceph object according to the storage address can be reduced, and the overall reading efficiency of the distributed object storage system can be improved.
In some implementations, as shown in fig. 2, the selecting a target Ceph object from at least two Ceph objects specifically includes:
s21, reading the storage time of each Ceph object in at least two Ceph objects;
s22, selecting the earliest stored Ceph object from at least two Ceph objects according to the storage time as a target Ceph object.
Specifically, the storage time refers to the time when the Ceph object is stored in the distributed object storage system, and the target Ceph object is the earliest stored Ceph object, that is, the storage time of the target Ceph object is earlier than other Ceph objects with the same HASH value. Of course, in practical application, since the distributed object storage system may store the Ceph objects in parallel, there may be two earliest stored Ceph objects in at least two Ceph objects, and at this time, one Ceph object may be randomly selected from the two earliest stored Ceph objects to be used as the target Ceph object. The embodiment of the application adopts the earliest stored Ceph object as the target Ceph object because the Ceph object is continuously stored in the distributed object storage system but not deleted, and the description is likely to be the frequently used Ceph object, so that the Ceph object can be reserved to bring convenience to use. Of course, in practical applications, the latest stored Ceph object may be used as the target Ceph object or the like.
Further, in practical applications, since the re-deletion operation for the target Ceph object is performed periodically, when a Ceph object having the same HASH value is acquired, there may already be a target Ceph object. Therefore, when the target Ceph object is selected from the at least two Ceph objects, whether the target Ceph object exists in the at least two Ceph objects or not can be checked, and when the target Ceph object exists, the target Ceph object is kept unchanged, and other Ceph objects are taken as reference Ceph objects; when the target Ceph object does not exist, selecting one target Ceph object from at least two Ceph objects. Therefore, when the target Ceph object exists, the target Ceph object is directly kept unchanged, the preset object storing the storage address of the target Ceph object is not required to be corrected, and the re-deleting process is simplified.
S30, constructing a preset object recorded with the storage position for each reference Ceph object except the target Ceph object in at least two Ceph objects, and deleting each reference Ceph object.
Specifically, the preset objects are Ceph objects having a preset format, that is, a respective Ceph object having a preset is constructed for each reference Ceph object. The preset format may be preset, for example, using a storage location as an object name, or using an object name of a reference Ceph object as a preset object name, and storing the storage location in a preset object, or the like. In addition, after a preset object is constructed for the reference Ceph object, the preset object is directly stored in the OSD as the reference Ceph object, and the reference Ceph object is deleted, wherein the preset object is stored in a storage position of the reference Ceph object. Therefore, on one hand, the storage position of the reference Ceph object can be kept unchanged, and on the other hand, the storage space required by the reference Ceph object can be reduced, so that the redundant space in the distributed object storage system is reduced, and the resource utilization rate of the distributed object storage system can be improved.
In some implementations, as shown in fig. 3, the configuring the preset object with the storage location recorded for each reference Ceph object except the target Ceph object in the at least two Ceph objects specifically includes:
s31, for each reference Ceph object except for a target Ceph object in at least two Ceph objects, creating an empty object for the reference Ceph object;
s32, recording the storage position in the empty object to obtain a preset object.
Specifically, the blank object refers to a Ceph object including object content, and the storage location may be recorded in the blank object by storing the storage location as object content in the blank object, adding the storage location to an object name, or adding the storage location as attribute information in an object attribute. In the embodiment of the application, in order not to affect the existing calling mode of the reference Ceph object, the storage address is stored as the object content in the empty object to obtain the preset object.
Further, during use of the distributed object storage system, the Ceph object in the distributed object storage system may be deleted, and when the target Ceph object is deleted, the reference Ceph object that needs to access the target Ceph object to obtain the object content may not obtain the object content. Therefore, in the embodiment of the present application, a deduplication identifier is set for a target Ceph object, and the target Ceph object is indicated to be associated with other Ceph objects through the deduplication identifier, and the associated other Ceph objects need to acquire object content by accessing the target Ceph object, so that when deleting the Ceph object, if the deleted Ceph object carries the deduplication identifier, the Ceh object is prompted to have the associated Ceph object.
In some implementations, the deduplication identification may employ a deduplication counter, and a count value of the deduplication counter may vary according to an increase or decrease of a reference Ceph object associated with the target Ceph object. Based on this, when there are at least two Ceph objects with the same HASH value, selecting a target Ceph object from the at least two Ceph objects, and reading a storage location of the target Ceph object, the method further includes:
constructing a deduplication counter for the target Ceph object;
reading the number of objects of the reference Ceph object, and configuring the number of objects in the deduplication counter.
Specifically, the deduplication counter is used for storing the number of times that the object content of the target Ceph object is deleted due to the duplication storage, and by the deduplication counter of the target Ceph object, it can be determined whether there is a Ceph object having the same object content as the target Ceph object in the distributed object storage system.
Because the target Ceph object may already carry the deduplication counter, based on this, before constructing the deduplication counter for the target Ceph object, checking whether the target Ceph object carries the deduplication counter, and when not carrying the deduplication counter, constructing the deduplication counter for the target Ceph object; when carrying the deduplication calculator, the step of reading the number of objects of the reference Ceph object is directly performed. Correspondingly, when the object number is configured in the deduplication counter, if the target Ceph object already carries the deduplication counter, the count of the deduplication counter of the target Ceph object is increased by the object number, and if the target Ceph object does not carry the deduplication counter, the object number is used as the count of the deduplication counter of the target Ceph object.
In some implementations, when deleting each reference Ceph object, only each reference Ceph object may be replaced by a preset object, or each reference Ceph object may be replaced by a preset object, and the HASH value of each reference Ceph object is deleted, so as to avoid being repeatedly selected in the next deduplication. Of course, it should be noted that, when only each reference Ceph object is replaced by a preset object, since the HASH value of each reference Ceph object can also be obtained, when determining at least two Ceph objects with the same HASH value according to the HASH value, all Ceph objects with the same HASH value are obtained first, then after removing the preset object, it is determined whether there are at least two Ceph objects with the same HASH value. For example, a Ceph object A, ceph object B and a Ceph object C with the same HASH value are obtained, where the Ceph object C is a preset object, and then the Ceph objects to be deleted again are a Ceph object a and a Ceph object B, that is, at least two obtained Ceph objects with the same HASH value are a Ceph object a and a Ceph object B.
In some implementations, a Ceph object in the distributed object storage system may be deleted. Accordingly, embodiments of the present application may further include:
when a preset object in the OSD is deleted, searching the target Ceph object based on a storage position of the preset object record, and subtracting 1 from a deduplication counter of the searched target Ceph object;
when a target Ceph object in a distributed object storage system is deleted, reading all preset objects recorded with storage addresses of the target Ceph object;
selecting a candidate object from all the read preset objects, and storing the object content of the target Ceph object into the candidate object to obtain a new target Ceph object;
and modifying the storage addresses of all preset object records except the candidate object in all preset objects into the storage address of the candidate object.
Specifically, when the preset object is deleted, it is indicated that the number of times of the target Ceph object corresponding to the deleted object is reduced by 1, at this time, the preset object may be directly deleted, the target Ceph object is found based on the storage location of the preset object record, and the deletion counter of the found target Ceph object is reduced by 1. When the target Ceph object is deleted, it is indicated that the target Ceph object cannot be acquired according to the storage address of the target Ceph object. Therefore, all preset objects corresponding to the target Ceph object need to be obtained, one preset object is selected from the preset objects to serve as a new target Ceph object, and then object content stored in the target Ceph object is stored in the new target Ceph object. In addition, a deduplication counter is configured for the new target Ceph object, and the deduplication counter configured for the target Ceph object is configured for the deduplication counter of the new target Ceph object after the count of the deduplication counter configured for the target Ceph object is reduced by 1.
In summary, the present embodiment provides a method for periodically deleting Ceph objects, which includes reading HASH values of the Ceph objects stored in OSD in a distributed object storage system; when at least two Ceph objects with the same HASH value are detected, selecting a target Ceph object from the at least two Ceph objects, and reading the storage position of the target Ceph object; and constructing a preset object recorded with a storage position for each reference Ceph object except the target Ceph object, and deleting each reference Ceph object. According to the embodiment of the application, the Ceph objects with the same object content in the distributed object storage system are searched at regular time, and then the reference Ceph objects are replaced by the preset objects recorded with the storage addresses of the target Ceph objects, so that repeated data in the distributed object storage system are reduced and the space utilization rate of the distributed object storage system is improved under the condition that the reference Ceph objects can be used.
Based on the above-mentioned periodic deletion method for the Ceph object, the present embodiment provides a periodic deletion apparatus for the Ceph object, as shown in fig. 4, where the periodic deletion apparatus for the Ceph object specifically includes:
a reading module 100, configured to read HASH values of the Ceph objects stored in the OSD in the distributed object storage system;
a selecting module 200, configured to select a target Ceph object from at least two Ceph objects when at least two Ceph objects having the same HASH value are detected, and read a storage location of the target Ceph object;
the construction module 300 is configured to construct a preset object recorded with the storage location for each reference Ceph object except for the target Ceph object in the at least two Ceph objects, and delete each reference Ceph object.
Based on the above-described periodic deletion method for Ceph objects, the present embodiment provides a computer-readable storage medium storing one or more programs executable by one or more processors to implement the steps in the periodic deletion method for Ceph objects as described in the above-described embodiments.
Based on the above method for periodic deletion of Ceph objects, the present application further provides a terminal device, as shown in fig. 5, which includes at least one processor (processor) 20; a display screen 21; and a memory (memory) 22, which may also include a communication interface (Communications Interface) 23 and a bus 24. Wherein the processor 20, the display 21, the memory 22 and the communication interface 23 may communicate with each other via a bus 24. The display screen 21 is configured to display a user guidance interface preset in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may invoke logic instructions in the memory 22 to perform the methods of the embodiments described above.
Further, the logic instructions in the memory 22 described above may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as a stand alone product.
The memory 22, as a computer readable storage medium, may be configured to store a software program, a computer executable program, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 performs functional applications and data processing, i.e. implements the methods of the embodiments described above, by running software programs, instructions or modules stored in the memory 22.
The memory 22 may include a storage program area that may store an operating system, at least one application program required for functions, and a storage data area; the storage data area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include high-speed random access memory, and may also include nonvolatile memory. For example, a plurality of media capable of storing program codes such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or a transitory storage medium may be used.
In addition, the specific processes that the storage medium and the plurality of instruction processors in the terminal device load and execute are described in detail in the above method, and are not stated here.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting thereof; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.
Claims (10)
1. The periodic deletion method for the Ceph object is characterized by specifically comprising the following steps of:
reading HASH values of the Ceph objects stored by the OSD in the distributed object storage system;
when at least two Ceph objects with the same HASH value are detected, selecting a target Ceph object from the at least two Ceph objects, and reading the storage position of the target Ceph object;
and constructing a preset object recorded with the storage position for each reference Ceph object except the target Ceph object in at least two Ceph objects, and deleting each reference Ceph object.
2. The method for periodic deletion of Ceph objects according to claim 1, wherein selecting a target Ceph object from the at least two Ceph objects comprises:
reading the storage time of each Ceph object in at least two Ceph objects;
and selecting the earliest stored Ceph object from at least two Ceph objects according to the storage time as a target Ceph object.
3. The method for periodically deleting a Ceph object according to claim 1, wherein said constructing a preset object recorded with the storage location for each reference Ceph object except for the target Ceph object of the at least two Ceph objects specifically comprises:
for each reference Ceph object of the at least two Ceph objects except for the target Ceph object, creating an empty object for the reference Ceph object;
and recording the storage position in the empty object to obtain a preset object.
4. The periodic deletion method for Ceph objects according to claim 1, wherein when at least two Ceph objects having the same HASH value are detected, one target Ceph object is selected from the at least two Ceph objects, and a storage location of the target Ceph object is read, the method further comprising:
constructing a deduplication counter for the target Ceph object;
reading the number of objects of the reference Ceph object, and configuring the number of objects in the deduplication counter.
5. The periodic deletion method for a Ceph object according to claim 4, further comprising:
when the preset object in the OSD is deleted, searching the target Ceph object based on the storage position of the preset object record, and reducing the deduplication counter of the searched target Ceph object by 1.
6. The periodic deletion method for a Ceph object according to claim 1, further comprising:
when a target Ceph object in a distributed object storage system is deleted, reading all preset objects recorded with storage addresses of the target Ceph object;
selecting a candidate object from all the read preset objects, and storing the object content of the target Ceph object into the candidate object to obtain a new target Ceph object;
and modifying the storage addresses of all preset object records except the candidate object in all preset objects into the storage address of the candidate object.
7. The method for periodically deleting Ceph objects according to claim 1, wherein the reading HASH values of the Ceph objects stored in the OSD in the distributed object storage system is specifically:
and reading the HASH value of the object recorded by the RocksDB in BuleFS at a preset time through the MGR process to obtain the HASH value of each Ceph object.
8. The device for periodically deleting the Ceph object is characterized by comprising the following components:
the reading module is used for reading HASH values of the Ceph objects stored by the OSD in the distributed object storage system;
the selecting module is used for selecting a target Ceph object from at least two Ceph objects when at least two Ceph objects with the same HASH value are detected, and reading the storage position of the target Ceph object;
the construction module is used for constructing preset objects recorded with the storage positions for each reference Ceph object except the target Ceph object in at least two Ceph objects, and deleting each reference Ceph object.
9. A computer readable storage medium storing one or more programs executable by one or more processors to implement the steps in the periodic deletion method for Ceph objects as in any one of claims 1-7.
10. A terminal device, comprising: a processor and a memory;
the memory has stored thereon a computer readable program executable by the processor;
the processor, when executing the computer-readable program, implements the steps in the periodic deletion method for a Ceph object as set forth in any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311786623.1A CN117762336A (en) | 2023-12-22 | 2023-12-22 | Regular deletion method, device and equipment for Ceph object |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311786623.1A CN117762336A (en) | 2023-12-22 | 2023-12-22 | Regular deletion method, device and equipment for Ceph object |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117762336A true CN117762336A (en) | 2024-03-26 |
Family
ID=90313991
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311786623.1A Pending CN117762336A (en) | 2023-12-22 | 2023-12-22 | Regular deletion method, device and equipment for Ceph object |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117762336A (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160117116A1 (en) * | 2014-10-27 | 2016-04-28 | Mediatek Inc. | Electronic device and a method for managing memory space thereof |
| US20180089183A1 (en) * | 2016-09-23 | 2018-03-29 | Amazon Technologies, Inc. | System for optimizing access to an indexed database |
| CN109522283A (en) * | 2018-10-30 | 2019-03-26 | 深圳先进技术研究院 | A kind of data de-duplication method and system |
| CN110427347A (en) * | 2019-07-08 | 2019-11-08 | 新华三技术有限公司成都分公司 | Method, apparatus, memory node and the storage medium of data de-duplication |
| US20210019067A1 (en) * | 2019-07-18 | 2021-01-21 | Pure Storage, Inc. | Data deduplication across storage systems |
| CN112544038A (en) * | 2019-07-22 | 2021-03-23 | 华为技术有限公司 | Method, device and equipment for compressing data of storage system and readable storage medium |
| KR20210085226A (en) * | 2019-12-30 | 2021-07-08 | 주식회사 안랩 | Method of controlling registration of data on blockchain network, method of controlling access to registered data, and peer performing methods |
| CN113467721A (en) * | 2021-07-22 | 2021-10-01 | 杭州海康威视数字技术股份有限公司 | Data deleting system, method and device |
| US20220075719A1 (en) * | 2020-09-08 | 2022-03-10 | EMC IP Holding Company LLC | Synchronizing garbage collection and incoming data traffic |
| US20220164316A1 (en) * | 2019-08-14 | 2022-05-26 | Huawei Technologies Co., Ltd. | Deduplication method and apparatus |
| US20220171676A1 (en) * | 2020-11-30 | 2022-06-02 | Samsung Electronics Co., Ltd | Storage device with data deduplication, operation method of storage device, and operation method of storage server |
| CN116578746A (en) * | 2023-05-19 | 2023-08-11 | 上海哔哩哔哩科技有限公司 | Object de-duplication method and device |
-
2023
- 2023-12-22 CN CN202311786623.1A patent/CN117762336A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160117116A1 (en) * | 2014-10-27 | 2016-04-28 | Mediatek Inc. | Electronic device and a method for managing memory space thereof |
| US20180089183A1 (en) * | 2016-09-23 | 2018-03-29 | Amazon Technologies, Inc. | System for optimizing access to an indexed database |
| CN109522283A (en) * | 2018-10-30 | 2019-03-26 | 深圳先进技术研究院 | A kind of data de-duplication method and system |
| CN110427347A (en) * | 2019-07-08 | 2019-11-08 | 新华三技术有限公司成都分公司 | Method, apparatus, memory node and the storage medium of data de-duplication |
| US20210019067A1 (en) * | 2019-07-18 | 2021-01-21 | Pure Storage, Inc. | Data deduplication across storage systems |
| CN112544038A (en) * | 2019-07-22 | 2021-03-23 | 华为技术有限公司 | Method, device and equipment for compressing data of storage system and readable storage medium |
| US20220164316A1 (en) * | 2019-08-14 | 2022-05-26 | Huawei Technologies Co., Ltd. | Deduplication method and apparatus |
| KR20210085226A (en) * | 2019-12-30 | 2021-07-08 | 주식회사 안랩 | Method of controlling registration of data on blockchain network, method of controlling access to registered data, and peer performing methods |
| US20220075719A1 (en) * | 2020-09-08 | 2022-03-10 | EMC IP Holding Company LLC | Synchronizing garbage collection and incoming data traffic |
| US20220171676A1 (en) * | 2020-11-30 | 2022-06-02 | Samsung Electronics Co., Ltd | Storage device with data deduplication, operation method of storage device, and operation method of storage server |
| CN113467721A (en) * | 2021-07-22 | 2021-10-01 | 杭州海康威视数字技术股份有限公司 | Data deleting system, method and device |
| CN116578746A (en) * | 2023-05-19 | 2023-08-11 | 上海哔哩哔哩科技有限公司 | Object de-duplication method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11068455B2 (en) | Mapper tree with super leaf nodes | |
| US20210152638A1 (en) | Data processing method, apparatus, and system | |
| US10127233B2 (en) | Data processing method and device in distributed file storage system | |
| US10331641B2 (en) | Hash database configuration method and apparatus | |
| US20200150890A1 (en) | Data Deduplication Method and Apparatus | |
| US11579777B2 (en) | Data writing method, client server, and system | |
| US9355121B1 (en) | Segregating data and metadata in a file system | |
| US11061788B2 (en) | Storage management method, electronic device, and computer program product | |
| CN107329704B (en) | Cache mirroring method and controller | |
| CN110888837B (en) | Object storage small file merging method and device | |
| US20190199794A1 (en) | Efficient replication of changes to a byte-addressable persistent memory over a network | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| US10572335B2 (en) | Metadata recovery method and apparatus | |
| US11449402B2 (en) | Handling of offline storage disk | |
| CN115525631A (en) | Database data migration method, device, equipment and storage medium | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| US9395930B2 (en) | Information processing system, control method of information processing system, and recording medium | |
| CN109508140B (en) | Storage resource management method and device, electronic equipment and system | |
| CN107422990B (en) | Dynamic migration method and device | |
| CN117762336A (en) | Regular deletion method, device and equipment for Ceph object | |
| CN108021562B (en) | Disk storage method and device applied to distributed file system and distributed file system | |
| US10762139B1 (en) | Method and system for managing a document search index | |
| CN111966845B (en) | Picture management method, device, storage node and storage medium | |
| US20220413733A1 (en) | mon service migration method, apparatus, and device, and readable storage medium | |
| CN115904211A (en) | Storage system, data processing method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |