CN115114232A - Method, device and medium for enumerating historical version objects - Google Patents
Method, device and medium for enumerating historical version objects Download PDFInfo
- Publication number
- CN115114232A CN115114232A CN202210832787.2A CN202210832787A CN115114232A CN 115114232 A CN115114232 A CN 115114232A CN 202210832787 A CN202210832787 A CN 202210832787A CN 115114232 A CN115114232 A CN 115114232A
- Authority
- CN
- China
- Prior art keywords
- version
- key value
- enumeration
- name
- value pair
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
Abstract
The application discloses a history version object enumerating method, a device and a medium thereof, which relate to the technical field of object storage and are used for enumerating history version objects, and aiming at the problems that the current method for enumerating the history version objects is low in efficiency and has large influence on the performance of a storage system, the history version object enumerating method is provided. All key value pairs in each index fragment do not need to be traversed, so that the time consumption of invalid traversal is avoided, the workload of the storage system during the enumeration of the historical version objects is reduced, and the influence on the performance of the storage system is reduced.
Description
Technical Field
The present application relates to the field of object storage technologies, and in particular, to a method, an apparatus, and a medium for enumerating historical version objects.
Background
With the development of internet technology, the data volume of internet, especially the data volume of unstructured data, is growing more and more, and distributed object storage is in the process of growing. The novel storage mode is suitable for storing unstructured data such as pictures and videos. The method has the characteristics of high-speed direct access to the disk for block storage and distributed sharing of file storage, and therefore, the method is widely developed and applied once being launched. The distributed object storage has a multi-version function, when the same-name files are repeatedly uploaded into the same storage bucket, a randomly generated version id is set for each same-name file to uniquely identify the objects of different versions, and meanwhile, the distributed object storage is different from a same-name file covering mechanism, so that the multi-version files cannot delete the previous same-name files, but store the same-name files in a system after the version ids are assigned.
Currently, when a user requests to enumerate historical versions of data stored in a distributed object storage system, each index slice in a bucket needs to be traversed, and each index slice stores a large number of key-value pairs (key-values), wherein a key includes an object name and a version id, and a value includes basic information of an object. And after traversing all the index fragments and all the key value pairs under each index fragment, returning the objects with the historical version marks as an enumeration result. This approach to traversing all storage objects is time consuming and inefficient, and can also severely impact the performance of the storage system when the number of storage objects is large.
Therefore, those skilled in the art need a history version object enumeration method to solve the problems of low efficiency and large impact on the performance of the storage system of the current history version object enumeration method.
Disclosure of Invention
The application aims to provide a history version object enumeration method, a history version object enumeration device and a history version object enumeration medium, so as to solve the problems that the existing history version object enumeration method is low in efficiency and has a large influence on the performance of a storage system.
In order to solve the above technical problem, the present application provides a history version object enumeration method, including:
when a history version object enumeration request is received, acquiring a target key value pair according to a history version prefix name; the historical version prefix name is a prefix name which is added and stored in a historical version object key value pair when a homonymous object is newly added;
traversing the obtained target key value pair to obtain object information of the historical version object;
and returning the object information of the history version object as an enumeration result.
Preferably, the method further comprises the following steps:
receiving an object file uploaded by a user side and storing the object file in a corresponding storage bucket;
if the object file is the same-name object file of the existing object in the storage bucket, newly adding a key value pair record for the existing object; wherein a key in the key value pair comprises: a history version prefix name, an object name, and a version id.
Preferably, the historical version enumeration request further includes: requesting an object name;
correspondingly, the method also comprises the following steps:
and screening the acquired target key value pairs according to the request object names, and filtering out the key value pairs of which the object names are not the request object names to acquire new target key value pairs.
Preferably, the historical version enumeration request further includes: requesting a version id;
correspondingly, the method also comprises the following steps:
and screening the obtained target key value pair according to the request version id, and filtering out the key value pair of which the version id is not the request version id so as to obtain a new target key value pair.
Preferably, after adding a key-value pair record to an existing object, the method further includes:
and deleting the original key value pair records of the existing objects.
Preferably, the storing in the corresponding bucket includes:
performing hash calculation according to the object name of the object file to obtain a hash value;
and determining a corresponding storage bucket according to the hash value.
Preferably, after returning the object information of the history version object as the enumeration result, the method further includes:
and returning the prompt information to the user side.
In order to solve the above technical problem, the present application further provides an apparatus for enumerating history version objects, including:
the request receiving module is used for acquiring a target key value pair according to the prefix name of the historical version when receiving the enumeration request of the historical version object; the historical version prefix name is a prefix name which is added and stored in a historical version object key value pair when a homonymous object is newly added;
the object traversing module is used for traversing the acquired target key value pair to acquire object information of the historical version object;
and the result generation module is used for returning the object information of the history version object as an enumeration result.
Preferably, the method further comprises the following steps:
the object adding module is used for receiving the object file uploaded by the user side and storing the object file in the corresponding storage bucket; if the object file is the same-name object file of the existing object in the storage bucket, newly adding a key value pair record for the existing object; wherein a key in the key value pair comprises: a historical version prefix name, an object name, and a version id.
And the object screening module is used for screening the acquired target key value pairs according to the request object names and filtering key value pairs of which the object names are not the request object names so as to acquire new target key value pairs.
And the version id screening module is used for screening the obtained target key value pair according to the request version id and filtering the key value pair of which the version id is not the request version id so as to obtain a new target key value pair.
And the space release module is used for deleting the original key value pair records of the existing object.
And the result prompting module is used for returning prompting information to the user side.
In order to solve the above technical problem, the present application further provides an apparatus for enumerating history version objects, including:
a memory for storing a computer program;
a processor for implementing the steps of the history version object enumeration method when executing the computer program.
In order to solve the above technical problem, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the history version object enumeration method as described above.
According to the method for enumerating the objects of the historical versions, because a specific record on an index fragment is acquired according to a specified prefix name in a bottom mechanism of object storage, when a file of an object with the same name is newly added and stored, the key value pair record of the existing object is newly added, the key value of the newly added key value pair is represented by the prefix name of the historical version, the name of the object name and the version id, the prefix name of the historical version is a specific prefix name and is only used by the object of the historical version, so when the object of the historical version is enumerated, all the key value pair records including the prefix name of the historical version can be acquired by using the bottom mechanism of the object storage, the basic information of the object is further traversed and acquired and is returned as an enumeration result, and the enumeration process of the object of the historical version is completed. According to the enumeration method provided by the application, all key value pairs in each index fragment do not need to be traversed, only after the objects of the historical version are screened out through the prefix names, the objects are traversed to obtain the basic information of the objects, so that the time consumption of invalid traversal is avoided, the workload of a storage system during the enumeration of the objects of the historical version is reduced, and the influence on the performance of the storage system is reduced.
The history version object enumerating device and the computer readable storage medium provided by the application correspond to the method, and the effect is the same as the effect.
Drawings
In order to more clearly illustrate the embodiments of the present application, the drawings needed for the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
FIG. 1 is a flowchart of a history version object enumeration method according to the present invention;
FIG. 2 is a block diagram of an apparatus for enumerating historical version objects according to the present invention;
fig. 3 is a block diagram of another history version object enumerating apparatus according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without any creative effort belong to the protection scope of the present application.
The core of the application is to provide a history version object listing method, a history version object listing device and a history version object listing medium.
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings.
In the default storage condition of the distributed object storage system, the object file is stored in a storage bucket in a key value pair mode, wherein key is the object name of the object file, and value is basic information of the object file. When the object files with the same name are uploaded to the same storage bucket, the files with the same name in the bucket are covered, and the object storage has no recycle bin function, so that the covered original files cannot be recovered after the original files stored in the object storage are covered due to certain error operation of a user, and the files are lost and cannot be recovered. Therefore, to avoid such an unexpected situation of data loss due to a malfunction, a multi-versioning function has been developed for the object storage system. When files with the same name are repeatedly uploaded in the same bucket, randomly generated version identity identification numbers (ids) are set for each file with the same name and used for uniquely identifying objects with different versions. Meanwhile, the method is different from a same-name file covering mechanism, the previous same-name file cannot be deleted by the multi-version, and the same-name file is stored in the system after being assigned with the version id. The key value of the history version object with the newly added version id has no change, and the key is still used for storing the basic information of the object file, but the key is changed into the form of object name + version id.
However, in an application scenario of the distributed object system, a large amount of data is mostly stored, and a user usually stores tens of thousands or even millions of objects in a bucket.
When storing files with the same name, the current object storage multi-version function stores data parts of files with all versions in a data pool, and records basic information of all files with the same name on index fragments in the index pool, wherein keys recorded on the index fragments are different from the default storage condition and are in the form of object name + version id, and value is the basic information of the object. When the historical versions of the storage bucket are enumerated, all key value pair records on all the bucket index fragments need to be traversed, whether the historical versions are the historical versions or not is distinguished according to the version id recorded on the index fragments, the historical version object is returned to the client side, and the enumeration process of the historical version object is completed. In this process, first all objects on all index shards need to be traversed, and then the object with the version id is returned to the client. If the number of the object files in the storage bucket reaches millions or even tens of millions and some files have historical versions, the traversal time is too long, and the performance of the storage system is seriously affected.
To solve the above problem, as shown in fig. 1, the present application provides a history version object enumeration method, including:
s11: and when a history version object enumeration request is received, acquiring a target key value pair according to the prefix name of the history version.
The historical version prefix name is a prefix name which is added and stored in the historical version object key value pair when the homonymous object is newly added.
Specifically, the key in the key value pair including the prefix name of the historical version is in the form of prefix name of the historical version + object name + version id. The historical version prefix name is one of prefix names supported by a bottom mechanism of the distributed storage system, and is essentially a specific prefix name used for indicating that an object corresponding to a key value pair with the prefix name is a historical version object. Therefore, all key-value pairs containing the prefix names of the historical versions, namely the target key-value pairs, can be quickly acquired through the prefix names of the historical versions.
S12: and traversing the acquired target key value pair to acquire the object information of the historical version object.
Specifically, the value of the target key-value pair is traversed to obtain the basic information of the object. It is easy to understand that, since most of the object key value pairs of the non-history version are filtered out by the prefix name of the history version, a large number of invalid traversals are reduced during the traversal, the data volume of the required traversal is also reduced by a large part, and the enumeration efficiency is improved.
S13: and returning the object information of the history version object as an enumeration result.
According to different enumeration requirements sent by a user side, the object information may be basic information of an object stored in a key value pair value, or may be a data part of the object found from a data pool through the basic information of the object.
In addition, it is easy to understand that, although the history version object enumeration method provided by the present application can effectively reduce the number of invalid traversals, thereby improving the enumeration efficiency, when the number of objects stored in the storage system is too large, a certain processing time is still required. Therefore, in order to improve the use experience of the user, the communication between the storage system and the user side can be realized in an asynchronous mode, and after the storage system receives the enumeration request sent by the user side, the prompt information is returned to inform the user that the request is successfully received and is being processed; then, the connection can be disconnected, and the process or program for completing the enumeration task runs in the background to avoid being perceived by the user; after the enumeration task is completed and the enumeration result is generated, the enumeration result is returned to the user terminal, and further, prompt information can be returned in the modes of mail, pushing, short message, voice broadcasting and the like so as to inform the user that the enumeration task is completed and please check and receive the enumeration result; thereby further improving the user experience.
According to the historical version object enumeration method, a prefix name mechanism in a bottom mechanism of a distributed storage system is utilized, and screening of key value pairs stored in a storage bucket is supported according to the prefix name. Therefore, when a new object with the same name is added, a specific prefix name is added to the key value pair of the new object, and the prefix name is used for indicating that the object corresponding to the current key value pair is a historical version object. Furthermore, when the history version object needs to be enumerated, all key value pairs containing the history version prefix name can be obtained in a prefix name screening mode, and then the obtained key value pairs are traversed to obtain object information to be returned as an enumeration result. In the enumeration process, a large amount of invalid traversals are saved in the traversal stage, the data volume required to be traversed is reduced, the traversal process is simpler and more direct, and the response is faster, so that the traversal efficiency is improved, the influence on the performance of the distributed object storage system is reduced, and the historical version object enumeration requirements of users are better met.
In view of the above, in order to achieve the beneficial effects of reducing invalid traversal, improving enumeration performance, and reducing the influence on the performance of the distributed object storage system, one of the technical solutions adopted in the history version object enumeration method is that when a new object with the same name is added, a prefix name of a history version needs to be added to a key value pair record of the new object. To further illustrate the enumeration method provided in the present application, the following further describes the steps for adding new objects in the storage system:
when an object is newly added to the storage system, the object file uploaded by the client is received and stored in the corresponding bucket, generally, the way of allocating the bucket to the object file is mainly implemented according to the object name of the object file, and this embodiment provides a preferred implementation scheme for this, including:
performing hash calculation according to the object name of the object file to obtain a hash value;
and determining a corresponding bucket according to the hash value.
The unique hash value can be determined in a hash calculation mode through the object name of the object file, and then the storage bucket stored in the object file is determined through the corresponding relation between the hash value and the storage bucket.
At this time, there are two cases, one of which is that the object file is a new object, the bucket has no object with the same name, and the object file is stored in a normal storage manner, that is, the key value pair record is added for storage, where the key is in the form of the object name.
Secondly, the object file is a homonymy object of an existing object in the storage bucket, at this time, if the multi-version function of the distributed storage system is started, a key value pair is newly added to store an index record of the existing object, a key of the newly added key value pair is stored in a form of a prefix name of a historical version + an object name + a version id, and the newly added object is stored in a default form.
After the storage of the newly added object is completed based on the steps, when the user side sends an enumeration request of the historical version object, the target key value pair can be quickly determined through the historical version, and the information of the historical version object can be obtained only by traversing the target key value pair subsequently, so that the information is returned as an enumeration result, and the task of enumerating the historical version object is completed.
Further, after adding a key-value pair to an existing object as an index record, the original key-value pair loses its utility, so to further improve the storage capacity of the storage system, this embodiment further provides a preferred implementation scheme, and after adding a key-value pair record to an existing object, the method further includes:
and deleting the original key value pair records of the existing object.
Because the index records of the existing objects are processed in a new mode through the steps, and the new key value pairs comprise the prefix names of the historical versions, the effect of improving the listing efficiency is achieved. Therefore, the content of the original index record of the existing object can be embodied by the new key value pair, the old index record is useless, and in order to save the storage space of the storage system, the preferred scheme provided by the embodiment deletes the original index record to release the storage space, thereby further improving the storage capacity of the distributed object storage system.
According to the preferred scheme provided by the embodiment, when a same-name object is newly added through the storage system, the newly added key value pair is newly added to the existing object to be used as an index record for storage, the newly added key value pair is in a format required by the existing multi-version function and also comprises the prefix name of the historical version, so that the needed key value pair can be screened out through the prefix name by utilizing the existing bottom mechanism of the distributed object storage system, correspondingly, the key value pair comprising the prefix name of the historical version only needs to be traversed during traversal, a large amount of invalid traversal is reduced, the enumeration efficiency is improved, and the influence on the performance of the storage system is reduced.
It is easily understood that, in the enumeration application scenario addressed by the foregoing embodiment, enumeration needs are mostly to enumerate all history version objects of the storage system. In actual use, however, there is also a need for personalized enumeration, for example, for historical versions of certain objects, or for some historical versions of certain objects. Therefore, in order to solve the above problem, this embodiment provides a preferred implementation, and the history version enumeration request further includes: requesting an object name;
correspondingly, the method further comprises the following steps:
and screening the acquired target key value pairs according to the request object names, and filtering out the key value pairs of which the object names are not the request object names to acquire new target key value pairs.
The filtering of the unneeded objects according to the request object names can be realized in the step of traversing the target key value pairs, and when the target key value pairs are traversed, the key of each key value pair can be naturally obtained, so that the object name and the version id of the current object can be obtained. Therefore, the enumeration result is filtered according to the request object name, and the returned enumeration result is more in line with the actual requirements of the user.
Further, if the user's enumeration requirement also includes a request version id;
correspondingly, the method further comprises the following steps:
and screening the obtained target key value pair according to the request version id, and filtering out the key value pair of which the version id is not the request version id so as to obtain a new target key value pair.
Therefore, after the multi-version function of the distributed storage system is started, when an object with the same name is newly added, the key value pair is newly added in the form of the object name + the version id, and the version id is randomly generated and used for uniquely identifying the objects with different versions. Therefore, according to the version id, the unneeded historical version object can be filtered, and the version id used for filtering the unneeded historical version object is the request version id.
It should be noted that the method provided by the above embodiment is mainly provided for the case that the user has a history version listing requirement for some specific objects, and has a listing requirement for specific versions of some objects. The reason is that in practical applications, the two common requirements of the user are described above, and for other situations, for example, the requirement for listing the objects of the specific historical version is not common in practical applications, so the specific screening method is not described in detail here, but the specific implementation scheme is the same as the above.
In the preferred embodiment, the target key value pair is filtered by enumerating the request object name and the request version id in the request, and the objects not needed by the user are filtered, so that the enumeration result is simplified, and the enumeration requirement of the user is better met.
To further illustrate the history version object enumeration method provided by the present application, the following description, in conjunction with examples, makes clear and detailed descriptions of the preferred schemes provided by the above embodiments:
after a user starts the multi-version function of the storage barrel through a control instruction, uploading an object file; therefore, a certain index fragment is determined in the index pool according to the hash calculation mode, and a normal index record is recorded on the index fragment. The normal index record is also the key is the object name + version id, and the value is the key value pair record of the object basic information.
When a user repeatedly uploads an object file with the same name to the same storage bucket, the object name is the same, so that the object file is divided into the same index fragment in a hash calculation mode. At this time, a key value pair record needs to be added for the previous homonymous object, wherein the key is the prefix name of the historical version + the object name + the version id, and the value is stored as the basic information of the previous homonymous object.
And deleting the original index record of the last object with the same name, namely, the key is the object name + the version id, and the value is the key value pair record of the basic information of the object.
And adding a normal index record on the index fragment for the newly written homonymous object, wherein key is the object name + the version id, and value is the basic information of the object.
And when the same-name object is repeatedly uploaded, the steps are repeated to finish the storage of the index record.
When a distributed object storage system receives a history version object enumeration request sent by a user side, traversing all index fragments in an index pool, then appointing prefix names of history versions when traversing to each index fragment, filtering index records, namely key value pairs, of all history version objects according to the prefix names, then reading basic information of the objects according to the key value pairs, returning the corresponding history version objects to the client side according to history version identifiers in the basic information of the objects, and completing the process of enumerating all history version objects.
The application provides a scheme for improving an object to store a multi-version function enumeration historical version object, and provides a historical version object enumeration method. The historical version prefix name is a prefix name exclusive to the historical version object, and the specified prefix name is supported in an object storage bottom layer mechanism to obtain a specific record on the index fragment. When the multi-version object is enumerated, the record corresponding to the historical version on the index fragment can be obtained by appointing the prefix name of the historical version, the basic information of the recorded object is read, and then the information of the historical version object is returned to the client, so that the enumeration process is completed. When the newly-added historical version is added, although the exclusive record is newly added on the index fragment, the space occupation of the index fragment is increased when the number of the objects is large, the time overhead of traversing all the objects on all the index fragments is avoided, the invalid traversal is reduced, and meanwhile, the required historical version object record is filtered out on the index fragment through the specified exclusive prefix, so that the searching is more accurate and the response is faster. The object storage gateway system (RGW) server traverses the method of all objects on all bucket index slices, as opposed to every enumeration request send before. On one hand, the index record of the historical version object can be directly obtained through the appointed prefix when the barrel index fragment is traversed, so that invalid traversal on the barrel index fragment can be reduced, and the traversal accuracy is improved; on the other hand, the historical version objects are directly read from the bucket index fragments, and compared with a mode of traversing all the objects and then screening the historical version objects, the method can obviously improve the enumeration speed and reduce the influence on the performance of the storage system.
In the foregoing embodiment, a history version object enumeration method is described in detail, and the present application also provides an embodiment corresponding to a history version object enumeration apparatus. It should be noted that the present application describes the embodiments of the apparatus portion from two perspectives, one from the perspective of the function module and the other from the perspective of the hardware.
Based on the angle of the function module, as shown in fig. 2, the present embodiment provides an apparatus for enumerating history version objects, including:
a request receiving module 21, configured to, when receiving a history version object enumeration request, obtain a target key value pair according to a history version prefix name; the historical version prefix name is a prefix name which is added and stored in a historical version object key value pair when a homonymous object is newly added;
an object traversing module 22, configured to traverse the obtained target key-value pair to obtain object information of the historical version object;
and the result generating module 23 is configured to return the object information of the history version object as an enumeration result.
Preferably, the method further comprises the following steps:
the object adding module is used for receiving the object file uploaded by the user side and storing the object file in the corresponding storage bucket; if the object file is the same-name object file of the existing object in the storage bucket, newly adding a key value pair record for the existing object; wherein a key in the key value pair comprises: a history version prefix name, an object name, and a version id.
And the object screening module is used for screening the acquired target key value pairs according to the request object names and filtering key value pairs of which the object names are not the request object names so as to acquire new target key value pairs.
And the version id screening module is used for screening the obtained target key value pair according to the request version id and filtering the key value pair of which the version id is not the request version id so as to obtain a new target key value pair.
And the space release module is used for deleting the original key value pair records of the existing object.
And the result prompting module is used for returning prompting information to the user side.
Since the embodiments of the apparatus portion and the method portion correspond to each other, please refer to the description of the embodiments of the method portion for the embodiments of the apparatus portion, which is not repeated here.
The history version object enumeration apparatus provided in this embodiment utilizes a prefix name mechanism in a bottom layer mechanism of a distributed storage system, and because the prefix name mechanism supports screening of key value pairs stored in buckets according to prefix names. Therefore, when a new object with the same name is added, a specific prefix name is added to the key value pair of the new object, and the prefix name is used for indicating that the object corresponding to the current key value pair is a historical version object. Furthermore, when the historical version objects need to be enumerated, all key value pairs containing the historical version prefix names can be obtained in a prefix name screening mode, and then the obtained key value pairs are traversed through the object traversing module so as to obtain object information to be returned as an enumeration result. In the enumeration process, a large amount of invalid traversals are omitted in the traversal stage, so that the traversal process is simpler and more direct, the response is faster, the data volume required to be traversed is reduced, the traversal efficiency is improved, the influence on the performance of the distributed object storage system is reduced, and the historical version object enumeration requirements of users are better met.
Fig. 3 is a block diagram of an apparatus for enumerating historical version objects according to another embodiment of the present application, and as shown in fig. 3, the apparatus for enumerating historical version objects includes: a memory 30 for storing a computer program;
a processor 31, configured to implement the steps of the history version object enumeration method according to the above embodiment when executing the computer program.
The history version object enumeration device provided in this embodiment may include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer.
The processor 31 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The Processor 31 may be implemented in hardware using at least one of a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), and a Programmable Logic Array (PLA). The processor 31 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 31 may be integrated with a Graphics Processing Unit (GPU) which is responsible for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 31 may further include an Artificial Intelligence (AI) processor for processing computational operations related to machine learning.
Memory 30 may include one or more computer-readable storage media, which may be non-transitory. Memory 30 may also include high speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory 30 is at least used for storing the computer program 301, wherein after being loaded and executed by the processor 31, the computer program can implement the relevant steps of a history version object enumeration method disclosed in any one of the foregoing embodiments. In addition, the resources stored by the memory 30 may also include an operating system 302, data 303, and the like, and the storage may be transient storage or permanent storage. Operating system 302 may include Windows, Unix, Linux, etc. Data 303 may include, but is not limited to, a historical version object enumeration method, etc.
In some embodiments, a history version object enumerator may further include a display screen 32, an input/output interface 33, a communication interface 34, a power source 35, and a communication bus 36.
Those skilled in the art will appreciate that the structure shown in FIG. 3 does not constitute a limitation on one historical version object enumeration apparatus and may include more or fewer components than those shown.
The history version object enumerating device provided by the embodiment of the application comprises a memory and a processor, wherein when the processor executes a program stored in the memory, the following method can be realized: a history version object enumeration method.
The history version object enumeration apparatus provided in this embodiment utilizes a prefix name mechanism in a bottom layer mechanism of a distributed storage system, and because the prefix name mechanism supports screening of key value pairs stored in buckets according to prefix names. Therefore, when a new object with the same name is added, a specific prefix name is added to the key value pair of the new object, and the prefix name is used for indicating that the object corresponding to the current key value pair is a historical version object. Furthermore, when the processor executes the computer program stored in the memory, all key value pairs containing the prefix names of the historical versions can be obtained in a prefix name screening mode, and then the obtained key value pairs are traversed through the object traversing module to obtain object information to be returned as an enumeration result. In the enumeration process, a large amount of invalid traversals are omitted in the traversal stage, so that the traversal process is simpler and more direct, the response is faster, the data volume required to be traversed is reduced, the traversal efficiency is improved, the influence on the performance of the distributed object storage system is reduced, and the requirement of users on historical version object enumeration is better met.
Finally, the application also provides a corresponding embodiment of the computer readable storage medium. The computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps as set forth in the above-mentioned method embodiments.
It is to be understood that if the method in the above embodiments is implemented in the form of software functional units and sold or used as a stand-alone product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium and executes all or part of the steps of the methods described in the embodiments of the present application, or all or part of the technical solutions. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
The computer-readable storage medium provided in this embodiment utilizes a prefix name mechanism in a bottom layer mechanism of a distributed storage system, and because the prefix name mechanism supports screening of key value pairs stored in buckets according to prefix names. Therefore, when a new named object is added, a specific prefix name is added to the key value pair of the new named object, and the prefix name is used for indicating that the object corresponding to the current key value pair is a history version object. Furthermore, when the computer program stored therein is executed, all key value pairs containing the prefix names of the historical versions can be obtained in a prefix name screening mode, and then the obtained key value pairs are traversed through the object traversing module to obtain object information to be returned as an enumeration result. In the enumeration process, a large amount of invalid traversals are omitted in the traversal stage, so that the traversal process is simpler and more direct, the response is faster, the data volume required to be traversed is reduced, the traversal efficiency is improved, the influence on the performance of the distributed object storage system is reduced, and the requirement of users on historical version object enumeration is better met.
The history version object enumeration method, the history version object enumeration device and the history version object enumeration medium provided by the application are described in detail above. The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A history version object enumeration method is characterized by comprising the following steps:
when a history version object enumeration request is received, acquiring a target key value pair according to a history version prefix name; the historical version prefix name is a prefix name which is added and stored in a historical version object key value pair when a homonymous object is newly added;
traversing the obtained target key value pair to obtain object information of a history version object;
and returning the object information of the history version object as an enumeration result.
2. The historical version object enumeration method of claim 1, further comprising:
receiving an object file uploaded by a user side and storing the object file in a corresponding storage bucket;
if the object file is the same-name object file of the existing object in the storage bucket, adding a key value pair record for the existing object; wherein a key in the key value pair comprises: the historical version prefix name, object name and version id.
3. The historical version object enumeration method of claim 1, wherein the historical version enumeration request further comprises: requesting an object name;
correspondingly, the method also comprises the following steps:
and screening the obtained target key value pair according to the request object name, and filtering out the key value pair of which the object name is not the request object name so as to obtain a new target key value pair.
4. The historical version object enumeration method of claim 3, wherein the historical version enumeration request further comprises: requesting a version id;
correspondingly, the method also comprises the following steps:
and screening the obtained target key value pair according to the request version id, and filtering out the key value pair of which the version id is not the request version id so as to obtain a new target key value pair.
5. The method for object enumeration of historical versions according to claim 2, further comprising, after adding a key-value pair record to the existing object:
and deleting the original key value pair record of the existing object.
6. The method for enumerating historical version objects according to claim 2, wherein the storing in the corresponding bucket comprises:
performing hash calculation according to the object name of the object file to obtain a hash value;
and determining the corresponding storage bucket according to the hash value.
7. The method for enumerating history version objects according to any one of claims 1 and 6, wherein after the object information of the history version object is returned as an enumeration result, the method further comprises:
and returning the prompt information to the user side.
8. An apparatus for enumerating historical version objects, comprising:
the request receiving module is used for acquiring a target key value pair according to the prefix name of the historical version when receiving the enumeration request of the historical version object; the historical version prefix name is a prefix name which is added and stored in a historical version object key value pair when a homonymous object is newly added;
the object traversing module is used for traversing the acquired target key value pair to acquire object information of the historical version object;
and the result generation module is used for returning the object information of the history version object as an enumeration result.
9. An apparatus for enumerating historical version objects, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the history version object enumeration method according to any of claims 1 to 7 when executing said computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, carries out the steps of the history version object enumeration method according to any of the claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210832787.2A CN115114232A (en) | 2022-07-15 | 2022-07-15 | Method, device and medium for enumerating historical version objects |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210832787.2A CN115114232A (en) | 2022-07-15 | 2022-07-15 | Method, device and medium for enumerating historical version objects |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115114232A true CN115114232A (en) | 2022-09-27 |
Family
ID=83332365
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210832787.2A Pending CN115114232A (en) | 2022-07-15 | 2022-07-15 | Method, device and medium for enumerating historical version objects |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115114232A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150093A (en) * | 2023-03-04 | 2023-05-23 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
| CN117033317A (en) * | 2023-09-28 | 2023-11-10 | 苏州元脑智能科技有限公司 | Enumeration information acquisition method, system, device, communication equipment and storage medium |
-
2022
- 2022-07-15 CN CN202210832787.2A patent/CN115114232A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150093A (en) * | 2023-03-04 | 2023-05-23 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
| CN116150093B (en) * | 2023-03-04 | 2023-11-03 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
| CN117033317A (en) * | 2023-09-28 | 2023-11-10 | 苏州元脑智能科技有限公司 | Enumeration information acquisition method, system, device, communication equipment and storage medium |
| CN117033317B (en) * | 2023-09-28 | 2024-02-02 | 苏州元脑智能科技有限公司 | Enumeration information acquisition method, system, device, communication equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109254733B (en) | Method, device and system for storing data | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| KR20160003682A (en) | Hydration and dehydration with placeholders | |
| US11093471B2 (en) | Large range lookups for Bϵ-tree | |
| CN109885577B (en) | Data processing method, device, terminal and storage medium | |
| US10983909B2 (en) | Trading off cache space and write amplification for Bε-trees | |
| CN111324665B (en) | Log playback method and device | |
| CN110888837B (en) | Object storage small file merging method and device | |
| CN113064859B (en) | Metadata processing method and device, electronic equipment and storage medium | |
| CN112416880A (en) | Method and device for optimizing storage performance of mass small files based on real-time merging | |
| CN111414424B (en) | Method, system, medium and equipment for automatically synchronizing redis of configuration data | |
| CN113806300A (en) | Data storage method, system, device, equipment and storage medium | |
| CN111831691B (en) | Data reading and writing method and device, electronic equipment and storage medium | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| CN112000971B (en) | File permission recording method, system and related device | |
| CN112965939A (en) | File merging method, device and equipment | |
| US11520818B2 (en) | Method, apparatus and computer program product for managing metadata of storage object | |
| CN114861003A (en) | Object enumeration method, device and medium under specified directory | |
| CN114385657A (en) | Data storage method, device and storage medium | |
| CN110222105B (en) | Data summarization processing method and device | |
| CN114443629A (en) | Cluster bloom filter data duplication removing method, terminal equipment and storage medium | |
| CN113297267A (en) | Data caching and task processing method, device, equipment and storage medium | |
| CN113806249B (en) | Object storage sequence lifting method, device, terminal and storage medium | |
| CN117369731B (en) | Data reduction processing method, device, equipment and medium | |
| CN117493276B (en) | Reading method and device for Ceph file, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |