CN113806249A - Object storage ordered enumeration method, device, terminal and storage medium - Google Patents
Object storage ordered enumeration method, device, terminal and storage medium Download PDFInfo
- Publication number
- CN113806249A CN113806249A CN202111069552.4A CN202111069552A CN113806249A CN 113806249 A CN113806249 A CN 113806249A CN 202111069552 A CN202111069552 A CN 202111069552A CN 113806249 A CN113806249 A CN 113806249A
- Authority
- CN
- China
- Prior art keywords
- objects
- barrel
- enumeration
- bucket
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 239000012634 fragment Substances 0.000 claims abstract description 105
- 238000001514 detection method Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- 230000003993 interaction Effects 0.000 abstract description 5
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0877—Cache access modes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0893—Caches characterised by their organisation or structure
Abstract
The invention discloses an object storage ordered enumeration method, a device, a terminal and a storage medium, wherein N objects are enumerated each time, each barrel is provided with M barrel fragments, and when an enumeration request of a client is received, whether a cache map contains a cache of a barrel corresponding to the object to be enumerated is detected; if not, reading M objects from M barrel fragments of a corresponding barrel at the bottom layer, reading N objects from each barrel fragment, orderly enumerating the N objects from the M objects to send to a client, and storing the rest (M-1) objects into a cache map; the remaining (M-1) × N objects are the caches of the bucket in the cache map; and if so, orderly enumerating the N objects from the cache of the corresponding bucket and sending the N objects to the client. On one hand, the invention can reduce the interaction with the bottom layer and reduce the cluster pressure; on the other hand, reading directly from the memory can significantly increase the speed of enumeration compared to the previous way of reading from the bottom layer (hard disk).
Description
Technical Field
The invention relates to the field of ordered enumeration of object storage, in particular to an ordered enumeration method, an ordered enumeration device, a terminal and a storage medium for object storage.
Background
With the development of the internet, the data volume (unstructured data) of the internet is larger and larger, and distributed object storage is generated at the same time. The new storage mode is suitable for storing unstructured data such as pictures and videos. The method has the characteristics of high-speed direct access to the disk for block storage and distributed sharing of file storage, and therefore, the method is widely developed and applied once being launched.
The object enumeration operation may facilitate a user in viewing objects and attributes of the objects within the bucket. And the object storage is distributed storage, and clients usually store millions and millions of objects in a storage bucket, so that for such large-scale data, it is very important to quickly view the objects in the bucket while ensuring that the cluster performance is not seriously affected. When the object storage creates the buckets, index objects with specified number (128 by default) are created in the index pool, and when the objects are uploaded, object names and object attributes are stored in key value pairs of omap of the bucket index in a hash mode. And traversing the objects in the bucket by traversing the objects recorded on the bucket index. At present, the object stores ordered enumeration and unordered enumeration, wherein the ordered enumeration needs to traverse objects from all barrel indexes, an ordered sequencing result is guaranteed to be obtained after the objects on all indexes are sequenced, and the unordered enumeration respectively traverses the objects in the barrels according to the slicing sequence.
When the number of the buckets is large, it is not practical from the viewpoint of cluster performance and network to traverse all the objects once and return results to the clients, and the sequencing of large-scale data causes memory explosion and increases the probability of network transmission failure. Therefore, the current object storage adopts a paging sorting method. For the object ordered traversal, 1000 objects are enumerated each time by default, 1000 objects are read backwards from the positions of the identified objects to all barrel fragments (128 by default) each time, and the top 1000 objects are selected from the 128 x 1000 objects in an ordered way as the result of the ordered enumeration. And taking the last object of the ordered 1000 objects as a new object identifier, and continuing the traversal from the new object identifier in the next traversal. According to the scheme, 1000 × 128 objects are read from the bottom layer each time, but only 1000 ordered results are obtained, 1000 × 128 objects are read from the bottom layer continuously in the next reading, the result of 127 × 1000 after the ordering of 1000 is not reasonably utilized, and a user usually identifies all the objects in the bucket by using a new object, so that the current scheme causes frequent access to the bottom layer (generally, a hard disk) and wastes system resources.
Disclosure of Invention
In order to solve the above problems, the present invention provides an object storage ordered enumeration method, apparatus, terminal and storage medium, where the remaining objects are stored in a cache map, the remaining objects are stored in a memory through the cache map, and when there is a new enumeration request, if the object to be enumerated is in the cache map, the object is extracted from the cache map, so as to fully utilize the remaining objects, reduce interaction with the bottom layer, reduce cluster pressure, and simultaneously directly moderate from the memory, and significantly improve enumeration speed.
In a first aspect, a technical solution of the present invention provides an ordered object storage enumeration method, where N objects are enumerated each time, each bucket has M bucket segments, and N and M are integers greater than or equal to 1, the method including the steps of:
when an enumeration request of a client is received, detecting whether a cache map contains a cache of a bucket corresponding to an object to be enumerated;
if not, reading M objects from M barrel fragments of a corresponding barrel at the bottom layer, reading N objects from each barrel fragment, orderly enumerating the N objects from the M objects to send to a client, and storing the rest (M-1) objects into a cache map; the remaining (M-1) × N objects are the caches of the bucket in the cache map;
and if so, orderly enumerating the N objects from the cache of the corresponding bucket and sending the N objects to the client.
Further, the cached data in the cache map is stored in a key-value pair form, wherein the key of the key-value pair is the bucket name, and the value is the combination of the bucket fragment address and the bucket fragment object.
Further, the enumeration request of the client includes an enumeration identifier, and the required enumeration objects are N objects behind the enumeration identifier; enumerating the identifiers as object names;
if the cache map contains the cache of the bucket corresponding to the object to be enumerated, the method comprises the following steps before orderly enumerating N objects from the cache of the corresponding bucket:
comparing the enumeration identification contained in the enumeration request with the head and tail objects of each barrel fragment of the barrel cache;
if the enumeration identifier does not satisfy the condition that the enumeration identifier is larger than or equal to the first object of the barrel fragment and is smaller than or equal to the last object of the barrel fragment, acquiring N objects from the corresponding barrel fragment at the bottom layer and storing the N objects into the cached barrel fragment; wherein the first object is a head object and the last object is a tail object;
if the objects newly stored in the barrel fragment still do not satisfy the condition that the enumeration identification is larger than or equal to the first object and is smaller than or equal to the last object, the N objects are continuously obtained from the corresponding barrel fragments at the bottom layer and stored in the cached barrel fragment until the objects of the barrel fragment satisfy the condition that the enumeration identification is larger than or equal to the first object and is smaller than or equal to the last object.
Further, enumerating N objects in order from the cache of the corresponding bucket specifically includes:
according to the enumeration identification in the enumeration request, a first object behind the list identification is taken out from each barrel fragment, M objects are taken out altogether, and the largest object is taken out from the M objects and is taken as the first enumeration object;
taking the name of the first enumerated object as a new enumerated identifier, taking the first object behind the new list identifier from each barrel fragment, taking M objects in total, and taking the largest object from the M objects as a second enumerated object;
and so on until N enumerated objects are fetched.
Further, the method comprises the following steps:
the N objects enumerated in order from the cache map at a time are deleted from the cache map.
Further, the method comprises the following steps:
if the object of a certain barrel fragment in the cache map is used up, acquiring N objects from the corresponding barrel fragment at the bottom layer again and storing the N objects into the cached barrel fragment;
or, detecting the number H of the remaining objects of each barrel fragment in the cache map in real time, and if the number H of the remaining objects of a certain barrel fragment is less than or equal to a threshold value, acquiring N objects again from the corresponding barrel fragments of the bottom layer to store the cached barrel fragments;
or, detecting the number H of the remaining objects of each barrel fragment in the cache map in real time, and if the number H of the remaining objects of a certain barrel fragment is less than or equal to a threshold value, acquiring N-H objects from the corresponding barrel fragments of the bottom layer again to store the cached barrel fragments.
Further, the N objects or N-H objects retrieved from the corresponding barrel slice at the bottom are the N objects or N-H objects after the last object in all the objects obtained last time.
In a second aspect, the present invention provides an apparatus for ordered enumeration of object storage, where N objects are enumerated each time, each bucket has M bucket slices, and N and M are integers greater than or equal to 1,
bucket buffer detection module: when an enumeration request of a client is received, detecting whether a cache map contains a cache of a bucket corresponding to an object to be enumerated;
bottom layer enumerates modules: if not, reading M objects from M barrel fragments of a corresponding barrel at the bottom layer, reading N objects from each barrel fragment, orderly enumerating the N objects from the M objects to send to a client, and storing the rest (M-1) objects into a cache map; the remaining (M-1) × N objects are the caches of the bucket in the cache map;
a cache enumeration module: and if so, orderly enumerating the N objects from the cache of the corresponding bucket and sending the N objects to the client.
In a third aspect, a technical solution of the present invention provides a terminal, including:
a processor;
a memory for storing instructions for execution by the processor;
wherein the processor is configured to perform any of the methods described above.
In a fourth aspect, the invention provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the method of any one of the above.
Compared with the prior art, the object storage ordered enumeration method, the object storage ordered enumeration device, the terminal and the storage medium provided by the invention have the following beneficial effects: the unused part of the N objects traversed from the bottom barrel fragment last time is cached in the memory by the cache map, so that in the most common object enumeration mode of a user, which is that the user continuously enumerates, the residual object traversal results read from the bottom layer before can be directly utilized in the next group enumeration. Compared with the method that the barrel fragment number N objects are obtained from the bottom layer when N objects are enumerated each time, the method can reduce interaction with the bottom layer and reduce cluster pressure on one hand; on the other hand, reading directly from the memory can significantly increase the speed of enumeration compared to the previous way of reading from the bottom layer (hard disk).
Drawings
For a clearer explanation of the embodiments or technical solutions of the prior art of the present application, the drawings needed for the description of the embodiments or prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic flowchart of an object storage ordered enumeration method according to an embodiment of the present invention.
Fig. 2 is a schematic block diagram of a structure of an object storage ordered enumeration device according to a second embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a terminal according to a third embodiment of the present invention.
Detailed Description
The following explains the related terms to which the present invention relates.
Object storage: the Network Storage architecture is a new Network Storage architecture, has the characteristics of high-speed direct access to a disk by an SAN (Storage Area Network) and the distributed sharing characteristics of an NAS (Network Attached Storage), and has wide application prospect and application value.
Barrel slicing: a slice of an index object of a bucket in RGW (radius Gateway) for recording objects in the bucket and information related to the objects.
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
For the object ordered traversal, 1000 objects are enumerated each time by default, 1000 objects are read backwards from the positions of the identified objects to all barrel slices (128 by default) each time, and the top 1000 objects are sorted from the objects in 128 x 1000 as the result of the ordered enumeration. And taking the last object of the ordered 1000 objects as a new object identifier, and continuing the traversal from the new object identifier in the next traversal. According to the scheme, 1000 × 128 objects are read from the bottom layer each time, but only 1000 ordered results are obtained, 1000 × 128 objects are read from the bottom layer continuously in the next reading, the result of 127 × 1000 after the ordering of 1000 is not reasonably utilized, and a user usually identifies all the objects in the bucket by using a new object, so that the current scheme causes frequent access to the bottom layer (generally, a hard disk) and wastes system resources.
Therefore, the invention provides an object storage ordered enumeration scheme, which stores the rest objects into a cache map, stores the rest objects into a memory through the cache map, and when a new enumeration request exists, if the needed enumeration object is in the cache map, the enumeration object is extracted from the cache map, so that the rest objects are fully utilized, the interaction with the bottom layer is reduced, the cluster pressure is reduced, and meanwhile, the enumeration speed is obviously improved by directly reading from the memory.
Example one
In this embodiment, an object storage ordered enumeration method is provided, where it is assumed that N objects are enumerated each time, each bucket has M bucket fragments, N and M are integers greater than or equal to 1, and of course, a value of N is generally 1000 and a value of M is generally 128 at present.
Fig. 1 is a flowchart illustrating an object storage ordered enumeration method according to a first embodiment of the present invention, which includes the following steps.
S101, when an enumeration request of a client is received, detecting whether a cache map contains a cache of a bucket corresponding to an object to be enumerated;
s102, if not, reading M objects from M barrel fragments of a corresponding barrel at the bottom layer, reading N objects from each barrel fragment, orderly enumerating N objects from the M objects to send to a client, and storing the rest (M-1) objects to a cache map; the remaining (M-1) × N objects are the caches of the bucket in the cache map;
and S103, if so, orderly enumerating the N objects from the cache of the corresponding bucket and sending the N objects to the client.
That is, when receiving an enumeration object sent by a client, first checking whether a cache of a bucket corresponding to the required enumeration object is in a cache map, if so, directly reading the required enumeration object in the cache map, and if not, reading the enumeration object from a corresponding bucket at the bottom layer. For example, when an object in a bucket is first enumerated, there is no cache for that bucket in the cache map. When the enumerated objects are read at the bottom layer, a traditional ordered enumeration method is used, namely M objects are taken out firstly, and then the M objects are sent to the client side from the M objects. Thus, the (M-1) × N objects are left, and the left (M-1) × N objects are stored in the cache map, i.e. the data in the cache map, i.e. the cache of the corresponding bucket, is formed.
In the object storage ordered enumeration method provided by this embodiment, the unused part of the N objects traversed from the bottom bucket fragment last time is cached in the memory by using the cache map, so that in the most common object enumeration manner in which a user continuously enumerates the user, the remaining object traversal results read from the bottom before can be directly utilized when performing the next group of enumeration. Compared with the method that the barrel fragment number N objects are obtained from the bottom layer when N objects are enumerated each time, the method can reduce interaction with the bottom layer and reduce cluster pressure on one hand; on the other hand, reading directly from the memory can significantly increase the speed of enumeration compared to the previous way of reading from the bottom layer (hard disk).
The cached data in the cache map is stored in a key-value pair form, and the remaining (M-1) × N objects in this embodiment are stored in the cache map to form a key-value pair, where the key of the key-value pair is a bucket name and the value is a combination of a bucket fragment address and a bucket fragment object. Specifically, < bucket name, < bucket shard address, bucket shard object > >.
In the step S101, whether the cache map includes the cache of the bucket corresponding to the object to be enumerated can be detected according to the key of the key-value pair. In the subsequent step S103, when N objects are listed in order from the cache of the corresponding bucket, the objects are also searched according to the key-value pair.
It should be noted that, when the remaining (M-1) × N objects are stored in the cache map, the original barrel fragments are still fragmented, that is, M barrel fragments are still included, and after the N objects are taken away for the first time, the number of each fragment is less than N. For example, if N is 1000 and M is 128, 990 objects may remain in a bucket fragment after taking 1000 objects for the first time, i.e., 10 objects are taken from the bucket fragment.
If the cache of the corresponding bucket exists in the cache map, when the enumeration command is received, it still needs to detect whether the required enumeration object can be completely fetched from the bucket cache. In specific implementation, each barrel fragment object is compared with the enumeration identifier of the object to be enumerated to judge whether the object to be enumerated can be completely taken out or not, and corresponding processing is required when the object cannot be completely taken out.
Specifically, the enumeration request of the client includes an enumeration identifier, and the required enumeration object is N objects after the enumeration identifier (that is, N objects after the enumeration identifier are sequentially taken out); the enumeration is identified as the object name. It should be noted that the object name is a character string, and objects are ordered and arranged in each bucket fragment according to the character string ordering.
Correspondingly, if the cache map in step S103 includes the cache of the bucket corresponding to the object to be enumerated, before sequentially enumerating N objects from the cache of the corresponding bucket, the method includes the following steps:
comparing an enumeration identifier contained in an enumeration request with head and tail objects of each barrel fragment of a barrel cache;
step two, if the enumeration identification does not satisfy the condition that the enumeration identification is larger than or equal to the first object of the barrel fragment and is smaller than or equal to the last object of the barrel fragment, acquiring N objects from the corresponding barrel fragment at the bottom layer and storing the N objects into the cached barrel fragment; wherein the first object is a head object and the last object is a tail object;
and step three, if the objects newly stored in the barrel fragment still do not meet the condition that the enumeration identification is larger than or equal to the first object and is smaller than or equal to the last object, continuously acquiring N objects from the corresponding barrel fragment at the bottom layer and storing the N objects in the cached barrel fragment until the objects of the barrel fragment meet the condition that the enumeration identification is larger than or equal to the first object and is smaller than or equal to the last object.
The N objects retrieved from the corresponding barrel slice at the bottom in the second and third steps are the N objects after the last object in the plurality of objects retrieved from the barrel slice last time. For example, when a certain barrel slice at the bottom layer contains the objects n1, n2, n3, n4 and n5, n1 and n2 are taken last time, and if 2 objects need to be taken this time, n3 and n4 should be taken.
Through the processing, each barrel fragment of the corresponding barrel in the cache map contains the object to be taken out, and the complete and ordered enumeration of the required objects can be ensured.
Then, N objects can be listed in order from the bucket cache, and in the above step S103 of this embodiment, the method for listing N objects in order from the cache of the corresponding bucket specifically includes the following steps:
step one, according to the enumeration identification in the enumeration request, a first object behind the list identification is taken out from each barrel fragment, M objects are taken out altogether, and the largest object is taken out from the M objects as the first enumeration object;
step two, taking the name of the first enumeration object as a new enumeration identifier, taking the first object behind the new list identifier out of each barrel fragment, taking M objects out altogether, and taking the largest object out of the M objects as a second enumeration object;
and step three, repeating the steps until N enumerated objects are taken out.
Through the steps, N objects can be listed in order from the barrel cache.
In this embodiment, N objects listed in order from the cache map at a time are deleted from the cache map, so that memory usage is reduced. Accordingly, the objects in the barrel fragments in the cache map are gradually used up, so that the objects can be continuously taken from the cache map in the follow-up process, and the cache objects are supplemented to the cached barrel fragments in time. In specific implementation, the replenishment of the barrel fragment cache object can be realized by the following methods:
in the first method, if the object of a certain barrel fragment in the cache map is used up, N objects are obtained from the corresponding barrel fragments at the bottom layer and stored in the cached barrel fragment again;
detecting the number H of the remaining objects of each barrel fragment in the cache map in real time, and if the number H of the remaining objects of a certain barrel fragment is less than or equal to a threshold value, acquiring N objects again from the corresponding barrel fragments at the bottom layer to store the cached barrel fragments;
and thirdly, detecting the number H of the residual objects of each barrel fragment in the cache map in real time, and if the number H of the residual objects of a certain barrel fragment is less than or equal to a threshold value, acquiring N-H objects from the corresponding barrel fragments at the bottom layer again to store the cached barrel fragments.
The user can select a proper method according to the needs. It should be noted that the N objects or N-H objects retrieved from the corresponding barrel slice at the bottom layer are N objects or N-H objects after the last object in all the objects obtained last time.
Example two
The second embodiment provides an object storage ordered enumeration device, which is used for implementing the storage ordered enumeration method. Similarly, each time N objects are enumerated, each bucket has M bucket slices, and N and M are integers greater than or equal to 1.
Fig. 2 is a schematic block diagram of a structure of an object storage ordered enumeration device according to a second embodiment, which includes the following functional modules.
Bucket buffer detection module: when an enumeration request of a client is received, detecting whether a cache map contains a cache of a bucket corresponding to an object to be enumerated;
bottom layer enumerates modules: if not, reading M objects from M barrel fragments of a corresponding barrel at the bottom layer, reading N objects from each barrel fragment, orderly enumerating the N objects from the M objects to send to a client, and storing the rest (M-1) objects into a cache map; the remaining (M-1) × N objects are the caches of the bucket in the cache map;
a cache enumeration module: and if so, orderly enumerating the N objects from the cache of the corresponding bucket and sending the N objects to the client.
The object storage ordered listing apparatus of this embodiment is used to implement the foregoing object storage ordered listing method, so that the specific implementation manner in the apparatus can be seen in the foregoing embodiment section of the object storage ordered listing method, and therefore, the specific implementation manner thereof can refer to the description of the corresponding embodiment section and will not be further described herein.
In addition, since the object storage and ordered listing apparatus of this embodiment is used to implement the foregoing object storage and ordered listing method, the role thereof corresponds to that of the foregoing method, and details thereof are not described here.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a terminal device 300 according to an embodiment of the present invention, where the terminal device 300 may be configured to execute the object store ordered listing method according to the embodiment of the present invention.
Among them, the terminal apparatus 300 may include: a processor 310, a memory 320, and a communication unit 330. The components communicate via one or more buses, and those skilled in the art will appreciate that the architecture of the servers shown in the figures is not intended to be limiting, and may be a bus architecture, a star architecture, a combination of more or less components than those shown, or a different arrangement of components.
The memory 320 may be used for storing instructions executed by the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile storage terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The executable instructions in memory 320, when executed by processor 310, enable terminal 300 to perform some or all of the steps in the method embodiments described below.
The processor 310 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by operating or executing software programs and/or modules stored in the memory 320 and calling data stored in the memory. The processor may be composed of an Integrated Circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs connected with the same or different functions. For example, the processor 310 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.
A communication unit 330, configured to establish a communication channel so that the storage terminal can communicate with other terminals. And receiving user data sent by other terminals or sending the user data to other terminals.
Example four
The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and the program may include some or all of the steps in the embodiments provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, and the storage medium can store program codes, and includes instructions for enabling a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, and the like) to perform all or part of the steps of the method in the embodiments of the present invention.
The same and similar parts in the various embodiments in this specification may be referred to each other. Especially, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the description in the method embodiment.
In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The above disclosure is only for the preferred embodiments of the present invention, but the present invention is not limited thereto, and any non-inventive changes that can be made by those skilled in the art and several modifications and amendments made without departing from the principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. An object storage ordered enumeration method, enumerating N objects each time, wherein each bucket has M bucket fragments, and both N and M are integers greater than or equal to 1, the method comprises the following steps:
when an enumeration request of a client is received, detecting whether a cache map contains a cache of a bucket corresponding to an object to be enumerated;
if not, reading M objects from M barrel fragments of a corresponding barrel at the bottom layer, reading N objects from each barrel fragment, orderly enumerating the N objects from the M objects to send to a client, and storing the rest (M-1) objects into a cache map; the remaining (M-1) × N objects are the caches of the bucket in the cache map;
and if so, orderly enumerating the N objects from the cache of the corresponding bucket and sending the N objects to the client.
2. The method for ordered enumeration of object storage according to claim 1, wherein the cached data in the cache map is stored in a key-value pair form, wherein the key of the key-value pair is a bucket name and the value is a combination of a bucket fragment address and a bucket fragment object.
3. The method for ordered enumeration of object storage according to claim 2, wherein the enumeration request of the client includes an enumeration identifier, and the required enumeration objects are N objects after the enumeration identifier; enumerating the identifiers as object names;
if the cache map contains the cache of the bucket corresponding to the object to be enumerated, the method comprises the following steps before orderly enumerating N objects from the cache of the corresponding bucket:
comparing the enumeration identification contained in the enumeration request with the head and tail objects of each barrel fragment of the barrel cache;
if the enumeration identifier does not satisfy the condition that the enumeration identifier is larger than or equal to the first object of the barrel fragment and is smaller than or equal to the last object of the barrel fragment, acquiring N objects from the corresponding barrel fragment at the bottom layer and storing the N objects into the cached barrel fragment; wherein the first object is a head object and the last object is a tail object;
if the objects newly stored in the barrel fragment still do not satisfy the condition that the enumeration identification is larger than or equal to the first object and is smaller than or equal to the last object, the N objects are continuously obtained from the corresponding barrel fragments at the bottom layer and stored in the cached barrel fragment until the objects of the barrel fragment satisfy the condition that the enumeration identification is larger than or equal to the first object and is smaller than or equal to the last object.
4. The method for ordered enumeration of object storage according to claim 3, wherein the ordered enumeration of N objects from the cache of the corresponding bucket specifically comprises:
according to the enumeration identification in the enumeration request, a first object behind the list identification is taken out from each barrel fragment, M objects are taken out altogether, and the largest object is taken out from the M objects and is taken as the first enumeration object;
taking the name of the first enumerated object as a new enumerated identifier, taking the first object behind the new list identifier from each barrel fragment, taking M objects in total, and taking the largest object from the M objects as a second enumerated object;
and so on until N enumerated objects are fetched.
5. The method for storing ordered lists according to claim 4, further comprising the steps of:
the N objects enumerated in order from the cache map at a time are deleted from the cache map.
6. The method for storing an ordered listing of objects of claim 5, further comprising the steps of:
if the object of a certain barrel fragment in the cache map is used up, acquiring N objects from the corresponding barrel fragment at the bottom layer again and storing the N objects into the cached barrel fragment;
or, detecting the number H of the remaining objects of each barrel fragment in the cache map in real time, and if the number H of the remaining objects of a certain barrel fragment is less than or equal to a threshold value, acquiring N objects again from the corresponding barrel fragments of the bottom layer to store the cached barrel fragments;
or, detecting the number H of the remaining objects of each barrel fragment in the cache map in real time, and if the number H of the remaining objects of a certain barrel fragment is less than or equal to a threshold value, acquiring N-H objects from the corresponding barrel fragments of the bottom layer again to store the cached barrel fragments.
7. The method for ordered enumeration of object storage according to claim 6, wherein the N objects or N-H objects retrieved from the corresponding bucket shards at the bottom are N objects or N-H objects after the last object in all the objects retrieved last time.
8. An object store ordered enumeration apparatus for enumerating N objects each time, each bucket having M bucket slices, N and M being integers greater than or equal to 1, the apparatus comprising,
bucket buffer detection module: when an enumeration request of a client is received, detecting whether a cache map contains a cache of a bucket corresponding to an object to be enumerated;
bottom layer enumerates modules: if not, reading M objects from M barrel fragments of a corresponding barrel at the bottom layer, reading N objects from each barrel fragment, orderly enumerating the N objects from the M objects to send to a client, and storing the rest (M-1) objects into a cache map; the remaining (M-1) × N objects are the caches of the bucket in the cache map;
a cache enumeration module: and if so, orderly enumerating the N objects from the cache of the corresponding bucket and sending the N objects to the client.
9. A terminal, comprising:
a processor;
a memory for storing instructions for execution by the processor;
wherein the processor is configured to perform the method of any one of claims 1-7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111069552.4A CN113806249B (en) | 2021-09-13 | 2021-09-13 | Object storage sequence lifting method, device, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111069552.4A CN113806249B (en) | 2021-09-13 | 2021-09-13 | Object storage sequence lifting method, device, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113806249A true CN113806249A (en) | 2021-12-17 |
| CN113806249B CN113806249B (en) | 2023-12-22 |
Family
ID=78941070
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111069552.4A Active CN113806249B (en) | 2021-09-13 | 2021-09-13 | Object storage sequence lifting method, device, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113806249B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101122885A (en) * | 2007-09-11 | 2008-02-13 | 腾讯科技（深圳）有限公司 | Data cache processing method, system and data cache device |
| US20130212208A1 (en) * | 2010-12-27 | 2013-08-15 | Limelight Networks, Inc. | Partial object caching |
| CN103914399A (en) * | 2012-12-31 | 2014-07-09 | 中国移动通信集团公司 | Disk cache method and device in parallel computer system |
| CN105718561A (en) * | 2016-01-20 | 2016-06-29 | 中国科学院计算技术研究所 | Particular distributed data storage file structure redundancy removing construction method and system |
| US20200195743A1 (en) * | 2018-12-14 | 2020-06-18 | Nutanix, Inc. | Systems and methods for accelerating object stores with distributed caching |
| US20200285542A1 (en) * | 2019-03-08 | 2020-09-10 | International Business Machines Corporation | Selective placement and adaptive backups for point-in-time database recovery |
| WO2020181820A1 (en) * | 2019-03-08 | 2020-09-17 | 平安科技（深圳）有限公司 | Data cache method and apparatus, computer device and storage medium |
| CN111857890A (en) * | 2020-07-24 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Service processing method, system, device and medium |
| CN113111033A (en) * | 2021-04-07 | 2021-07-13 | 山东英信计算机技术有限公司 | Method and system for dynamically redistributing bucket indexes in distributed object storage system |
-
2021
- 2021-09-13 CN CN202111069552.4A patent/CN113806249B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101122885A (en) * | 2007-09-11 | 2008-02-13 | 腾讯科技（深圳）有限公司 | Data cache processing method, system and data cache device |
| US20130212208A1 (en) * | 2010-12-27 | 2013-08-15 | Limelight Networks, Inc. | Partial object caching |
| CN103914399A (en) * | 2012-12-31 | 2014-07-09 | 中国移动通信集团公司 | Disk cache method and device in parallel computer system |
| CN105718561A (en) * | 2016-01-20 | 2016-06-29 | 中国科学院计算技术研究所 | Particular distributed data storage file structure redundancy removing construction method and system |
| US20200195743A1 (en) * | 2018-12-14 | 2020-06-18 | Nutanix, Inc. | Systems and methods for accelerating object stores with distributed caching |
| US20200285542A1 (en) * | 2019-03-08 | 2020-09-10 | International Business Machines Corporation | Selective placement and adaptive backups for point-in-time database recovery |
| WO2020181820A1 (en) * | 2019-03-08 | 2020-09-17 | 平安科技（深圳）有限公司 | Data cache method and apparatus, computer device and storage medium |
| CN111857890A (en) * | 2020-07-24 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Service processing method, system, device and medium |
| CN113111033A (en) * | 2021-04-07 | 2021-07-13 | 山东英信计算机技术有限公司 | Method and system for dynamically redistributing bucket indexes in distributed object storage system |
Non-Patent Citations (1)
| Title |
|---|
| 王明婷;孙江涛;吕绍华;: "一种数据仓库操作的优化方法", 计算机工程与设计, no. 11 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113806249B (en) | 2023-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109254733B (en) | Method, device and system for storing data | |
| CN106657213B (en) | File transmission method and device | |
| US10210191B2 (en) | Accelerated access to objects in an object store implemented utilizing a file storage system | |
| US9507821B2 (en) | Mail indexing and searching using hierarchical caches | |
| CN108228799B (en) | Object index information storage method and device | |
| CN110347651A (en) | Method of data synchronization, device, equipment and storage medium based on cloud storage | |
| CN107426041B (en) | Method and device for analyzing command | |
| CN109445861A (en) | System start method, device, computer installation and storage medium | |
| CN110888837B (en) | Object storage small file merging method and device | |
| US11681770B2 (en) | Determining whether to process identified uniform resource locators | |
| JP2022159405A (en) | Method and device for appending data, electronic device, storage medium, and computer program | |
| CN111338688B (en) | Data long-acting caching method and device, computer system and readable storage medium | |
| CN104035822A (en) | Low-cost efficient internal storage redundancy removing method and system | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| CN110352410A (en) | Track the access module and preextraction index node of index node | |
| US11520818B2 (en) | Method, apparatus and computer program product for managing metadata of storage object | |
| US8818970B2 (en) | Partitioning a directory while accessing the directory | |
| CN112306957A (en) | Method and device for acquiring index node number, computing equipment and storage medium | |
| CN113806249A (en) | Object storage ordered enumeration method, device, terminal and storage medium | |
| CN111309471A (en) | Data processing method and device and distributed system | |
| CN115129789A (en) | Bucket index storage method, device and medium of distributed object storage system | |
| CN114443583A (en) | Method, device and equipment for arranging fragment space and storage medium | |
| US11144593B2 (en) | Indexing structure with size bucket indexes | |
| CN111625500A (en) | File snapshot method and device, electronic equipment and storage medium | |
| CN106528577B (en) | Method and device for setting file to be cleaned |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |