CN113806249B - Object storage sequence lifting method, device, terminal and storage medium - Google Patents

Object storage sequence lifting method, device, terminal and storage medium Download PDF

Info

Publication number
CN113806249B
CN113806249B CN202111069552.4A CN202111069552A CN113806249B CN 113806249 B CN113806249 B CN 113806249B CN 202111069552 A CN202111069552 A CN 202111069552A CN 113806249 B CN113806249 B CN 113806249B
Authority
CN
China
Prior art keywords
objects
barrel
enumeration
bucket
cache
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111069552.4A
Other languages
Chinese (zh)
Other versions
CN113806249A (en
Inventor
程晓煜
陶桐桐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jinan Inspur Data Technology Co Ltd
Original Assignee
Jinan Inspur Data Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jinan Inspur Data Technology Co Ltd filed Critical Jinan Inspur Data Technology Co Ltd
Priority to CN202111069552.4A priority Critical patent/CN113806249B/en
Publication of CN113806249A publication Critical patent/CN113806249A/en
Application granted granted Critical
Publication of CN113806249B publication Critical patent/CN113806249B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0877Cache access modes
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0893Caches characterised by their organisation or structure

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method, a device, a terminal and a storage medium for storing a sequence of objects, wherein N objects are enumerated each time, each barrel is provided with M barrel fragments, and when an enumeration request of a client is received, whether a cache map contains a cache of a barrel corresponding to the object to be enumerated is detected; if not, M x N objects are read from M barrel fragments of the corresponding barrel at the bottom layer, N objects are read from each barrel fragment, N objects are sent to the client side from sequences in the M x N objects, and the rest (M-1) x N objects are stored in the cache map; the remaining (M-1) N objects are the cache of the bucket in the cache map; if the object is included, the N objects are sent to the client side from the buffer memory of the corresponding bucket in sequence. On one hand, the invention can reduce interaction with the bottom layer and reduce cluster pressure; on the other hand, the method can read from the memory directly, and can remarkably improve the enumeration speed compared with the method of reading from the bottom layer (hard disk) before.

Description

Object storage sequence lifting method, device, terminal and storage medium
Technical Field
The invention relates to the field of object storage sequence lifting, in particular to an object storage sequence lifting method, device, terminal and storage medium.
Background
With the development of the internet, the amount of internet data (unstructured data) is increasing, and distributed object storage is generated. The new storage mode is suitable for storing unstructured data such as pictures, videos and the like. The method has the characteristics of high-speed direct access to the disk by the block storage and the characteristics of distributed sharing of the file storage, so that the method is widely developed and applied once being proposed.
The object enumeration operation may facilitate a user's view of objects within a bucket and attributes of the objects. Whereas object stores are distributed stores, clients typically store millions and tens of millions of objects in a bucket, for such large-scale data, it is important to quickly look at the objects in the bucket while ensuring that cluster performance is not severely impacted. When creating the bucket, the object storage creates index objects with specified numbers (128 default) in the index pool, and when uploading the objects, the object names and the attributes of the objects are stored in the key value pair of the omap of the bucket index in a hash mode. Traversing the objects in the bucket is accomplished by traversing the objects recorded on the bucket index. At present, the object stores two types of ordered enumeration and unordered enumeration, the ordered enumeration needs to traverse the objects from all bucket indexes, the ordered results of the ordered results are ensured to be obtained after the objects on all indexes are ordered, and the unordered enumeration traverses the objects in the bucket respectively according to the slicing order.
When the number of the buckets is large, traversing all the objects at a time and returning the results to the clients is not realistic, and the sorting of large-scale data can cause memory storm and increase the probability of network transmission failure. Therefore, the current object storage adopts a paging ordering method. For the ordered traversal of the objects, 1000 objects are enumerated each time by default, 1000 objects are read back from the identified object position to all bucket fragments (128 by default) each time, and the first 1000 objects are sorted out of the objects in the 128×1000 as a result of the ordered enumeration. The last object of the 1000 ordered objects is taken as a new object identifier, and the next traversal is continued from the new object identifier. In this scheme, 1000×128 objects are read from the bottom layer every time, but only 1000 ordered results are obtained, and 1000×128 objects continue to be read from the bottom layer when the next reading is performed, and for 127×1000 results after 1000 orders are not reasonably utilized, a user usually can identify all objects in the bucket by using new objects, so the current scheme can cause frequent access to the bottom layer (generally, a hard disk), and waste of system resources.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method, an apparatus, a terminal and a storage medium for storing a sequence of objects, storing the remaining objects in a cache map, storing the remaining objects in a memory through the cache map, extracting the remaining objects from the cache map if a new enumeration request is required, fully utilizing the remaining objects, reducing interactions with a bottom layer, reducing cluster pressure, and simultaneously improving enumeration speed substantially from the memory.
In a first aspect, the present invention provides a method for storing a sequence of objects, each time enumerating N objects, each bucket having M bucket fragments, where N and M are integers greater than or equal to 1, the method comprising the steps of:
when an enumeration request of a client is received, detecting whether a cache map contains a cache of a barrel corresponding to a required enumeration object;
if not, M x N objects are read from M barrel fragments of the corresponding barrel at the bottom layer, N objects are read from each barrel fragment, N objects are sent to the client side from sequences in the M x N objects, and the rest (M-1) x N objects are stored in the cache map; the remaining (M-1) N objects are the cache of the bucket in the cache map;
if the object is included, the N objects are sent to the client side from the buffer memory of the corresponding bucket in sequence.
Further, the cached data in the cache map is stored in the form of key-value pairs, wherein the keys of the key-value pairs are bucket names and the values are a combination of bucket fragment addresses and bucket fragment objects.
Further, the enumeration request of the client comprises an enumeration identifier, and the required enumeration objects are N objects after the enumeration identifier; enumerating the identification as an object name;
if the cache map contains the cache of the corresponding bucket of the object to be enumerated, before the sequence of N objects is lifted from the cache of the corresponding bucket, the method comprises the following steps:
comparing an enumeration identifier contained in the enumeration request with the head-to-tail object of each bucket fragment of the bucket cache;
if the list identification of a certain barrel slice is not more than or equal to the first object of the barrel slice and is less than or equal to the last object of the barrel slice, acquiring N objects again from the corresponding barrel slice at the bottom layer and storing the N objects into the cached barrel slice; wherein the first object is a head object and the last object is a tail object;
if the object re-stored in the bucket fragment still does not meet the requirement that the enumeration identifier is greater than or equal to the first object and less than or equal to the last object, continuously re-acquiring N objects from the corresponding bucket fragment at the bottom layer and storing the N objects into the cached bucket fragment until the object of the bucket fragment meets the requirement that the enumeration identifier is greater than or equal to the first object and less than or equal to the last object.
Further, there are N objects in sequence from the cache of the corresponding bucket, specifically including:
according to the enumeration identifier in the enumeration request, the first object after the list identifier is taken out of each barrel slice, M objects are taken out altogether, and the largest object is taken out of the M objects as the first enumeration object;
taking the name of the first enumeration object as a new enumeration identifier, taking the new list identifier from each barrel slice, taking M objects out, and taking the largest object out of the M objects as a second enumeration object;
and so on until N enumerated objects are fetched.
Further, the method comprises the following steps:
n objects with sequence lifting in the cache map at a time are deleted from the cache map.
Further, the method comprises the following steps:
if the object of a certain barrel slice in the cache map is used up, N objects are obtained again from the corresponding barrel slice at the bottom layer and stored in the cached barrel slice;
or detecting the number H of the remaining objects of each barrel slice in the cache map in real time, and if the number H of the remaining objects of a certain barrel slice is smaller than or equal to a threshold value, re-acquiring N barrel slices of the corresponding barrel slices at the bottom layer, wherein the N barrel slices are stored in the cache;
or detecting the number H of the remaining objects of each barrel slice in the cache map in real time, and if the number H of the remaining objects of a certain barrel slice is smaller than or equal to a threshold value, re-acquiring N-H barrel slices of the corresponding barrel slices at the bottom layer for storing the cache.
Further, the N objects or N-H objects retrieved from the underlying corresponding bucket partition are N objects or N-H objects after the last object in all the objects retrieved last time.
In a second aspect, the present invention provides an object storage sequence lifting apparatus for enumerating N objects each time, each bucket having M bucket fragments, N and M being integers greater than or equal to 1, the apparatus comprising,
bucket cache detection module: when an enumeration request of a client is received, detecting whether a cache map contains a cache of a barrel corresponding to a required enumeration object;
the bottom layer enumerates the module: if not, M x N objects are read from M barrel fragments of the corresponding barrel at the bottom layer, N objects are read from each barrel fragment, N objects are sent to the client side from sequences in the M x N objects, and the rest (M-1) x N objects are stored in the cache map; the remaining (M-1) N objects are the cache of the bucket in the cache map;
cache enumeration module: if the object is included, the N objects are sent to the client side from the buffer memory of the corresponding bucket in sequence.
In a third aspect, a technical solution of the present invention provides a terminal, including:
a processor;
a memory for storing execution instructions of the processor;
wherein the processor is configured to perform the method of any of the above.
In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements a method as described in any one of the preceding claims.
The object storage sequence lifting method, device, terminal and storage medium provided by the invention have the following beneficial effects compared with the prior art: by caching the unused portions of the N objects last traversed from the underlying bucket segment into memory by means of the cache map, in the most common object enumeration manner of users in which users continuously enumerate, the remaining object traversal results previously read from the underlying can be directly utilized in the next group enumeration. Compared with the method that N objects are listed each time before, the method needs to obtain the barrel fragmentation number of the N objects from the bottom layer, on one hand, the interaction with the bottom layer can be reduced, and cluster pressure is reduced; on the other hand, the method can read from the memory directly, and can remarkably improve the enumeration speed compared with the method of reading from the bottom layer (hard disk) before.
Drawings
For a clearer description of embodiments of the present application or of the prior art, the drawings that are used in the description of the embodiments or of the prior art will be briefly described, it being apparent that the drawings in the description that follow are only some embodiments of the present application, and that other drawings may be obtained from these drawings by a person of ordinary skill in the art without inventive effort.
Fig. 1 is a flowchart of a method for storing a sequence of objects according to a first embodiment of the present invention.
Fig. 2 is a schematic block diagram of an object storage sequence lifting device according to a second embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a terminal according to a third embodiment of the present invention.
Detailed Description
The following explains the related terms related to the present invention.
Object storage: the network storage architecture is a new network storage architecture, has the characteristics of SAN (Storage Area Network ) high-speed direct access disk and the distributed sharing characteristics of NAS (Network Attached Storage ), and has wider application prospect and application value.
Barrel slicing: fragments of index objects of buckets in RGW (Rados Gateway) are used to record the objects in the buckets and the object related information.
In order to provide a better understanding of the present application, those skilled in the art will now make further details of the present application with reference to the drawings and detailed description. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
For the ordered traversal of objects, 1000 objects are enumerated each time by default, 1000 objects are read back from the identified object location to all bucket fragments (128 by default) each time, and the first 1000 objects are sorted out of the objects in 128 x 1000 as a result of this ordered enumeration. The last object of the 1000 ordered objects is taken as a new object identifier, and the next traversal is continued from the new object identifier. In this scheme, 1000×128 objects are read from the bottom layer every time, but only 1000 ordered results are obtained, and 1000×128 objects continue to be read from the bottom layer when the next reading is performed, and for 127×1000 results after 1000 orders are not reasonably utilized, a user usually can identify all objects in the bucket by using new objects, so the current scheme can cause frequent access to the bottom layer (generally, a hard disk), and waste of system resources.
Therefore, the invention provides a sequence lifting scheme for storing the objects, wherein the rest objects are stored in a cache map, the rest objects are stored in a memory through the cache map, when a new enumeration request is needed, if the needed enumeration objects are in the cache map, the needed enumeration objects are extracted from the cache map, the rest objects are fully utilized, interaction with a bottom layer is reduced, cluster pressure is reduced, and meanwhile, the objects are directly read from the memory, so that the enumeration speed is remarkably improved.
Example 1
In the first embodiment, a method for storing objects in sequence is provided, in which N objects are enumerated each time, each bucket has M bucket fragments, where N and M are integers greater than or equal to 1, and of course, the current value of N is typically 1000, and the value of M is typically 128.
Fig. 1 is a flowchart of an object storage method according to the first embodiment, which includes the following steps.
S101, when an enumeration request of a client is received, detecting whether a cache map contains a cache of a barrel corresponding to a required enumeration object;
s102, if not, M x N objects are read from M barrel fragments of the corresponding barrel at the bottom layer, N objects are read from each barrel fragment, N objects are sequentially lifted from the M x N objects and sent to a client, and the rest (M-1) x N objects are stored in a cache map; the remaining (M-1) N objects are the cache of the bucket in the cache map;
and S103, if the object is included, the N objects are sequentially sent to the client from the cache of the corresponding bucket.
When receiving an enumeration object sent by a client, firstly checking whether a cache of a bucket corresponding to the needed enumeration object is in a cache map, if so, directly reading the needed enumeration object in the cache map, and if not, reading the enumeration object from a corresponding bucket at the bottom layer. For example, the first time an object in a bucket is enumerated, there is no cache for that bucket in the cache map. When the bottom layer reads the enumerated objects, the traditional sequential lifting method is used, namely M x N objects are firstly taken out, and then N objects are sequentially lifted from the M x N objects to be sent to the client. Thus, (M-1) N objects remain, and the remaining (M-1) N objects are stored in the cache map, i.e. the data in the cache map, i.e. the cache of the corresponding bucket is formed.
In the method for continuously enumerating the most common objects of the user, the method can directly utilize the residual object traversal result read from the bottom layer before the next group enumeration. Compared with the method that N objects are listed each time before, the method needs to obtain the barrel fragmentation number of the N objects from the bottom layer, on one hand, the interaction with the bottom layer can be reduced, and cluster pressure is reduced; on the other hand, the method can read from the memory directly, and can remarkably improve the enumeration speed compared with the method of reading from the bottom layer (hard disk) before.
The cached data in the cache map is stored in the form of key-value pairs, and in this embodiment, the remaining (M-1) N objects are stored in the cache map to form key-value pairs, where the keys of the key-value pairs are bucket names, and the values are combinations of bucket fragment addresses and bucket fragment objects. Specifically, < bucket name, < bucket fragment address, < bucket fragment object > >.
In step S101, it is detected whether the cache map includes the cache of the bucket corresponding to the object to be enumerated, and the cache map can be searched according to the key of the key value pair. In the subsequent step S103, when there are N objects in sequence from the caches of the corresponding buckets, the N objects are also searched according to the key value pairs.
It should be noted that, when the remaining (M-1) N objects are stored in the cache map, the original bucket fragments are still used for performing the fragmentation, that is, M bucket fragments are still included, and after the N objects are removed for the first time, the number of each of the bucket fragments is less than N. For example, N is 1000 and m is 128, after 1000 objects are first removed, 990 objects may remain in a bucket fragment, i.e., 10 objects are removed from the bucket fragment.
If the cache of the corresponding bucket exists in the cache map, when the enumeration command is received, it still needs to detect whether the needed enumeration object can be completely fetched from the bucket cache. In the implementation, by comparing the enumeration identifier of each barrel slice object with the enumeration identifier of the required enumeration object, whether the required enumeration object can be completely fetched or not is judged, and corresponding processing is required when the required enumeration object cannot be completely fetched.
Specifically, the enumeration request of the client includes an enumeration identifier, and the required enumeration object is N objects after the enumeration identifier (i.e. N objects after the enumeration identifier are sequentially fetched); the enumeration is identified as the object name. It should be noted that, the object names are character strings, and the objects are orderly arranged in the sequence of the character strings in each barrel slice.
Accordingly, if the cache map includes the cache of the corresponding bucket of the object to be enumerated in step S103, before the N objects are sequentially lifted from the cache of the corresponding bucket, the method includes the following steps:
comparing an enumeration identifier contained in an enumeration request with head and tail objects of each barrel slice of a barrel cache;
step two, if the list identification of a certain barrel slice is not more than or equal to the first object of the barrel slice and is less than or equal to the last object of the barrel slice, acquiring N objects again from the corresponding barrel slice at the bottom layer and storing the N objects into the cached barrel slice; wherein the first object is a head object and the last object is a tail object;
and thirdly, if the object re-stored in the barrel segmentation still does not meet the requirement that the enumeration identifier is greater than or equal to the first object and less than or equal to the last object, continuously re-acquiring N objects from the corresponding barrel segmentation at the bottom layer and storing the N objects into the barrel segmentation of the cache until the object of the barrel segmentation meets the requirement that the enumeration identifier is greater than or equal to the first object and less than or equal to the last object.
The N objects retrieved from the corresponding bucket fragments at the bottom layer in the second and third steps are N objects after the last object in the plurality of objects retrieved from the bucket fragments last time. For example, a certain bucket slice at the bottom layer contains objects n1, n2, n3, n4 and n5, where n1 and n2 are removed last time, and if 2 objects need to be removed this time, n3 and n4 should be removed.
Through the processing, each barrel fragment of the corresponding barrel in the cache map contains the object to be taken out, so that the complete sequence can be ensured to lift the required object.
Then, N objects can be sequentially lifted from the bucket cache, in step S103, the method specifically includes the following steps:
step one, according to an enumeration identifier in an enumeration request, a first object after the list identifier is taken out of each barrel slice, M objects are taken out altogether, and the largest object is taken out of the M objects as the first enumeration object;
taking the name of the first enumeration object as a new enumeration identifier, taking the new list identifier from each barrel slice, taking M objects out, and taking the largest object out of the M objects as a second enumeration object;
step three, and so on until N enumerated objects are fetched.
Through the steps, N objects can be sequenced from the barrel cache.
In this embodiment, N objects with sequences in the cache map are deleted from the cache map each time, so as to reduce memory occupation. Accordingly, the objects in the barrel fragments in the cache map gradually run out, so as to ensure that the objects can be continuously fetched from the cache map later, and timely supplement the cached objects for the cached barrel fragments. In specific implementation, the bucket fragmentation cache object can be supplemented by the following methods:
the method comprises the steps that firstly, if a certain barrel of the object in the cache map is used up, N objects are obtained again from the corresponding barrel of the bottom layer and stored in the barrel of the cache;
detecting the number H of the residual objects of each barrel slice in the cache map in real time, and if the number H of the residual objects of a certain barrel slice is smaller than or equal to a threshold value, re-acquiring N barrel slices of the corresponding barrel slice at the bottom layer, wherein the N barrel slices are stored in the cache;
detecting the number H of the residual objects of each barrel slice in the cache map in real time, and if the number H of the residual objects of a certain barrel slice is smaller than or equal to a threshold value, re-acquiring N-H barrel slices of the corresponding barrel slice at the bottom layer for storing the cache.
The user can select the proper method according to the needs. It should be noted that, the N objects or N-H objects re-acquired from the corresponding bucket partition at the bottom layer are N objects or N-H objects after the last object in all the objects acquired last time.
Example two
The second embodiment provides an object stored sequence lifting device, which is used for implementing the stored sequence lifting method. Similarly, each time N objects are enumerated, each bucket has M bucket fragments, N and M are integers greater than or equal to 1.
Fig. 2 is a schematic block diagram of an object storage and lifting device according to a second embodiment, which includes the following functional modules.
Bucket cache detection module: when an enumeration request of a client is received, detecting whether a cache map contains a cache of a barrel corresponding to a required enumeration object;
the bottom layer enumerates the module: if not, M x N objects are read from M barrel fragments of the corresponding barrel at the bottom layer, N objects are read from each barrel fragment, N objects are sent to the client side from sequences in the M x N objects, and the rest (M-1) x N objects are stored in the cache map; the remaining (M-1) N objects are the cache of the bucket in the cache map;
cache enumeration module: if the object is included, the N objects are sent to the client side from the buffer memory of the corresponding bucket in sequence.
The object of this embodiment stores a sequential lifting device for implementing the foregoing object stores a sequential lifting method, so that the specific implementation in the device can see that the foregoing object stores an example portion of the sequential lifting method, so that the specific implementation can refer to the description of the examples of the respective portions, and will not be described herein.
In addition, since the object storage sequence lifting device in this embodiment is used to implement the foregoing object storage sequence lifting method, the function of the object storage sequence lifting device corresponds to that of the foregoing method, and will not be described herein.
Example III
Fig. 3 is a schematic structural diagram of a terminal device 300 according to an embodiment of the present invention, where the terminal device 300 may be used to execute the object storage sequence method according to the embodiment of the present invention.
The terminal apparatus 300 may include: a processor 310, a memory 320 and a communication unit 330. The components may communicate via one or more buses, and it will be appreciated by those skilled in the art that the configuration of the server as shown in the drawings is not limiting of the invention, as it may be a bus-like structure, a star-like structure, or include more or fewer components than shown, or may be a combination of certain components or a different arrangement of components.
The memory 320 may be used to store instructions for execution by the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile memory terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The execution of the instructions in memory 320, when executed by processor 310, enables terminal 300 to perform some or all of the steps in the method embodiments described below.
The processor 310 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by running or executing software programs and/or modules stored in the memory 320, and invoking data stored in the memory. The processor may be comprised of an integrated circuit (Integrated Circuit, simply referred to as an IC), for example, a single packaged IC, or may be comprised of a plurality of packaged ICs connected to the same function or different functions. For example, the processor 310 may include only a central processing unit (Central Processing Unit, simply CPU). In the embodiment of the invention, the CPU can be a single operation core or can comprise multiple operation cores.
And a communication unit 330 for establishing a communication channel so that the storage terminal can communicate with other terminals. Receiving user data sent by other terminals or sending the user data to other terminals.
Example IV
The present invention also provides a computer storage medium in which a program may be stored, which program may include some or all of the steps in the embodiments provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (random access memory, RAM), or the like.
It will be apparent to those skilled in the art that the techniques of embodiments of the present invention may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solution in the embodiments of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium such as a U-disc, a mobile hard disc, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk or an optical disk, etc. various media capable of storing program codes, including several instructions for causing a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the method described in the embodiments of the present invention.
The same or similar parts between the various embodiments in this specification are referred to each other. In particular, for the terminal embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and reference should be made to the description in the method embodiment for relevant points.
In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The foregoing disclosure is merely illustrative of the preferred embodiments of the invention and the invention is not limited thereto, since modifications and variations may be made by those skilled in the art without departing from the principles of the invention.

Claims (8)

1. An object storage sequence lifting method, each time enumerating N objects, each bucket having M bucket fragments, N and M being integers greater than or equal to 1, characterized in that the method comprises the steps of:
when an enumeration request of a client is received, detecting whether a cache map contains a cache of a barrel corresponding to a required enumeration object;
if not, M x N objects are read from M barrel fragments of the corresponding barrel at the bottom layer, N objects are read from each barrel fragment, N objects are sent to the client side from sequences in the M x N objects, and the rest (M-1) x N objects are stored in the cache map; the remaining (M-1) N objects are the cache of the bucket in the cache map;
if the object is included, the N objects are sent to the client side in sequence from the cache of the corresponding bucket;
the cached data in the cache map is stored in the form of key value pairs, wherein the keys of the key value pairs are bucket names, and the values are combinations of bucket fragment addresses and bucket fragment objects;
the enumeration request of the client contains an enumeration identifier, and the required enumeration objects are N objects after the enumeration identifier; enumerating the identification as an object name;
if the cache map contains the cache of the corresponding bucket of the object to be enumerated, before the sequence of N objects is lifted from the cache of the corresponding bucket, the method comprises the following steps:
comparing an enumeration identifier contained in the enumeration request with the head-to-tail object of each bucket fragment of the bucket cache;
if the list identification of a certain barrel slice is not more than or equal to the first object of the barrel slice and is less than or equal to the last object of the barrel slice, acquiring N objects again from the corresponding barrel slice at the bottom layer and storing the N objects into the cached barrel slice; wherein the first object is a head object and the last object is a tail object;
if the object re-stored in the bucket fragment still does not meet the requirement that the enumeration identifier is greater than or equal to the first object and less than or equal to the last object, continuously re-acquiring N objects from the corresponding bucket fragment at the bottom layer and storing the N objects into the cached bucket fragment until the object of the bucket fragment meets the requirement that the enumeration identifier is greater than or equal to the first object and less than or equal to the last object.
2. The method according to claim 1, wherein the step of sequentially lifting N objects from the cache of the corresponding bucket comprises:
taking out the first object after the enumeration identifier from each barrel slice according to the enumeration identifier in the enumeration request, taking out M objects in total, and taking out the largest object from the M objects as the first enumeration object;
taking the name of the first enumeration object as a new enumeration identifier, taking the new enumeration identifier from each barrel slice, taking M objects out, and taking the largest object out of the M objects as a second enumeration object;
and so on until N enumerated objects are fetched.
3. The object storage sequential lift method of claim 2, further comprising the steps of:
n objects with sequence lifting in the cache map at a time are deleted from the cache map.
4. A method of storing a sequence of objects according to claim 3, the method further comprising the steps of:
if the object of a certain barrel slice in the cache map is used up, N objects are obtained again from the corresponding barrel slice at the bottom layer and stored in the cached barrel slice;
or detecting the number H of the remaining objects of each barrel slice in the cache map in real time, and if the number H of the remaining objects of a certain barrel slice is smaller than or equal to a threshold value, re-acquiring N barrel slices of the corresponding barrel slices at the bottom layer, wherein the N barrel slices are stored in the cache;
or detecting the number H of the remaining objects of each barrel slice in the cache map in real time, and if the number H of the remaining objects of a certain barrel slice is smaller than or equal to a threshold value, re-acquiring N-H barrel slices of the corresponding barrel slices at the bottom layer for storing the cache.
5. The method according to claim 4, wherein the N objects or N-H objects retrieved from the underlying corresponding bucket fragment are N objects or N-H objects after the last object in all the last acquired objects.
6. An object storage sequence lifting device for enumerating N objects each time, each barrel having M barrel fragments, N and M being integers greater than or equal to 1, characterized in that the device comprises,
bucket cache detection module: when an enumeration request of a client is received, detecting whether a cache map contains a cache of a barrel corresponding to a required enumeration object;
the bottom layer enumerates the module: if not, M x N objects are read from M barrel fragments of the corresponding barrel at the bottom layer, N objects are read from each barrel fragment, N objects are sent to the client side from sequences in the M x N objects, and the rest (M-1) x N objects are stored in the cache map; the remaining (M-1) N objects are the cache of the bucket in the cache map;
cache enumeration module: if the object is included, the N objects are sent to the client side in sequence from the cache of the corresponding bucket;
the cached data in the cache map is stored in the form of key value pairs, wherein the keys of the key value pairs are bucket names, and the values are combinations of bucket fragment addresses and bucket fragment objects;
the enumeration request of the client contains an enumeration identifier, and the required enumeration objects are N objects after the enumeration identifier; enumerating the identification as an object name;
before the N objects are listed in sequence from the caches of the corresponding buckets, the cache enumeration module is further configured to:
comparing an enumeration identifier contained in the enumeration request with the head-to-tail object of each bucket fragment of the bucket cache;
if the list identification of a certain barrel slice is not more than or equal to the first object of the barrel slice and is less than or equal to the last object of the barrel slice, acquiring N objects again from the corresponding barrel slice at the bottom layer and storing the N objects into the cached barrel slice; wherein the first object is a head object and the last object is a tail object;
if the object re-stored in the bucket fragment still does not meet the requirement that the enumeration identifier is greater than or equal to the first object and less than or equal to the last object, continuously re-acquiring N objects from the corresponding bucket fragment at the bottom layer and storing the N objects into the cached bucket fragment until the object of the bucket fragment meets the requirement that the enumeration identifier is greater than or equal to the first object and less than or equal to the last object.
7. A terminal, comprising:
a processor;
a memory for storing execution instructions of the processor;
wherein the processor is configured to perform the method of any of claims 1-5.
8. A computer readable storage medium storing a computer program, which when executed by a processor implements the method of any one of claims 1-5.
CN202111069552.4A 2021-09-13 2021-09-13 Object storage sequence lifting method, device, terminal and storage medium Active CN113806249B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111069552.4A CN113806249B (en) 2021-09-13 2021-09-13 Object storage sequence lifting method, device, terminal and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111069552.4A CN113806249B (en) 2021-09-13 2021-09-13 Object storage sequence lifting method, device, terminal and storage medium

Publications (2)

Publication Number Publication Date
CN113806249A CN113806249A (en) 2021-12-17
CN113806249B true CN113806249B (en) 2023-12-22

Family

ID=78941070

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111069552.4A Active CN113806249B (en) 2021-09-13 2021-09-13 Object storage sequence lifting method, device, terminal and storage medium

Country Status (1)

Country Link
CN (1) CN113806249B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101122885A (en) * 2007-09-11 2008-02-13 腾讯科技(深圳)有限公司 Data cache processing method, system and data cache device
CN103914399A (en) * 2012-12-31 2014-07-09 中国移动通信集团公司 Disk cache method and device in parallel computer system
CN105718561A (en) * 2016-01-20 2016-06-29 中国科学院计算技术研究所 Particular distributed data storage file structure redundancy removing construction method and system
WO2020181820A1 (en) * 2019-03-08 2020-09-17 平安科技(深圳)有限公司 Data cache method and apparatus, computer device and storage medium
CN111857890A (en) * 2020-07-24 2020-10-30 苏州浪潮智能科技有限公司 Service processing method, system, device and medium
CN113111033A (en) * 2021-04-07 2021-07-13 山东英信计算机技术有限公司 Method and system for dynamically redistributing bucket indexes in distributed object storage system

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2010276462B1 (en) * 2010-12-27 2012-01-12 Limelight Networks, Inc. Partial object caching
US10715622B2 (en) * 2018-12-14 2020-07-14 Nutanix, Inc. Systems and methods for accelerating object stores with distributed caching
US10977132B2 (en) * 2019-03-08 2021-04-13 International Business Machines Corporation Selective placement and adaptive backups for point-in-time database recovery

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101122885A (en) * 2007-09-11 2008-02-13 腾讯科技(深圳)有限公司 Data cache processing method, system and data cache device
CN103914399A (en) * 2012-12-31 2014-07-09 中国移动通信集团公司 Disk cache method and device in parallel computer system
CN105718561A (en) * 2016-01-20 2016-06-29 中国科学院计算技术研究所 Particular distributed data storage file structure redundancy removing construction method and system
WO2020181820A1 (en) * 2019-03-08 2020-09-17 平安科技(深圳)有限公司 Data cache method and apparatus, computer device and storage medium
CN111857890A (en) * 2020-07-24 2020-10-30 苏州浪潮智能科技有限公司 Service processing method, system, device and medium
CN113111033A (en) * 2021-04-07 2021-07-13 山东英信计算机技术有限公司 Method and system for dynamically redistributing bucket indexes in distributed object storage system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
一种数据仓库操作的优化方法;王明婷;孙江涛;吕绍华;;计算机工程与设计(第11期);全文 *

Also Published As

Publication number Publication date
CN113806249A (en) 2021-12-17

Similar Documents

Publication Publication Date Title
CN108604239B (en) System and method for efficiently classifying data objects
US10169391B2 (en) Index management
CN109445861A (en) System start method, device, computer installation and storage medium
CN110019873B (en) Face data processing method, device and equipment
CN110888837B (en) Object storage small file merging method and device
CN106909595B (en) Data migration method and device
CN112148217B (en) Method, device and medium for caching deduplication metadata of full flash memory system
CN114327278A (en) Data additional writing method, device, equipment and storage medium
CN115114232A (en) Method, device and medium for enumerating historical version objects
CN111338688B (en) Data long-acting caching method and device, computer system and readable storage medium
CN111930305A (en) Data storage method and device, storage medium and electronic device
CN104035822A (en) Low-cost efficient internal storage redundancy removing method and system
CN111831691B (en) Data reading and writing method and device, electronic equipment and storage medium
CN117369731B (en) Data reduction processing method, device, equipment and medium
CN110352410A (en) Track the access module and preextraction index node of index node
CN110427394B (en) Data operation method and device
CN113806249B (en) Object storage sequence lifting method, device, terminal and storage medium
US10761940B2 (en) Method, device and program product for reducing data recovery time of storage system
CN111913913A (en) Access request processing method and device
CN106055640A (en) Buffer memory management method and system
CN115576947A (en) Data management method and device, combined library, electronic equipment and storage medium
WO2019143967A1 (en) Methods for automated artifact storage management and devices thereof
CN114443583A (en) Method, device and equipment for arranging fragment space and storage medium
WO2022001133A1 (en) Method and system for improving soft copy read performance, terminal, and storage medium
US11226778B2 (en) Method, apparatus and computer program product for managing metadata migration

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant